Understanding Cybersecurity Measures and Metrics for Medical Devices | Ep. 31

In this episode of The Med Device Cyber Podcast, hosts Christian Espinosa and Trevor Slattery of Blue Goat Cyber tackle the often-misunderstood topic of measures and metrics in the context of medical device cybersecurity and FDA submissions. They begin by establishing a clear distinction between the two terms, which is critical for manufacturers preparing 510(k) or PMA submissions. A 'measure' is defined as a direct, quantifiable attribute of something, such as the time it takes to apply a software patch or the total number of security incidents that have occurred. In contrast, a 'metric' is a calculation derived from one or more measures, typically expressed as a percentage or ratio. For instance, while the time to apply a patch is a measure, the percentage of systems patched within a defined timeframe is a metric. This fundamental difference forms the basis of the entire discussion, as the FDA has specific expectations for both. The hosts then pivot to what the FDA specifically looks for in premarket submissions. The FDA is interested in how a manufacturer collects the necessary figures to produce these measures and metrics, covering the total product lifecycle. The core of the FDA's focus is on vulnerability management. Espinosa and Slattery outline the three key data points the FDA requires. The primary metric is the percentage of identified vulnerabilities that are updated or patched. This is supported by two critical measures: the duration from the initial identification of a vulnerability to when a patch becomes available, and the subsequent duration from patch availability to its full deployment across all fielded devices. They clarify that for brand new devices without a predicate, manufacturers are not expected to have historical data but must present a comprehensive plan for how they will collect this data post-market. For devices with a predicate or those undergoing PMA annual reporting, the expectation is that this data will be available and presented. The conversation also touches upon the importance of context and a risk-based approach. The risk profile of a device can change drastically depending on its environment of use—a device in a connected hospital network faces different threats than one used in a home environment. Furthermore, measures like device downtime and recovery time are crucial, especially for critical care systems where availability is paramount. A surgical robot that takes a minute to reboot after an issue is far more concerning than an accessory device with the same reboot time. The hosts conclude by reinforcing that while the FDA outlines a minimum baseline for compliance, a robust cybersecurity posture requires going beyond simply checking a box. The goal isn't just to collect data but to make it actionable, ensuring that the information gathered is used effectively to reduce risk and enhance patient safety throughout the device's lifecycle.

Christian: Hi, welcome back to the Med device Cyber podcast. Today we're going to talk about measures and metrics, a commonly misunderstood topic. Uh measures are different than metrics and the FDA is looking for very specific things in a 510k or PMA submission. Uh so what we'll cover these those in enough detail where you can have a decent understanding of them. And I'm Christian Espinosa, your co-host here with Trevor Slattery. Uh how you doing today Trevor? Trevor: Doing good. Getting some nice weather right now. Christian: Are you in Sedona? Trevor: I am in Sedona. Yeah. Christian: You know, I I I did I used a AI once to create a snippet of one of our pod podcast and it mentioned the guy with the the the bull in the in the background, it referenced you as that and that's where you are. I see the bull picture in the background. Is it a bull or what is that? Trevor: Uh, I think it's a yak or a bull. Something like that. Christian: Yeah, it was funny AI referred to you as the the young man with the the bull in the background or something like that. Trevor: There you go. Christian: Cool. All right, so I think it's important to start with the definition of a measure and then the definition of a metric. So a measure is just like a tape measure, right? You it's like a quantifiable attribute of something. Like how long did it does it take to apply a patch? How many incidents occurred? That's a measure. Whereas a metric is some sort of calculation derived from a couple measures typically. Usually usually a percentage. Uh, you you have anything to elaborate on those two? Trevor: Yeah, so essentially what we're saying is that the metrics are derived from measures. And so they go hand-in-hand, but they are distinct, separate items. Christian: Yeah, so an example would be a measure how long does it take to apply a patch. A metric could be like your patch management efficiency. The percentage of systems patched within a defined time. So that's a percentage, the metric versus the duration, which is the measure. Trevor: Spoiler for what we're going to talk about when we say what the FDA wants to see. Christian: It's a spoiler? Trevor: We'll get there in a few minutes, I'm sure. Christian: All right. So now that we understand uh a measure versus a metric. What does the FDA want to see in terms of measure, we'll start with measures first. Trevor: Well, I think before we get into the specific measures and metrics, we talk about we should talk about how the FDA wants to see us collect the figures that go into this. So, the measures and metrics are these quantifiable figures. How long does it take to apply a patch? How many vulnerabilities are we remediating? And they want to see now that we're understanding, you know, these different measures and metrics, how are we getting the information to provide this. And so the first thing when we're looking at the required FDA documentation, they combine the total product lifecycle considerations into the measures and metrics when you're looking at the headings throughout FDA guidance and through the e-star submission process. And so, the first area that the FDA wants to see you covers, how are you identifying, addressing, and mitigating vulnerabilities? Which it's essentially three separate questions. But it should talk about your vulnerability management process, any tools, any processes that you're using for collecting these vulnerabilities, and then how you're tracking it once you've identified them. Christian: Well, I think the the measures that go with that are once you've identified a vulnerability and you've verified it actually is a real vulnerability, like what's the risk associated with it? And then if the risk is critical, how quickly should that be passed versus if the risk is low, as an example. That that duration would be a measure. Trevor: Right. Yeah, and that feeds directly into we can, well, we can talk about what the measures and metrics are and then work backwards from how we get to these figures. So, the FDA wants to see, as one of the first points, what is your percentage of identified vulnerabilities that are updated or patched? And so, especially with a complicated device, you can get in hundreds if not thousands of vulnerabilities at a single time. And so seeing these come in, how are you triaging them based on severity? Like you mentioned, it's going to be more important to handle these critical vulnerabilities as opposed to these low vulnerabilities. What is your timeline for remediation and just general patch cycles. And so if you're following sprints, which a lot of engineers and manufacturers like to follow that development cycle. What are your sprint cycles looking like? How many vulnerabilities are you remediating per sprint cycle? You should be looking at close to as 100% of critical vulnerabilities as possible. And then you can start tapering off as you go down lower in risk. Christian: So, what what are the typical measures then for what you just talked about? Uh cause this comes up quite often. If I have identified a high-risk item or a critical risk item, what are the typical timelines that should be applied for to get that that patch rolled out. Trevor: It's going to depend a lot on your device, how the implementation works, how the actual vulnerability works. And so, if we have a device capable of supporting over-the-air updates, it's likely going to be a lot faster than if we need to roll out service technicians to handle the update process. So this should be a scaling figure based on the actual architecture of your device. Um, whatever the figures end up being for a critical vulnerability, we want to keep these as tight as possible. For over-the-air updates, if we can get it within a single patch cycle, that would certainly be ideal. And then handling any advisories or sort of banda-id fixes while we're dealing with rolling out service technicians, uh, can help alleviate the risk while we deal with potentially longer times for remediation. So it can really depend. Whatever these timelines are, there's something that needs to be documented. So this is another measure that the FDA wants to see is what is our duration from identification to patch availability? And then the final one the FDA is really concerned about is the duration from when a patch or update is available to when it's rolled out into all fielded products. So this all, all of these feed into each other to make sure that we have the process start to finish from identifying a vulnerability, how many of them are we fixing, how long does it take to fix it, how long does it take to get it in our devices, and then verify everything's been fixed. The FDA wants to collect these measures and metrics, and they want manufacturers to collect them to see what that total flow looks like. Christian: Um, what are the the questions I see quite a bit that people ask uh is what are the measures is like about detecting the uh incidents on their device. Uh so that goes that that ties back to do we have logs on our device and does somebody actually look at the logs? Uh and we've had a few clients ask about this um because most devices are not monitored in real time like by a SOC or a traditional like a traditional IT infrastructure. Um, what are your thoughts on that? Trevor: I think designing alerting mechanisms into your device is a good way to bypass that problem. So, sure, if you're collecting logs about any security events all day long, that's great. But if nobody's looking at them, it doesn't really mean anything. So, having a mechanism in place for devices to notify users that there's a problem is a great way to work around this. Let's say that an easy example is we can talk about software as a medical device. If there's a pop up saying, there's been a log in at a different location under your account, that's something that should alert most users to something's going on or they say, oh I just logged in on my phone while I was on my laptop, so that's normal, no big deal. So having a way to notify users in a clear to understand way what's happening in these backend logs is a great way to work around that problem. Christian: So we would have um as a measure uh the incident is discovered via the log in real time basically or maybe in a couple minutes and the user is alerted and then it's up to the user though to alert the manufacturer. Trevor: Exactly. Yeah there and that goes into part of the labeling considerations and how we're the device to essentially guide users around the process that they need to follow. But when we're providing this labeling documentation it should contain clear steps for users to follow. If an anomaly is identified in the device who do you notify how do you notify them what information needs to be captured things like that. and so this all ties in this is why there are so many different requirements from the FDA they all tie together they're all there for a reason and using them in combination allows to have more comprehensive security when problems like this come up Christian: Okay so before we move on to metrics what are a couple other measures that the FDA is most concerned with I know what they're we could have like thousands of measures but like what are the couple the the main ones we talked about uh the time to patch a system we talked about the time to if an incident is uh detected to respond to it what other measures are there Trevor: So those are the only measures that the FDA as long as we're drawing the line between measures and metrics those are the only ones that the FDA is really concerned about. Uh when they're looking at how we're capturing these measures and metrics the three that they explicitly call out are the percentage of identified vulnerabilities that are updated or patched uh the duration from identification to patch availability and duration from patch availability to patch deployment. So those are the three things the FDA is looking at. Christian: So the durations are the the measures. Um the percentage of vulnerabilities patched um Trevor: That's going to be the metric. Christian: That'll be the metric. So that's like um I don't know that that metric kind of bothers me because if I've got 8,000 like critical vulnerabilities and 97,000 like like informational vulnerabilities you know I really it only matters if I patch the criticals right so that percentage is a misleading number, I think. I mean what are your thoughts on that? Trevor: Yeah and that's something that we provide recommendations for as well with these. If you have well first off if you have 8,000 critical vulnerabilities in your device you have other problems than the percentage of this you're fixing you have serious, serious problems you need to figure out there. Christian: Yes. Trevor: But we have seen it, you know, we've seen I don't know about 8,000, but we've seen probably past a thousand critical vulnerabilities in a product before. And so there's no easy way to fix those all in one go. Usually that's going to be more in the pre-market phase and so where these metrics start coming in to play these measures of metrics is once we have a field of device we're assuming there's a level of baseline security. vulnerabilities are going to pop up in the post-market phase how are we addressing them as they come up? If we're getting hundreds of informational vulnerabilities coming in what's the actual impact? It's all should be a risk-based approach. And so if we say well sure you can identify the headers on our web application what can you actually do with that information? It's really not going to be that big of a deal. That can be essentially on a when I get to it basis. If we're looking at something that can provide a direct impact to a patient to a user, to their data, that's when we need to say, now there is far less tolerance for this. This can't be when we get to it. This needs to be pushed forward as a priority. So that percentage needs to get driven up significantly. Christian: Yeah, I agree. And now as you're talking, I was thinking um uh I I get my weight measured quite often. And your your weight and your height are like a measure, right? It's just a number. And then you have things like BMI which is actually a metric, right? But BMI kind of like the percentage of of of vulnerabilities patched is is a misleading metric because it's purely based on your height versus your weight and you could have a lot of muscle uh and be relatively short, which would give you a high BMI versus a body fat percentage, which would uh is a more accurate metric. Trevor: Right. Yeah, and I think that unlike the case of BMI I guess sort of like the case of BMI. It's a little bit of an odd metric at times. I feel like this identifying the percentage of patch vulnerabilities is a lot more effective in the post-market phase. Um but in the pre-market phase, it's not going to be very effective. And this does lead into when are these measures and metrics applicable since they aren't applicable for every device and they aren't applicable for every manufacturer which is that's a big area of misconception. I've seen in the past. If you are submitting a new device and you don't have these measures and metrics available from your substantially equivalent product, you don't need to provide them as part of your submission. You need to provide your plan to capture them. And then if you're submitting a PMA, you need to provide them as part of your PMA annual reporting, but you don't need to submit them during the initial submission. This is when you have previously filed devices to collect these from that you provide them as part of your an initial FDA submission. Christian: So you're saying if uh there's a predicate device and I'm doing a 510K, I probably have an idea of what measures and metrics should be included for my device. Um but if there's not a predicate device, I don't need to include the measures and metrics, but I need to have a plan for those for post market. Trevor: Correct. And even with the predicate device considering the guidance is still fairly new as of 2023, it may be the case that you weren't collecting these for the predicate device. You you're doing a resubmission, but you weren't collecting them since it wasn't a requirement under the old guidance. So you don't have these readily available. If you've done a submission under the current guidance and you're actually adhering to what you're supposed to in the post-market phase, you should have these readily available to provide as part of any resubmissions or any submissions where you're using the old device as a predicate. And so this is something where in the coming years this is going to become a lot more important but I feel like a lot of devices are still just saying we don't have these for our predicates or we're still a new device we don't have this. Christian: And what about uh because I there's an article that came out a couple days ago or talks about medical devices and when they fail, it says 43% of them have failed for up to 4 hours of downtime. Uh should that be a measure like the anticipated amount of downtime before the system can recover. Um I'm just I'm just curious because I was trying to tie what you said back to that article I I read a couple days ago. Trevor: Yeah, I think it can depend, well it'll depend on the device functionality. Sometimes availability isn't really a big concern from a security or safety perspective. Um if we're looking at an accessory device for a non-essential therapy, something for convenience, then it's likely not going to be the biggest deal. If it is something for critical care, for immediate patient treatment, that downtime can be life or death and so that can feed into how are you addressing different metrics, different measures, across different devices. Everything goes back to what's the actual level of risk and so seeing what is your expected downtime? What is the downtime before you can expect adverse results? Um, what is, you know, how often does downtime come up? Those are good things to capture especially when you're looking at something that really needs maximum uptime to be an effective product. Christian: Yeah, I mean if you got something that you got a thousand patients relying on this device to give a diagnosis and it goes down for 12 hours, uh that's probably not acceptable. Trevor: Right. Christian: So you need to design the device in a manner where what is the actual amount of downtime we can anticipate? And can the device recover within that measurement? Trevor: Ideally, how can we prevent downtime all together? Christian: Exactly. Not always possible, but you know, we can dream. Trevor: Yeah. Christian: Well, often the the way to prevent that is to reboot the system. You know if the system if something goes wrong in the labeling and it says reboot it, but what is the measure of how long it takes to reboot a system that is critical for patients? You know that's something I I consider because I actually there's a movie I watched the other day and um, it took like an hour to reboot this this system and the whole time it was rebooting, you know, people were unable to get care. Um, I don't remember the exact movie, but you know it was interesting because the guy came around with this work around and rebooted in like five minutes. Actually this movie is about um, you may have seen it's uh it's about these divers that are stuck under the water and it's about the ship that lost automated control to stay stationary. That's what it's about. So the system to keep the all the motors running to keep the ship stationary took like an hour and a half to reboot or something and this guy had to bypass it. But the whole time these the people that were connected on the dive platform or whatever way down in the water were running out of air and everything else. It was a major problem. Trevor: Yeah, I think I know the movie you're talking about. Yeah, and that's especially with some Christian: last breath I think it was what it was called. Trevor: Yeah. It sounds familiar and I know I've seen some movie around I feel like there are a couple of movies with a similar similar theme but yeah, using an example like that usually you know, you've done a lot of diving you see some of the equipment that they're using on these dive boats and it looks like it hasn't been touched since the '80s and that can be the same case with a lot of medical equipment it's very old you think about how long it took to boot up an old desktop system from 15 years ago compared to a modern laptop. The laptop I'm using right now probably goes from powered off to start it up in 10 seconds or less. I think about I have some older laptops that are running on aren't using solid state drives or using an older version of Windows and those can take a matter of you know multiple minutes to boot up. When we're looking at a critical care system, those multiple minutes can be important and if we're using a really old system which unfortunately a lot of hospitals still are using these really old systems, take even longer than that to boot up. Christian: Yeah, that's a good point. Like uh MRI machine may take uh quite a quite some time to reboot. Hopefully like a a surgical robot that's performing surgery if it if it goes haywire, you have to reboot it doesn't take, you know, an hour to reboot because if you got, you know, open open heart surgery or open spine surgery going on and you know, an hour is a problem. Trevor: I think I'd freak out if it took a minute to reboot. I want that thing back up immediately. Christian: Exactly. Because it also affects like, you know, surgeon fatigue and other things as well if they're using if a surgeon uses a robot. Trevor: Right. Christian: Cool, anything else we should discuss here on measures and metrics? Trevor: I think be sure you're getting a good plan to collect this information ahead of time. Once you're going through the submission process, if an FDA audit pops up or if you're going through a PMA submission especially, you need to have this information on hand at all times. So as you're, as soon as you're cleared by the FDA, you should start collecting this information. You should have a very good plan in place to do so beforehand. Christian: And I would add one thing to that. Collect the information is like step one but being able to make the information actionable in a way to reduce the risk is the the part that I think a lot of people have challenge with. Christian: Right. It's just like, you can measure your BMI but if you don't change your diet or don't do anything or don't have the the right action plan behind it, it doesn't really matter. Trevor: Right. Christian: All right, well thank you everyone for tuning in to this episode of the Med Device Cyber Podcast. I hope you found it useful and please tune in next time.