Understanding Cybersecurity Measures and Metrics for Medical Devices | Ep. 31

Hi, welcome back to The Med Device Cyber Podcast. Today we're going to talk about measures and metrics, a commonly misunderstood topic. Measures are different than metrics, and the FDA is looking for very specific things in a 510k or PMA submission. We'll cover these in enough detail so you can have a decent understanding of them. I'm Christian Espinosa, your co-host here with Trevor Slatterie. How are you doing today, Trevor? Doing good. Getting some nice weather right now. Are you in Sedona? I am in Sedona. Yeah. You know, I did use AI once to create a snippet of one of our podcasts, and it mentioned the guy with the bull in the background. It referenced you as that, and that's where you are. I see the bull picture in the background. Is it a bull or what is that? I think it's a yak or a bull. Something like that. Yeah, it's funny. The AI referred to you as the young man with the bull in the background or something like that. There you go. Cool. All right. So, I think it's important to start with the definition of a measure and then the definition of a metric. A measure is just like a tape measure, right? It's like a quantifiable attribute of something, like how long does it take to apply a patch? How many incidents occurred? That's a measure. Whereas a metric is some sort of calculation derived from a couple of measures, typically usually a percentage. Do you have anything to elaborate on those two? Yeah, so essentially what we're saying is that the metrics are derived from measures, and so they go hand in hand, but they are distinct, separate items. Yeah. So, an example would be a measure: how long does it take to apply a patch? A metric could be your patch management efficiency: the percentage of systems patched within a defined time. So that's a percentage, the metric, versus the duration, which is the measure. Spoiler for what we're going to talk about when we say what the FDA wants to see. That's a spoiler. We'll get there in a few minutes, I'm sure. All right. So now that we understand a measure versus a metric, what does the FDA want to see in terms of measures? We'll start with measures first. Well, I think before we get into the specific measures and metrics, we should talk about how the FDA wants to see us collect the figures that go into this. So, the measures and metrics are these quantifiable figures: how long does it take to apply a patch? How many vulnerabilities are we remediating? And they want to see, now that we're understanding these different measures and metrics, how are we getting the information to provide this? And so the first thing, when we're looking at the required FDA documentation, they combine the total product life cycle considerations into the measures and metrics when you're looking at the headings throughout FDA guidance and through the eSTAR submission process. And so the first area that the FDA wants to see you cover is how you are identifying, addressing, and mitigating vulnerabilities, which it's essentially three separate and comprehensive questions, but it should talk about your vulnerability management process, any tools, any processes that you're using for collecting these vulnerabilities, and then how you're tracking it once you've identified them. Well, I think the measures that go with that are once you've identified a vulnerability and you've verified it actually is a real vulnerability, what's the risk associated with it? And then if the risk is critical, how quickly should that be patched versus if the risk is low, as an example, that duration would be a measure. Right? Yeah. Yeah. And that feeds directly into, we can, well, we can talk about what the measures and metrics are and then work backwards from how we get to these figures. So, the FDA wants to see as one of the first points what your percentage of identified vulnerabilities that are updated or patched is. And so, especially with a complicated device, you can get hundreds, if not thousands, of vulnerabilities at a single time. And so seeing these come in, how are you triaging them based on severity? Like you mentioned, it's going to be more important to handle these critical vulnerabilities as opposed to these low vulnerabilities. What is your timeline for remediation in just general patch cycles? And so if you're following sprints, which a lot of engineers and manufacturers like to follow that development cycle, what are your sprint cycles looking like? How many vulnerabilities are you remediating per sprint cycle? You should be looking at close to 100% of critical vulnerabilities as possible, and then you can start tapering off as you go down lower in risk. So, what are the typical measures then for what you just talked about? Because this comes up quite often if I have identified a high-risk item or critical risk item. What are the typical timelines that should be applied to get that patch rolled out? It's going to depend a lot on your device, how the implementation works, how the actual vulnerability works. And so if we have a device capable of supporting over-the-air updates, it's likely going to be a lot faster than if we need to roll out service technicians to handle the update process. So this should be a scaling figure based on the actual architecture of your device. Whatever the figures end up being for a critical vulnerability, we want to keep these as tight as possible. For over-the-air updates, if we can get it within a single patch cycle, that would certainly be ideal. And then handling any advisories or sort of band-aid fixes while we're dealing with rolling out service technicians can help alleviate the risk while we deal with potentially longer times for remediation. So, it can really depend. Whatever these timelines are, there's something that needs to be documented. So this is another measure that the FDA wants to see: what is our duration from identification to patch availability? And then the final one the FDA is really concerned about is the duration from when a patch or an update is available to when it's rolled out into all fielded products. So, all of these feed into each other to make sure that we have the process start to finish from identifying a vulnerability. How many of them are we fixing? How long does it take to fix it? How long does it take to get it in our devices and then verify everything's been fixed? The FDA wants to collect these measures and metrics, and they want manufacturers to collect them to see what that total flow looks like. One of the questions I see quite a bit that people ask is what measures are about detecting incidents on the device. So, that goes, that ties back to do we have logs on a device and does somebody actually look at the logs? And we've had a few clients ask about this because most devices are not monitored in real time like by a SOC or traditional, like a traditional IT infrastructure. What are your thoughts on that? I think designing alerting mechanisms into your device is a good way to bypass that problem. So sure, if you're collecting logs about any security events all day long, that's great. But if nobody's looking at them, it doesn't really mean anything. So, having a mechanism in place for devices to notify users that there's a problem is a great way to work around this. Let's say that an easy example is we can talk about software as a medical device. If there's a popup saying there's been a login at a different location under your account, that's something that should alert most users to something's going on. Or they say, "Oh, I just logged in on my phone while I was on my laptop," so that's normal, no big deal. So, having a way to notify users in a clear to understand way what's happening in these backend logs is a great way to work around that problem. So, we would have as a measure, the incident is discovered via the log in real time basically, or maybe within a couple minutes, and the user is alerted, and then it's up to the user though to alert the manufacturer. Exactly. Yeah. Then, and that goes into part of the labeling considerations and how we're designing the device to essentially guide users around the process that they need to follow. But when we're providing this labeling documentation, it should contain clear steps for users to follow: if an anomaly is identified in the device, who do you notify? How do you notify them? What information needs to be captured? Things like that. And so, this all ties in. This is why there are so many different requirements from the FDA. They all tie together. They're all there for a reason. And using them in combination allows for more comprehensive security when problems like this come up. Okay. So, before we move on to metrics, what are a couple other measures that the FDA is most concerned with? I know there, we could have like thousands of measures, but like what are the couple, the main ones? We talked about the time to patch a system. We talked about the time for an incident, if detected, to respond to it. What other measures are there? So, those are the only measures that the FDA, as long as we're drawing the line between measures and metrics, those are the only ones that the FDA is really concerned about. When they're looking at how we're capturing these measures and metrics, the three that they explicitly call out are the percentage of identified vulnerabilities that are updated or patched, the duration from identification to patch availability, and duration from patch availability to patch deployment. So, those are the three things the FDA is looking at. So, the durations are the measures. The percentage of vulnerabilities patched, that's going to be the metric. That'll be the metric. So, that's like, I don't know, that metric kind of bothers me because if I've got 8,000 critical vulnerabilities and 97,000 informational vulnerabilities, really, it only matters that I patch the criticals, right? So, that percentage is a misleading number, I think. I mean, what are your thoughts on that? Yeah, and that's something that we provide recommendations for as well with these. If you have, well, first off, if you have 8,000 critical vulnerabilities in your device, you have other problems than the percentage that you're fixing. You have serious, serious problems you need to figure out there. Yes. Um, but we have seen it. You know, we've seen, I don't know about 8,000, but we've seen probably past a thousand critical vulnerabilities in a product before. And so there's no easy way to fix those all in one go. Usually that's going to be more in the pre-market phase. And so where these metrics start coming into play, these measures of metrics, is once we have a fielded device, we're assuming there's a level of baseline security. Vulnerabilities are going to pop up in the postmarket phase. How are we addressing them as they come up? If we're getting hundreds of informational vulnerabilities coming in, what's the actual impact? It should all be a risk-based approach. And so if we say, "Well, sure, you can identify the headers on our web application. What can you actually do with that information?" It's really not going to be that big of a deal. That can be essentially on a "when I get to it" basis. If we're looking at something that can provide a direct impact to a patient, to a user, to their data, that's when we need to say, "No, there is far less tolerance for this. This can't be when we get to it. This needs to be pushed forward as a priority." So, that percentage needs to get driven up significantly. Yeah, I agree. I know, as you're talking, I was thinking, I get my weight measured quite often, and your weight and your height are like a measure, right? It's just a number. And then you have things like BMI, which is actually a metric, right? But BMI, kind of like the percentages of vulnerabilities patched, is a misleading metric because it's purely based on your height versus your weight. And you could have a lot of muscle and be relatively short, which would give you a high BMI versus a body fat percentage, which would be a more accurate metric. Right. Yeah. And I think that unlike the case of BMI, well, I guess sort of like the case of BMI, it's a little bit of an odd metric at times. I feel like this identifying the percentage of patch vulnerabilities is a lot more effective in the postmarket phase. Um, but in the pre-market phase, it's not going to be very effective. And this does lead into when are these measures and metrics applicable, since they aren't applicable for every device and they aren't applicable for every manufacturer, which is, that's a big area of misconception I've seen in the past. If you are submitting a new device and you don't have these measures and metrics available from your substantially equivalent product, you don't need to provide them as part of your submission. You need to provide your plan to capture them. And then if you're submitting a PMA, you need to provide them as part of your PMA annual reporting, but you don't need to submit them during the initial submission. This is when you have previously fielded devices to collect these from that you provide them as part of your initial FDA submission. So, you're saying if there's a predicate device, I'm doing a 510(k), I probably have an idea of what measures and metrics should be included for my device. Um, but if there's not a predicate device, I don't need to include the measures and metrics, but I need to have a plan for those for postmarket. Correct. And even with the predicate device, considering the guidance is still fairly new as of 2023, it may be the case that you weren't collecting these for the predicate device, or you're doing a resubmission, but you weren't collecting them since it wasn't a requirement under the old guidance. So, you don't have these readily available. If you've done a submission under the current guidance and you're actually adhering to what you're supposed to in the postmarket phase, you should have these readily available to provide as part of any resubmissions or any submissions where you're using the old device as a predicate. And so this is something where, in the coming years, this is going to become a lot more important, but I feel like a lot of devices are still just saying we don't have these for our predicates or we're still a new device, we don't have this. Yeah. So, what were, I think you mentioned two measures and one metric that the FDA is requiring. Could you go over those one more time? Yep. So, the metric the FDA wants to see is your percentage of identified vulnerabilities that have been updated or patched, and then the measures that sort of feed from that are the duration from identification to a patch and then the duration from patch availability to deployment. And these are like really the minimum set. And I, I, we talk about this quite often, but just because the FDA says these are the minimum things you should do, that doesn't mean that's all you should do, because I'm a firm believer that cybersecurity goes much further than just compliance or checking a box. Correct. And even with going through that FDA submission process, it's a good look to point out any additional metrics or measures that you're covering. So, this is the bare minimum, the bare baseline that the FDA will tolerate. Anything less than that, and they're not going to accept it. Anything more than that shows that you are seriously addressing security in your device, you're putting thought into how you're going to secure your product throughout its life cycle. And that's a good look to the FDA. The FDA wants to see manufacturers thinking that way. They want to see those devices get cleared. So, while this acts as the minimum baseline, it shouldn't be thought of as the finish line for how you're addressing security. And what about, because I there's an article that came out a couple days ago where it talks about medical devices and when they fail. It says 43% of them have failed for up to 4 hours of downtime. Should that be a measure, like the anticipated amount of downtime before the system can recover? I was just, I was just curious. I was trying to tie what you said back to that article I read a couple days ago. Yeah, I think it can depend. Well, it'll depend on the device functionality. Sometimes availability isn't really a big concern from a security or safety perspective. If we're looking at an accessory device for a non-essential therapy, something for convenience, then it's likely not going to be the biggest deal. If it is something for critical care, for immediate patient treatment, that downtime can be life or death. And so that can feed into how are you addressing different metrics, different measures across different devices. Everything goes back to what's the actual level of risk. And so seeing what is your expected downtime, what is the downtime before you can expect adverse results, what is, you know, how often does downtime come up, those are good things to capture, especially when you're looking at something that really needs maximum uptime to be an effective product. Yeah. I mean, if you've got something that you've got a thousand patients relying on this device to give a diagnosis, and it goes down for 12 hours, that's probably not acceptable, right? So, we need to design the device in a manner where what is the actual amount of downtime we can anticipate and can the device recover within that measurement, like you said. Ideally, how can we prevent downtime altogether? Exactly. Not always possible, but you know we can dream. Well, often the way to prevent that is to reboot the system, you know? If the system, if something goes wrong, in the labeling it may say to reboot it, but what is the measure of how long it takes to reboot a system that is critical for patients? You know, that's something I, I consider, because there's some movie I watched the other day, and it took like an hour to reboot this system, and the whole time it was rebooting, people were unable to get care. I don't remember the exact movie, but you know, it was interesting because the guy found this workaround, rebooted it in like 5 minutes. Actually, this movie is about, um, you may have seen it, it's about these divers that get stuck under the water, and it's about the ship that lost automatic control to stay stationary. That's what it's about. So, the system to keep all the motors running to keep the ship stationary took like an hour and a half to reboot or something. And this guy had to bypass it. But the whole time these, the people that were connected on the dive platform or whatever, way down in the water, were running out of air and everything else. It was a major problem. Yeah. I think I know the movie you're talking about. Yeah. And that's especially with some breath, I think is what it's called. Last breath. Yeah. Yeah. It sounds familiar. And I, I know I've seen some movie around, I feel like there are a couple of movies with a similar, similar theme. But yeah, using an example like that, usually, you know, I know you've done a lot of diving. You see some of the equipment that they're using on these dive boats, and it looks like it hasn't been touched since the 80s. And that could be the same case with a lot of medical equipment. It's very old. You think about how long it took to boot up an old desktop system from 15 years ago compared to a modern laptop. The laptop I'm using right now probably goes from powered off to started up in 10 seconds or less. I think about, I have some older laptops that are running on, aren't using solid state drives, or using an older version of Windows, and those can take a matter of, you know, multiple minutes to boot up. When we're looking at a critical care system, those multiple minutes can be important. And if we're using a really old system, which unfortunately a lot of hospitals still are using these really old systems, they take even longer than that to boot up. Yeah, that's a good point. Like an MRI machine may take quite, quite some time to reboot. Hopefully like a surgical robot that's performing surgery, if it, if it goes haywire and you have to reboot it, doesn't take, you know, an hour to reboot, because you got, you know, open heart surgery or open spine surgery going on, and, you know, an hour is a problem. I think I'd freak out if it took a minute to reboot. I want that thing back up immediately. Exactly. Because it also affects like, you know, surgeon fatigue and other things as well, if they're using, if a surgeon uses the robot. Right. Cool. Anything else we should discuss here on measures of metrics? Yeah, I think there's one other point that feeds into these conversations that is good to look at, and that's how we're covering a risk profile. And so that's usually part of the information that'll lead into collecting these measures and metrics. But a risk profile talks about if you have one device in different environments or different use cases, how does the level of risk change? We'll say you have a device that you can use in the hospital and you can use in a home environment. Home environments are typically subject to less risk than a hospital environment. So, how are you evaluating risk differently? Might you consider a high or critical in a different environment? Things like that. And so that's a very important consideration when collecting this information, storing this information, saying how it all goes back to what's the actual risk. How can the risk change depending on how you're using the product? 100%. And that's a good, that's something I think a lot of people don't quite understand. Like if a device is deployed at a house versus a healthcare organization that's interconnected, the risk profile is very different obviously. And we'll even see a lot of devices that'll have a cloud deployment solution that's maintained by the manufacturer, and then they say, "Well, optionally, we'll also let you have your own on-premise deployment." And so then dealing with security on these two different setups can take a, take a few different considerations as well. So, I think it's something very important to tie into this whole conversation, how you're collecting this information. Awesome. Well, we're about out of time here. Any last-minute words of wisdom on measures and metrics? I think be sure you're getting a good plan to collect this information ahead of time once you're going through the submission process. If an FDA audit pops up or if you're going through a PMA submission especially, you need to have this information on hand at all times. So, as soon as you're cleared by the FDA, you should start collecting this information. You should have a very good plan in place to do so beforehand. And I would add one thing to that. Collecting the information is like step one, but being able to make the information actionable in a way to reduce the risk is the part that I think a lot of people have challenge with, right? It's just like you can measure your BMI, but if you don't change your diet or don't do anything or don't have the right action plan behind it, it doesn't really matter, right? All right. Well, thanks everyone for tuning in to this episode of The Med Device Cyber Podcast. I hope you found it useful and please tune in next time.