Episode 67 · August 5, 2025 · 23m listen · 3,769 words · ~19 min read
Understanding Cybersecurity Measures and Metrics for Medical Devices | Ep. 31 - Full Transcript | The Med Device Cyber Podcast
Read the complete, searchable transcript of Episode 67 of The Med Device Cyber Podcast - expert conversations on medical device cybersecurity, FDA premarket and postmarket guidance, SBOM management, threat modeling, and penetration testing.
Prefer the listening experience? Open the episode page for the synopsis, key takeaways, topics, and Apple / YouTube listen links.
Episode summary
In this episode of The Med Device Cyber Podcast, hosts Christian Espinosa and Trevor Slattery of Blue Goat Cyber tackle the often-misunderstood topic of measures and metrics in the context of medical device cybersecurity and FDA submissions. They begin by establishing a clear distinction between the two terms, which is critical for manufacturers preparing 510(k) or PMA submissions. A 'measure' is defined as a direct, quantifiable attribute of something, such as the time it takes to apply a software patch or the total number of security incidents that have occurred. In contrast, a 'metric' is a calculation derived from one or more measures, typically expressed as a percentage or ratio. For instance, while the time to apply a patch is a measure, the percentage of systems patched within a defined timeframe is a metric. This fundamental difference forms the basis of the entire discussion, as the FDA has specific expectations for both.
The hosts then pivot to what the FDA specifically looks for in premarket submissions. The FDA is interested in how a manufacturer collects the necessary figures to produce these measures and metrics, covering the total product lifecycle. The core of the FDA's focus is on vulnerability management. Espinosa and Slattery outline the three key data points the FDA requires. The primary metric is the percentage of identified vulnerabilities that are updated or patched. This is supported by two critical measures: the duration from the initial identification of a vulnerability to when a patch becomes available, and the subsequent duration from patch availability to its full deployment across all fielded devices. They clarify that for brand new devices without a predicate, manufacturers are not expected to have historical data but must present a comprehensive plan for how they will collect this data post-market. For devices with a predicate or those undergoing PMA annual reporting, the expectation is that this data will be available and presented.
The conversation also touches upon the importance of context and a risk-based approach. The risk profile of a device can change drastically depending on its environment of use—a device in a connected hospital network faces different threats than one used in a home environment. Furthermore, measures like device downtime and recovery time are crucial, especially for critical care systems where availability is paramount. A surgical robot that takes a minute to reboot after an issue is far more concerning than an accessory device with the same reboot time. The hosts conclude by reinforcing that while the FDA outlines a minimum baseline for compliance, a robust cybersecurity posture requires going beyond simply checking a box. The goal isn't just to collect data but to make it actionable, ensuring that the information gathered is used effectively to reduce risk and enhance patient safety throughout the device's lifecycle.
Key takeaways from this episode
Measures and metrics are distinct concepts: a measure is a direct quantification (e.g., time to patch), while a metric is a calculation derived from measures (e.g., percentage of systems patched).
The FDA specifically requires manufacturers to report on key measures and metrics related to vulnerability management in their submissions.
Three crucial data points for the FDA are: the percentage of vulnerabilities patched, the time from vulnerability identification to patch availability, and the time from patch availability to deployment.
For new devices without a predicate, a detailed plan to collect post-market data is required, whereas historical data is expected for devices with a predicate.
A device's risk profile, and therefore its security requirements, can vary significantly based on its intended environment of use, such as a hospital versus a home setting.
Beyond vulnerability patching, measures like device downtime and recovery time are critical, especially for life-supporting or critical care systems.
Simply collecting measures and metrics to meet FDA requirements is insufficient; the data must be made actionable to genuinely reduce risk and improve security.
Meeting the FDA's minimum requirements should be seen as a baseline, not the finish line, for comprehensive medical device cybersecurity.
Full episode transcript
Page 1 of 5· Paragraphs 1 - 26
Christian: Hi, welcome back to the Med device Cyber podcast. Today we're going to talk about measures and metrics, a commonly misunderstood topic. Uh measures are different than metrics and the FDA is looking for very specific things in a 510k or PMA submission.
Uh so what we'll cover these those in enough detail where you can have a decent understanding of them. And I'm Christian Espinosa, your co-host here with Trevor Slattery. Uh how you doing today Trevor?
Trevor: Doing good. Getting some nice weather right now.
Christian: Are you in Sedona?
Trevor: I am in Sedona. Yeah.
Christian: You know, I I I did I used a AI once to create a snippet of one of our pod podcast and it mentioned the guy with the the the bull in the in the background, it referenced you as that and that's where you are. I see the bull picture in the background. Is it a bull or what is that?
Trevor: Uh, I think it's a yak or a bull. Something like that.
Christian: Yeah, it was funny AI referred to you as the the young man with the the bull in the background or something like that.
Trevor: There you go.
Christian: Cool. All right, so I think it's important to start with the definition of a measure and then the definition of a metric.
So a measure is just like a tape measure, right? You it's like a quantifiable attribute of something. Like how long did it does it take to apply a patch? How many incidents occurred? That's a measure.
Whereas a metric is some sort of calculation derived from a couple measures typically. Usually usually a percentage.
Uh, you you have anything to elaborate on those two?
Trevor: Yeah, so essentially what we're saying is that the metrics are derived from measures. And so they go hand-in-hand, but they are distinct, separate items.
Christian: Yeah, so an example would be a measure how long does it take to apply a patch. A metric could be like your patch management efficiency. The percentage of systems patched within a defined time. So that's a percentage, the metric versus the duration, which is the measure.
Trevor: Spoiler for what we're going to talk about when we say what the FDA wants to see.
Christian: It's a spoiler?
Trevor: We'll get there in a few minutes, I'm sure.
Christian: All right. So now that we understand uh a measure versus a metric. What does the FDA want to see in terms of measure, we'll start with measures first.
Trevor: Well, I think before we get into the specific measures and metrics, we talk about we should talk about how the FDA wants to see us collect the figures that go into this.
So, the measures and metrics are these quantifiable figures. How long does it take to apply a patch? How many vulnerabilities are we remediating? And they want to see now that we're understanding, you know, these different measures and metrics, how are we getting the information to provide this. And so the first thing when we're looking at the required FDA documentation, they combine the total product lifecycle considerations into the measures and metrics when you're looking at the headings throughout FDA guidance and through the e-star submission process.
And so, the first area that the FDA wants to see you covers, how are you identifying, addressing, and mitigating vulnerabilities?
Which it's essentially three separate questions. But it should talk about your vulnerability management process, any tools, any processes that you're using for collecting these vulnerabilities, and then how you're tracking it once you've identified them.
Christian: Well, I think the the measures that go with that are once you've identified a vulnerability and you've verified it actually is a real vulnerability, like what's the risk associated with it? And then if the risk is critical, how quickly should that be passed versus if the risk is low, as an example. That that duration would be a measure.
Trevor: Right. Yeah, and that feeds directly into we can, well, we can talk about what the measures and metrics are and then work backwards from how we get to these figures. So, the FDA wants to see, as one of the first points, what is your percentage of identified vulnerabilities that are updated or patched? And so, especially with a complicated device, you can get in hundreds if not thousands of vulnerabilities at a single time. And so seeing these come in, how are you triaging them based on severity?
Like you mentioned, it's going to be more important to handle these critical vulnerabilities as opposed to these low vulnerabilities. What is your timeline for remediation and just general patch cycles. And so if you're following sprints, which a lot of engineers and manufacturers like to follow that development cycle. What are your sprint cycles looking like? How many vulnerabilities are you remediating per sprint cycle? You should be looking at close to as 100% of critical vulnerabilities as possible. And then you can start tapering off as you go down lower in risk.