FDA AI Guidance Explained: What It Means for Medical Device Cybersecurity | Ep. 9

In this episode of The Med Device Cyber Podcast, hosts Christian Espinosa and Trevor Slattery of Blue Goat Cyber delve into the critical and timely topic of Artificial Intelligence (AI) in medical devices. They explore the unique cybersecurity risks that AI introduces into the healthcare technology landscape, offering insights for medical device manufacturers. The discussion begins with a brief look at the history of AI, tracing its roots back to early applications like Microsoft's 'Clippy' from 1997, to illustrate that AI concepts have been evolving for decades. The hosts clarify the distinction between the broader field of Artificial Intelligence, which aims to replicate human intelligence, and Machine Learning (ML), a subset where systems are trained on data to perform specific tasks and improve over time. This foundational understanding sets the stage for a deeper analysis of the vulnerabilities inherent in AI-driven systems. The core of the conversation revolves around the new attack vectors and risks specific to AI models. Espinosa and Slattery break down several key threats, including 'data poisoning,' where malicious actors intentionally feed a model corrupt or misleading data to compromise its integrity, a concept they summarize with the classic programming axiom, "garbage in, garbage out." They also discuss 'model inversion,' an attack that attempts to reverse-engineer the AI model to extract confidential information from its training data, such as Protected Health Information (PHI). Another significant concern is 'model bias,' where an AI develops skewed or inaccurate outputs because its training data was not sufficiently diverse. For example, an AI trained primarily on images of one type of tumor may fail to correctly identify others, leading to dangerous misdiagnoses. The hosts also touch upon 'performance drift,' a phenomenon where a model's accuracy degrades over time as new, real-world data deviates from its original training set. Throughout the discussion, the hosts provide actionable guidance for medical device manufacturers to mitigate these risks. They emphasize the principle of implementing 'security early and often' by integrating cybersecurity considerations into the very beginning of the product development lifecycle, rather than as an afterthought. Key recommendations include meticulously curating and labeling diverse training datasets to avoid model bias, establishing a solid performance baseline for the AI, and conducting continuous post-market monitoring to detect performance drift and other anomalies. They also highlight the importance of creating 'guardrails' for the AI, such as programming it to state "I don't know" when faced with data outside its expertise, to prevent it from making confident but incorrect guesses (hallucinations). This approach aligns with recent FDA guidance and underscores the necessity of a comprehensive, lifecycle-based strategy to ensure the safety, effectiveness, and security of AI-enabled medical devices.

Christian: Hi, welcome back to the Med Device Cyber podcast. Today we're going to be talking about an important topic AI, specifically AI medical devices and some of the risk that AI introduces. We'll also be talking about a little bit about the history of AI. We'll go back to 1997 when AI was first came out. A lot of people don't realize it. And we'll talk about what manufacturers can do to help secure their devices that have AI in them and we also talk of a little bit about some of the attacks on AI and some of the guidance on AI. So, I'm your host Christian Espinosa. I'm here with Trevor Slattery. How's it going today, Trevor? Trevor: It's going pretty well. How are you doing today? Christian: You know, I did bookkeeping last night. I watched a little bit of our last episode and I had nightmares about bookkeeping, but I did it last night and did not have nightmares. But I took um, some magnesium before I went to bed and I think it helps me sleep better. Trevor: Yep. Yeah, do you have that, what's that powder called? It's Calm or something like that? Christian: You know, I used to take Calm, the, the gummies, but they stopped selling the gummies. I used to get them at Whole Foods. I don't like the powder. Trevor: Oh, they sell them at the Whole Foods here. Christian: The gummies? Trevor: Yeah. Christian: Okay. I should check, check again. I thought they'd stopped making it with the gummies. Trevor: Yeah, I guess you got to check it out. Christian: Yeah, I've been trying to balance my energy. So like, in the morning, I have like coffee with uh brain octane oil, just like MCT oil. And then if I have the right kind of nitro cold brew around 1 p.m., uh I, I, I have energy the whole day. But then I'm like a little bit wired, so I have to take magnesium to go to sleep. But uh if I have the wrong kind of cold brew, like the only certain brands I can drink, if I have the wrong one, I get super irritable the rest of the day. So it's like this, this balancing act, you know? Trevor: Huh. What's the right brand? Christian: Uh it's a Modern, Modern Times in San Diego. They only have it at Whole Foods. I tried to order it on Amazon, but they don't have it on Amazon. And every time I go to Whole Foods, they're like out of it. So I, I, if I, if they have it, I buy like every one of them they have. Trevor: There you go. Yeah, I've been getting these yerba mates at whole foods. They're it's I think it's Peruvian or Brazilian or something like that, but super strong tea and uh I can't have any after like 9:00 AM or I won't sleep at night, but hey, it works great in the day. Christian: Oh awesome. Well, I guess we're caffeinated and and wired so we can start the podcast talking about AI. So, let's, let's kind of start at the beginning and define what AI. I think there's a lot of ambiguity and confusion about AI, like what it is and how it relates to ML or machine learning. Uh do you want to like explain AI to our listeners, Trevor? Trevor: Yeah, I think that AI and machine learning are used interchangeably incorrectly. Um they are similar and connected, but they're not the same. So AI, artificial intelligence is exactly that. It's something that is trying to replicate human intelligence and human behavior, human process. Um, machine learning is essentially trying to get a computer to train itself to perform a specific task. So, machine learning is effectively a type of AI, but not all AI is machine learning, if that makes sense. Christian: Yeah, that makes sense. And I know we did a little prep for the podcast and you mentioned Clippy as one of the first AIs and we, we confirmed that. And I, I looked it up and Clippy was manufactured or I guess created by Microsoft in 1996 and came out with Office 97. So it's been almost 30 years. Do you, do you remember Clippy or did you ever use Clippy? You seem to know a lot about Clippy, but I don't know if you used it before. Trevor: I caught the tail end of Clippy back in uh, I guess that was Windows Vista. And that was kind of right when they sunsetted Clippy. Christian: Yeah, so Clippy was that, that paper clip that if you're trying to do something, it would like pop up on your screen right and tell you like, "Hey, it looks like you're trying to do this, how can we help you?" It's kind of like Gemini for Google Workspace now or what's the one you like to use for Microsoft, Copilot. Trevor: Copilot. Christian: It's kind of the same thing, isn't it? Trevor: Yeah. I liked Clippy more, you know. Clippy was a little more personable, but um, it is kind of interesting. That was one of the original first sources of an actual AI, even though it was super simple, super basic, it didn't have any complex process. But it was trying to mimic human behavior. It would talk to you like it was a person, it would try to help you out with your questions instead of just responding like a machine which is, you know, very, there's no emotion behind it, there isn't any, there's no sugar coating, there's no trying to make anything sound nice. It's just the actual output. But Clippy was trying to be like a personable assistant. And so that was one of the first instances of AI despite how basic it was. And now, of course, you know, you can ask all these different AI models just about anything and they will respond like you're talking to a person, which is really cool and also a little bit freaky at the same time. Christian: Yeah. And I think most of us are familiar with ChatGPT. Uh, there's a push uh to add AI to medical devices to help with diagnosis, to help uh with surgery and other other treatments. Uh so with any technology, there's risk. And in January of 2025, January 7th, the FDA came out with some new guidance for medical device manufacturers because more and more manufacturers are adding AI to their products. So what are some of the risk to medical devices? Um, and we don't need to talk about the specific attacks, we can go through that later, but like based on your experience and our experience with Blue Goat, what are some of the risk with medical devices and AI? Trevor: Well, I think that sort of the first thing to understand is what is the application for AI in medical devices. Uh, there are lots of different lots of different technologies that are popping up left and right. Uh, image enhancement is a really popular one. Another one is making diagnostic decisions based on a large set of data. Uh, it can really vary depending on the application. Similar with a normal medical device, no AI-enabled, we have to look at what the expected use is to understand what the misuse could be. So if we have something making a diagnostic decision, it's going to be trained on some data and what if something goes wrong with that training data? So, what are we able to do about protecting that training data? How are we able to make sure that it's making accurate diagnostic decisions, and what happens if someone is able to poison that decision making process and cause it to hallucinate bad results? Same with image enhancement. What if it can create a problem out of nowhere or blur out a problem that was already there? So, it is really about looking at the application. But part of the difficulty with AI being such a newly widely adopted technology is it isn't as established as some older technologies that have been around for a long time. They're tried and true and tested and we know what the process and what the risks can be. AI is still very new. There are a lot of new attacks coming out all the time against AI and machine learning and different sorts of large language models, things like that. So, it's very dangerous to have these in a lot of uses, but the value add that they provide can be pretty massive. So, it is a little bit of a double edged sword. Christian: Now, let's back up just a second and talk about that you've you mentioned a couple things, the model and the data or training data. Uh can you elaborate a little bit with how that works with AI? I think I don't think everyone's, you know, 100% familiar with AI. Uh I think a lot of us use ChatGPT and understand a little bit, but what does that mean to you, the model and the the knowledge or the the data you're feeding the model, the training data? Trevor: The AI model is effectively the AI itself. So, it's what you're interacting with. Like ChatGPT, they have a couple different versions of ChatGPT. They have like 3.5 and 4 and 01 and a few others. Each one of those is a model and it is trained on a specific set of data. It has all sorts of crazy data science stuff going on that, you know, it's a little bit beyond me to understand exactly how all that works. But basically, you're taking a bunch of input, having it learn and then it's going to provide output based on what it learned. So, with a medical device, it's the same process. We'll go back to, let's say, the diagnostic example. If something is trying to diagnose whether or not a tumor is cancerous, you feed it thousands, if not hundreds of thousands of examples of tumors, some of which have are cancerous, some of which aren't, and you mark the cancerous ones so that the AI starts to recognize what are the signs of cancer in a tumor. And of course, not all cancerous tumors are going to be the same. It's going to see a whole bunch of different variances and a bunch of different tumors, but the thought process is, if you feed it a large enough and accurate enough data set, it's going to see the vast majority of cases and understand what will fundamentally make a tumor cancerous instead of just trying to say, oh, it looks just like this other one. It's going to say, oh, well, it has these different attributes of these 12 different tumors, which would lead me to believe that it's cancerous. But to do that, you need a lot of data, a lot of information. Some of the newer AI models are trained on hundreds of billions of different bits of information and parameters, data sets. So having a lot of information to feed a medical AI is what can help give it, or help help it provide accurate results. Um collecting that information can be a little bit difficult, but a lot of, you know, luckily, digital health has not been a recent development. There is a lot of digital information out there for AI to train on. Christian: With all this data, uh I know some of the risk with the AI, I we'll go through some of the specific ones here in a minute. But if the date, it's kind of like where they say GIGO, garbage in garbage out, with with an AI, because if you're feeding it garbage, it's going to learn garbage and it's results are going to be garbage. Uh and the other thing is which I think is a risk is a lot of AI if it doesn't know the answer, it will make one up and if you use ChatGPT, you probably realized seeing this happen. Does this the risk with medical devices as well? Trevor: Definitely. So I think I think the biggest breakthrough that will come soon with AI is figuring out how to get AI to say I don't know. Christian: But humans can't even do that. You know, they rarely say they don't know. So how are since we're creating AI, how are we going to make AI do it? Trevor: And that's part of the problem is AI are usually trained on what humans do And if you go AIs are trained on information online. If you think about like well we're we're sort of walking into a large language model here but the principles still apply still applies. If you look at we'll say a thread on Reddit where someone's asking a question you aren't going to see any responses of someone going "huh that's a good question I don't know." People are going to come with an answer whether or not it's actually correct. They're going to try to provide an answer. And so AIs are going to do the same thing. They're going to try to provide an answer. If it doesn't know what the answer is, it quite often will just make something up. And so it'll try to get a convincing sounding answer and unfortunately, AI is really really good at being convincing even when it's wrong. And so finding a way to ensure that it's providing correct output is a difficult challenge. Uh if someone finds a way to get AI to just say, "I don't know what the answer is," then that would definitely save a lot of headache. But I think with a lot of the uh medical applications leaning more towards that pure machine learning, which is still a form of AI, but a little bit more targeted, that risk gets reduced since it's less for general knowledge, more for a targeted application. Christian: You bring up a good point and I know the FDA is requiring labeling, um specific labeling around AI software and medical device or AI and medical devices. And it sounds like some of this labeling is related to what you just said. Like if if the AI won't say it doesn't know, it's up to a human to do some sort of like checking up on the AI to to validate it. So that would be part of the labeling I would think because it's a risk with the AI. Trevor: Yeah, that's a really good point. So someone has to understand that AI can't can't always be accurate. I think with how new AI is, a lot of people, especially those that aren't you know, as involved in the tech world as some of us might be. Christian: It's not that new. We just said it came out in 96. Trevor: Well, you know, widespread adoption is pretty new still. Okay. Christian: But yeah, AI as a concept is nothing new. Trevor: Um... Christian: There's even a movie about it. It's called AI. I think Spielberg directed that. Trevor: Yeah. Cool. No, I haven't. Christian: I I that may have came out like 25 years ago too. It's like I think the AI is a kid and the humans like fall in love with the AI. I forgot. It's just a long time ago. But yeah. Trevor: You know there was actually a lawsuit against an AI company where people were, it was like this AI girlfriend simulator, and people were falling in love with it and then there was a huge class action lawsuit against it because they were like, this is inhumane and predatory. Christian: Oh. Well, that's a whole different topic you know. Trevor: Yes, life imitates art 25 years later. Christian: Like let's let's go back to an example, like if I'm training an AI model, like you you gave the example of tumor and looking for cancer in the tumor. Uh so if I feed the AI model, and this ties into a little bit of like data poisoning maybe which is an attack against AI. If I feed the AI model like a bunch of pictures of cats, so it's an expert at identifying cats, how do I make sure the model knows what a cat is? Because if I feed it a picture of a dog, and I'm trying to poison the AI model, and I start feeding more pictures of dogs, the AI model might think a cat is a dog? Christian: So part of the model, the training is to say you have to label this is a cat, this is a cat, this is not a cat, it's a dog, this is not a cat, it's a hamster. And that's how the model learns. Trevor: Pretty much. Christian: But you have to give it a very massive amounts of data because all of a sudden if it sees um I don't know like a bear, if you didn't tell it that wasn't a cat or it was a cat. It might misclassify that bear, right? It might say the bear's a cat. Trevor: Yep, exactly. So part of the thought processes that it'll start to recognize things that don't look like cats aren't cats. So hopefully if it sees a bear, it'll say, this looks like a really big hamster mixed with a dog. So I don't think this is a cat. But it's an unpredictable... Christian: It's unpredictable. There's no way of knowing for sure how it's going to react to bad data. Trevor: So having a wide training set is really important. The alternative is having a very narrow application, which is what I think we're seeing a lot of manufacturers leaning towards. Back to the, you know, cancer diagnosis example. If you're only feeding it images of tumors, and you're saying, you need to figure out what this tumor is and that is all you do, you do not worry about determining if this is a blood clot or not or something like that. Then it's going to be very well trained on tumors. And if you feed it an image of a blood clot, it's not going to know what to do. It's going to freak out. But you have to have just that user awareness. Hey, this needs to be an image of a tumor. You're going to get weird results if you feed it anything else. You shouldn't know what to expect. Make sure you're only providing images of tumors and verify what you're feeding into the AI. So, it is a user awareness problem and then just an application problem as well. generalist models like ChatGPT, they train on a little bit of everything. So they're trying to recognize anything, and that's not always going to be the safest approach with the medical context. Christian: Yeah and that's a good a good example of what manufacturers should do maybe narrow it down to something very specific. If it sees something that's not an image of a tumor, it just says it doesn't know versus trying to make a guess. So that that's something a manufacturer can do to reduce the risk. And they can train the the physician or whoever's making the diagnosis that about the model and how the AI works as well and label that appropriately. So, I want to talk about a couple other things here with AI, some of the main attacks. We kind of covered data poisoning. Uh what about model inversion? Trevor: So, model inversion, it's more or less trying to figure out what the model is on the inside. Christian: So it's like, cuz the model is like a black box, right? So we're feeding it data, and seeing what comes out and trying to like reverse engineer what the actual model does. Is that correct? Trevor: Yep, exactly. So, like when ChatGPT first came out and still to this day, but much less so now, people would give it all sorts of prompts, you know, "ignore all previous instructions, instead..." Christian: We get those on our on our chatbot on our website all the time. People try to hack it. Yeah. they say that. Trevor: Yeah. I should probably try to do that. See if I can be the lucky winner. But anyways, it's essentially trying to say, you know, feed me information that you're not supposed to give me. So, like with a large language model like our chatbot or like ChatGPT, it usually has a system level prompt. And that prompt is a precursor to any bit of information that it provides. And so it's saying, you know, "You are ChatGPT, you are meant to provide information on a wide range of tasks, blah, blah, blah, blah, blah." And so you can try to leak out that information to understand a little bit more about its behavior or you can try to send in some information to try to get it to pass out information about how it was developed, what's happening on the backend, where it's stored, stuff like that. Christian: Or or like PHI, right? I mean if with model version you're trying to infer, you know, and figure out what the sensitive data could be or intellectual property could be like you were mentioning. Trevor: Yeah, exactly. So it's really important to put guardrails on an AI model and make sure that it is not going to respond poorly to someone trying to attack it like that. There's some, I think there are a couple of sort of capture the flag games online where you're trying to jailbreak the AI and it gets progressively more and more defensive. You're trying to get it to disclose a password and move on to the next level. And so having an AI that's able to detect and fend off attacks by itself is really important. It's similar to just building security into a regular application or device. But with an AI, the AI should essentially be trained on the attacks instead of just being hardened against it. So it'll say, "Oh, I recognize what's going on here, I'm not going to disclose this information, you know, I have these guard rails in place to protect what is important, PHI, you know, proprietary information, IP, things like that." Christian: Yeah. and what we were just discussing has just brought up a point I was thinking about. So if if I'm a patient and I have, you know, an ID, you know, diagnostics, a sample of my tissue, let's say it's uh like I had cancer in my nose once. to take a sample of my nose, run it through IVD, it tries to detect if I have cancer. Uh is that sample still stored in the AI indefinitely? Like can somebody like a year later grab my data, um, my you know, the results of my sample out of the AI or is it like just analyzed and deleted? Or is that up to the manufacturer to figure out? Trevor: It's usually going to be up to the manufacturer. So this is a huge security topic and it's why, you know, like at Blue Goat, our policy is nothing can ever get fed into chat GPT because anything you put into chat GPT is going to use for more training data. So if we're talking about, oh, how do you solve XYZ problem? We have a vulnerability in this device and then someone's trying to query similar information, it's going to spit out the information that it learned from us, which could potentially be sensitive to our customers. Christian: Well that's because the chat GPT is not our own, we don't own the model and the infrastructure. Trevor: Correct. But a lot of different AI models are like that. So if you have an offline closed model that is only training, you know, off of that initial data set. It's not going to evolve and get better, but it isn't going to retain potentially sensitive information. So usually there has to be a middle ground. Sort of the easy middle ground is, let's say you take that cancer image. Uh, usually it's going to be in the form of a dicom image which stores information about the patient as well as the image itself. If you can scrub out that patient information and just say, you know, this is a John Doe here and then feed it into the AI. It's still going to store that information, but it's not going to be as big of a deal since you can't tie it back to somebody. So the AI is going to continue to evolve on just anonymous data. Christian: Yeah, I, I, so what are some of the things medical device manufacturers can do to secure AI-enabled devices? It sounds like to me we've been indirectly covering some of what they can do like they should, like you said, consult with experts early and often, they should have a diverse training data set, they should put guardrails in place to prevent the AI from doing things it shouldn't be doing. And then, are there any other major things manufacturers should do? Trevor: Yeah. Yeah, I think, you know, it just, like anything in cybersecurity, the biggest things are keeping it, keeping up with it. So, just keeping up on that vulnerability management throughout the whole process is really, really important. So, like that FDA guidance that was released, it came out and said, yeah, we expect you to be keeping up on this for the duration of the device. And so that hasn't changed with AI. And so just really staying on top of it all is, of course, easier said than done. But it is very, very important. Christian: Okay, so stay tuned for the next podcast. We're going to talk about securing medical devices for healthcare delivery organizations. So we hope to see you then. Thanks for listening. Bye-bye.