What Happens When AI in Medical Devices Make Mistakes? | Ep. 40

About "What Happens When AI in Medical Devices Make Mistakes? | Ep. 40"

"What Happens When AI in Medical Devices Make Mistakes? | Ep. 40" is episode 49 of The Med Device Cyber Podcast, published on October 7, 2025. Host Christian Espinosa and the guest dig into the practical realities of shipping and maintaining secure connected medical devices - the kind of detail you only get from people who have done the work.

While the AI-generated summary, key takeaways, and full searchable transcript for episode 49 finish processing, the audio and video are already live on YouTube, Spotify, and Apple Podcasts using the links above.

Explore the full episode catalog to find more on FDA premarket and postmarket cybersecurity, SBOM management, threat modeling, and medical device penetration testing.

Hello and welcome back to another episode of the Med Device Cyber podcast. I'm your co-host Trevor Slattery joined by our co-host Christian Espinosa. And today we're going to look at something really interesting. What happens when AI gets it wrong? In the medical context, this can mean someone's life is on the line. So AI making a decision trying to step in as the place of, you know, diagnosis, therapy, provision is a little bit of a dangerous territory. Of course, AI provides a lot of great innovation, but we want to make sure sure but it's handled safe safely. How are you doing today, Christian? Christian: I'm doing well. I'm doing well. It's uh, it's Thursday today, I think it is. And it's uh hot in Phoenix. I'm still recovering from last week. Last week I was in New Jersey doing a Formula 4 uh race course. Trevor: So what's the difference between the different, I guess formulas going up four to one? Christian: Well, Formula 4, Formula 3 and Formula 2, the cars are all the same, so it's purely up to the driver. as far as skill set. Formula one, the cars each team constructs their own cars um based on a set of specifications from the FIA. So the cars are different, like the team cars are different. Some are faster, some are slower, some are faster than straight, some are faster on turns and the drivers are different. So from a driver perspective, it's an equal playing field in like F4, 3 and 2, F1, you have the manufacture of the car that comes into play as well. And then F4 cars aren't quite as fast or powerful as F3 or F2 or F1. They they go up in order from, you know, one is the fastest obviously. And also the most expensive. Trevor: Yep. Yeah, we're uh unpacking into our new apartment right now and I can't see where it is, but we have a signed photo from Charles Leclerc, which I'm gonna put right back here on the wall. Christian: Cool. Um, yeah, so I'm I'm hoping to do a F4 race at some point also. Probably uh maybe later this year or early next year. Trevor: That'd be pretty cool. Any uh preferred location for the track? Christian: Whatever track I'm I'm I'm good with. I I I did that course at the New Jersey Motorsports Parkway. Um so I'm very familiar with that course. I watched a couple races there on F4 on that course. I feel like I could kick ass at that course, but a new a new course, you know, I have to learn the course and all that stuff, which is part of the challenge. Trevor: Well there you go. Maybe it'll be back to New Jersey for your uh championship trophy. Christian: Well, I'll be pretty ambitious to win a championship on the first race, but we'll see. Trevor: Awesome. Well, let's jump right into some of these AI considerations. So, this comes up as a little bit of, you know, some existing guidance but with some changes here. So, right now what we're looking at is the EU AI act and then some of the new guidance pushed out by the medical device coordination group in the EU. Uh now while this is a little bit separate from our standard focus on around the FDA problems, FDA considerations, it all ties in to medical device safety and of course we do handle a lot work within the EU and IVDR so it is especially relevant. I know we're looking at just how to bring in a little bit more security in AI systems which the EU has been more on top of than a lot of other agencies I would say, making sure that AI is pushed out securely, safely and their regulations in place to try to add some guard rails to what can be in some cases a little bit of a risky technology. Christian: Yeah, and I think it's good to provide a little context here with a real case that's an active case going on right now. Uh and I think you're familiar with this case. There was a medical device manufacturer that has a a mental health application that has an AI-based chatbot. So from a mental health perspective, it's supposed to, you know, help with the mental health of the patient. And the case that is being examined today is there was a suicidal patient that over the course of several months was interacting with this AI-based mental health chatbot. And after three months, for some reason, the AI-based chatbot told the patient, well, you might as well go ahead and kill yourself. Uh, and the patient ended up killing themselves. So the patient's family is now suing the uh, manufacturer of this product. Uh, and this brings up a whole like line of questioning. Like, are we really to the point where we should be using AI in medical devices in this context? Because it seems like we haven't, at least from my perspective, we don't have a very good handle on all the edge cases for AI and medical in this medical device space. What are your thoughts? Trevor: I agree. I I think that the technology is too new to understand exactly where it can go and what it can do. And it's going to take a little bit of time for us to fully research and understand what are the implications of using these these different technologies in a medical context. Now, health care, medical devices, anything involving patient information is very tightly regulated and very tightly controlled compared to other industries. I think that using a broad, wide technology such as AI that is more generally intended for a general purpose. You think of something like chat GPT. It's meant to be a general knowledge source sort of replacing a search engine. That's what we're really trying to do there. Maybe add some automation in. But it goes far and wide and it isn't really regulated what it can do. And so trying to fit it into a regulated industry is very difficult. And I think that there are going to be some guard rails that need to be put in place onto existing AI to make it fit into this framework or we're going to see the creation and this is what we're already seeing for a lot of different use cases of purely medical AI, which is difficult with a chatbot and a little bit more applicable for say like image enhancement or diagnosis, something around that nature. But I think that we need to have it tailored for this exact purpose, built with security in mind, built with safety in mind most importantly, and ensuring that we aren't getting a risky product or a risky technology into a very high-risk industry. Christian: When you're talking about having, like, pure AI, uh, that implies that the AI is making the diagnosis for, like, let's say image enhancement, looking at your vascular system versus a physician taking the output from the AI and making the diagnosis. Is that what you're suggesting? Trevor: So that is where it becomes very tricky. If you have a system, no matter what it is, if it's AI enabled or not, that is making a diagnostic decision, that is very tightly controlled compared to, um, like clinical decision support tooling. So a clinical decision support is meant to add context, provide additional information to a physician to make their own decision, but it is not meant to diagnose or treat any illness, disease, anything. When we draw that line, uh it becomes pretty clear to see where AI can get a little bit complicated. If AI is providing additional context, that burden is still going to be on the clinician to ensure that it's providing accurate information. They are making the best most informed decision for the patient. If the AI itself is making these decisions. If AI is trying to diagnose or prescribe, then it is going to get quickly very complicated since the liability is going to be a bit more on the AI side of things, on the manufacturer, on the healthcare delivery organization for integrating that AI, on the regulators for approving that AI. There are so many different things that can go wrong in that context and so it needs to be done very well and that can be hard sometimes. Christian: So with this AI-based mental health chatbot where we had the issue with what what is the solution for that? We're just two immature to even be using that or we need a physician like approving the AI messages to the patient. You know, like if we just look at that particular case, which is something I'm sure the developers of this product didn't think of this case uh and it's an ed- edge case they just simply didn't think of. Uh like what is the solution for something like that? Trevor: Well, we'll start with, you know, if they aren't thinking about this, that's going to be that's effectively going to be just neglectful of what could happen to the patient. Under such a sensitive topic as, you know, mental health treatment, therapy. Obviously when a human's doing that, a human an's able to make some more informed decisions. They're able to think with emotion and, you know, and logic and go down that path. It's not going to be very likely that you're going to see a human therapist take those same actions. When an AI is trying to make these decisions, it doesn't understand the context that it's sitting in. It just understands what it's trying to do and what it was trained on. If this is a general usage chatbot, there's no way of knowing exactly what it was trained on. And so I think this will and we've talked about this before, garbage in, garbage out. If the AI is not properly trained, it does not have the right guard rails that it learns and it doesn't know what's appropriate and what's not to discuss. It doesn't know what it can use and what it can't use, then it's going to produce very strange output. Now having said that, even if it is trained well, it is ultimately up to the manufacturers and up to the developers to ensure that they are looking for these edge cases. Tying into how this will work with cybersecurity. I like to use threat modeling as an example. So threat modeling is our hypothetical exercise what can go wrong. We think about if you look at, take my apartment for an example, I can say, well, what if someone leaves the window to the fire escape open? What if someone leaves the front door open? What if someone in my building props open, you know, the front door which normally has a lock on it and then someone can get in. You're going through these hypothetical with threats against a device, against a system and that should be applied to AI as well. What if the AI doesn't understand what someone is saying and gives bad advice? What if the AI gives malicious advice? What if the AI, you know goes off the rails completely and isn't trained properly with good data. There are a lot of things that need to be considered and that burden is on the manufacturers. Christian: That's a good point, that's a big burden. Uh, I remember it was probably like 15 years ago when AI was kind of first started to be adopted, I was at a hacker conference going over the implications of AI and one of the scenarios I came up with was if we have an autonomous driving car and it is and it's in a situation where it has no choice but to run into something, like it can't stop quick enough. And there's a car in front of it with a woman and a and a baby in the car that it can detect, and there's a elderly woman walking on the road, which one does it hit if it has to hit one. You know, it has to make these like moral decisions which means a human had to program those decisions, which which introduces a whole another set of challenges, right? And I never thought about those perspectives, but we're talking about like threat models and all the use cases, like that is a scenario that may occur, right? Trevor: Totally. Uh on that self-driving car example, I recently saw a video of a self-driving Tesla where it was going through a crowded street and a woman, a woman fell right in front of the Tesla as it was going at a speed that would have caused pretty significant damage. The car was moving fairly fast, not crazy fast, but you know, think 20, 25 miles an hour. And the self-driving Tesla saw the woman fall in front of the car and so it jerked away into the other lane and caused a direct head-on collision. Now, obviously at that speed, it's unlikely that someone's going to die from a head-on collision, but they can still get hurt. They can still get bruised, they might even break a rib depending on, you know, if they hit the steering wheel hard enough. And so there was certainly some harm caused. Now, the AI, in my opinion, did make the correct decision if it ran over that woman, it would have probably caused far more harm than this head-on collision. But there are a lot of situations that you brought up that aren't going to be so black and white that'll be a lot harder for the AI to make a decision. What if the cars were going 75 miles an hour and it caused that head-on collision, it could have caused deaths to both the driver and whoever was in the other car. So how is the AI going to know what is the correct decision at this point? Christian: Yeah, and it's good that the EU is pushing this AI Act. It's been out for a while actually. uh and it does classify medical devices as high risk. uh and high risk is like these things we've been talking about through used for diagnosis and other like the mental health app, I would consider high risk as well. And just recently, a couple months ago, the MDCG came out some new guidance. Uh I don't know if you're familiar with the new guidance, but it's trying to basically, and I've got the guidance over here, uh, it it the MDCG is the Medical Device Coordination Group, but it's looking at the EU MDR, the medical device devices regulation, the IVDR, In vitro diagnostic medical device regulation and the artificial intelligence Act and and looking at the intersection of all three of those because we are seeing AI become predominant in medical devices, maybe not predominant, is becoming fairly common now. Uh and a lot of IVD, in vitro diagnostic systems which take a sample of your tissue and make some sort of diagnosis on it, uh about what bacteria you have, what disease you have, and then recommend a course of treatment. These are starting to have AI as well. And given some of the scenarios we just went through with AI, there can be some major ramifications if the AI is not implemented properly and that's why I believe the MDCG is come out with the this document and we're starting to like increase the awareness like, hey, maybe we should really consider the implications for AI in the medical device space. Trevor: Totally. And I think it's great that we're getting a little bit more granular with this. So, the AI Act would classify what the AI level of risk is. And so minimal would be something like a spam filter where there's not really that much that can go wrong. Worst case scenario, you get a few more emails that you don't want. Limited would be a chat bot like Chat GPT or consumer electronics would fall under that as well. So like a toy. Um and then high risk is going to be like hiring decision software, medical devices. This is where I think we needed some more granularity which is what MD or MDCG is trying to provide by assigning the AI act a little bit more granular. We look at the different classes of device risk and it doesn't make sense that say an oxygen pump is going to have the same level of risk as a surgical robot with assuming there's AI enabled in both of them. So I think that breaking that out a little bit more is important. And then finally, the AI Act talks about unacceptable usage for AI. So like social scoring, predictive policing, things like that are banned all together. And you know, they're not banned here. I don't think they're banned in the United States. I think some countries are still doing are doing that actually. No, the United States you can do whatever you want. AI is the wild West. Um but they're trying to, you know, prevent situations like, oh, what was, what was that movie where, you know, that you were assigned some score on how likely you were to commit a crime and then, the Minority Report. Christian: Oh, is that that was in that movie? That was a long, that's an old movie with Tom Cruise, right? Trevor: Yeah, yeah. It's an old movie. They're trying to prevent situations from that and they're also trying to prevent people from making Skynet for obvious reasons. And so, a little bit more safety classification around there is never going to be a bad thing. But I think that once we're in this high risk category, that's when we need to break it out a little bit more. And so I mentioned some examples on different devices are going to have different risk profiles and we need to have a little bit more, I guess, understanding of that in this context. Christian: Yeah, so it sounds like if I'm a med- Medical Device manufacturer, I've got a and I have AI as part of my product, I've got a lot more to think about than just uh meeting the requirements for a 510(k) checklist from a cybersecurity perspective. I actually had to think about all these AI specific risk if I'm going to do my due diligence. Or or do you think the the procedures are in place to enforce that or is it really up to the manufacturer? What do you, what do you think about that? Trevor: I think that it's, AI is always going to be a little bit of a shared burden. So, the regulators need to make sure that they're doing their due diligence to vet these devices appropriately. Manufacturers need to build these devices appropriately. Healthcare delivery organizations need to have their vendor control and supplier control process in place and the users need to be informed of what risk they may be exposed to. So it goes all the way down the chain and this is going to apply to any type of medical device. But with AI, especially, um there needs to be in my opinion, the US regulations are a little bit too thin. There's some draft guidance out right now for AI and medical devices. But for the most part, it is pretty much the wild West. manufacturers, we're seeing manufacturers use AI in a massive range of applications and a lot of them are truly groundbreaking, amazing technologies, but of course we're introducing risk through that process. Christian: Yeah, and it's also interesting from a uh just a the the the ecosystem between investors and innovators, it seems like I talked to a lot of investors and innovators. Innovators will sometimes say that they have AI as part of their system just in order to get funding. It seems like there's more of a propensity for an investment to fund something with AI for some reason. Uh, and it's almost like if I'm an investor, given these recent scenarios and you know, that patient um, committing suicide, I would be like leery of funding anything with AI. I have the opposite perspective, but it it seems like the industry is not doesn't have my perspective. They're like, oh, it's got AI, we should invest in it. Trevor: Yeah, well the AI boom is very real. It's the reason why rent is so bad in this city. It's, everyone wants to do something with AI. They're trying to push out new AI products. It's the hot topic, it's the buzz word on everyone's mind. And it is, you know, I do think it is partially a marketing effort more than, you know, more than not in a lot of cases. Say, just labeling something with AI, slapping it on to it, even if it's just like a decision tree in the source code. Um, often times I don't think that people are drawing the line between what is AI and what is not AI. They're just looking for that marketing term for that vendor or for that funding from the investors. And I think that I heard a similar example once of around 2008 when cell phones, like smartphones became common place. Everyone wanted a mobile app attached to their medical device. And then the FDA started regulating medical apps and then suddenly no one wants a mobile app attached to their medical device. There are too many regulations involved. It's no longer just a marketing term. I'd imagine we're going to see the same thing with AI in um, in the healthcare space. is that draft guidance out for the FDA. Once that's finalized, I'm sure we'll see fewer and fewer AI enabled systems. Christian: Yeah, it's like the boom is going to go away like the stock market and the floor is going to drop out at some point. Trevor: I've heard people say, oh, AI's a bubble, it's gonna pop and then go back to zero. I don't think that's gonna happen, but I do think the hype is going to slow down. The train's the train's gonna slow down, but it's not gonna stop. Christian: Yeah, so I think from a manufacturer perspective there's if you do AI right, there's definitely an advantage, but you have to consider a lot more and from uh road map perspective, uh these things definitely need to be on your road map because there there have been some serious incidents with AI recently that um have caused major legalties. And in the market, I think is getting a little bit lery about AI because of some of these incidents. So how do you provide some assurance that your device actually has AI implemented properly. I think that's a big challenge that some of these manufacturers going to face as well. What do you think about that? Trevor: I think it's similar to enforcing, you know, proving you have cybersecurity implemented correctly. So if we're looking at, you know, the cybersecurity guidance from the FDA came out in September of 2023. Before that, you weren't really required to prove that you had cybersecurity, some healthcare delivery organizations would ask for it, but they weren't asking for this evidence. And then it caught everyone off guard. Even though the draft guidance was out for a while, people knew it was coming, when it finally was finalized, it caught a lot of people off guard. FDA rejections were going through the roof. And eventually the industry started to catch up and I think they're still catching up on what needs to be done, what documentation needs to be provided, testing needs to be provided. It'll be very similar with AI, I believe. So, we'll see as these, as these regulations get more enforced and more strict, uh, a lot of manufacturers are going to have some rude awakenings on some new controls they need to put in place, new guidelines and safety around their AI models. But they'll start to evolve, they'll start to adapt and get used to the evolving regulations. Christian: Yeah, I agree. Uh and we're coming up here on time. I think one of the important takeaways is that the regulators aren't asking if your AI works, they're asking how it fails. And that's the exact case that happened with the person that's suicidal to suicide. The AI failed in a manner that caused patient harm and patient death in the scenario. Trevor: Yep, yeah, I definitely agree there. Christian: And I think we need to look at that Yeah, we need to stop looking at like, oh my AI works. so, you know the question should be what happens when it doesn't work and what is the harm? you know That's the lens we start need to start looking through as well. Trevor: Yeah, everyone likes to talk about the 99% success rate, but not about the 1% failure rate. And often times, and especially in the medical context, that 1% of the time can lead to death, can lead to harm, can lead to, you know, terrible situations, like that AI chatbot example, where someone ends up committing suicide based off of bad advice. Yeah, it's an edge case, but it is a case that came up. Christian: Cool, well we'll wrap up the episode here. I hope everyone enjoyed this episode of the Med device Cyber podcast. We hope to see you on the next one. And as always, if you need any help with medical device cybersecurity, penetration, testing, threat modeling, software build materials, full service, documentation, whatever help you need, let us know. Hope to see you on the next one.