What Happens When AI in Medical Devices Make Mistakes? | Ep. 40 - Full Transcript | The Med Device Cyber Podcast

Hello and welcome back to another episode of the Med Device Cyber podcast. I'm your co-host Trevor Slattery joined by our co-host Christian Espinosa. And today we're going to look at something really interesting. What happens when AI gets it wrong? In the medical context, this can mean someone's life is on the line. So AI making a decision trying to step in as the place of, you know, diagnosis, therapy, provision is a little bit of a dangerous territory. Of course, AI provides a lot of great innovation, but we want to make sure sure but it's handled safe safely. How are you doing today, Christian? Christian: I'm doing well. I'm doing well. It's uh, it's Thursday today, I think it is. And it's uh hot in Phoenix. I'm still recovering from last week. Last week I was in New Jersey doing a Formula 4 uh race course. Trevor: So what's the difference between the different, I guess formulas going up four to one? Christian: Well, Formula 4, Formula 3 and Formula 2, the cars are all the same, so it's purely up to the driver. as far as skill set. Formula one, the cars each team constructs their own cars um based on a set of specifications from the FIA. So the cars are different, like the team cars are different. Some are faster, some are slower, some are faster than straight, some are faster on turns and the drivers are different. So from a driver perspective, it's an equal playing field in like F4, 3 and 2, F1, you have the manufacture of the car that comes into play as well. And then F4 cars aren't quite as fast or powerful as F3 or F2 or F1. They they go up in order from, you know, one is the fastest obviously. And also the most expensive. Trevor: Yep. Yeah, we're uh unpacking into our new apartment right now and I can't see where it is, but we have a signed photo from Charles Leclerc, which I'm gonna put right back here on the wall. Christian: Cool. Um, yeah, so I'm I'm hoping to do a F4 race at some point also. Probably uh maybe later this year or early next year. Trevor: That'd be pretty cool. Any uh preferred location for the track? Christian: Whatever track I'm I'm I'm good with. I I I did that course at the New Jersey Motorsports Parkway. Um so I'm very familiar with that course. I watched a couple races there on F4 on that course. I feel like I could kick ass at that course, but a new a new course, you know, I have to learn the course and all that stuff, which is part of the challenge. Trevor: Well there you go. Maybe it'll be back to New Jersey for your uh championship trophy. Christian: Well, I'll be pretty ambitious to win a championship on the first race, but we'll see. Trevor: Awesome. Well, let's jump right into some of these AI considerations. So, this comes up as a little bit of, you know, some existing guidance but with some changes here. So, right now what we're looking at is the EU AI act and then some of the new guidance pushed out by the medical device coordination group in the EU. Uh now while this is a little bit separate from our standard focus on around the FDA problems, FDA considerations, it all ties in to medical device safety and of course we do handle a lot work within the EU and IVDR so it is especially relevant. I know we're looking at just how to bring in a little bit more security in AI systems which the EU has been more on top of than a lot of other agencies I would say, making sure that AI is pushed out securely, safely and their regulations in place to try to add some guard rails to what can be in some cases a little bit of a risky technology. Christian: Yeah, and I think it's good to provide a little context here with a real case that's an active case going on right now. Uh and I think you're familiar with this case. There was a medical device manufacturer that has a a mental health application that has an AI-based chatbot. So from a mental health perspective, it's supposed to, you know, help with the mental health of the patient. And the case that is being examined today is there was a suicidal patient that over the course of several months was interacting with this AI-based mental health chatbot. And after three months, for some reason, the AI-based chatbot told the patient, well, you might as well go ahead and kill yourself. Uh, and the patient ended up killing themselves. So the patient's family is now suing the uh, manufacturer of this product.