The Hidden Cybersecurity Risks When Doctors Use AI Diagnostics | Ep. 58

In this episode of the Med Device Cyber Podcast, hosts Trevor Slattery and Christian Espinosa are joined by special guest Jun Xiang Tan, the owner of TuringLabs, who is currently working with a health-tech startup in Singapore. Jun Xiang brings a unique perspective, with a background in military cybersecurity and network forensics before transitioning into the AI and health-tech space. The conversation centers on the burgeoning use of Artificial Intelligence in healthcare and the significant, often overlooked, cybersecurity challenges it presents. The discussion kicks off by highlighting the alarming trend of 'shadow IT' in clinical settings. Christian Espinosa points out that studies show almost 25% of clinicians use unauthorized AI tools like ChatGPT for diagnostic support. Jun Xiang elaborates on this, noting the convenience for doctors to quickly input patient symptoms, text, or even upload X-ray images to get instant feedback. This practice, however, introduces massive data privacy and compliance risks, as sensitive Protected Health Information (PHI) is fed into public models that may use it for future training, essentially creating a major data breach. The podcast then delves into the core vulnerabilities of AI systems themselves. A primary concern raised is data poisoning, a type of adversarial attack where the AI's training data is manipulated. The hosts discuss a case where poisoning just 0.001% of the training data resulted in a 5% increase in incorrect outputs. In a healthcare context, such inaccuracies could lead to misdiagnoses and severe patient harm, underscoring the 'garbage in, garbage out' principle. The conversation also scrutinizes the quality of AI-generated code. Citing recent statistics, Christian notes that nearly 50% of code written by AI introduces new security vulnerabilities, such as cross-site scripting. This is largely because AI models are trained on vast repositories of public, open-source code from platforms like Stack Overflow, much of which is outdated, insecure, or written by inexperienced developers. The AI, therefore, learns and replicates these poor security practices, creating bloated and vulnerable codebases that require significant manual effort to clean up and secure. The episode contrasts the rapid, often unregulated development of commercial AI with the stringent, safety-critical standards of the medical device industry, such as IEC 62304. This standard dictates a rigorous, safe development lifecycle that current AI tools cannot replicate. The hosts and guest conclude that while AI offers powerful capabilities as a support tool—a 'pair programmer' or a clinical decision support system—it cannot be trusted to operate autonomously. The risk of hallucinations, biases, and security flaws necessitates constant human oversight. The ultimate message is to guide the AI, not let it guide you, by providing it with clear requirements and verifying its output, ensuring that patient safety remains the paramount concern.

A lot of physicians or clinicians, almost 25% are using AI in an unauthorized manner without any real controls around that. It is so convenient to just take up your phone while you are going around rounds in a hospital or even as a general practitioner in the clinic. To take out ChatGPT, just type in a few phrases, to either diagnose the patient using text or even you have an X-ray imagery, you just send it to ChatGPT and ask it to spot any anomalies. .001% of training data resulted in a 5% increase in wrong outputs. If you're training your AI on bad data, it's going to give you bad output every single time. Almost 50% of AI generated code introduces vulnerabilities such as cross-site scripting. In the medical space IEC 62304 dictates the way that medical software needs to be developed in a safe fashion. Is this becoming a bigger problem than we think? Hello and welcome back to the Med Device Cyber Podcast. We have here your usual co-hosts, Trevor Slattery and Christian Espinosa. And then we have a very special guest coming in from Singapore as well. Today we're going to be talking about some really exciting things with code security as well as how AI has helped it and in some ways how it's hurt it. And what we can do to make sure that we're developing safer code within the medical space. I want to start by turning it over to you, Jun, to do a little bit of an intro and some background on yourself and then we can go ahead and jump right in. Yeah. Uh very good morning to Christian and Trevor. So, thank you for having me on the podcast. Uh so a bit of background of myself. I was actually not from the health and med tech space. I started off with the military, spent four years there, did network forensics or cybersecurity space. Decided I uh wanted to do something new, so I went out, I did prototyping uh AI systems in a software agency that I run on my own. And now I'm currently with a health tech startup, Caregiver, which uh helps doctors make more informed decisions on their patients' data using AI. Isn't it mandatory in Singapore to join the military for a while? Oh yes, definitely. Yeah, we have to serve two years. Usually that it's before our university. I would spend two years before, yeah, we go to spend the four years on our university. I think they need to make it mandatory in the United States. It would probably solve a lot of our problems with our younger generations, I think. Is there a manpower uh gap in the US or usually I I do see a lot of people joining the military. There's no gap of manpower in the US, but there is a Christian's alluding to the gap in a gap in a few other areas about the US. Ah, okay, got it. Discipline, hard work, ownership. Yeah, just a few. Yes, the military does train you up on that. I learned quite a bit uh makes you, I mean, the first two weeks we would uh how they do it is the first two weeks you just stay in camp throughout. That's kind of like the first touch point for many young people where they leave their families for a two weeks, yeah before they return home uh once every week on the weekends. Yeah, I was part of the military in the United States. Trevor wasn't part of the military, but his parents kind of threw him out in the jungle and made him track for himself I think, right? Yeah, I got the, I got to grow up in Central America out in Belize and so it was a very fun, very interesting environment compared to the typical US upbringing. Oh, that's interesting. How long were you, did you have to stay out for or were you by yourself or? Yeah, they, um, I remember I was probably 12 years old and my dad got me this big pole spear and he was like, "Okay, go out and go hunt fish and lobster." And I was just like, "Awesome. See you in five years." And then became obsessed with spearfishing from there and so it was a, it was a fun, fun time for sure. Awesome. So let's uh get into AI a little bit because I know one of your specialties uh Jun is uh AI. And before we were hit record, we were talking about some of the challenges with AI, and some of the emerging AI threats to healthcare. I think our audience would find some of those interesting. I think one of the things you brought up is that a lot of physicians or clinicians, almost 25% are using AI in an unauthorized manner without any real controls around that. Maybe could you elaborate a little bit on that? Uh yeah, regarding uh, I mean, as part of the article itself that mentioned that 23% of them were using unauthorized AI tools. Actually, through our own research as well when we spoke to uh clinicians and doctors, it is so convenient to just take up your phone while you are going around rounds in hospitals or even as a general practitioner in the clinic, right? Uh to take out ChatGPT, just type in a few phrases, to either diagnose the patient using text or even you have an X-ray imagery, you just send it to ChatGPT and ask it to spot any anomalies in this thing. So this would this are increasing trends that we are spotting uh in interviews, uh actual interviews that we are doing. So so so clinicians are actually using images, uh ultrasound or X-ray images, and uploading them to ChatGPT to help with a diagnosis now? That is correct. That is correct. In a completely unauthorized manner? Yes. I shall not name names, but yes. That's what we observe. Yeah, because I mean, the capabilities of ChatGPT are, I mean, when it first come out to the state, I mean, it came out almost, almost five years now, it's about coming six years now, and uh it has grown a lot and uh it's capabilities are amazing. I mean, you get real-time feedback, right? Compared to you having to go down to a specialist just to get an outcome. And even patients are doing that themselves, uploading their own clinical data or their own medical history onto such platforms. What, so what is the problem, just for our listeners with clinician or doctor taking an X-ray or ultrasound image, uploading it to ChatGPT and having it give them a diagnosis. Are there any security issues with that? Yeah, definitely. I mean, uh so, I mean, one one form of cyber attack it's manipulating, uh just manipulating a few pixels within the image itself which might not uh be present to the human eye, right? And this could trigger a specific response by the AI to represent the the data in a certain way. Like, for example, inserting or removing certain disease signs and that could be a false positive in itself. Even another big part of it, I think... are feeding this information into an AI, it's then feeding that back into its own training. And so that information becomes effectively part of OpenAI's domain when you're just feeding it someone else's health records, which is a major compliance violation. Right, because if they're my medical records and I don't authorize them to be given to OpenAI or ChatGPT, that that's uh that's a problem, right? Because they may have uh personally identifiable information on there too. That's correct. So, I mean, recently, I think two weeks ago, they released uh, three weeks ago, they released ChatGPT health. I think it's still slowly rolling out, supposed to deal with some of these issues, but we'll still need to wait and see how the trend pans out. I know, it's a trend is the data poisoning. Trevor and I have talked about data poisoning in a couple episodes. And something that, I think is interesting like a case study you brought up earlier, offline, was that .001% of corrupted data, training data resulted in a 5% increase in wrong outputs. So that's a pretty like broad stew from the amount of training data that resulted in a much larger percentage of wrong outputs. Is this becoming a bigger problem than we think? I I know Trevor's answer on this, but I'll ask, I'll ask you, Jun, as well. It really depends on the training. I think the paper clearly draw that for those smaller models where we are talking about like 1.3 billion billion parameter models, those are more susceptible to these attacks, but the larger models as we are scaling it up, uh might not might not be so. And also if the larger data set itself is is clean and fine tuned, I I suppose I'm more positive on this note that uh such things will be cleaned up over time than more negative that it's going to cause more misinformation and harm. Yeah, as AI tools. But of course it's a double-edged sword in itself. Yeah, I think there are a couple of angles with it that are important. So the first being, and we've talked about this before, we always say garbage in, garbage out. If you're training your AI on bad data, it's going to give you bad output every single time. And so I think you bring up a great point, which is how much context does this AI have? If it's on a lower subset, you know, you're saying something like a billion parameters or, you know, even 10 billion parameters, but you're poisoning it with 100 or 1,000 bad bits of information. It doesn't seem like all that much compared to a billion, but that's still a significant amount, enough amount of information to start manipulating output. If we're looking a little bit closer to a trillion data points that we're training it on, then you're going to see a little bit less of that. It has such a bigger context, it's going to understand how to filter out some of the bad points there. Now, having said that, I think that a lot of medical companies trying to spin up their own models are not necessarily going to have the research or maybe even the expertise to train these large models with large amounts of data points. It's a very expensive and a very time consuming process. And so I think that it is something to make sure that people have at the top of their minds while they're working through these problems. But I completely agree with your view that as the AI industry gets a little bit more mature, as these models get a little bit more complex, I think this is a problem that we'll see start to diminish. Where do the people training the model get the data? That's that's always a question of mine because it's not easy to get data from Europe. It may not be easy to get the data from the United States or other countries. So aren't you going to introduce some bias anyway into the model because the data you're getting is not an accurate representation of humanity and diseases? Well, right now they're paying $100 an hour in San Francisco to train AI models manually. And so just giving it a bunch of information in highly technical fields such as healthcare, uh they're hiring contractors specifically for that purpose. From KOLs? From KOLs, the key opinion leaders in the space. Just learn what that, that acronym stood for. Oh. I've heard it like 9,000 times since I figured it out last week. It was one of those viral ones, I guess. Uh yeah, I hear the KOL. As soon as I heard it. Yeah. It's difficult in medical law, areas like that, where it's taking such a huge amount of information and then applying it based on connecting all of this context. And so a buy the book representation of exactly what you learn during your residency or during med school is going to be different from how an actual hospital operates. And so seeing a little bit of that difference with these KOLs training the AI models themselves based on what's seen in the real world, I think does actually have a lot of value. I was just in San Francisco last week at the JP Morgan event. And Trevor, you're right, every billboard I saw was some AI company. We saw like some competitors to Waymo driving their AI vehicles around test driving them. And it just seems like a bunch of hype, to be honest over there. I think to some extent, so they are seeing that a little bit of the mindset is that we're gearing up for a new era in technology. The same way that immediate viewed returns on the internet were not actually that huge from a business perspective compared to the way that it's scaled up over time. And so there's a lot of buzz into directions. The first is that we are seeing a turn and a revolution in the way that technology fundamentally works. It takes time to get traction, but all these trillions of dollars being pumped into those companies is eventually going to show that output. Who is putting this trillions of dollars in these companies? Are there really investors just like if it's got AI on it still going to just invest in the company? Pretty much. A couple of them, you know, swung it out of the park with OpenAI or Anthropic and then now they're flushed with cash to give to anyone. That's like two out of a 200,000 probably. The odds are probably not that great. Well, when you're going for a trillion dollar IPO, I'm sure you're making some pretty good money. But I think the other angle is that we are not seeing immediate return from AI. It is right now a little bit of a hype. And so from an actual business perspective, it does not provide the same amount of value that it costs right now. The other angle is that, it is what it is. We tried, it didn't work out. AI is not going to be this huge groundbreaking thing and while it definitely provided value, did it provide the trillions of value that it was expected to provide? No. So, those are the two different angles that the prediction it could go. I guess uh time will only tell who's correct. What about using AI to write, help write the code for your medical device or your application? Uh, is that a safe way of doing it? I'm going to say as a blanket statement, no, but I'm curious to hear your thoughts, Jun, on that. I mean, right now the industry honestly is figuring itself out. But I think ever since coding agents have come out, I mean, the no code and no code tools like lovable, v-zero versus the fully coding agents like Claude Code, and a few other IDEs that came out in the recent years, cursor, anti-gravity, Winter, the code quality has been improving. But I mean, based on what I, I I've been using agents almost like at least four to five hours daily. But if you stop monitoring it, if you let it go wild for let's say 15 minutes, the code itself, either you do not understand what it does. It has written like 200 or 300 lines of code to do what it's supposed to do in five lines. So it's got some room for improvement. I also heard some stats from Veracode, almost 50% of AI generated code introduces vulnerabilities such as cross-site scripting, which cross-site scripting has been around forever. It seems like the code should be smart enough to test for cross-site scripting and things like that. Yes, that is correct. But I mean, I I I I would think that this could happen because it's training on a large volume of open source data as well, right? So if you think about it, a lot of public data could be coming from hackathons or school projects where this programmers are fresh out of the market, right? So... what you're saying is AI is training on code that was poorly written to begin with, so of course it's going to write poor code and vulnerable code. Is that what you're saying? Yes, yes. I would I would think so. I mean, uh, this is probably, I mean, the foundation of large amounts of code, right? I mean, corporate, large code bases that are very well cleaned in corporate setting, I think those would be governed as a IP in itself and this would not be part of the training set itself. That's why, uh, they are, it is a huge market today of AI tools trying to either rewrite code or vet, vet the code and even give you a code review, like telling you what's wrong in the code, and then the human developer goes in to evaluate if that it's worthwhile for a fix. So that there is a huge market right now. There are companies such as, uh, CodeRabbit, SonarQube, that is doing this, uh, code quality checks using AI. So, we're looking at it through an admittedly very narrow lens with our focus. Our focus is what risk is introduced into the system, not how good is or how quality is your code. And so for us, you know, you brought up a perfect example. Anytime I mess around with, um, just building like a little custom project to see how some of these projects work, I created this little monitoring dashboard for some internal stuff and then mostly to test some of our workflows for source code testing, automation, and just to see how vulnerable it was. And I noticed that to create a small dashboard, small front end, really minimal back end, with just a couple of API calls to one third-party service, I was using, um, Cursor, and it introduced almost 500 third-party components into the system when I just let it go, go nuts and do whatever it wanted. And so from a functional perspective, it does what I asked it to do. Is it massively and ridiculously bloated? Yes, it is. But I looked at from there, what level of effort would be involved with getting it, you know, manually patched up for security and it wasn't that much. It was a weekend's worth of work. So I patched it all up for security and then I look and I have this secure but ridiculously massive, unclean, you know, zero documentation code base. And so what we're looking at is making sure that the software is safe and secure. If we can prove that with a hundred lines of code or five lines of code, it's not really too much of a concern from the security perspective outside of the fact that it can quickly become unmaintainable. The big concern is around that code quality and that's something that's a little bit hard and that's usually why, you know, that's why advanced 15 years of experience software engineers command such ridiculous salaries in these big tech hubs is because they have the expertise to look and go, is this good? Not just is that functional. So using an AI to do that, I think we're still a little bit in its infancy, but I think it'll get there eventually. But you bring up a great point. All these big corporations keep all their IP protected and private. And so even their code review cycles are going to be private compared to scraping what some 17-year-old is doing off of the internet. Oh yeah, made me recall uh last week itself, there was an open source repo itself. It's a bit out, out of scope. It's a whiteboarding software which is open source. They actually stopped accepting PR reviews that were AI written. So these are certain trends that are starting to happen in the open source community where this they there's a term for it now, they call it AI's law that once it gets pushed in and you're actually spending more time to do the clean up more than you just did it correctly the first time as a manual developer. Yeah, that's a good point about AI being able to fail safely. That's something we always look at as well. Everyone always thinks about how great AI is when it gets everything right, but rarely do people think about it when it gets things wrong. And that's usually where consequences happen and patient safety could be impacted as well. All right, we're coming up here on time, so I'd like to go around the room and ask for last minute words of wisdom for our audience. So I'll start with you, Trevor, as usual, and you got to say something new instead of the same thing you always say. Guide the AI, don't let the AI guide you. I feel like it is, it's way too easy and it's way too prevalent for someone to just say, hey, I make me a medical device. I'll check back in an hour. And then utilize whatever code it's spitting out and you have no idea unless you understand that AI coding tools are an incredible way to increase an engineer's output tenfold, if not a hundredfold, but you need to be very careful guiding the AI through this process. Do it in a compartmentalized fashion. You should still build out your requirements to feed in the AI. Don't let the AI invent its own requirements. So, it's in a careful fashion and don't let the AI just run out of control. All right. Jun, what do you got? Uh, yeah, I agree with that uh entirely with Trevor. And uh for my working model itself, I treat AI as my pair programmer. How I generally start things off with it is usually doing a trade off. I have this three software options, help me evaluate or generate a matrix table for me of what they do or what they do not do and how it suits my needs. Uh, it is very good at abstraction as well, bringing it one level higher. So, things like thinking about building integrations, building plugins, and lastly for me, it's also helping me with my repetitive work. So, this is the outcome that I know that I need to derive, but now I'm going to do it 300 times in a similar fashion, but of course there are some variations within it, and AI is good at that, and I can vet the work, vet the output, or vet the process of it doing the work correctly. Awesome. Something I'll add is, we kind of touched upon a little bit is using AI for like a clinical decision support system. I think we're okay with that, but we've gone through a lot of problems with AI like hallucination, model poisoning, data poisoning, a lot of these things that can cause problems with AI. So I still think we need a human involved to interpret the results, but I feel like we're trying to push really quickly for AI to be a diagnostic tool and even a therapeutic tool without fully understanding all these ramifications. Well, thanks for joining us Jun from uh Singapore and Trevor from San Francisco. I'm in Phoenix, Arizona today, sunny and warm and nice. Not quite as hot and humid as uh Singapore, but it is uh nice here today. And I hope everyone found value in this episode, so thanks for tuning in and I hope to see you on the next one. Thank you so much Christian and Trevor for today's episode. Yes, thank you. Thank you.