Using consumer Large Language Models (LLMs) like ChatGPT for medical diagnosis is unsafe and...
Using consumer Large Language Models (LLMs) like ChatGPT for medical diagnosis is unsafe and breaches their license agreements, as they are not FDA-cleared for clinical use.
A key distinction exists between "frontier" AI models
which are developed in-house with full control, and "harness" products, which are built on third-party LLMs and introduce uncontrollable supplier risks.
The use of machine learning in FDA-cleared medical devices is not new
the first such device was cleared in the 1990s, indicating a long history of this technology in healthcare.
For AI as a Medical Device (AIaMD) to be safe and effective
it is critical to keep a "human in the loop" where the AI provides recommendations but a clinician makes the final decision.
The FDA's Predetermined Change Control Plan (PCCP) is a regulatory tool that allows for planned...
The FDA's Predetermined Change Control Plan (PCCP) is a regulatory tool that allows for planned updates to an AI model without requiring a new submission for every change.
Developing a safe and effective AIaMD is a lengthy
rigorous, and expensive process, often taking many years of research and clinical validation before market clearance.
Companies should prioritize building AI that is demonstrably better and safer than existing algorithms
rather than simply incorporating AI for its own sake.
The intellectual property around AI software is often protected as a trade secret rather than through patents
as the patent process requires a level of public disclosure that can compromise the proprietary nature of the algorithms.
Episode Summary
In this episode of the MedDevice Cyber Podcast, host Christian Espinosa welcomes guest Tyler Harmon, CEO and co-founder of IASO Automated Medical Systems, to discuss the critical safety and regulatory considerations surrounding Artificial Intelligence in MedTech. Tyler Harmon, a biomedical engineer with over a decade of experience as a medical device consultant, provides a deep and nuanced perspective on the burgeoning field of "AI as a Medical Device" (AIaMD). He begins with a strong cautionary note, specifically addressing medical students and practitioners who might be tempted to use publicly available Large Language Models (LLMs) like ChatGPT or Claude for diagnostic purposes. Harmon argues passionately that these tools are not FDA-cleared as medical devices and using them for clinical decisions is a misuse of the technology that could lead to significant liability and patient safety risks, as their license agreements explicitly state they are not for medical use.
The conversation delves into the distinction between different types of AI development. Harmon explains that his company, IASO, is a "frontier model" developer, meaning they build their AI algorithms entirely in-house. This gives them full control over the data, training, validation, and risk management processes. He contrasts this with "harness" products, which are applications built on top of third-party LLMs. While these can be developed more quickly, they rely on a model whose inner workings and training data are not fully transparent to the developer, introducing a layer of uncontrollable risk. Harmon emphasizes that while the concept of AI in medicine feels new, machine learning has been a component of FDA-cleared devices since the 1990s. The current challenge lies in navigating the rapidly evolving technology within a framework that prioritizes patient safety and efficacy.
Harmon and host Christian Espinosa also explore the regulatory landscape, including the FDA's guidance and the importance of a Predetermined Change Control Plan (PCCP). This plan allows developers to define a scope of acceptable modifications to an AI model post-market without needing to file a new 510(k) for every update, a crucial aspect for a technology that is designed to learn and evolve. Throughout the discussion, a central theme is the necessity of maintaining a "human in the loop." Harmon asserts that at the current stage of technology, AI should serve as a powerful tool to augment and inform clinicians, but the final decision-making authority must rest with a qualified human. He believes this is the key to creating safe and effective AIaMD products, mitigating risks while still harnessing the technology's potential to improve workflows and, ultimately, save lives.
Frequently Asked Questions
Quick answers drawn from this episode.
In this episode of the MedDevice Cyber Podcast, host Christian Espinosa welcomes guest Tyler Harmon, CEO and co-founder of IASO Automated Medical Systems, to discuss the critical safety and regulatory considerations surrounding Artificial Intelligence in MedTech.
Using consumer Large Language Models (LLMs) like ChatGPT for medical diagnosis is unsafe and breaches their license agreements, as they are not FDA-cleared for clinical use. A key distinction exists between "frontier" AI models, which are developed in-house with full control, and "harness" products, which are built on third-party LLMs and introduce...
He begins with a strong cautionary note, specifically addressing medical students and practitioners who might be tempted to use publicly available Large Language Models (LLMs) like ChatGPT or Claude for diagnostic purposes. It's most useful for medical device manufacturers, cybersecurity engineers, regulatory affairs professionals, and...
Using consumer Large Language Models (LLMs) like ChatGPT for medical diagnosis is unsafe and breaches their license agreements, as they are not FDA-cleared for clinical use.
A key distinction exists between "frontier" AI models, which are developed in-house with full control, and "harness" products, which are built on third-party LLMs and introduce uncontrollable supplier risks.
Share this episode
Pre-fills with: "Using consumer Large Language Models (LLMs) like ChatGPT for medical diagnosis is unsafe and breaches their license agreements, as they are not FDA-cleared for clinical use."
AI is moving quickly, but medical devices cannot afford to treat every new model like an ordinary software update. In this episode of the Med Device Cyber Podcast, Christian Espinosa speaks with Tyler Harmon, CEO and co-founder of IASO Automated Medical Systems, about the risks and responsibilities surrounding AI as a medical device. Tyler explains why consumer tools such as ChatGPT and Claude should not be treated as clinical diagnostic systems, even when a doctor remains involved in the final decision. He also explores why a human in the loop does not automatically make an AI system safe when the underlying model was never designed or cleared for that medical use. The conversation examines how predetermined change control plans can support safer AI updates, why developers must understand the difference between frontier models and AI harnesses, and what can go wrong when a medical-device company builds on top of a third-party model. In This Episode 00:48 Tyler Harmon and IASO Automated Medical Systems 03:34 What qualifies as AI as a medical device? 04:44 Predetermined change control plans 06:08 When should an AI medical device be updated? 09:00 Why medical AI is not new 11:32 Why medical AI still needs human oversight 17:40 Doctors using ChatGPT to interpret X-rays 20:09 Frontier models and AI harnesses 24:16 The challenge of developing proprietary AI 29:21 Why medical AI can take years to reach the market 34:28 Using consumer LLMs outside their intended licence 39:25 Licensing third-party AI for medical-device use 40:41 Final lessons for medical-AI developers Check out Tyler Harmon’s LinkedIn here: https://www.linkedin.com/in/tyler-h-938bbb43/ The Med Device Cyber Podcast is brought to you by Blue Goat Cyber, cybersecurity experts providing essential security solutions for the medical device industry. Learn more by visiting https://bluegoatcyber.com . If you’re interested in our services or partnering with us, schedule a Discovery Session: https://go.bluegoatcyber.com/meetings/blue-goat-cyber/discovery-session Christian Espinosa is the CEO and founder of Blue Goat Cyber. Christian Espinosa on LinkedIn: https://www.linkedin.com/in/christianespinosa/ Blue Goat Cyber on LinkedIn: https://www.linkedin.com/company/blue-goat-cyber/ Blue Goat Cyber on Instagram: https://www.instagram.com/bluegoatcyber/ Blue Goat Cyber on Facebook: https://www.facebook.com/bluegoatcyber/ Blue Goat Cyber on YouTube: https://www.youtube.com/@BlueGoatCyber/?sub_confirmation=1 Download MP3 — full interview
Guest: any students listening to this podcast who have aspirations of going into medicine, please don't do this because these tools are not AI as a medical device cleared. As a consultant, I've worked on what we would call AI devices since way back in 2015. We've been doing this type of math for decades at this point. In fact, the first cleared FDA device happened way back in the '90s.
Christian: I've heard of cases where doctors are having trouble reading their X-ray, so they upload it to ChatGPT or to Claude and say, diagnose this for me. I don't know, I feel like we're kind of headed where as a society, people are relying more and more on AI and less on their own abilities. And I don't know if we're headed that way in medical. Maybe, maybe not.
Christian: Hi, welcome back to another episode of the Med Device Cyber Podcast. Today we're talking about a topic that is evolving, and there are some risk with that topic. It's about AI and some of the safety concerns with AI in MedTech.
And we've got a guest today that I think is probably one of the most, probably the smartest person I've talked to about AI in a long time. Tyler, he's comes from New Zealand. I met him in Singapore, uh, a few weeks ago.
I drank some champagne with him, but I quickly realized uh, I I can't handle my champagne after a while. But uh, great to have you, Tyler. Maybe you could give a little bit of background and uh, like what the main thing we're going to talk about today is, because I know you're got a lot of passion and energy about AI and some of the concerns with it uh, just from a efficacy concern, safety concern, and cybersecurity concern as well.
Tyler: Certainly, and thanks so much for having me, Christian. Um, so my name is Tyler Harmon. I'm the CEO and co-founder of IASO Automated Medical Systems. We are a frontier AI as a medical device company based in uh, Auckland, New Zealand, but we have offices in the US as well.
And uh, we are focusing on AI as a medical device in, in the ICU. And so our objective is to detect acute respiratory distress syndrome, uh, which is the full heart lung failure of the of the patient in the ICU and help them get treatment faster. That's sort of everything in a nutshell.
And we're one of the, you know, the companies that you'll find out there that's doing our own AI development. Um, we don't use anybody else's models, we don't, you know, use any harnesses on top of anybody else's models. Our full stack is within our own development uh, suite.
And my background is in biomedical engineering. So before starting IASO, I was a uh, medical device consultant for more than, you know, 10 years in the space. Worked in a lot of software as medical device, a lot of software enabled medical devices, never worked outside the medical device field.
Um, graduated Georgia Tech in 2014, had been working in medical device since even before I graduated as part of the translational research lab at in under Dr. Yoganathan in the Cardiovascular Fluid Mechanics Lab. So that's sort of my background in a nutshell. Um, if you have any questions about, we talked a lot about the specific projects I worked on, but I feel like some of the highlights are, you know, the, the CardioMEMS acquisition by St. Jude Medical back in 2014, 2015. That was a huge experience. Uh, really lucky to have been a part of that.
I worked on a Johnson & Johnson divestiture project in 2018. Uh, and I also worked for Apple for a brief stint and there's a lot of startups and mid markets kind of sprinkled throughout there as well.
Christian: Everyone thinks if you work for Apple, you have you must have been pretty smart. Is it, is it true? Like Apple or Google? Which one, who who on average is smarter? If you work for Apple or Google?
Tyler: Uh, that's hard to say because there's a lot of good Yellow Jackets at both those companies. And you know, I'm a, I'm a Yellow Jacket through and through. I bleed white and gold. So I think that I definitely, you know, respect and really enjoyed working with everybody I worked with at Apple, but I do have some colleagues, um, from my time in undergraduate. In fact, folks that were in the biomedical engineering program who went on to work at Google.
Uh, so, you know, I, I know Apple's processes. There's a lot that I like about what they do, um, and and even some of the, you know, behind the scenes stuff that I'm not sure how much I can talk about because uh, that NDA is still in effect. Uh, but, you know, I'm sure the things that let them know, like, hey, you're doing a good job, they might not mind having someone talk about.
But yeah, because they, they were very early adopters as AI as a medical device before anybody was really calling it that. They had transformer technologies embedded in some of their SaMD products on the Apple Watch and they did a really, really good job with it. I was always very impressed with working with those teams.
Christian: So, when you say AI is a medical device, is that just a subset of software as a medical device or are you talking about a different sort of classification altogether?
Tyler: So there are FDA separate FDA guidance for an AI-enabled medical devices. It's not super clear from my reading of the guidance, and you know, I'll defer to my chief quality and regulatory officer here, give her a shout out, uh, Ladavia Fowler. Um, uh, she, her reading as she's explained it to me is this, they, it is effectively treated just like any SaMD product, but there are special considerations you need to have, especially for the, the cadence of change for, um, software as a medical device that is AI enabled.
So most of the time, you know, it's treated just as a normal SaMD. However, they strongly recommend, although it's a guidance so it's not a requirement and there's nothing that's been changed in 21 CFR 820 to reflect this, but there's a strong recommendation that I tend to follow where you submit a thing called a predetermined change control plan, which is where as a part of your 510(k) or a pre-sub or any interaction you have with FDA, you can hand them a document that, you know, that they're all trained on it, they know exactly what to expect in those documents.
And uh, it essentially it walks the FDA through or any regulatory body through, here's what how we plan on managing the AI throughout the life cycle of this device. Because I think there was a realization at FDA by some very smart people, um, some of whom I'm sure you've also met, that, uh, AI has a different, you know, required tempo of change in order to be safe and effective.
You know, I've always felt that some one of the things that inherently makes software as a medical device different from medical device overall, is the fact that, you know, it's actually sometimes safer to allow fast changes to to software as a medical device. The AI as a medical device, uh, you know, really leans into that and really, you know, uh, really pushes people to make those changes faster and to have that predetermined change control plan so that you're not having to submit a letter to file or a new 510(k) every time there's a relatively small change, and it helps kind of draw those barriers, um, between that.
And so we're, we're actually working on the predetermined change control plan for our product as a part of our work today.
Christian: And how how often do you plan, I know you you want to put the frequency in the PCCP, how often do you plan to update the product? I know it ideally it's continuous, but you know there's there's probably some exceptions to that.
Tyler: Yeah, so in our our architecture planning, um, and this is a, an, a, kind of a net new invention. I haven't seen this approach taken, um, by anybody in, in software as a medical device. Uh, Paul McNeal is our head of, of product, uh, and he, he implemented a system that I can't go into the exact details about because as you know, for part of cybersecurity is keeping, uh, confidential information confidential.
But, uh, at, at the gist of it is that it's not so much a time threshold, it's a data threshold. So when there's a, a certain bolus of net new information that we feel like adds risk to the device without doing retraining, because at the end of the day, you know, you should only be updating the device to make it safer. Don't make updates just to make updates, right?
I know there's certain, you know, in the commercial side and the commodity, you know, uh, in the customer facing side of, of software in general, we have a tendency to just feel like, oh, well, you have to put an update out every two weeks. It's what it's what users expect, right? Um, it's what power users of even any given app expect.
But the special considerations for safety and software as a medical device. So, at the end of the day, you know, you need to look at what risk am I introducing, what is the actual substance of this change? Because you are going through change control at this point if your product is on the market.
And, you know, I feel like there's some significant safety risks that emerge from that decision. And so we took that under consideration, and we're going to continue to take that under consideration as we develop our risk management file.
Christian: Do you feel like this trend with AI and MedTech, uh, like where are we on the trajectory? Because from my perspective, there's like this influx of prospects that have, they take your an ECG reading, or what is it? ECG or ECG?
Tyler: We, we call, we call it ECG in the Southern hemisphere, but you know, one of the tells whenever you come here for to New Zealand is that, uh, if a doctor starts saying EKG, it's like, ah, American, got him.
Christian: I call it an ECG because I think I feel like most of the prospects we, we talk to call it ECG. The rest of the world calls it ECG. Yeah, that's one example. We, we've got a lot of prospects recently they're taking a traditional ECG, analyzing it with AI, and then doing some sort of predictive analysis to what your underlying condition may be or what you may may have later on, uh, based on that. And that's like one thing.
And then we have other people using AI to monitor your brainwaves and detect these other patterns. It's like, I feel like, I don't know, like, if we're like just at the beginning or like 50% through like this AI wave or like, I don't know, because if I feel like we're sort of at the beginning, but I was curious because you work in AI more than I do, like what's your perspective is?
Tyler: As a consultant, I've worked on what we would call AI devices since way back in 2015. And that, that particular product was for CardioMEMS, um, that that a company that was acquired by St. Jude Medical. And, uh, they, they were doing, you know, extrapolative machine learning applications. We wouldn't have called it AI back then because, you know, your classic frontier model companies weren't on on the scene, obviously.
But, I, we've been doing this type of math for decades at this point. In fact, the first cleared FDA device for with a machine learning component that I personally would call an AI component, um, happened way back in the '90s. So we've been, we have had an understanding of this from as a medical device industry for a long time.
And I personally started working on it in 2014. So, but there's a whole, there's a new class of of products that are coming on the market. This is probably reflected in the prospects you're referring to. I think we may know the same, some of the same companies.
Um, and some of them are doing a great job of this. Uh, is, there are some of them that are now integrating large language models into their product as, as a whole. Um, and so I think, and there's, there's distinctions between is it a, a clinical healthcare, you know, component, where it's more of a workflow LLM device.
And so those, you know, you can look at the definitions of a medical device and say, hey, this is definitely not one. This is definitely not one. You know, it is not a value add or a value to public safety for this to go through the FDA process. Then there's some
Christian: Meaning like if it's an AI scribe that listens in a room...
Tyler: Bingo. Exactly. And then there's a whole lot of other clinical workflow things. They're doing admin work. They're saying, hey, our admins are overwhelmed in private practice or even in the, in the ER. You know, how do we make this faster? How do we make the, you know, more effective?
Because at the end of the day, there are certain things that, you know, AI and machine learning are inherently going to be a little bit faster at in terms of doing, uh, interpolations and making conclusions from that information than the human brain.
And I, I personally feel like we're at a point where you can add safety and efficacy to a device if you have a human in the loop. Um, where, you know, for folks who aren't familiar with the art, term of art is at the end of the day, it's the human that's making the decision. While the AI is providing a lot of information and context and additional, you know, methodologies for the for the person to make an evaluation. Um, whether those are are from existing protocols or net new protocols, the, you know, the human in the loop kind of cures what ails you from a risk management standpoint.
I'm sure you and I have looked at some risk management files, uh, in our time, um, with, you know, some of your clients, some of my friends who ask me for for help every so often, where the risk of having the AI do something is inherently more, you know, more of an exposure from a safety and efficacy standpoint than than just having a human give it a quick once over.
Christian: You had doctors on the podcast and talk about this scenario. And they would say as their responsibility, and this is just one of them in particular, a heart surgeon, that at the end of the day, it didn't matter what the devices around the the operating table were telling him, it mattered what he saw in the patient. So it's ultimately his decision. Correct.
So, you know, you're he he was arguing for your point of view, but I've I've also heard the inverse where I've had, I've heard of cases where doctors are having trouble reading their, um, X-ray, so they upload it to ChatGPT or to Claude and say, diagnose this for me. You know, so I've, so that's a human loop too, but it's like the wrong way, right? So it's like, I don't know, I feel like we're kind of headed where as a society, people are relying more and more on AI and less on their own abilities. And I don't know if we're headed that way in medical, maybe, maybe not.
Tyler: So I do a good bit of teaching at various medical schools and schools where, you know, teaching their graduate programs for biomedical engineers that are hopefully going to go into the field and work on this stuff.
Um, and I, I would say, you know, in my evaluation as an expert, it's always funny to, you kind of look up one day and you're like, oh, you know, just a few years ago, I was, I felt like I was just a kid, you know, being, being told like, oh, this is what you should do. And all of a sudden, I'm, I'm turning around and it's like, oh, people are calling me an expert. Oh no. I can't be, I can't be the person who knows the most about this.
But you know, uh, people look to different levels of experience and so in my opinion as an expert, um, when you, when you're involving those third party LLMs in a process that they're not licensed or designed for, you're kind of doing a disservice to that company.
And I actually dissuade the students, you know, that I work with because there's not anything inherently illegal if you're de-identifying the, the X-ray. Um, you know, there's a whole lot of legality things that I don't necessarily, I'm not a lawyer, so I don't want to get into the nitty gritty of it.
But if you're, I I want to say any of our doctors listening to this podcast, please don't. Um, I any any students listening to this podcast who have aspirations of going into medicine, please don't do this because these tools are not AI as a medical device cleared. Um, you know, you upload it to ChatGPT, you upload it to Grok, you upload it to to Anthropic, to or to Claude, uh, or any of their harnessed products, right?
Um, and this is a safety implication that I wanted to go into today. Um, but you, whether you're using the frontier model directly or you're using it through a harness,
Christian: Hey, I want to, I want to, I want to back up a second because I'm not sure all of the users understand harness and frontier. Maybe you could like explain that a little bit before you go the next step.
Tyler: Yeah, good, good catch. So, a, when I say frontier model, I just mean someone who develops their own LLM, right? There's a lot of, you know, for, there we have a few different companies in the, in the US that are doing their own, you know, LLM development, whether, whether that's Microsoft's Copilot, Google's Gemini, uh, you know, and then there's companies that have relationships with those large compute providers, but aren't, you know, they're totally separate entities, like Anthropic and OpenAI.
Um, and then their products, you know, OpenAI's main product being ChatGPT and then Anthropic's main product being Claude and all the harness products that go along with that. And when I, so that's what I mean by frontier model, is you develop it yourself, really.
And we bandy about that term, it gets given different definitions, but we in medical devices, we're in a regulated industry. We need to have hard definitions that you can write at the top of the document and hand to the FDA, plain and simple. That's how I, I, you know, identify frontier models, and that's why when I talk about our product suite, um, we are going to be using frontier models because we aren't depending on other people's models. We aren't integrating from, you know, Anthropic as a supplier of Claude, um, and, you know, and then integrate that into our device. That's just not what we do. We have a different approach than most people, uh, in the, in the AI space.
Christian: And the harness, uh, what's the harness? You kind of alluded to it, but I'd like to get a little more explicit.
Tyler: Yeah, sure. So a harness really, once again, there's a lot of different definitions, but when I take a look at it, the most simple way to explain it is it's a deterministic layer that is sitting on top of your LLM. So, or it's, sometimes people call harnesses where it's like, hey, I have an agent that is then calling the LLM, and then they're calling that a harness.
That's a, that's a bit of a misnomer in my opinion because at the end of the day, a harness is just a layered approach to accessing the LLM. So when I talk about harnesses and LLMs, they're functionally equivalent, but maybe there's a layer added in between how the LLM's getting used and how the user is actually experiencing it.
Christian: So this would be like Claude, if I'm using Opus or Sonnet or Haiku or Fable, like those would be harnesses that affect how I interact with the LLM.
Tyler: Well, some, you could define it that way. I personally, I would say those are more so interacting with different model versions of the of the underlying LLM. Um, because those are, you know, famously Fable was recently released and it got reeled back because there was a lot of buzz about how this is actually an unsafe level of of access.
Um, you know, I know people who actually got to use Fable for a brief amount of time, and, uh, you know, they, they didn't feel like there were safety concerns, but I'll leave that to other types of experts. I'm not a computer scientist or a software engineer.
The harness piece to me is something like a Claude Code, where, you know, uh, there, there's a pretty famous, uh, I, I wish I could give the journalist credit who did it, you know, off the top of mind. But there were some famous pieces where for a very brief, you know, snapshot in time, Claude Code's sort of architecture was made available and some of like the actual guts of how Claude Code worked, right?
And it really was just a whole bunch of deterministic calls and different loops looking at Claude itself and and asking Claude to do certain tasks and, you know, doing different system prompts on top of the Claude model itself. Now, I feel like Claude Code and Claude Design and, you know, Cursor and these all these other great products that are accelerating, you know, uh, the, the efficacy of our software engineering teams. Uh, those are great examples of great uses of harness products. Um, I really advocate for their use, uh, even in a AI as a medical device company. We, we use them in our workflows for certain parts of our team.
Um, but then there's other instances where it doesn't make sense. So, just know when I say harness, that means that there's something in between the LLM and you that is, you know, making calls, doing deterministic, you know, algorithms, adding a system prompt that is helping you use the LLM.
Christian: That's an interesting scenario because we always see of AI as like a very quickly evolving field, and you're saying it took them at Updoc five years to get cleared. And I know that's because we're in a regulated industry, and my understanding is it takes an average of seven years and $35 million to bring a medical device to market.
Uh, whereas in Silicon Valley or I could create my little game or my app or something or my website with interaction, I could do it like in, you know, one afternoon. So it's a very different, uh, it's very different than, uh, the rest of the world in MedTech because it's it's about patient safety really, and it's highly regulated.
Tyler: Yeah, yeah, and I'd be remiss, you know, once again, uh, my, to, to, shout out to, to Ladavia, she keeps me on my toes and and makes sure I behave right. I, if I didn't mention that our device is not cleared, is not on the market available for purchase. So, you know, if anybody from the FDA or any doctors are listening to this, don't, don't try and come and ask for access to our model. This is not, not marketing of our model.
But, you know, we, I feel we still have a ways to go in that process. I don't know how far that is. Right? But there's still a journey in front of us. And so going through that journey, maybe it takes less than seven years for AI as a medical device products. You know, because when you, when we say that seven years, where is that crossover point? Is that at the point from where it spins out of a university and we say, hey, this is now, you know, how many years of research is now being sold off or rented off at pennies on the dollar to this startup company? And then they can go translate that into a technology.
Christian: Oh, you're saying when does the seven years start?
Tyler: Yeah, when, when do we consider that starting?
Christian: Yeah, yeah, that's a good point. Like what's the basis? What, what was the things that predated the start of the seven years, right?
Tyler: Yeah, because because, you know, we're, we're all standing on the shoulders of giants, right? AI, you know, all the way going all the way back into Turing, we can really say that that's the beginning of AI research as we talk about it today. Um, you know, sometimes that's used as a misnomer, I believe, but
Christian: You're talking about Alan Turing, the Turing test. Isn't that how you test to something, like you could test to something's an AI or not by doing that test, right?
Tyler: It's a little bit more nuanced than that. It was, it's really a philosophical exercise, which I think is really interesting because I would say that there's a lot of the tools that we claim as AI today don't pass that philosophical exercise because for the for those who may not be familiar, it's effectively saying, hey, put the AI on the other side of a table from me, right? In whatever interactive format that takes.
And can I sit on the other side of that table and, me as a person with an average level of knowledge, you know, and, uh, and ability to interact with it, do I know whether or not it's a person? That's sort of the essence of the Turing test. Um, and you know, there was publications he put forth like in the, you know, during the 1940s after the war, after World War II.
Um, and, and, uh, I can't, don't quote me on that. That's not an area of computer science history I'm particularly well versed in. Um, but if, if I can sit on the other side of the table and it, and it can fool me effectively, and I, no one's trying to fool you when you're using an AI product, you should have, we should all know that there's not a human, you know, on the other side of the world typing back to you when you type something into ChatGPT.
But because ChatGPT and all these other LLM products are the first most convincing version of people to be able to interact with an algorithm that is, the, the actual use case of it is it's trying to interact with you as if a human was on the other side of the table, right?
And so that is really the general, like sort of the the pivotal change of this technology ever since they came out in the 2023, 2024. You know, I, we've made leaps and bounds and the products have gotten really interesting, but they are, they're being used in regulated industries, sometimes against the license agreements. Um, you know, I'm always quick to point out that, you know, OpenAI has great lawyers, Anthropic have, have really smart lawyers and they put in their licenses, hey, don't give us electronic health records, don't give us X-ray images, don't feed this to us because we are not equipped to handle it. It's not what they market themselves as.
And if they did market themselves as that without going through the FDA's processes, they could get in a lot of trouble. And so I feel, you know, those lawyers are very smart to not, to not put themselves in that position. But you're actually breaking your licenser agreement if you're a clinician or any other type of scientist and you're doing this without some kind of adjustment to that, to if you're just using the general license agreement, then, you know, it opens you up to some liability that once again, I'm not a lawyer, but it's my expert opinion that it really is not best practice to be doing that.
Christian: It could be common practice, but not the right practice.
Tyler: Yeah, yeah. Um, I, I know some doctors who are really gung ho and, you know, and and once again, I feel like this circles back to the harness product question because there's also a lot of products that we don't give them credit as being AI as a medical device, right? Um, there's a company, one that I'm relatively familiar with is there's a company based in Australia called Harrison AI. They've been in existence for several, several years now and they do radiology enablement products. Some of them are SaMD, some of them are not.
Um, and, you know, I, I've talked to some of the folks who do those evaluations, uh, and, you know, they, there's a lot of smart people working over there, and, uh, at the end of the day, they, they've made the correct decision in my mind on whether or not to market these things as SaMD or not.
And, and then also products like Viz.ai's product. Um, that was a company that was acquired by Google kind of before this AI boom that we're looking at. Uh, and they're, I would call them AI as a medical device because they develop their own model, it uses transformers on the backend mathematically, and it's providing a direct, you know, indication to the clinician on what to do next. And their device has definitely saved lives because it helps people diagnose strokes more safe safely and effectively.
And then Harrison AI looks at a broad suite of radiology products. And that component of it, uh, is really up to the company that's that's marketing the product. And if you, if you as a developer, you know, feel like you're doing the right thing, lean into that because I think we have a lot of great developers who have good instincts in this space.
Those harness products themselves can be safe and effective, right? Um, but you also have to look at what the company is marketing it for. What who, who are the people that are actually building this thing? What are they advocating for the use case of theirs?
And I think that the LLM product companies have, most of the time, been very responsible. Um, and saying, hey, we don't want this information, we don't need this information, you know, is a misuse of our license. And going back to Updox just for a minute because I, I really commend these guys. Um, I wish I had their 510(k) in front of me so I could give them direct credit. But, you know, if, if you Google Updox, you'll find them and you'll find their study. They're very clear in their study that their methodology is that they worked on top of Alexa, using Alexa as their, you know, AWS's Alexa product as their core, you know, LLM and then they built a harness on top of that that made it safe and effective for use in their particular case, which is helping people manage their insulin levels at home.
You know, it saves them a call to the doctor, it saves the doctor a telehealth visit, it saves, it saves all these costs and is a net public good. You know, it is a good thing that the FDA cleared this device when they did. At the end of the day, you know, they've been very clear, hey, we're a harness on top of a frontier model. And I think we're going to see a lot of companies come through that should be classified that way. Um, that's not our product suite, that's not what we're going to be doing.
But it's very possible to do that safe, safely and and effectively just like the Updox folks have and have shown and proven objectively.
Christian: Yeah, that's a I'm gonna have to go read their 510(k) submission. I haven't read it yet. But it sounds like, uh, they do it, the way they did it is a a good model to follow.
Tyler: Yeah, and and they have a special license to AWS, right? When I talk about, you know, when you're uploading this data directly into the LLMs, this isn't what their licenses really tell you you can or should do, they have specifically worked directly with AWS, um, to to have specific licenses. They call that out in the in their publications and in several, you know, public announcements that, hey, AWS is a partner in this.
And so I think that when you're building these harness products that are, that are going to be AI MD, I really recommend, what if you're a frontier model developer yourselves, you know, make sure you have proper internal controls around that model. But then, you know, all the design control elements of 21 CFR 820, those apply to us, um, as frontier model developers.
But if you're using someone else's and you're using them as a critical supplier under your supplier management, you know, process, make sure you have really clear cut license deals. Talk to someone at those companies. Um, because they're, they're going to help steer you in the right direction. They've got smart people and smart lawyers who can help you out.
Christian: 100%. Well, we're coming up on time, so I'd like to summarize, uh, Tyler, like what you think some of the key takeaways are from our conversation, maybe like the top two or three, and then I'll, I'll go, and then I'll go and, uh, we'll wrap it up.
Tyler: Sure. So I'd say everybody who's, you know, what I want people to take away from this conversation is that it is, it is possible to use frontier models safely and effectively in, uh, in your AI as a medical device products. You know, but then to our clinicians and our scientists out there, be very cautious about using just the, the sort of, uh, consumer versions of these models in your work because sometimes that's not what they're built for.
And kind of the third thing is that we as frontier model developers have a responsibility to the broad, you know, to the patients that'll be using, you know, have the product used on us and to the clinicians that will use our products to make sure we're following design controls and make sure we're considering the cybersecurity elements. I know we talked more about general safety, but there's a lot of specific cybersecurity elements that that roll into this. Maybe we can talk about that another time. But the, you know, those are sort of the three pieces I'd want people to take away from this discussion.
Christian: Cool. I'll give two. Uh, I talk about PCCPs, the predetermined change control plan, quite often when people are doing, uh, software with AI. I think it's important to understand how you're going to update the device and update the model and all that.
And the second thing you, you mentioned, which I'm an advocate for as well, I think AI is great in MedTech, but I think it's still a little immature. Uh, and you're talking about the human in the in the loop. I'm definitely a proponent of the human in the loop. Even here in Phoenix, Arizona, where I live, we have autonomous driving cars that have AI, Waymos, but there's always a human in the loop that can take over the car at any point in time. Uh, so it's even in other industries, like, um, automotive, they have humans in the loop. So I think we still need them probably for quite some time in MedTech.
So thanks, uh, Tyler, for joining us. I appreciate you coming all the way across the internet from New Zealand.
Tyler: Yeah, thanks so much for having me, Christian. Really enjoyed the conversation.
Christian: Yeah, awesome. And thanks everyone for tuning in. I hope you found value in this episode, and we hope to see you on the next one.