AI in Medical Devices: Opportunities & Regulation with Matt Lemay | Ep. 22

This episode of The Med Device Cyber Podcast features hosts Trevor Slattery and Christian Espinosa in conversation with Matt Lemay, the CEO of Lemay AI, to discuss the integration of artificial intelligence into medical devices and other regulated industries. The discussion provides a comprehensive overview of the opportunities and challenges associated with AI in medtech, emphasizing the critical need for a structured, engineering-driven approach to ensure safety, compliance, and cybersecurity. Matt Lemay introduces his company, which specializes in guiding organizations through their AI adoption journey, from initial strategy to implementation and regulatory approval. He draws heavily on his background in a medical device startup, where he was responsible for implementing the ISO 13485 quality management standard. This experience, he explains, has deeply influenced his company's methodology, which prioritizes the rigorous, risk-aware principles of engineering to build AI systems that are robust and verifiable, a necessity in high-stakes environments like healthcare. The core of the conversation revolves around the practical challenges of developing and deploying AI in a regulated context. Matt argues that the time for medtech companies to engage with AI was "somewhere between not yet and five years ago," highlighting the rapid evolution of both the technology and its regulatory landscape. A significant point of discussion is the emerging ISO 42001 standard for AI management systems, which Lemay presents as a vital, certifiable framework for creating traceable and compliant AI. The speakers differentiate between AI applications based on their intended use, noting that systems for exploratory data analysis face different regulatory hurdles than those making autonomous diagnostic decisions. This distinction leads to a broader discussion on the critical cybersecurity risks associated with AI. Several key risks are identified, with the most prominent being the "garbage in, garbage out" problem, where the quality of training data directly impacts the model's accuracy and safety, potentially leading to dangerous misdiagnoses. Matt also raises concerns about "data drift"—the degradation of a model's performance over time as real-world data evolves away from the training data. The conversation addresses the implications of deployment architecture (on-device vs. cloud), which affects everything from performance and latency to data privacy and sovereignty. A major unresolved topic is liability; using the example of ticketing autonomous vehicles, the speakers explore the complex question of who is at fault when an AI system fails. The episode concludes by touching on the need for AI to better communicate its own uncertainty, moving away from the tendency to "hallucinate" answers and toward a more collaborative interaction with human users, which will be essential for its responsible adoption in medicine.

Hello and welcome back to another episode of the Med Device Cyber Podcast. Today we're going to talk about artificial intelligence in medical devices as well as AI in regulated industries. We have a special guest on today, Matt LeMay, who is the CEO of LeMay AI. How are you doing today, Matt? Matt: I'm good. Thanks for having me. It's going to be a good conversation. Trevor: Looking forward to it. Christian: Yeah, it's a little bit of context uh, I met Melissa and I met Matt on the cultural tour, I believe it was at MedTech World Dubai. And then Matt gave me this awesome book here, the 50 inventions that shaped the modern economy. I haven't started reading it yet. I just got it a couple days ago, but I've kind of thumbed through it and it looks pretty awesome. Matt: Absolutely. So Christian and I were in the lobby of the Intercontinental at Festival City and you just see a group of people that are all CEOs and co-founders and engineers and medical devices and everyone's in shorts and flip flops and polos going on a tour. It was it was great conversations all around. We're I'm definitely looking forward to the next MedTech world event. Christian: Yeah, for sure. So you want to give us a little context about what you do Matt, what your organization does and and kind of and spin spin it a little bit on MedTech since we're focused on MedTech. Matt: Absolutely. Well, I think MedTech definitely shaped the entire growth of our organization. So, my team, Lemay AI, we're a team of 30 people, about 85% engineers that specialize in helping clients on their AI adoption journey specifically in regulated industries, including MedTech. We do this by delivering tailored AI solutions at whatever stage you're at. So, if you need strategic guidance on which projects you should implement, if you need some core implementation support, if you need some scale up or even now regulatory approval as AI is becoming more and more relevant in each one of these regulated industries, we have a framework for helping clients along this journey. Specifically about myself, I actually came from a medical device startup where we implemented ISO 13485 from scratch where I met my now co-founder, Daniel. And we did a lot of work back in the day not just in systematizing our design and development processes to be able to achieve a CE mark, FDA approval processes, Health Canada approval. Each one of those compliance mechanisms requiring a certain amount of of oversight and that's what help shape how we actually do AI. So, part of what we do now and how we do it is heavily influenced with engineering principles that are required to comply with a lot of these emerging standards. Christian: So when would an organization uh, like say a MedTech manufacturer. When would they want to engage with you if they're going to do, let's say image enhancement using AI on software as a medical device? Matt: That's a fantastic question. The there's a lot of ways of approaching it, and the simple answer is somewhere between not yet and five years ago. So, at the not yet level, what we're seeing in the regulatory changes and landscape is a lot of people have been pushing for various governance frameworks on how to do safe AI. The thing with a lot of these governance frameworks is that they're hard to audit and they're hard to verify and therefore hard to include in a lot of these medical device oversight frameworks and audit processes. What we're seeing right now is that there's actually a lot of work happening under one particular standard, ISO 42001, which specifically prescribes how to manage your AI systems. So, 42001 being called AI management systems. What we find very, very interesting with that standard is that it is certifiable. So for the first time, you can have an AI included in your medical devices that is able to be verified by an external third party. When it comes to image recognition, what you also have to look in mind is what is the purpose of this image recognition which will immediately impact the strategy that you want to pursue and whether or not a team like ours actually makes sense. So, if you say, I want to do exploratory data science, and I want to look at the pictures of dermatology, x-ray, I want to look at cells at the microscopic level to understand how they're moving, how they're understanding. And your intent at that level is much more investigative, it's much more open-ended, then you really don't need a lot of certification. You can actually engage in that direction as long as you're in respect with GDPR and a lot of these uh, data protection mechanisms, you're going to be fine. Where you have to tread a bit more carefully is when you start making diagnostic calls or when you start having a lot of decisions being automated through the information that you see. Now you want to make sure that there's a series of systems in place to ensure that whatever you're prescribing to a client in terms of recommended treatment, or if there are some particular categorization that you want to pursue, you at least want a bit more structure in in your processes that touches not just a code, but also the policies and procedures on how you do the work. Christian: So if a client comes to you and follows the standards and and does it does AI from a proper engineering perspective? Do you feel like the AI would be more secure from a cyber cybersecurity perspective? Matt: There's a strong overlap. So if you look at a lot of these cybersecurity standards, like I saw 27,000, if you look at GDPR, there's a lot of prescriptivity around how the data should be handled. Now with the AI management systems that are designed to properly fit into a lot of these frameworks, they tend to have a lot of prescription of what you do with the data. So now you have how you manipulate the data, you have what to do, and together you have a sense of why you should go in a particular direction. It'll help prescribe what are what is your business logic, it'll help dictate what can and can't you do with the data that you have. Do you need more data? So we find that together they mesh quite nicely. Christian: Hm. Trevor, what do you think uh, some of the biggest threats that we've noticed with our clients about AI are? Trevor: I think it depends on the exact type of AI if we're looking at, you know, like a large language model as opposed to sort of a pure machine learning application. But I think the biggest thing that can come up, especially in that diagnostic space like you were mentioning, Matt, is if the AI is not properly trained on the correct data. So, an issue can be if you're feeding in, you know, some grainy images into the AI and it's not getting a good understanding of how to look for something like cancer. Uh it's obviously not going to have the information that it needs to make the correct diagnostic call and so you're going to get inconsistent results and you might trigger a misdiagnosis. So I think the training data is a pretty critical thing that needs to be considered. Matt: Oh, absolutely. We've seen a lot of examples of people that start working on a particular open dataset only to see that the size, the resolution, the color scheme of the images that they were using was completely different. Not something that is easily uh, perceivable when you start looking at the data, but unless you actually do an audit and a verification of what happens post-deployment, what type of data drift you might be manifesting, that absolutely has been a big issue. The other big cyber security risk that often people forget when there isn't a good meshing between the data scientists and the machine learning engineers is where does a model actually exist? Is it going to be on the device? Is it going to be on the local computer inside the clinic? Is it going to be in the cloud somewhere? If it's in the cloud, under which geopolitical segment it doesn't fall under. All of these start touching on data sovereignty issues, cloud sovereignty issues, um, privacy and security. And depending on where it resides, you're actually going to see a lot of changes. One thing that we've seen as well is sometimes depending on API calls that people like to use early on because you can just send the data and it takes care of itself. There isn't a strong compliance or version control mechanism. So without you knowing, it is possible that whatever cloud provider you use updates the models, starts changing the results and now suddenly your device is less performing than it was before without ever, uh, having received the type of notification or some notice that there was a change actually happening. So anything that you can do to maintain sovereignty is absolutely effective. But that's also something that you see just at the communication of the data level, right? Like what how do you guys go about making sure that when you have these images get captured, how does it go back and forth between the devices, the cloud, the cell phone, and everywhere? Trevor: Yeah, I think that making sure the data is protected is another huge concern as far as AI goes. Um, had a call not too long ago with someone who had an idea for an AI product that was going to handle a lot of PHI. They said, "Well, can't we do this with just open AI API calls?" I said, "Well, you can. It's probably not the best idea if you're holding people's uh, you know, health information. So I wouldn't recommend it." Matt: Yeah. There's, um, that's a super interesting uh, topic because we've seen personal health information, personal, uh, personally identifiable information, and also the annoying Venn diagram where they start overlapping quite aggressively. So, we've seen, for example, in the past, we did a lot of work on progressive anonymization. So, very relevant if you want to understand statistical representations of various population pools, it it's kind of relevant to start shifting people's birthday by a few days. That usually doesn't have a lot of impact. Where it starts having a lot of impact is, for example, cancer medication, HIV medication. How do you continue representing that information in the statistical data without making that original data set traceable to identify the core individuals if you're in a certain geography? So, even when you start looking at basic calculations, if you open the file in excel, there could be a risk if you're not careful with this type of data. So there's certainly a lot of these considerations early on that you need to take into place. Christian: You know, one thing, you touched up on medication there. So, there's a bill proposed that is, you know, getting back and forth in the government right now on allowing AI to prescribe medication to patients. I'd be curious to hear your thoughts on what some of the implications could be there and, you know, what are the risks of letting an AI make that decision, prescribe medication to someone based on its own assessment of a problem? Matt: Let's start at the the system level of just what happens when you have an autonomous agent making decisions. Whatever be self-driving cars, whether it be controlling boilers, or in this case prescribing medications. There's the first question, which is, if something goes wrong, who's at fault? And with doctors, it's pretty clear. Ultimately, the doctor has a certain insurance that they work under. The same thing with other large manufacturers, there's a lot of coverage in making sure that there is the attribution of blame. What the AI conversations when it comes to automation, what they've opened up is a lot more complexity without necessarily attribution of blame potential. So, if you're working with cloud providers and they have their own APIs, is it the person that issued the change order? Is it the person that tried to increase the amount of data, or is it you ultimately the person that built a device that is responsible for having used this algorithm? So that's kind of the very simple question of what happens when something goes wrong. The other thing that we tend to see is what happens at the feedback loop? So if people are unhappy with their prescription, is there sufficient responses in the rest of the system to be able to make some adjustments, to make some, uh, some tweaks. So, uh, I have a few, uh, acquaintances that are in the, um, uh, addiction management space and they tend to have a pool of patients of 400 to 500 people and continuously see what they do is monitor, adjust, tweak, and intervene if necessary to be able to help these people return to normal lives. And what we tend to find is without the particularities or the the additional curiosity on certain questions, it's hard to get the feedback. The last part that is actually the most interesting for a company like ours is actually, how did you achieve the decision process for issuing a certain type of medication in a certain amount of time. Uh, why this medication versus another and the entire medical rationale? Is it because it's the most relevant, um, prevalent medication in the literature? Is it because it's the best medication? Is it the latest one? Or, uh, cynically, is it possible that there are some third parties that are both investors in your new LLM startup that you're using that also have a stake in a particular, um, uh, pharmaceutical company? So, there it needs to be a bit of adjustments there, but typically we tend to see that as long as people are pursuing the best type of model, then you have a lot of advantages in making sure that your solution is delivered successfully. When you're talking about the best type of model too, that's where you have to be careful. Are you just chasing the highest F1 score? That's good. Are you making sure that there aren't any false positives or false negatives in what you're prescribing? Very sensitive when it comes to cancer diagnostics. Are you simply trying to have a system that is efficient and quick and doesn't cost too much to run? That also may have some implications on the type of models and the types of design structures that you have. But overall, we think that this type of bill needs to take into account the entire pipeline, not just the fact that some people can over their cell phone have a quick, um, chat with a chatbot and then receive pills in the mail. We want something that's that has a bit more oversight. Christian: It sounds like it's a bit premature based on what you're saying. Matt: So, let's open up that, uh, that conversation because I find that often there's, there are regulations in place in order to protect the public, which are absolutely necessary, but there are also needs to be some permissiveness and some aggressiveness in letting new technologies find their way into the real world. There's a lot of mechanisms in place, whether it be journalism, whether it be public complaints, whether it be reporting channels to ensure that these companies that operate in contact with the public do have a means of providing a certain level of, um, oversight and feedback if their product is not working successfully. But what we find is that the regulations will always try to find what is the most amount of protection while balancing innovation, permissiveness with public safety. So together, there's a fine line that you can, you can navigate. Trevor: You know, kind of a little bit of an anecdotal story on your initial point on who do you blame for AI. So, like Christian and I live in Arizona, and Phoenix is, you know, one of the three cities with Waymos. And Waymos can pick you up at the airport. It's pretty cool now. So whenever I'm getting back from somewhere, I leave my car at a hotel, get a Waymo to go there. Christian: Can they actually pick you up at the terminal though? Or you still have to take the? Trevor: Oh, they can. They they can pick you up at uh, in Phoenix airport. It's the only one. But I was wondering what happens if a Waymo's in the wrong lane, or there's a crash, can you ticket a Waymo? And so I look it up and in Arizona, you cannot give a Waymo a ticket. You have to give a ticket to an individual and not a car. So if the car makes a traffic violation, even if it creates a hazardous situation, there is no one to blame. So you cannot hold the car at fault, and so you can't give it a ticket. It just kind of goes away. Matt: That's super interesting. And it's there is a shortcoming in the the legal reporting structure. The one the one structure that I have seen that is that is possible is the same thing with what happens if a bridge collapses. Well, if a bridge collapses, there was a chief civil engineer, typically and usually, a professional engineer that signed off on the design, the implementation, and the overall safety of this solution. And this is true in multiple contexts. I remember even when I was a lifeguard, we had a chief medical officer for the overall allowance of letting lifeguards perform paramedic activities such as CPR and under certain circumstances defibrillators, additional injection when they were offsite. There's a lot of complexity with those types of structure but it ultimately reported to the top with a single individual that was able to sign off on a lot of these activities. So, when we look at a lot of these other technologies, there is something that follows any other industry, whether it be mechanical engineering, whether it be civil engineering, there is a mechanism through professional engineering to certify these devices and hold people accountable when something goes wrong. Christian: So, in that case, the car would not get a ticket but whoever certified the car would effectively get a ticket. Matt: That's correct. The individual that is in charge of the project, the individual that is licensed by the state to build intelligent agents and deploy them into the real world would be held responsible. So Christian: That's a whole lot of responsibility. It's especially if you have like thousands of these Waymo cars driving around in three different cities in the United States. Matt: It is and and that's something that we've seen happen quite a bit in different circumstances. So, the if you look at the concept of professional engineer, something that I'm quite proud of, myself and a lot of people that I know do sign off on a certain amount of responsibility when they deploy designs and solutions. So, engineering can simply be called the risk-aware application of scientific principles, emphasis on risk-aware. So, same thing with medical devices. How do you know that if something goes wrong, there's a reporting mechanism to ensure public safety? So, if your device goes on the fritz and a doctor complains, you need to be certified and licensed as a company to be able to effectively handle and report that complaint in a way that makes sense. So there's a simple question, why if a taxi cab that has its front camera covered and it just starts plowing through a crowd, why hasn't that design been licensed or certified? What I believe is also part of the case and I would like some additional verification on that. I believe in Arizona with Waymo, there are special licenses and special permissions for that vehicle to be on the road. However, there needs to be also a ticketing mechanism if the vehicle does not operate adequately. Trevor: One interesting thing is they can get parking tickets, but they can't get traffic tickets. Christian: What what made you look this up, Trevor? Trevor: I was in a Waymo and I was wondering what's going on. Like how does this thing work? I'm sitting in the back of a driverless car. This is weird, what what's going on here? Matt: Doesn't it give you more time to think? I love those interactions where you're like, huh, now that I don't have to spend all my focus on not crashing, look at all the three thoughts I have. Trevor: Oh yeah. I think it's I think the regulations are a little bit different in California than Arizona, but yeah, regardless, having a system where you can't like Waymo should get the ticket. You know, the car obviously can't. There's no driver but Waymo as an organization should probably be the ones to get the ticket. Christian: Well there's a, yeah and this is kind of a point of contention I think in in MedTech and healthcare. And I just had someone asked me about this today in a podcast, if I'm a medical device manufacturer and I create this this product with AI, let's say, and I'm a healthcare provider and I buy that product and accept the risk of that product and something happens, who is actually liable in that scenario? Matt: I think that's on a jurisdictional basis because right now we've seen, for example and and there's something that we're expecting to happen in Europe over the next few weeks now that they're a new government in place. Specifically in the MedTech space, we believe that AI-enabled devices are going to require additional certification and therefore through that lens, the ability to have AI-enabled capabilities on a licensed basis. As in, I built a solution, my solution has AI, my solution has been deemed safe and the AI within that system has also been been deemed sufficiently safe in order to to be exposed to the public. With that context, we find that there's going to be a new mechanism to allow companies to be certified. The reason last year we had our leadership team be certified as ISO 42001 lead auditors is we believe that the only way that you can really achieve that is not going to be through uh various governance standards or governance frameworks that exist, but it's going to be through some sort of certifiable standard. Not to kind of rehash what we talked about at the beginning of the podcast, but I think it's very context-relevant here. If you look at what uh states are putting forward across the US, every major state whether it be New York, Texas, California, a lot of these uh these other states. I think even Wyoming was talking about a new AI legislation just to make sure they're part of the conversation. You know that there's there's going to be a lot of changes coming. We did a project a few years ago with uh Justice Canada here in Canada where we've actually helped uh update laws and regulations using AI. So try and to scan the stock of documentation to identify all third party references to see if these references were still relevant. And through that, what we found is how these laws actually get put into place. So you have this big law, the EU AI Act, CCPA, whatever law gets established, there's a set of criteria and there isn't solutions, there's always penalties. They they always talk about fines and jail time, they never talk about how to actually do it. So those laws then reference a regulation. So in Canada we have the the medical device act, I believe that references the medical device regulations and a lot of the states work on the same fashion. And then the regulation prescribes the use of um, ISO 13485. So that link is called corporation by reference. And what happens throughout that chain, you have the law, you have the regulation, you have the incorporation the standard. The standard can now be audited. So, think of every industry now having that requirement for verifying the AI with full traceability. So, if you're a medical device manufacturer, what you have to look at now is, I already have a quality management system in place. So this this process is not new to me. It's going to be a nice, very natural extension of what I'm already doing. In fact, we've recommended not even getting certified in this standard. We've just walked our clients through what is the process to get certified and they're like, oh, this actually makes sense. This is this is going to help our our team members work together and uh cohesively across all these different channels. So why not have everybody work the same way within our organization? Together, we find that all of these standards and quality management systems, the laws does provide sufficient oversight at the manufacturer level. So going back to your question, if you're a provider who's ultimately at risk, that manufacturer will have built an AI-enabled device that will have permission from the state, from the country, from the jurisdiction to be used for a specific purpose. So that's what we think is going to happen, uh over the next few months. Christian: We we talked about the one of the biggest threats to AI is training the model properly. Uh in addition to that, like what what's something else, like if I'm a a healthcare delivery organization or I'm a manufacturer, what's like one of the other top things I should be concerned about with AI? Matt: There's a few. So one of the the big risks that we've seen is uh the this is more at the at the manufacturer level, but it's understanding, uh, how will the AI be actually operated? Where is it going to live? How is it going to work? The if you have all of your uh, your data processing on all your big machines, is it going to be deployed to your wearable? Is it going to be deployed to your smartphone? Is it going to be deployed in the cloud? You have to think about that at front. And that's actually a risk because if you start thinking about building this pristine, beautiful, perfect model, and then it doesn't fit on whatever device you run it on, then that's going to be a problem. Same thing if you leave it in the cloud. Is it a critical application? Does it need just in time results? Do you need extremely low latency? Internet connectivity is good. It's never great and it's certainly never perfect. So, are you going to recommend that a surgery room robot has all of its inference running into the cloud? No. So immediately you're going to have to think about what are the costs to making sure that there's sufficient performance on site on premise to be able to provide the value that I promise to my clients. So that's something that we want to make sure that get that gets discussed up front. The other thing that we're seeing is uh close but not identical use cases. So if you build your model a certain way, and if you apply it in a particular uh for a particular process, it's going to be very, very hard to simply repurpose it or move laterally to be able to pursue off-label uses. So the device that we built a few years ago which was transcranial dry current simulation, whatever uh treatment that we were recommended, whether it were the official treatments, for example, chronic pain treatment, and we wanted to use off-label use treatments such as smoking cravings reduction, those protocols were the exact same. You just had to press start on the machine and it would do its thing. So it's the exact same thing. We just had a recommendation of changes where the electrodes were placed on the brain whenever that treatment happened. But that's going to be very, very difficult to do with AI unless you've taken those use cases under consideration. The last thing that we're seeing is how do you ensure that whatever you deploy can maintain performance year-over-year even as the libraries change? So, we've seen for example, whether it be Python, there's a major revision every few months or every 12 to 18 months approximately. Well, if your device is going to have a shelf life of 10 to 15 years, how do you ensure that the protocols are always up to date? Do you have a clear ICD between the device and your cloud? The uh, that risk in particular, we've seen that the right deployment strategy makes a huge difference. So, instead of having just an image recognition model, or let's say, we do a time series pattern recognition when it comes to EEG signals, EKG signals, or other types of ongoing monitoring, then what you actually should do is convert a a very boutique, a very niche, uh library piece of code and convert it back to polynomials because polynomials would be basic math and you can actually run that on any device, any microcontroller, anywhere in the world. Christian: Can you explain what the polynomial is to Matt: Oh, it's just the like, oh, you remember your high school algebra? So, a squared plus b uh, x plus c equals something. Well, you can actually convert a lot of complex models to a to a chain of variables that gets computed. What we like about that is that is verifiable. And that's actually what we did with one of our devices, we um, one of the patents that we wrote was on neurological energy delivery safety systems. So, how do you make sure that the electrodes that are in contact in the skin have sufficiently good contact in order to have a safe and effective treatment? So, what we did, we actually wrote, uh, we were investigating micro pulses, so small electrical pulses that based on the amount of contact of the sponge on the skin, it would change the RC curve for you electrical engineers out there, and it would actually be able to change the pattern of the the curve as we're recording it. And originally we were using very, very complex models to trace the shape and just optimizing it through converting that crazy model to very simple math. We actually reduced the compute requirements of the microcontroller from something like, I think we went from 13,000 clock cycles down to 400 clock cycles down to seven. And so it was just squeezing the most amount of performance, but we were able to mimic that shape into something that was much more tenable for these uh, extremely low power microcontrollers to do calculations on. So again, just to recap, we find that those uh intelligent conversions and moving away from uh pure uh AI libraries and back into something that is more math-based does provide um, some long-term viability to the solutions. Christian: Yeah, for sure. Especially if you have like an implantable with limited battery life and everything else. Matt: Exactly. You want to distinguish what should be processed on the chip, what should be processed off chip. You also have to think about there's going to be very limited firmware updates, especially if it's implantable. I think a few years ago, there was also a lot of risk and cybersecurity issues in low energy Bluetooth, for people that were hijacking people's pacemakers. Christian: Yeah. So one of the things Trevor likes to talk about, he talks about, he likes to talk about a lot of things, but one of the things is that it says AI is created by humans, AI has a hard time admitting it doesn't know the answer to something. I'm just curious like what in the context of like a medical diagnosis or or recommended course of treatment, what are your thoughts on that, Matt? Matt: Oh, there's, there's a lot there. So the first one is at the problem level. So let's, let's contextualize this to LLMs. So, at the prompt level, there's a lot of experience that you can even run with that you can do with your friends. I think I have uh, one of the old uh CIA playbooks on how to actually recruit people, and there's certain ways of styling questions to make sure that people do come by with what you do. So the simple example is, um, do you want to do A or B? And it's a false economy, right? there's millions of choices that you can make at any second of the day. But if I just present you with A or B, you're kind of on the spot and you're like, oh man, I need to evaluate A versus B. Well, I need to move forward with either A versus B. And it usually takes, um, that's how bullying works, that's how gaslighting works, that's how very intimidating meetings get run when in reality that's not the right conversation. So, accidentally I find a lot of people do introduce that in their prompts where they go, uh, would it be category A or category B? And you're like, well, no, that's the wrong context. So, even when we do, let's say, LLM-based, um, dataset synthesis automation where we create a dataset and then convert it to more complex models, we always give the LLM the option, it could be category A, could be category B, or it could be something else. We always give it that buffer. That's that's one thing. So, we find that right now people are learning more and more on uh prompt engineering, how do I actually write the description in a way that they can interact effectively with with the model. There's also the flip side, which is how can builders of LLMs and people that build these soft these pieces of software actually improve the interaction with uh with their users. And there was a very interesting paper that came out a few weeks ago, which was, um, confidence adjusting prompts. And I'm not quoting it correctly, and I'll, I'll show you the link after so we can, uh, we can reference it. But the idea was to increase the type of interactions that the LLM would have with the human in order to shape its self-confidence. Self-confidence shaping, I think is what the paper was called. So, if someone came in and was uh, prompting something very exact, very precise, and very direct, the model would kind of slow the person down a bit and say, are you sure? Are there different ways? Have you consider everything? And provide a certain level of assurance to make sure that that individual at least had thought about all of these possibilities. And also on the flip side, if someone wasn't sure or they were unclear, the LLM interaction was there to boost that confidence to say, no, no, like you, you're here to make a decision, let's, you know, you have the facts, which one would you like to do? So that type of interactivity with the models, we're seeing a lot of a lot of research on the psychology side coming up. Trevor: Yeah, I think it's, you know, super fascinating stuff, the direction that AI is going. You compare it to I mean, even a year ago, even six months ago, just it's completely night and day. I always think that it's interesting during those early phases of AI, anytime you'd ask it something, I mean, the hallucination rates must have been over 50%. It was ridiculous. You know, when ChatGPT first came out. and I would always think why is that? And you look at what that's training it's training itself based on the internet it's training itself on a bunch of information out there. And how often do you Google a question online and you open up a thread on Reddit or something and the response is, huh, good point. I don't know. It's always some answer, whether it's correct or not. and so that's, you know, it's going to respond the way it's trained to behave. It's going to respond by saying, you know, either an incorrect answer or a correct answer, but it's going to give you an answer. Matt: Yeah, you see that also with humans. I call it, I think the karaoke complex, also aka the idiot with a microphone. You puts someone with a microphone in a channel, they'll find something to say. It's I think kind of that we're talking about judging different podcasts. But I think there is something to be said about what happens when you ask someone just to produce sounds with their mouth for a certain duration amount of time. Is it all coherent? Is it all well thought out? Is it all available? Same thing with a lot of those, right? They sometimes they need a bit more time to make sure that the underlying content is is proper. And to your point, there have been a lot of improvements on not just the ability for them to have better internal representations of the training data, but we're also seeing a big change in maximum context length. Maximum context length is super important because right now part of the limitations and the latest information I I have is it's really good or LLM's seem to be getting prototype right now at the 1 million to 3 million uh token input length. So we're talking about 1 to 3 million words, like for reference the full King James Bible, I think has 700,000 words or tokens. So that's a lot, but and you can certainly analyze for example, a few public filings if you're doing stock market analysis, but you can't look at all of the 10K filings at once. So we're still very, very far from that. So if we're looking at simply not just the training data, but also looking holistically at the amount of information that is available in order to inform that decision, then there's still some limiting um strong limits as to what a system might be capable of doing. Hopefully a few things that are going to change over the next few weeks. Christian: I like the uh karaoke complex. I've never heard that before. Trevor: But we're the good ones, right? We're not we're not like them. Matt: Exactly. This is top notch, top notch. Christian: Well, we're we're running up on time here. I always ask people if they have any uh last words of wisdom. I'll start with you, Trevor, and then I'll throw it over to you, Matt. Trevor: Yeah, I think, uh, you know, kind of the rule applies to many different things, but AI especially, garbage in garbage out. Make sure that you're getting the right training, make sure that you're taking care of your AI on the front end, and then you're going to start getting quality results on the back end. Matt: Absolutely. Uh, what are my recommendation, just like think like an engineer. There's been a lot of data science that came from software in order to build these systems. And often people are focused on the problem space and the definition of problems. But instead, engineering is based on identification of objectives, requirements, and delivery of a solution. So if you look at what are the limits of AI, think of it in terms of problem solving and suddenly it's going to become a lot easier to understand what you can and can't do with your system. Christian: Awesome. Well, we appreciate you uh, being a guest on the Med Device Cyber Podcast, Matt. Where can people find you or your company? Matt: Uh, you can Google us at lemay.ai, l e m a y.aii. Uh we also write quite a bit on medium and our own internal uh blogs and we participate in a lot of conferences around the world. So check out our LinkedIn page to find out where we're next. We just came back from Dubai, Miami, Montreal, and now I think we're going to Zurich. So, Christian: You're all over. Matt: We're uh, we're traveling and you can catch us. uh as we're flying over or flying next to you. Christian: All right, awesome. Well, thank you so much everyone for tuning into the Med Device Cyber Podcast. We hope to see you on the next one.