Total Product Lifecycle Security: From Design to Disposal | Ep. 27

In this episode of The Med Device Cyber Podcast, hosts Trevor Slattery and Christian Espinosa delve into the critical but often overlooked concepts of Total Product Lifecycle (TPLC) security and the Secure Product Development Framework (SPDF). They begin by defining these terms, explaining that TPLC encompasses the entire lifespan of a product, from the initial concept phase all the way through to its final decommissioning. The SPDF, which is closely related to the Secure Software Development Lifecycle (SSDLC), is presented as an essential component of the TPLC. It provides a structured approach to ensure security is integrated into every stage of the product's ongoing development, rather than being treated as an afterthought or a final checklist item before release. The hosts argue that neglecting these frameworks is a primary reason why many software products, especially in the medical device field, end up with significant security vulnerabilities. The discussion further explores the nuanced relationship between these frameworks. Trevor clarifies that the SPDF and SSDLC are subsets of the broader TPLC philosophy. While TPLC demands a holistic view of security from cradle to grave, the SPDF/SSDLC focuses on the cyclical, iterative process of development, including code reviews, static testing, and verification at each step. Christian provides compelling real-world examples to illustrate the dangers of ignoring the full lifecycle. He recounts instances where decommissioned medical devices and even classified government printers were sold online with unencrypted hard drives, exposing sensitive data like Protected Health Information (PHI) or classified documents. This highlights the vital importance of the decommissioning phase, a part of the lifecycle that manufacturers frequently forget. The hosts also address the considerable challenges, particularly for startups, in implementing these comprehensive security measures. They acknowledge that creating a robust SPDF and adhering to the TPLC is expensive, time-consuming, and may not add immediate, tangible value in the eyes of a company focused on getting a product to market. This often leads to security being pushed to the back burner. However, they stress that this is a shortsighted approach, as the value of security becomes painfully apparent only after a breach occurs or when regulatory scrutiny begins. The conversation also touches on securing the development environment itself, the risks associated with third-party contractors, insecure update mechanisms (like using a public kiosk to manage updates), and the simple human errors that can undermine even the most sophisticated systems.

Trevor: Hello and welcome back to the Med Device Cyber podcast. We're joined here by your co-host Christian Espinosa. And today we're going to be talking about total product life cycle security and developing a secure software development life cycle. How are you doing today, Christian? Christian: I'm doing good. This is one of my favorite topics. It's something that is uh the TPLC or the SPDF uh is something that's commonly neglected and it's often why software ends up unsecure or insecure in my opinion. Christian: But yeah, I'm doing pretty good. I did put on some, not enough, it wasn't SPDF. It's uh, what what do you call that? The S, the sunscreen. I had I put some of that on the other day. I went to the beach. Trevor: What? Christian: SPF. Trevor: SPF. Christian: Yeah. Trevor: There you go. Christian: So I still got a little bit sunburned. Um but yeah, I'm stuck uh stuck in Florida for a couple of days. Uh worst place to be stuck I guess, but uh, you know, travel delays going through Dallas. I figured I'd rather be stuck here than Dallas. So that's what's going on with me. Trevor: Seems like there's a delay anytime you fly through Dallas. I think they're just not used to weather at all there. So if the wind shifts the wrong direction, they shut down the airport. Christian: Yeah, so maybe I'll avoid Dallas next time. But yeah. Christian: There you go. Christian: So, let's go back to TPLC and SPDF. Um, SPDF. What what, are those like the same or are they really different? What do you think? Trevor: So, TPLC and SPDF, I think that um, the SP, the Secure Product Development Framework is part of the Total Product Life cycle. Trevor: So when we're looking at total product life cycle, total is kind of the key word there, it goes from the concept phase all the way to decommissioning whenever you're done supporting the product. So it needs to cover everything. Trevor: The secure product development framework, product development is an ongoing effort. It's not a one-and-done situation. It keeps going throughout the life of the product. So, it's a framework that ensures you are managing security at every step of the way. You aren't missing any big considerations. Um, you're designing it with security at the front of mind, you're performing regular code checks going through the design process and you aren't leaving security to just a time block at the end. Trevor: Um, that, I think the synonymous part is the SSDLC, the Secure Software Development Life Cycle, which goes into that cyclical process. So you're making a change to it, you're reviewing the change, you're implementing the change, you're testing the change, and then you go in again, you're making another change. Trevor: All of that needs to have security at the front of mind. There are a lot of processes, a lot of tools, lots that goes into a secure software development life cycle. Christian: So the SPDF, the secure product development framework is really synonymous with an SSDLC, a secure software development life cycle. Would you say that? Trevor: Just about, yeah, they're pretty similar. And then all of that ties into the total product life cycle. So it's a component of, you know, the full, the full product. Trevor: Obviously, the development is the main part of the product. You have an initial device, you keep making tweaks and changes, you keep developing it, you keep changing it. And so that's what we're looking at with the total product life cycle and that's what we're looking at with the secure product development framework. Christian: Yeah, and I think the total product life cycle is something that needs more emphasis and I think that's why it's a requirement now. I know in the past, I've worked with a medical device manufacturer that had the assumption, which is a true assumption, that the device would be in a secure uh room in a hospital. Christian: But what they did not consider is when the device is decommissioned and the hospital no longer wants it, what were they going to do with the device? And these devices did not have encrypted hard drives. So the hospitals were getting rid of these devices, people were able to purchase them off of eBay and other sources and grab all the PHI off the hard drive. So they totally kind of forgot about that whole decommissioning and the security involved with that. Christian: And this even applied, like back in the day when I worked for the government, the DOD, they would get rid of like printers, and these, some of the printers were classified printers, and they would just sell them to whoever wanted to buy them. And a lot of these printers had hard drives on them with classified documents. So I I think it's extremely important to think about from, like you said, concept to decommissioning the security in that entire process because it's often forgotten about once the product is sold. Christian: And that's even like, there there's post-market management, which is once it's sold, but there's also the decommissioning aspect and both of those need to be considered for the TPLC. Trevor: Yeah. Yeah, it all goes in all the way down the line. And a lot of times, yeah, manufacturers aren't thinking about that. They say, "Well, when the device is done, it's done." But if you throw that in a dumpster and it has patient records on it, um, I know now everyone's pushing to like cloud EHR systems, but before when EHR was coming into play, everything was on-prem. And then when you have these on-prem EHR systems, people would just leave them on the computer and throw them out, and someone could go dumpster diving and then pull out thousands of patient records. Trevor: And it seems like a bit of a silly way to hack the organization, but often times the easiest way is the path of least resistance. Going into the dumpster is a lot easier than trying to hack from the outside in to a hospital network. So, everything needs to be considered start to finish here. Trevor: I think a really important part of it that's also often overlooked is what is the development environment like. So, not just how you're building the product, not just the code going into it, but who's interacting with that code? How are you securing the code? How are you securing your workspaces? Are you just renting a WeWork office and leaving your laptop there when you go for lunch with an unlocked door? Anyone can make any changes that they want to it. Are you enabling multifactor authentication on your GitHub? A lot of this stuff, manufacturers aren't really thinking about, and that can be a very, very easy way for their device to be tampered with or compromised. Christian: Yeah, and I think something that's also not considered very often and we've had a couple clients um have some challenges with this area is the update environment as well. Because when we're talking about the TPLC, we're talking about and the SPDF as well, is if we need to patch our system because there's a vulnerability that came out, how do we securely develop that patch, which is one part of it. That's more the SPDF portion. But then, how do we securely deploy that patch, which is the other part? And I think a lot of people just don't bother to consider that aspect of an attack. So because of one of our clients, and I I think you're familiar with this client, one of the challenges where they, they're able to do over-the-air updates to their products, but the environment they were doing the over-the-air updates from was basically their corporate environment. So they're, like, engineers are writing the patches right, we're connected to the same corporate network as HR and the accountants and other people that could easily click on a phishing email and contaminate the engineer's laptop, which could then therefore contaminate all the products via the over-the-air update. Christian: So, I mean, all these things need to be considered and it's really a lot to consider actually. Trevor: Yeah, we had um, kind of a similar situation where the client had a device and the device itself was extremely secure. We were helping test it, helping secure it, and we were just kind of drawing blanks trying to hack into it. There was not really an easy way to hack into this product. Trevor: And what it was doing, it was taking in some information and then it was receiving updates, it was performing analysis on a local machine, receiving updates from the cloud anytime there was a change. Trevor: And you can see, you can capture that update process. It's encrypted, of course, because it's going over the open internet under HTTPS. You even look at the cipher suites, that's a common problem. So what type of encryption are you using on that transfer? Everything was secure. Trevor: And then we looked at the sort of the password for that transfer and it was pulled straight out of Stack Overflow with a publicly disclosed vulnerability in the key itself for the encryption for the data transfer. So the encryption was secure, but the key was something that could be easily guessed and had been easily guessed before. Christian: It doesn't matter how great the encryption is if the key is very simple and you can break it, right? Trevor: And so, we were able to just guess the key and then we pushed out, we could push out our own updates across all of the fielded devices with whatever we wanted on it by tampering what's going on in that update server. Christian: So that's one of the vulnerabilities with uh, I I think like there's always like the convenience factor of OTA, over-the-air updates, people like to say. But there's also a major pathway for attackers. Like the scenario you just gave, if I can hack in the infrastructure, I can push a malicious update to everybody, very easily. Trevor: Exactly. And you're not going to have that consideration if you have service techs take a USB thumb drive and upload it directly into the device, but then you have to deploy service technicians to all-fielded locations. So, trade-offs on both sides. Um, are you more worried about convenience or likely easier security? But even having service techs go out, what if, you know, they go to the bathroom, someone swaps the USB drives, things like that. There are always considerations there. Um, so there's never going to be perfect security. Christian: Yeah, like I'm in a hotel and I had to go print something. Um I was supposed to like wet-sign it, like, you know, with a pen. So I have a USB key and that there's a kiosk computer, but I was like, I'm just going to like sign it electronically and see if they notice it's electronic versus a wet signature because I didn't want to go and put this USB key in the kiosk computer. And if I am like, let's say I was the update engineer, I've got the the the update on here, but then I go use a kiosk in a hotel, that computer, which those are notoriously infected, you know, then now my update is going to be infected. So you have to have a whole process around that as well to make it secure. Trevor: Mhmm. Yeah, and so that goes into the total product life cycle. You have to have all of these processes covered. You have to touch up on every single area of security. You can't leave any base uncovered. And so, you know, who is doing these updates? Are are you outsourcing it to contractors to do these updates? How can you trust these contractors? Who is doing your development? Is it done in house? Do you know if, you know, you're outsourcing to a contract manufacturer? How are they developing this code? Are they doing it right? So there are so many questions that can go into this and it's always going to depend on the device, the environment, how you're developing it, where you're developing it, but all of that needs to have an answer. Christian: It's almost like a table-top exercise you sit down and think of all the scenarios and the data flow for everything about your product and how you update it, how you decommission it. You have to like, think through like every use case basically to make, make sure this is, uh, I guess, totally covered, right? Trevor: Yeah, exactly. And it's something where we actually do that exercise. It's something that the FDA wants to see for medical devices, not specifically for this, but that's threat modeling. And so when we're doing threat modeling, of course we're saying, you know, what if someone uses a bad password on the device? What if there's bad encryption in the device? What if all these things go wrong? And we're also saying, what if developers don't have multifactor authentication on their GitHub accounts? What if their GitHub account password is "password"? What if they are leaving laptops open and working in a coffee shop and then they don't lock their laptop when they leave? All of that goes into consideration. How are their controls that are going to prevent things from happening in the development environment? In the manufacturing environment, in the research environment. And so, threat modeling covers that whole total product life cycle outside of just looking at the device. And, you know, I'm sure we've talked about threat modeling a lot and we've talked about all the common pitfalls, but that's a really big one with threat modeling is keeping the lens too narrow, focusing on the device instead of widening it and looking at the product and the systems involved. Christian: Yeah, and that's a good point. My cousin used to work for a major staffing firm. He was a software developer. He had all the information on a laptop. He left the laptop in his vehicle. He ran into a convenience store in the Houston area and someone stole his laptop. So there was a major disclosure. There was like 80,000 client records on his laptop, unencrypted. So it's no different than a service tech going to update one of these devices. That laptop may have sensitive data, that USB key may have uh a patch on it, but how do we know that patch wasn't modified which goes back into that integrity in the hash and maybe a digital signature to make sure the patch wasn't somehow modified. And then I think the other aspect that's often overlooked, uh, like when I did work on the DOD, and I even commercial work, a lot like you mentioned, a lot of people move things to the cloud, a lot of software of medical devices is deployed in the cloud. But the question I think people often neglect is where, I mean the cloud is the cloud, but physically in the cloud, like what country is my data in? Because in the DOD, we we we had people in commercially too, they'll be using the cloud, but they chose a site that's like in a country where, what do you do if uh, that country no longer likes the United States or there's a riot in the country or they someone breaks in the data center there, and now your US government data only is in another country as an example. So I mean there's a lot of things to consider. Trevor: Yeah, and even just considering where in the US is your data? Like, a little bit of a fringe example, but how are you, if you're storing all of your information in a singular cloud location based in, we'll say Oklahoma or something or Florida? The US is prone to a lot of natural disasters. We get hurricanes, tornadoes, fires, floods. And so, are you protecting your data? Are you protecting your product in the event of an event, something like that, a natural disaster happening? And it seems like a fringe case, but it happens all the time. There were all of the fires in LA, and of course there are a lot of data centers in LA. There hurricanes in Florida, hurricanes even reaching up into Atlanta. Um I know that we even use data centers in Atlanta, Georgia. and so there are problems that can happen out there. I'm sure about the world's data at this point is in San Francisco. And someday the ocean's going to, you know, a big enough earthquake's going to happen and San Francisco is going to get swallowed up by the ocean. And so what's happening to all the data then? Christian: Well, that goes to the planning portion of that TPLC. I think if you need to have the data always be available, uh, then you need to find two data centers that are, the data is mirrored on, and the data centers are geographically pretty far apart. I think that the, in the DOD, I think it would have to be like 150 miles apart. I think in general, like a natural disaster is contained to like 150-mile radius was the, the rationale. Trevor: It sounds about right. Christian: I would probably put them more than 150 miles apart, though. Trevor: Yeah, I figure you'd want to be in two pretty distant places. Like if you have a data center in the southeast and you'd want one in the northwest or stick it in the middle of the desert of Nevada or something where nothing really happens. Christian: Right. Yeah, and... Host: Even down to static testing as a part of continuous integration, it's a standard best practice with development. But the issue is we're interacting with a lot of startup, startups who may not have the expertise or the, you know, background in this to understand how to develop all of these cyber security concepts in the pipeline. And so when we're looking at something like static testing, which is essentially going to be secure coding 101, they may just be too new in the space, they may not have the resources, they may not have the manpower to do this. And so it's pretty uncommon that we'll see a good, it's pretty uncommon that we'll see any sort of secure product development framework and it's extremely uncommon that we'll see a very strong one. Now, having said that, usually when we're working with bigger company strategics in the space, they have the maturity and they have the resources to have engineers focused purely on the development architecture. And so someone is maintaining all of this infrastructure, they have it stood up. Every single piece of code goes through all these different layers of checks and they have full-time employees just making sure that infrastructure is in place. And so, it's not really that people are, manufacturers are being lazy or they're forgetting about it. It's just that it's very hard to do and very time-consuming, and there are a million things happening with a startup so sometimes it can get pushed to the back burner. Christian: It's interesting because, uh, I know a lot of startups, we work with a lot of startups and a lot of them have contracted out their software development to a software development organization. And then when we pull back the, the covers a little bit, that software development organization, which is all they do, they don't do secure software development, which I always find interesting. Uh, unless it's like a highly regulated industry like nuclear, uh, it's still like in its infancy with, with Medtech and in, I think, software development in general. Trevor: Yeah. Now, I definitely agree, and it's, it's really interesting because I think that creating software documentation, that's been something that's been happening since the 90s. And so, contract manufacturers, contract engineers, are pretty used to creating good software artifacts at this point. That's why when we're looking at, if you look at a, uh, contract manufacturer for MedTech, they'll often say, "We are ISO 13485 compliant," which is quality systems in medical devices. What that means is they're preparing all the documentation and artifacts to feed into your quality system, saying, you know, "Here's our architecture, here's our requirement specs, here's our V&V and traceability back to the requirement specs." Now, what we want to look for is are you IEC 62304 compliant? And 62304 is medical device secure, or secure medical software development. And so, that is when you're moving into that CI/CD pipeline. They have all of these checks in place, they have all the static testing, the analysis of your software bill of materials, they're building in all of the security requirements, and they're testing them, which is required to show the FDA security requirement testing. And so, but interestingly enough, if you look around for contract manufacturers in the MedTech space, you're not going to see 62304 compliance as much as you think you would. But you'll always see secure software, or not secure, but good software documentation. Good cyber security documentation is a little bit newer. Christian: Yeah, I wonder if this, uh, challenge is ever going to, this gap is ever going to close because I, I've been involved in cyber security for like 30 years and it's always been a challenge getting software developers to develop code securely. Even though there's plenty of documentation today, like OASP has a whole devsec ops playbook, you know, it's not like there's a lack of what to do. I think like you said, is costly, like a lot of these SAS tools are extremely costly. And, um, it requires someone that's looking at the a bigger picture than maybe the person developing the code because they're looking at this overall framework and how it should all tie together. Um, yeah, so I think it's, uh, just a complexity of implementing, maybe the cost or a couple of the reasons it's not being done as much as it should be, should be. Trevor: Mmm-hmm. Trevor: That's right. Christian: Yeah, it's an interesting point because I think the number, like I've heard from, uh, investors is like 93% of MedTech startups fail. And imagine like you're trying to spend all this money on setting up a secure product development framework, a TPLC, and decommissioning, but your product actually never gets launched. You know, it's kind of the irony is you need all these things set up to get approved by the FDA or the UMDR, but then if you haven't actually figured out how to commercialize your product, these things may have all been for nothing. I guess just like the development of the product or launching the company, it's the same concept, but, yeah, it's, it's interesting, like if a product never gets out there, the decommissioning aspect is irrelevant. Trevor: Right. But you're not going to get your product out there if you haven't considered the decommission aspect. Christian: Right. Christian: Yeah, so I, I could understand like, why why these MedTech startups are like, we don't even know if our product's going to make it, but we're we're having to spend money and to figure out how to de-commission it if it makes it. But we're just trying to get it to market. You know, it's, it's, it's a tough, it's a tough decision and it but you have to like do it for the regulatory requirements, but from a a funding and a getting something to market, I can understand the the mindset around that, around like cyber security in general, like why they, why people don't want to do it and why there might be some resistance about even these things, like the SPDF for the total product life cycle. Trevor: Totally, yep. Christian: Cool. Christian: All right, well, I think we're coming up on time here. Christian: Any last words of wisdom, Trevor? Trevor: Yeah, I think that uh, cyber security should start at the concept and then it should finish at your exit. So, it's not something you do once, it's something you're always considering, and that's the total product life cycle. You have to figure out how you're handling cyber security all the way up until, you know, the very end. Even something that we're preparing documentation for as part of our post-market plan is, what happens if the company goes under to any fielded devices? And so, it's a hard conversation, but it's something you have to think about. And so, cyber security needs to be addressed throughout everything. That's why they call it the total product life cycle instead of partial product life cycle. Christian: It's interesting because people don't want to think about their own death either, right? Because I know I was talking to some people like not too long ago and their this woman's father died, did not have a will, did not have a trust, nothing was figured out. So it was a mess. The family uh didn't know what to do with anything and it caused more stress, right? And then about a year has gone by with this this person and I've asked her if she's got a will and she doesn't. So I'm like, you just went through this with your parent. It was very stressful, but now you're going to put your kids through it. So like, you know, it's like, it's like I'm saying this is like the total product life cycle decomissioning ourselves I guess but it's like, you know, we don't don't think about it. It's like but but it makes it, it makes it more challenging for everybody else. There's resistance though. Trevor: Oh yeah. Yeah, nobody, nobody wants to have that conversation. Nobody wants to talk about their company failing. Nobody wants to talk about themselves failing. Christian: But, but statistically, we're all going to fail at some point, so we might as well figure it out. At least for me, I'm doing, uh, you know, revocable living trust and a few other things, I'm getting those things done because I don't wanna leave all this confusion to people if I die. Uh, and I've witnessed other people that have had that happen to them, so I'm I'm just taking care of business. I guess I'm thinking of the TPLC in terms of me. Trevor: Yeah, with all the, uh, dangerous sports and extreme whatever that I do, I should probably figure that out sooner or later. Trevor: Yeah, you should probably figure it out, too, yeah. Trevor: What happens if I go down free diving but don't come up? Who, who gets the car? All that good stuff. Christian: Exactly. Otherwise it's up for, it's up for, uh, decisions and people like don't actually know what your intentions were at some point. Trevor: Yeah. Christian: To guess. Cool. Um, all right, well, I think we'll wrap up here. Thanks everyone for tuning in to the Med Device Cyber podcast. I hope you found value in this podcast about the TPLC and a little bit of a deep dive into the SPDF as well. And hope to see you on the next one.