Vulnerability, Penetration & Other Cybersecurity Testing Types Explained | Ep. 33

In this episode of The Med Device Cyber Podcast, hosts Trevor Slattery and Christian Espinosa provide a comprehensive overview of cybersecurity testing specifically for medical devices. They begin by differentiating between vulnerability testing and penetration testing—two terms that are often used interchangeably but have distinct meanings. Vulnerability testing is described as the process of identifying potential weaknesses, often through automated tools and static analysis, similar to creating a map of potential entry points. Penetration testing, on the other hand, is the active process of trying to exploit those identified vulnerabilities to determine their real-world impact, simulating the actions of a malicious attacker. The hosts emphasize that while people often consider penetration testing the 'sexy' part of cybersecurity, both it and the underlying documentation and vulnerability assessments are critical for creating a secure product. The discussion delves into the various types of testing required by regulatory bodies like the FDA. This includes Static Application Security Testing (SAST), which analyzes the source code without running it to find coding errors and potential security flaws, and Dynamic Application Security Testing (DAST), which tests the application while it is running. A significant portion of the conversation is dedicated to Software Composition Analysis (SCA) and the creation of a Software Bill of Materials (SBOM). They explain that an SBOM is a complete inventory of all software components, including third-party and open-source libraries, used in a device. This is crucial because these external components are a major source of vulnerabilities, and without a complete inventory, manufacturers cannot effectively manage their risk. The hosts also detail different methodologies for penetration testing, including Black Box (where the tester has no prior knowledge of the system), Grey Box (partial knowledge, such as user credentials), and White Box (full access to source code and documentation). They recommend a comprehensive White Box test for medical devices to ensure thorough coverage and note that the FDA requires this testing to be conducted by an independent third party for regulatory submissions. They conclude by stressing the importance of testing all interfaces and considering 'abuse cases'—scenarios where the device is used in unintended ways—to build a robust and secure medical device that meets regulatory expectations and protects patient safety.

Welcome back to the Med Device Cyber podcast. This is going to be uh exciting episode. We're talking about the main fun part of cyber security that everyone wants to go over which is cyber security testing. I'm your co-host Trevor Slattery, joined by the other co-host Christian Espinosa. How are you doing right now, Christian? I'm doing well. I I think it's interesting you think this is exciting part. I guess it is. People think penetration testing and testing is sexy and documentation is boring typically in cyber security. So maybe that's what you're referring to. I think documentation is sexy because without the documentation, the testing doesn't really matter. Yeah. No, that's what I get every time I say, yeah, I'm in cyber security. They go, wow, are you a penetration tester? That's so amazing. No, people usually go, oh, cyber security, okay. Yeah. Awesome. Well, I'm doing pretty good. I'm a little bit tired. I've uh recently traveled uh about 28 hours I think from door to door, maybe like 30. Kind of like a six-hour layover in San Francisco and slept a little bit and slept in a little hour chunks and yeah, it is what it is. And I was 15 hours time difference from where I am today, 15 hours ahead in the future. But uh that's part of um you know, going in business and going to conferences and events is like dealing with jet lag. Yep. And uh guess you, you're down in Singapore so I guess you're getting used to the dry heat instead of the humid heat now. Yeah, Phoenix is a much drier heat, which is much more tolerable than uh the Singapore heat, which reminded me of growing up in Arkansas, just uh the humidity is like is miserable. I don't miss the humidity at all. I feel the opposite way. When I was living in Malaysia, I love the heat. It would be, you know, 95 degrees, 80% humidity all the time. When I go step outside in Phoenix, I mean, I'll start my car, it says it's 125 degrees inside the car. Feel's like your soul's leaving your body when you step outside. I feel like my soul's leaving my body in the humidity when it's just like sweating out my soul. It's like a shower. You step outside it's like it's like you lose a pound of sweat in like three three minutes in Singapore. And you never feel clean, you feel sticky all the time. You do. Yes. And I can't sleep in humidity either. I remember in Arkansas, actually in St. Louis, I got a ear infection and I'd went camping and I was sweating so much my ear was like in water the whole time basically and I got ear infection from it. So, yeah, I'm not a big fan of humidity. Oh, there you go. All right, so let's jump in our topic here. We're not talking about humidity. We're talking about uh our topic today is on cyber security testing for medical devices and kind of what falls under that overall umbrella of cyber security testing, which are a lot of things that the FDA asked for and other regulatory authorities ask for. So what do what what do you want to start? Like what is like one of the main types of testing. Maybe we'll start vulnerability testing. Let's start with that. That's one of the things the FDA asked for and there's some common misconceptions between vulnerability testing and penetration testing. Uh how would you explain like vulnerability testing first? So, vulnerability testing, we're really looking for any risks through just various methods of information collection, through threat collection. This can be through automated tooling, this can be through manual review, but we're more looking at problems from I guess a static and automated perspective where penetration testing is going to be a little bit more of a dynamic and manual perspective. Um I think that the line is often drawn a little bit too explicitly between the two since they often blend together very nicely. And vulnerability testing should be used in many cases as an input into penetration testing. But there is still a distinction in the tooling and in the process there. Yeah, and that's a good point. Vulnerability testing you typically identify the vulnerabilities and then that is often the first step in penetration testing because you have to identify a vulnerability to exploit it, and then penetration testing takes it a step further to see once you exploit one vulnerability, what else can you therefore exploit like once you're in a system or devices, for example. Exactly. And a lot of vulnerability assessments, so if we're looking at what the FDA wants to see, we'll pick one of the examples that they want to see, static and dynamic testing of the source code. And so if we're looking at static testing of the source code and we identify that we did a bunch of findings back saying they aren't handling any uh input sanitization in this code base and so we're making sure that they are making sure that bad input can't go into an input field like a username or a password field. Then that can be a good clue for the penetration tester to go, oh, well, if we're identifying this during the vulnerability testing phase, during our penetration testing phase, we're likely going to want to drill in a little bit deeper to these input fields and see what we can do with that lack of sanitization. Maybe we can find an injection attack or try to extract some sensitive information through those fields. And so that's where they definitely have that blending in together. But there is, I feel like some of the main distinctions are penetration testing is usually done from a very manual perspective. And so a good penetration tester is hands-on keyboard or hands-on the device interacting with it directly trying to have a little bit more of an in-depth approach to really see how far you can take a problem. Um where vulnerability assessments that are quick, they're fast, they cover a lot of ground, but they aren't going down to those next layers. They aren't going all the way down to try to cover every bit of ground that's possible. It's a bit more of quick information collection, cover as much ground as possible and look for the low hanging fruit. And you mentioned like static code analysis under uh vulnerability testing uh as a sort of subset. Uh there's also software composition analysis, which is effectively generating the software bill of materials. Um can you elaborate a little bit on that on the SBOM and how that is used for vulnerability testing? Definitely. So when we're looking at the software bill of materials, we'll start with looking at that source code. The first step for looking at all these different risks, we're going to look at that static testing. So that'll go back to an example like I said with the um lack of input sanitization. We can find hard-coded credentials, we might find memory problems with that static testing. So that's looking at poor coding practices introducing risk. Now what we want to do after is generate that software build materials as that software composition analysis. So what what is the software composed of? What bill, what goes into building the software? What third-party components are used? We have to cover questions like that. So we're generating this bill of materials. That's a list of everything in the device. And then we say, are there any risks in that bill of materials. So it's a little bit separate from looking at what are the problems directly in the device. It's more looking at what are the problems with components you're including in the device. And this can go down deep as many layers of recursion as you want. For any one of those components in the device what components are they using? What risk is involved there? As you can imagine, the further down this chain you get, the less likely those vulnerabilities are going to make their way into the final system, but it's still something that you want to look at. Yeah, and this is also with the software composition analysis where it's a sort of a subset of the SBOM. Um if it's not an identified third party, uh if it's software that came from somewhere uh that's that we don't know that would be the soup or the software of unknown provenance. And there's often risk associated with that as we've discovered. A lot of developers will go to some forum and just copy software with hardcoded password and put it directly into the the source code. Stack Overflow has been the downfall of many an engineer. Stack overflow is the most common form? Yep. Yeah, I've seen uh we've seen on tests before vulnerabilities from breached credentials or hard-coded credentials somewhere and then we're able to actively exploit it and decrypt sensitive information, passwords, PII, lots of really, really nasty stuff. And then we look down to the source code to see what's actually going on there. And then you search for that source code and sure enough you find it verbatum on a stack overflow paste or post from 15 years ago and then someone leaves the hard-coded credential in there. It then gets decrypted. And so it's more common than you think really. Um engineers don't want to have to reinvent the wheel if there's a code out there for it, that's why software libraries exist. That's why these alternative solutions exist. You shouldn't have to write everything from scratch, but you make sure that what you're using to build this need is safe and secure and you are introducing insecure code into your product. Uh yeah, it's kind of interesting because uh technically, I don't know if that would be soup because we know we know where it came from. It's from Cape of Stack Overflow. Yeah, I guess that's fair. Now, that that opens up an interesting debate, where how can you determine, well, first off, if you have no idea where the code came from, how did the engineers find it? Maybe that's Google or ChatGTP probably. It could I guess now today a lot of code could come from ChatGTP. Just have it try to write your code for you. Yeah and ChatGTP is pulling it from who knows where. From stack overflow. Stack overflow, yeah. All right, so we talked about um, software composition analysis, which is the S bomb, the soup. And we talked about static code analysis, part of vulnerability testing. Uh, we touched upon penetration testing which is exploiting the vulnerabilities and there's quite a few subcategories of pen testing. Uh, let's just zoom out a little bit and talk about black, gray and white box penetration testing because typically with a medical device we do white box penetration testing which means we have access to the source code, documentation, the developers if necessary. Um, you want to explain the three, the difference between those three real briefly? So these are talking about different levels of access. With black box testing, you're acting as an attacker would, no insider information going in completely blind. Uh there's definitely merit to each approach. Uh moving into gray box testing is somewhere in between. Often times this will be a set of credentials to test an authenticated perspective, but not really that much more information or maybe just a high-level overview of how the application works or the product works. And then white box testing, you're fully looking under the hood. You have full access to source code, credentials, architecture documentation, uh software requirements. Like you mentioned, you can talk to engineers, say, tell me exactly how this function works, show me what the data flow is in the code. So that's really scrubbing in and trying to find everything that you can, which is our preferred approach when we're looking at this FDA process since it leaves very little room for anything to be missed. Right, the white box is the most comprehensive. But the FDA asks for close box as well, which is equivalent to black box. Yep. And there's certainly merit to the black box approach as well and that's usually going to be a more authentic test, I guess you could put it as since that's taking the approach that most actual hackers are going to be taking. They aren't going to have this insider information that we have in an ideal situation. Uh they're looking at it from the outside in. So if you're getting a black box penetration test done, I think this is especially valuable for network testing. You have to go in completely blind. Try to get any bit of information you can, do your own reconnaissance, try to find your own credentials and work your way into the system. For a medical device, it's great to have this approach, but you're usually going to see a lot more value going all the way in with that white box testing, which is why we like to have the full start to finish coverage. And I think it's important to understand a lot of people with penetration testing and this testing in general, cyber security testing, they um just assume we don't have to test certain things. A lot of of our clients or prospects will say, well, our USB port is only used for our maintenance crew uh or maintenance updates, right? By our maintenance uh staff. So they assume that that's out of scope, but if it's exposed on the device, we have to test that. Right. Or the another one is saying, oh, it's only for updates. I say, well, is there no way we can interact with data? Guess we've never really, or not for updates, it's only for charging. But there's, is there any way we can interact with data? Guess we don't really know, we just use it for charging and then you try and sure enough, you can, you know, download something onto it or interact with the OS underneath. And so using this will go back into what the FDA wants to look at for the vulnerability testing, which again, that line gets really blurred really quickly. They want to see abuse cases. And so the use case, what you should do under normal operation, the abuse case is, you know, well, you say it's only for testing, but what if I plug a hacky into it? And see what happens. What if I have an HDMI port and you're using it to connect to a third-party monitor, what if I see if the CEC protocol is enabled and I can shut the device off while it's operating through the HDMI port. And so the abuse case is essentially what shouldn't you do to the device which is probably the first thing an attacker is going to try to do is whatever you're not supposed to do, whatever you can do to make something go wrong. So that's when it gets really important to get super creative with being a penetration tester and think what is everything that we could do here? Yeah, so a scope of a device uh I know in a previous episode you talk about the three-piece suit which is a lot of devices have uh a medical device has a device physical which has interfaces, uh some sort of app on a mobile device and then a cloud component. So all of those would need to be tested. They're all in scope uh for vulnerabilities and pen tested and everything else including APIs that the mobile app may need to uh or use to connect the cloud app. And I think a lot of people, a lot of um manufacturers don't understand like every single thing that makes up your system if there's an entry point into it has to be tested from a vulnerability and penetration testing perspective. Right. I think about even the stove that we have here. You can control the stove, turn it on and off from your phone. And yeah. Well I can, you know, it's got you can even look and see, you know, if you're baking bread or whatever, go, "Oh, the bread's done," turn off the stove. And I think about what if someone hacks into that and just leaves it running and we're not here? And then the stove runs for four days and burns down the house. Exactly, that seems ridiculous. The risk doesn't seem um, the risk of that feature I'll definitely is outweighs the benefit in my opinion. I mean like, why does the stove need to be turned on remotely? I think I've used it once. Well it's nice if you have something that you want to warm up, have ready for when you get home. Then I'll say, okay, I'll be home in 30 minutes, turn on the stove, leave the food already in there. Yeah and then I get back and it's ready to go. But yeah, it's ridiculous. And so when and you know all the stuff connects to the internet. And so I look at it and I go, yeah, I don't trust any of this stuff. And then I'll talk to my girlfriend for example and she goes, oh, that's so cool. You can turn on the stove with your phone. I go, no, it's terrifying. Everything is scary. You should be frightened of this. I could probably look on like showno and find out if your stove or your oven is a compromisable and just turn it on remotely. Probably. Don't do that. I want to sleep well tonight. But you know, you cook. I'm I'm kind of I might do it. I'm kind of curious now. Not because I have malicious intent against you. I'm just like I I'm sure there's probably a lot of cuz I remember one time I found a lot of windmills that were available you could compromise. Um so you probably find these stoves too. Yeah. He's on. This is uh now that we're talking about the different the path on showed in and even just talking about that you know paranoid mindset that a lot of penetration testers have. If you go on showda and you look for electronic health records systems or a dicom server which stores health records, uh images, very sensitive PII. There are over 1000 of them with no password exposed to the open internet from healthcare delivery organizations in the United States. Yeah, I'm not surprised. And so you can just go look at them remotely and there are no protections on them, anyone can look at them from the open internet and it's terrifying to think about. And so how many people's health records are just exposed? How many people's imaging results are just out there on the internet for anyone to look at without protections in place. Yeah, I just assume all my data has been compromised and it all of it has. Uh I have, you know, I used to have a a clearance with the government and all that was compromised, so all my health records have been compromised, so it's just a matter of time before someone uses it against me, I feel like. Yep. I know everyone always talks about the concern, the panic. Oh, China steals all of our data. No, everyone steals all of everyone's data. Everyone knows everything about everyone. It's not a specific problem. You don't have any secrets, believe me. Yeah. So we've under the umbrella of security testing, uh we talked about pen testing, uh we talked about showed a little bit which is more like uh looking for open source intelligence and what vulnerabilities may exist uh that somebody else has already discovered. Um, I know the FDA talks about abuse cases, uh we've talked about that. Malformed packets. I think they talked about malformed data injection as well. Uh and that kind of ties into fuzz testing, which fuzz testing is really sort of a an abuse case. If there's an expected type of input on a form or a, um, data connection, like a TCPIP connection. Uh, what happens if we send a bunch of random data, uh, you know, with different characters, different links, uh, to that form or that um TCP port. You know, how does the system respond? And that's fuzz testing. It's just a bunch of fuzz data which could take a long time because we're generating data from every combination of characters and and and special characters and different links or different bites. Um, what what do you have to elaborate on those areas of testing? fuzz testing is super interesting type of testing and it's a great way that a lot of security researchers identify very unique they call them zero-day vulnerabilities. So previously unknown vulnerabilities in a specific technology or product. And what fuzz testing is doing is like you said, it's throwing tons of malformed input at a connection, at an input field and seeing how the application responds. So it might be the case that as soon as you send something over 10,000 characters, the application crashes and you go, well, why did it behave that way? So you're looking to see what input triggers weird behavior and then pin down why it triggered that behavior. So it may be the case if you identify 10,000 characters crashes the application, well, the memory is only set to handle so much and they aren't properly checking that it's enough to fit in that memory space. It goes over and it overflows the stack. Immediately you know that you have a memory vulnerability which can be extremely dangerous in devices. That can essentially be the keys to the kingdom for an attacker. So what we're trying to do here is really identify what is everything that the device can handle and how is it rejecting any unsafe input. Devices that are handling a data connection, handling an input field, which is pretty much all of them. Should only take expected data. Should only take an expected connection. It shouldn't be anything weird that we wouldn't normally see during standard operation. So if we have an email field, it should only take emails. It should follow the RFC standard for what an email looks like. It's usually going to be letters, numbers, special characters, and then an add sign something.com.net, whatever it may be, it should follow that specification. anything outside of that should be rejected as bad input. But and this should apply to any input field or data connection on the product. Well, you're assuming um, this is a common thing that does a neglected by developers, uh, input validation or standardization and bounce checking. I think both of those are often, um overlooked because they're assuming people are not going to abuse the system, but we always assume we're going to abuse the system. Right. I have kind of a funny story around that. So I have a friend whose name is just the letter G. That it's a long story how that's his name, but it is and so he goes online to any government website to, you know, renovate his passport or his driver's license. No government websites except a one character input for your first name. And so every time he has to call or go in person and say, you know, here's my, here's my birth certificate, here's evidence, my name is one letter. Can you please put it in manually on the back end? And so, yeah, that's a case of the government not understanding some edge cases as far as their input validation there. Well, they're doing some input validation by not accepting one character. Yeah, but they forgot about the people in the United States, all probably 15 of them who have one letter for their name. Yeah, I'm sure you have the same problem with people that have like 27 characters for their name. Exactly. And so I think there is a balance between understanding all these edge cases, how many people do you really have with one letter for for their name. And so should you work around that as your standard input or say it's very unlikely we're going to see that. We can deal with it on a case-by-case basis and we want to deal with what we know is going to be safe data. So there's a little bit of a balance there when we're addressing security, but it's something very important. Yeah, and I think asking the questions of who you're trying to hire for a pen test, um their experience with hardware testing, which a lot of medical devices have different interfaces that we have to test. And I think a lot of pen testers may have like cloud experience but not specific medtech experience like you mentioned. Yep. Hardware testing is a rare skill for sure. All right, any anything else you want to go over? I think we're about done with the episode here. I was going to mention one thing because we're talking about tools. It's it's a little bit off topic, but when I was in Singapore, you know, they had they have the ERP, the electronic road pricing. Uh, this guy was a driver for us for for like going from the to the airport. He he referred to it as a everyday rob people. I thought it was pretty pretty funny. That's funny. They had their all little acronym for stuff over there. And the tolls at Singapore and no joke. That's what he said, a lot of people buy a car, but then they forget about the tolls and they can't drive the car because it costs too much to drive it or they have to skip on their sleep and drive like 6:00 in the morning because the tolls started at 7:00 so they can avoid the tolls. It's like ridiculous the prices on these tolls. He also said Singapore is a fine country. A fine country? There's a fine for everything. Yep. That sounds about right. Yeah. Awesome. Well, thanks everyone for tuning in to this episode on Cyber Security Testing for Medical devices and a little bit about Singapore as well. So hopefully we'll see you on the next episode and hopefully you found this one valuable. Take care.