The Differences Between Black, Gray, and White Penetration Testing | Ep. 50

In this episode of the Med Device Cyber Podcast, hosts Christian Espinosa and Trevor Slattery from Blue Goat Cyber delve into the critical topic of penetration testing for medical devices. The discussion centers on clarifying the distinctions between the three primary methodologies: black box, gray box, and white box testing. Also known as ethical hacking, penetration testing is a vital component of medical device cybersecurity, and the hosts explain why understanding the differences is crucial for manufacturers seeking regulatory approval from bodies like the U.S. Food and Drug Administration (FDA). The episode breaks down each testing type based on the level of information provided to the security tester. Black box testing is presented as a scenario where the tester has no prior knowledge of the device's internal workings, simulating an external attacker who might stumble upon the device. This approach is realistic for opportunistic threats but is the least comprehensive. Gray box testing represents a middle ground, where the tester is given partial information, such as user-level credentials or high-level architecture diagrams, mimicking an attacker with some insider knowledge. Finally, white box testing is described as the most thorough and in-depth approach. In this scenario, the testers are granted full access to all relevant materials, including source code, detailed documentation, and direct communication with software developers, giving them complete visibility into the system. The core argument of the episode is geared towards medical device manufacturers navigating the regulatory landscape. While the FDA and other global bodies may not explicitly mandate a specific type of penetration test, they require a justification for the chosen methodology and often reject submissions due to 'insufficient' testing. Espinosa and Slattery strongly advocate for a white box approach, presenting it as the most reliable way to ensure due diligence and satisfy regulatory expectations. They caution that opting for a cheaper, less comprehensive black box test often proves to be a false economy. Such tests risk missing critical vulnerabilities, leading to regulatory rejections, costly delays in getting to market, and the eventual need to conduct a more thorough test anyway. They use the adage 'buy once, cry once' to emphasize that investing in a comprehensive white box test from the outset is the most efficient and effective strategy for ensuring both regulatory compliance and patient safety.

Christian: Hi, welcome back to another episode of the Med device Cyber podcast. Today we're today we're talking about penetration testing specifically what are the differences in black, gray and white penetration testing. Penetration testing is also known as ethical hacking. Christian: And we're talking about this in the context of medical device cyber security and what the FDA and other regulatory bodies are really looking for because sometimes black is not enough, gray is not enough. White might be the preferred, but we'll dig into that topic here in a second. Christian: I've got these cool uh glasses. They don't look so cool on screen, but they Trevor claims I will sleep like a baby tonight because I'm wearing these glasses even though I haven't slept in three days because I just got back from Singapore and leaving for Europe tomorrow. So I feel like I have permanent jet lag. Trevor: Well, they won't help with that, but they will make it easier to fall asleep when you're staring at a screen for 14 hours a day as we typically do. Christian: So we'll start with black. Black the device is like a black box. We don't know anything about it. We don't have any really not much documentation other than maybe a user manual. We don't have a lot of visibility into it. We can't talk to the software developers. We have user level access. Is that a good explanation for Black Box penetration testing? Trevor: Exactly. It would be thought of as an attacker walks into a room, they want to cause some damage to something, but they don't have any prior insight into whatever it is. They see a device sitting on a table, they just grab it and try to hack into it. Trevor: So, that's the perspective that we're coming into a black box penetration test from. It's a little bit difficult at times as, you know, penetration testers, of course, we're doing this differently from actual bad guys. We are contracted to do it. People are willingly paying and asking us to hack into their products. So, they know we're doing it, but they still have to try to keep as many secrets from us as possible from when we're trying to understand what are we testing, what are we allowed to do, what aren't we? They have to set some guy some guidelines on that without giving up too much information. Trevor: So it can be a little bit interesting and sometimes it's funny too, working with clients on these engagements for a Black Box test. We'll say, oh well, could you explain what this process looks like and they go, no, you have to figure it out yourself. So it's a little bit of a more exploratory type of testing. Trevor: It is also going to be the most realistic from like a grab and go attacker. So if you're looking at what is typical for malicious hackers, they're trying to look for low-hanging fruit, they're trying to grab on to the first thing that they can see that they think they can hack into. That is most indicative of a black box testing approach. The first thing that someone can see that they can try to attack without any prior knowledge. Trevor: As opposed to this, sometimes a bit of a misconception where attackers are doing a ton of background research, really trying to find a way in, try to really focus on a single target. Usually they're more looking at it from just grab and go. So a little bit less depth of the testing, but a bit more of a realistic scenario. Christian: Grab and go. I haven't heard that term before, but basically you're saying if I'm a hacker in my home, I've got my whatever that TV is in the room. Samsung I think it is. If I try to hack in that TV, that's a black box type of penetration test, right? Trevor: Exactly. Now, if you had let's say the password to the admin settings on the TV. Or if you knew about all of the parts inside the TV that builds it out or going further past that, you had access to the actual source code that's running on the processors within the TV, then you're no longer coming in from that black box outside perspective. So that's where you'll see a little bit of that difference. Christian: So let's bring a better a better example. I have this uh iHealth um measures your pulse and your uh heart rate and all that stuff. your um blood pressure. Because I therefore I my blood pressure is a little high because Trevor was stressing me out about stuff. It's it's better now. Christian: So, but yeah, if I this has Bluetooth connection to my cell phone though, if I were to try to hack into this, uh, I only have the user manual that's black hat, right? Trevor: Well, if you have the user manual, that's where it's a little bit of uh tying into the gray box testing is a little bit of a gray area. That could be thought of as additional information. Now, generally user manuals are often just going to be provided online. You could probably Google that device, look for the user manual and it would be readily available. Trevor: Sometimes manufacturers will keep that information a little bit more secretive. They'll only provide it to specific users or you know, authorized resellers, whoever this person might be where they are expected to have that user manual. So it can depend a little bit, but within your case, you're probably able to just Google that user manual, so this is nothing that an attacker wouldn't be able to find normally. So yeah, that would be that black box perspective. Christian: Yeah, I haven't tried to hack into it, but it does have uh the device itself. It connects uh Bluetooth to an app and the app connects to the cloud. So I can log on the cloud and check my history of blood pressure and everything else. Trevor: So, now there's a great point where this would start blending into the gray box. You said you log into the cloud to check the history. When we're talking about that device, we would think of the scope of the device as the actual blood pressure monitor, the device that is actually taking in these measurements, the mobile application as well that you're connecting over Bluetooth, and that cloud server. So all of these are considered part of one device. Trevor: Now, if you were to try doing a black box test against all of those, you would attack each one of them from the perspective that you would expect to see it. So maybe you would just download the mobile app off the App Store and try to hack into it with no insight. You would just go onto that website, try to get in. But as soon as you say logging into that website, that's usually where we start to segway into gray box. Trevor: So, we'll talk in a little bit about what white box is, but gray box should be thought of as the intersection between black box and white box being no information versus the keys to the kingdom effectively. We'll usually have maybe some architecture diagrams explaining a little bit about what's going on. We'll have some credentials. So we would have a username and password for that application. Gives us a little bit more access or maybe we'll even have access to one of the engineering members of the team where we can ask them questions on the way that the system is working to get a deeper understanding of the product. So that's where we start to shift outside of that black box testing into a bit more of a comprehensive approach. Christian: So from a knowledge perspective of black box, we have the least knowledge about the device. Grey box, we have some in the middle, white box we have the most, which also translates to from a completeness perspective, black box is going to be less complete and less accurate than white box because we're we know more things to test. We have access to more people in white box than black box. Trevor: So with gray box were generally just splitting the difference there. It's some information, but it's not absolutely everything. I guess white box with the technical exactly what white box means and white box in effect can be a little bit different as well. Trevor: So what white box means is any bit of information you need. Whatever that might be, uh access to any documentation, processes, people, source code. Generally, in effect, the last point is the most important. If you say white box testing, the assumption is you're testing against the source code. Trevor: And as penetration testers, source code access is pretty much the deepest amount of information that we can get. It lets us know exactly how everything inside the product is working. It lets us create very specific targeted exploits against the system. And so it's going to be the most covered and widespread, the deepest type of testing that we would see there. Christian: I like to give the analogy that black box, you have like no user level access, no access at all. Gray box, you have user level access and then white box you have administrator level access for full visibility. Do you agree with that analogy? Trevor: Yeah, yeah, that would be a pretty good way to put it. Often times with gray box, you can have like technically administrative visibility, so you would have a set of credentials for testing the product. But white box, you would have administrative visibility around the company, the process, everything. Christian: The code, all the documentation as well, yeah. What do the regulatory bodies globally prefer? Because I know prior to recording the podcast, uh we were talking a little bit about this and they're not as explicit as they probably should be. Trevor: They aren't. So, when we're talking about, obviously the FDA is a great example for this, they tend to lead the charge on a lot of these cyber security efforts within medical devices. So we'll use their guidance as a big example here. Trevor: When you read through the FDA's guidance, the requirement is that you specify what type of testing was conducted. They do not say what type of testing is approved. Technically any one of the three would be approved. Trevor: Now, where this gets into a little bit of an area of nuance is would you be confident that you did enough testing under a black box perspective? Is there anything that could have been missed because an attacker or the tester didn't have sufficient information or sufficient time? Trevor: As a little bit of a side note, I do want to talk about this time aspect as well. As penetration testers, we're usually doing this over the course of a week, maybe two weeks. Attackers have forever. From a black box perspective, they're going to have more time to uncover information than we will. So it is something that can factor in a little bigger. Trevor: But when we're going through this test, it is a question of whether or not white box or black box testing would be sufficient. Now, having said that, the FDA does require that you perform static application security testing against your source code. They require SBOM generation out of your source code as well as analysis against any safety and security risks that may or may not be present within that SBOM. Trevor: So, while the penetration testing itself can technically be done from any one of the three perspectives, typically our recommendation is to just consider this all white box testing. Wrap it up into a single process, you'll go through and tick all of the boxes that the FDA requires in one fell swoop as opposed to having to go through every process individually and then try to restrict information from the testing team and try to get them to then do it differently for the static testing, it doesn't always work very smoothly from that end. Christian: I think if I were a medical device manufacturer and I'm concerned about patient safety as a ramification of somebody hacking into my device, I would want to do a white box to make sure I've done my completeness and as most accurate as possible and my due diligence. Trevor: Generally, that's what we see the regulators prefer as well. Now, part of the frustration that a lot of people experience when dealing with regulator concerns is I'm sure anyone who's handled the submission with the FDA has seen a letter saying something to the tune of while we recognize that you provided X, Y and Z, it is insufficient with no supporting information on why it's insufficient. Trevor: This is especially common with penetration testing. We very frequently get prospects coming to us, asking about some of the issues that they might have, saying, hey, we submitted to the FDA, they kicked it back, here's this letter. We don't even know where to start because the FDA didn't tell us. They just said, you didn't do well enough, start over. Trevor: What we generally see is this comes back more if testing was done from a black box perspective or against an incomplete scope of the product or if the testing methodology was not adherent with FDA expectations. So more aligned with just general IT considerations. That's when the FDA does not feel that there's been enough coverage. Trevor: It's ultimately up to the reviewers and up to the FDA to say you're sufficient. Um, we're considering this device substantially equivalent or approved for initial market access and you're good to go on the U.S. market. So, white box testing is the most sure fire way to do that. You're providing sufficient assurance to the regulators that you have done your due diligence and it is always ultimately going to be up to their discretion whether or not that is good enough. Trevor: We have never had a single submission get so much as a question when we do white box testing. Around our penetration testing process, coverage and procedures, the FDA has been happy with it every single time. And that's because of our extremely in depth comprehensive approach. We leave no stone unturned going through that. So, having that in mind when conducting these assessments removes a lot of the uncertainty that can come with this filing process. Christian: So the short answer is, if you're a medical device manufacturer, you should be demanding from your penetration testing firm white box penetration testing. Otherwise, you run the risk of getting a deficiency from the FDA or other global regulatory bodies. Trevor: Exactly. And it just streamlines the process too. If you're doing white box testing, you're ticking some of the additional boxes that the FDA calls out and you don't have to do them as a separate process. So, that's what we see is the most efficient in our workflow. We'll work in multiple different capacities. Sometimes we're working with a manufacturer that wants us to just do this black box portion and then they handle all of the source code access, so that more white box testing. Trevor: And the back and forth adds time on to the submission. It's not always as smooth to get everything in place. It's very successful, but it is not a smooth process. When manufacturers come to us and say, do the white box test, here's our source code, here's our product, here's all the documentation you can need and here's the email of every engineer on our team. Then we dive right into it. We go through all of the testing, integrate everything into one process, integrate all of the output into a single risk assessment. It's very streamlined and very efficient doing it like that. Christian: I agree. I think one of the challenges is a lot of people look at the upfront cost of things and if we're looking at cost, typically black box of a scale is the least expensive, white box is more expensive because there's more testing involved, more access to information, it's more in depth. And I think people often choose black box thinking this is all I need to get approved, but then they get the rejection and they have to end up doing white box anyway. So they're basically paying for a black box and a white box and the delayed time to market because they have to re-submit to the FDA or whichever regulatory body, would you agree with that? Trevor: Yep, definitely. And especially factoring in any potential review cycles for the FDA as well as the time that it takes to spin up and wrap up a completely new penetration test. Getting it right the first time is just a far safer option. Trevor: So, we'll we'll picture the scenario that you're talking about. And this is something that we've seen happen in the past and we've even had device manufacturers come to us with this exact same problem. They do a black box test, the FDA says, we don't like the way you did it for one reason or another. It could be insufficient scope, insufficient detail, whatever it is, but not enough information was provided so that the testers could not do a good enough job. Trevor: The manufacturer says, okay, we need to find a good solution. They reach out to us, they say, hey, we got this letter back from the FDA, we're on a timer, we need to make this happen. And we go, okay, not a problem. We can test really fast. We can get these initial reports out, but we start that test and then we start digging up more and more findings since we have this deeper level of access and understanding to the system, and we start finding problems where it comes to a point where we have to say, well, we're probably going to have to shift back the time frame just because of how much remediation is required from all of these findings that were missed from your first round of testing. Trevor: So that time frame can really push up on the submission, uh, on the submission deadline. Especially since once the FDA rejects part of a submission and they want to see additional information, you're on a timer. You don't have the luxury of taking as long as you want to do these remediations. Often times, we'll have manufacturers come to us with a matter of a couple of months or even a couple of weeks left before the FDA is going to reject their submission and they have to go through a new filing. So, getting into that buzzer beater situation is always, should be a last resort as opposed to something that you should roll the dice on. Christian: Yeah, I agree. I think we covered black, gray and white box testing pretty decently. Uh and we're coming up on time here. So any last minute words of wisdom on this topic, Trevor. Trevor: I think that uh the age old adage of buy once, cry once really does apply to penetration testing. The deeper and more comprehensive testing you conduct, it's going to prevent the need from going through this cyclical process. where one round of testing was insufficient, you need to go through a second round or even a third round, which is a situation we see happen all the time. So, getting it right the first time is really important for the FDA. It's also just showing that you have quality in your product, documentation and submission material. And avoiding situations where you have to go through multiple rounds of changes or remediation is always ideal. Christian: You know, I not sure I've actually heard that analogy buy once, cry once. Trevor: That was uh the analogy that my girlfriend's been using on the new couch that we got that was about twice our expected couch budget. Christian: That's a big cry, though. I think Melissa's out clothes shopping right now. So does it count like every item she buys, she's going to cry? Because I don't see, she doesn't seem to cry when she buys clothes. Trevor: Well, that's the thing. Uh she does not cry and you're the buyer in that situation and you are not crying. Christian: I'm crying. I don't want to buy it at all. Trevor: So now you're crying every time. Christian: Yes. All right. So now I have a new perspective. I'm going to have to explain that to her tonight that I'm the one crying, maybe not externally, but internally, I'm definitely crying about the clothes and the shoes and the purses and all the other stuff. It's like, uh, All right, but I get what you're saying. Buy once, choose the right white box penetration test, pay for it once, which is much better than paying for it multiple times and cry once if you feel like crying about it. Trevor: Exactly. Christian: Well, thanks so much everyone for tuning into the Med Device Cyber podcast. We hope to see you on the next one and we hope you found value in this one about black, gray and white box penetration testing in the context of medical device cyber security.