Defending Manufacturing: Best Practices in OT Security

00:02

This is CyberSound, Your simplified and fundamentals-focused source for all things cybersecurity.

Jason Pufahl  00:12

Welcome to CyberSound. I’m your host, Jason Pufahl, joined in a surprise gesture by Steve Maresca?

Steven Maresca  00:18

A surprise?

Jason Pufahl  00:19

Yeah, you looked at me like, look at you go today.

Steven Maresca  00:22

It’s more like eyebrow reading.

Jason Pufahl  00:23

Maybe it’s all my energy. I’m telling you I’m dragging a little bit, and I come at it with gusto. Right. And, and Brian Brehart, it’s nice to for you to join again, I appreciate you. I appreciate you taking the time to be willing to participate.

Brian Brehart  00:36

Thanks for having me.

Jason Pufahl  00:38

So we did, we did talk, I think our last episode about the fact that we’re going to have a little mini series on sort of the Internet of Things, operational technology, security things that exist around that. We spent a bunch of time in a previous podcast talking about, I’ll call them home-related challenges here. The Smart Home issues,

Steven Maresca  00:58

The land of broken toys.

Jason Pufahl  01:00

Yeah, for. Yeah, no doubt. I say actually, I think we drove that home. Maybe not exactly that language, but probably like, boy, this is all complicated stuff. But I think we want to talk a little bit more about, like operational technology, probably a little bit more on the in the manufacturing space, we do a fair amount of work with manufacturing, I think we see a lot of kind of purpose built technology. And it’s got a whole really a whole variety of problems. I know, Steve, you mentioned, you know, seeing mail servers on old I think you said old thermostats. Temperature enclosures, right? I mean, these have a real, real wide set of challenges. But they are, unlike the home where a lot of it’s nice to haves. These are all critical things, right? They run machining, they run critical processes, they can’t get away from them, yet. They have all these security challenges.

Steven Maresca  01:53

And I’d go beyond manufacturing. I mean, some of our customer base has shipboard technology out in the middle of the ocean, that is, you know, monitoring some subsystem that’s not manufacturing, but it’s got the same kind of problem domain, right and criticality, right. And there are lots of organizations that have HVAC stuff that’s monitored, they have valves that are remotely controllable, things like that. So operational technology is Internet of Things, but more serious at the end of the day. Right? There’s revenue attached to it is probably the right way to think about it. And, yeah, manufacturing is really the biggest sensitive industry, right. And I don’t necessarily mean big widgets, you might be talking about water treatment plants, there have been lots of items in the news about waterflow waste treatment being attacked by adversaries United States, that certainly comes into play here. The bottom line is the things we’re talking about the technology we’re talking about, is fragile. And yes. What do you do about that?

Jason Pufahl  03:03

So I think the one thing that jumps as I’m thinking about the previous podcast, even, the home stuff, all reasonably new, people are buying things within the last year, year to five, what not, not a really long period of time, the the OT space, I think one of the challenges are people might have something in production for the last 15 plus years. And it hasn’t been updated. It hasn’t changed at all right?

Steven Maresca  03:28

So let’s talk examples, just to frame it. For folks that may have this stuff and not know it. Electronically controlled valves, solenoids, temperature sensors, pumps, programmable logic controllers, microcontrollers that do some sort of automation task related to an industrial process, manufacturing line, measurement equipment, anything that might be in that neighborhood, including security, locks, access controls, these are all in the domain of discrete little devices that get purchased, you know, in 2010, and then don’t get replaced until 2025. Because they didn’t stop working. So why bother? If there’s something on the shelf? Well, it was put there in inventory at the time that contract was fulfilled initially, and the same old device gets replaced with the exact set of vulnerabilities that might have been there at time of purchase, right.

Brian Brehart  04:28

And, you know, and it comes down to, you know, they don’t get replaced because they have such a critical role. Like I used to work at a large logistics company, that will go unnamed, and we had systems that control you know, moving boxes from A to B, that this the hardware was at the time already 10 years old, and the operating system that supported the software we wrote. Like they wouldn’t, you know, we would say, well, why don’t we upgrade this box? And they weren’t messing with it, right? I mean, yeah, that’s it works, you know?

Steven Maresca  05:17

I’ve been in bowling alleys that have relay controlled equipment that is decades back decades beyond what you would believe is a reasonable lifetime. It’s not the same. But it let’s say,

Jason Pufahl  05:28

You take your one pin drop away, and all of a sudden you win a game. That’s big stuff.

Steven Maresca  05:33

I mean, certainly would be irritating if it stopped working and I was winning. So, is that is that internet connected? No, of course not. But the point of the matter is that longevity is both a desirable issue, a desirable facet, and a problem in this area. So here’s the problem. If a device has a vulnerability, how do you assess that? Well, typical, IT run some sort of a scan, measures that risk in situ, that is not a viable practice with operational technology. Because of robustness. Yeah, if you breathe on these pieces of equipment, from a network perspective, if they are network connected at all, they will fall over. If they are not network connected, they are accessible through gateways of some kind. And we’re talking about potentially inducing physical realm impacts, if you make a pump fall over, you may void a tank, if that is 30,000 gallons of caustic fluid on the factory floor, that is not an outcome that is tolerable. And if that line is down, it might be talking about millions of dollars of revenue, and many staff, you know, not productive. So, you know, a strategic way of assessing this risk requires a very different practice from what is the norm, we’re talking labs, we’re talking isolated copies of equipment, we’re talking about, effectively, isolated islands, to compartmentalize risk. Yeah. And that’s okay. As long as it’s known, it can be controlled. I think,  I was gonna say another thing that we also have to think about is a lot of these control systems, especially the ones that, like you said, that have been around for 15 plus years, did not originally come out with connectivity in mind. So you’re already behind the eight ball. And then then to your point, they’re so fragile that. Okay, so now what, and then you have to do that risk assessment, you have to say, well, it’s running. You know, a lot of times they’re running like a version of Linux that sometimes that that distro actually doesn’t exist anymore, right? In the modern, Right, we’re talking about 2000-1999 era code that will never be updated and the manufacturers out of business or something to that effect.

Brian Brehart  08:09

Yes.

Jason Pufahl  08:10

So yes, we’re talking, we spend a little bit here, the last couple of minutes talking about the risk would being proactive, with trying to assess the vulnerabilities, right, don’t run run with a vulnerability scan, because it might actually, you know, cause the device to crash, etc. But they’re targeted it right. So I think there’s there’s big news events that occurred around this. Yeah,

Steven Maresca  08:30

One of the most memorable for your average person would be Stuxnet, which is many years past now, but was the targeting of Siemens programmable logic controllers that were located in many locations, but the main targets were in Iran, and it was related to uranium enrichment facilities. It was a very complex attack path. These were not directly accessible from the internet. But it enabled the destruction of equipment. And, frankly, some very specific weapons research was interrupted. That is espionage, that is nation state level consideration. But I mentioned water control and water treatment, it’s happening in a more domestic and mundane sphere every day. The fact is that some of this equipment is connected, whether directly or indirectly, to the internet at large, or on networks within an environment that they should not be. It’s a convenience thing. It’s a monitoring thing. It’s done with good intentions. But if you can get to it and you’re an adversary, the odds are very good that you will be able to invoke some sort of damage by interacting with it.

Jason Pufahl  09:48

And I think that’s a bit that’s a challenge, right? Because, again, look, looking back at that podcast, we talked about home devices. We talked about the security risk, but if you’re really going to stack them up against the operational tech risks, totally different. You’re really talking about public health, your water supplies, the electrical grid, you know, things that are that everybody depends on, that would be expensive if they went down, hugely disruptive if they went down. So sort of knowing that, we probably should talk a little bit about strategies, because the, for me, the difficulty with this conversation a little bit was, hey, these things are real or not that robust, it’s really difficult to scan them and validate whether they’re secure or not. And by the way, soft times are so old, you don’t have any recourse or ability to update them anyway. So what is what’s reasonable strategy to then protect this?

Steven Maresca  10:42

Well, first, I would suggest, most generally, that we think of this from a audience perspective as more of a business continuity, revenue protection kind of mentality, because the strategies that you employ to protect that type of area actually help from a security standpoint, anyone know. Bottom line, the very first thing is please, oh, please don’t connect it remotely, to any sort of outside vendor, to the extent that you can manage that, or to the internet at large. If you need a third party vendor to reach this stuff, make sure it’s done through protective controls, like VPNs, through network segments, and all of the other typical trappings that require extreme validation of the authenticity of the other. Some sort of gateway. Absolutely. And that gateway also translates to structural lee architecting operational technology appropriately. Many vendors, most vendors have some sort of a gateway that bridges between network segments, translates between protocols and things of that variety. Those are the appropriate devices to put in front of any other communications. Beyond that, if you don’t have those available, isolation is the name of the game.

Brian Brehart  11:56

Yeah, yeah, isolation. Things we did in some other companies I worked in. There was, there was the idea of it as as as things started getting connected to the internet, and we started hearing more and more about, you know, these control systems getting attacked, there was a legit, look at, okay, we need to replace this. What is that going to take? It’s not, it’s, you know, simple is okay, isolation. But it was something that, you know, this is our business, this is what makes us go. And so I mean, it’s modernization is what you’re talking? Yeah, it is, yeah, modernization, exactly. And one of the things that really helps, believe it or not, as I’d been seeing on the YouTubes. And reading, is the advent of the microcontrollers, like a, or single board computers, like a Raspberry Pi, or Raspberry Pi Pico, where these things are, there’s a lot of companies that are using these devices to go to these companies that have control systems and say, we can with a reasonable amount of effort replace your system with a modern version using this device. We’ve tested it here, which has, you know, and a lot of people and a lot of people are looking at it, such as a logistics company I worked for, they were looking at building their own, right, yeah, using it, to build their own systems and maintain it. So you don’t have to connect to a vendor, and you don’t have to have a VPN, we just handle it. So those are, those are some things that I’ve been experienced with.

Steven Maresca  13:50

And certainly, you know, if you’re an individual in process engineering, or if you have those folks that report to you, those individuals may not be able to customize something like a Raspberry Pi or another microcontroller. But during procurement, you can at the very least, kick the tires, so to speak, test the equipment that’s provided in a realistic scenario and see how resilient it is. building that into the actual validation of the equipment while it’s being evaluated for purchase is, in my opinion, a really brilliant idea. Because that might be the only time that you would just be realistically have the ability to test it with rigor. It’s not deployed yet, you have sample equipment, you know, it’s going to be configured in a reasonably similar way. Throw it in a lab environment and bring in security professionals to do some destructive tests. That’s a practice that many organizations don’t bother performing. They may have labs for testing, you know, the processes that are about to implement, and they only go through the actual sequencing.

Jason Pufahl  14:58

So, I’m going too throw one more, maybe wrench in this then. So how do you convince a small manufacturing company to go through the level of effort to replace these things or upgrade these things when in their, it’s sort of in their view, they probably feel like they aren’t necessarily significant targets for attack. They feel like the control systems that they have are working, right. They haven’t demonstrated any real problems. And it’s really nothing but a potential expense to address your kind of an unrealized risk, right? What’s the conversation you have there? Because it because it’s a hard one, it’s, it’s easier with your, like, an Eversource, where the entire grid potentially is impacted, and they probably have a reasonable budget for it.

Steven Maresca  15:43

And they have a regulatory expectation of doing that, right? This is a risk management decision. If you’re talking about a CNC machine, in the manufacturing sphere, it is hundreds of thousands or millions of dollars, depending upon its complexity, you’re not updating that device. Yep, you need it to be on the network to some degree to have G code sent to it to manufacture parganas instruction, right? That right there, it’s, it’s an unavoidable acceptable risk just because of the revenue associated with it that will stay static because it needs to, and then you manage it as best they can you manage it as best you can. Everything else, you know, if it’s tied to revenue, or tied to reputation or tied to productivity, those are the reasons where you don’t want to have something fall over. And you might want to work around that from a risk perspective.

Jason Pufahl  16:36

I mean, that seems like the right answer, you can’t ignore it, you have to at least understand what your risks are and make reasonable determination about what you’re going,

Steven Maresca  16:43

If you’re going to lose customers or lose, lose the lot of revenue, probably warrants some attention.

Brian Brehart  16:50

Yeah, it’s, you run the gamut of all the risk, right? It’s either risk avoidance, risk acceptance, you know, you look at your like, like Steven said, it’s, you look at the equipment, it’s millions of dollars, okay, we just accept that, or we mitigate as best we can. But for a smaller company, they’re a little more flexible, I worked for a manufacturer who made slot machines. And while we were international, we were a smaller company. I mean, I don’t think we had more than 1000 people who work there. And, and most of them were game designers. So so we were a little more flexible in what we could do with our manufacturing process there. It wasn’t unusual. I mean, to give you an idea, they went from making pinball machines, to video games, and then from video games, they went to making slot machines, so they changed the entire or modified, I don’t know, they, they didn’t replace it. But so they were nimble, and they could do those things. And they done it before because they had to because and they were affected by regulation. So you know, they would come and talk to you about manufacturing games. And, you know, how do you lock it? And how do you? So that would, as that evolves, so would the company. So it does, it’s, you know, your situation dictates? And so you just make those decisions to fit.

Steven Maresca  18:22

So take advantage of product life cycles, and the changes of the business to effect change as well.

Jason Pufahl  18:28

Yeah, and it’s a risk. I think you’re right, that it’s a risk discussion. But, you know, with CMMC, a lot of manufacturers now have regulatory requirements. Absolutely, you know, half dozen years ago, so we are seeing regulations creep down into smaller businesses, for sure it’s true, these may or may not be in scope, perhaps, but it’s going to be a guiding factor, right?

Steven Maresca  18:50

If you don’t, if you’re regulated, start there. If you’re not right, risk is where you begin. That’s fair. That’s fair, isn’t it? Yeah.

Brian Brehart  18:56

One final one. If you’re a very large company, you can put in a new, I mean, this, I know, it’s a lot of money. And you know, it’s not something you do commonly, but it’s something that we had done, which was you just create a new facility with all the modern stuff, move all of your manufacturing to that, and then retrofit or upgrade the original one, and now you have both, but again, you have to have a certain amount of revenue to make that go. So but that was the kind of company we were we were doing that anyway, that was part of our, you know, business model is we would create new because it just will grow. Yeah, do evolve. Need will grow. Right? Yeah.

Jason Pufahl  19:43

That’s, I feel like we’ve covered everything,

Steven Maresca  19:44

I think so. Don’t, don’t ignore it, assess it. Think reasonably think critically, and you can control the risk,

Brian Brehart  19:53

And keep up on the risks and threats that are affecting your specific area.

Jason Pufahl  19:59

Well, so we’ve done a two part we may we may, I think we did talk about possibly running a discussion around the privacy risks and some of these devices, which is probably a really worthwhile subject. Yeah. Heavily security based in the last in the last couple. So I think we will find, we’ll find ourselves a good privacy person to talk with about that.

Steven Maresca  19:59

Absolutely. I wonder who have we, you know, i mind.

Jason Pufahl  20:21

The thing is, once you find it, there’s more and more of them, right. And as always, we hope we hope people got some value out of this learned a little bit, started to think about, you know, some of the risks perhaps within your own environments. As always, Brian, we’re gonna I’m gonna throw this to you because you said it the first time I still love it. What do we need people to do?

Brian Brehart  20:40

We need people to smash that like button.

Jason Pufahl  20:44

We do I mean, honestly, the more you engage, the more sort of people see this. And hopefully, from an education standpoint, we get the word out better. So we appreciate everybody listening. Hope you got value and look forward to you tune in for the next one. Thank you.

20:59

We’d love to hear your feedback. Feel free to get in touch at Vancord on LinkedIn. And remember, stay vigilant, stay resilient. This has been CyberSound.