Security Challenges Facing Operational Technology

Matt Fusaro 00:21
Hey.Steven Maresca 00:47
Sure, I mean, it’s a it’s a general category, it’s more of a newer term for stuff that’s existed for a long time, just so we’re clear. But stuff that tends to be included would be industrial controls, temperature sensors, valves, project, programmable logic controllers, things that have changed logic, but aren’t necessarily programmed in a way that, you know, higher level languages will be used. These are things that you know, deal with manufacturing, movement of goods and materials across an assembly line, things of that sort of sequencing, metering. measurement types of devices.Jason Pufahl 01:27
So I’m gonna go out on a limb, IT stuff; servers, desktops, you know, network switches, things like that, easier to maintain, probably right, a lot more publicly available, patches from vendors, maybe just generally broader support. So what makes OT more difficult?

Steven Maresca 01:47
Well, there’s actually two answers to that, and both of them are conflicting. They’re easier to maintain on one level, because the protocols that are used, the wires that are used, they’re built for robustness. They work in environments that are high vibration, high temperature, acidic, you know, things of that sort, these are designed to last, which is,

Jason Pufahl 02:08
IT?

Steven Maresca 02:08
OT.

Jason Pufahl 02:09
Ok, let’s be specific.

Steven Maresca 02:10
Right.

Matt Fusaro 02:11
And usually only have a couple of functions.

Steven Maresca 02:13
Right, maybe they actually do all of their communication over literally one wire.

Jason Pufahl 02:18
So is longevity akin to maintenance, because that sounds what you’re talking about is something that can last a long time, ignored?

Steven Maresca 02:26
Yeah, I mean, it reduces maintenance requirements. From a typical IT context, what that means is that you have material that was put into place with technology that may not be changed for 10 years. That could be a risk. Therefore, a maintenance headache, it depends on the context here. Some people would see it as a maintenance boon, to not need to touch it for a long time. Others would consider it a maintenance headache, because of the fact that it’s it’s not received a patch.

Matt Fusaro 02:55
Or the manufacturer of that device just isn’t around anymore.

Jason Pufahl 02:58
Right.

Steven Maresca 02:59
Super common.

Matt Fusaro 03:00
Yeah, I mean, AT&T is probably one of your best examples, large phone companies, they’ve got technology from the 80s still running right now.

Jason Pufahl 03:09
Right, routing calls and just hoping it works.

Matt Fusaro 03:12
Yeah, routing calls, testing data circuits, that stuff is ancient.

Jason Pufahl 03:17
So, so then where are our challenges if it, you know, one, you get longevity out a lot of the stuff. And frankly, you can’t say the same out of a lot of your your more mainstream IT infrastructure, right? So you get longevity. But because there’s maybe reduced or more challenging patching, or perhaps in some cases, no updates available at all, you probably introduce a fair amount of security, security risks, maybe just security concerns to stuff like this as as other technologies have evolved?

Matt Fusaro 03:47
Yeah, a lot of the risk that comes into especially OTs, what these OT systems are being hooked up into, right, so to gain efficiencies, to automate how they’re actually used, they’re hooked up to actual server farms, in some cases, or, you know, small devices that are on site that may actually be using a web server to take calls from an application somewhere. So that you can centralize your HVAC team, so you could centralize anyone controlling valves for anything like that. There’s so many examples out there.

Steven Maresca 04:23
Right. But other examples would be like, sensors on a pipeline, literally in the middle of nowhere, hundreds of miles away from a tech that happens to connect over the cell network, or something to that effect. There are lots of situations of that sort that are challenges. Maintenance for a lot of operational technologies actually, replacement, it broke, you’re going to swap it out with another part. It’s never actually being updated per se. It’s being swapped.

Jason Pufahl 04:54
Right, with something that you hoarded that you found previously or it’s something new.

Steven Maresca 05:00
Pretty common as an old part.

Jason Pufahl 05:02
So is it, is there anything different in protecting these, though, than you might with your more standard IT infrastructure, right?

Matt Fusaro 05:12
Well, a lot a lot of times. So for example, in IT infrastructure, we’ll do active scanning against them, to find, you know, hey, does it need updates, is it vulnerable to any of these types of protocols or attacks that we know about? If you do that with some OT, you can end up actually causing a problem, right? You don’t necessarily want to be actively probing some of these things and, you know, turn on a gas pipe, for example, that shouldn’t be turned on. So how do you, how do you do these things now, right? How do you check for threats? How do you identify traffic that should be or shouldn’t be going in and out of the networks that they’re connected to? It becomes more of a challenge, because you have to have somewhat of a hands off approach to it. Otherwise, you start interacting with physical things in the world, right?

Steven Maresca 05:59
Yeah, but a lot of risk management and risk discovery in IT involves probing live equipment. In OT, the safest way is in a lab, in isolation, in control conditions, something to that effect. A lot of these devices are, they have such low computational power, that merely interacting with them, distracts them from their very real time oriented tasks that they’re obligated to perform, and therefore you’ve created a problem because they don’t function. So, very different types of strategies attached to how you maintain them and assess them.

Jason Pufahl 06:39
How much of this conversation are we talking about 20 year old devices versus maybe something that, you know, performs that same kind of function, right? You know, controlling the HVAC systems, for example, but something that’s been installed in the last year, I mean, there’s a certain set of regulatory requirements that sort of define how these things are designed now, I believe, to some degree what their sort of vendor maintenance requirements might be and things like that. So you know, has OT gotten easier to protect, as as you know, in more recent times, or not necessarily?

Matt Fusaro 07:16
I think it’s gotten easier to protect, there’s more tools out there, there’s more detection technology, I think more organizations are starting to understand that they can’t just connect these things into their corporate network without any thought behind it. But there are still organizations out there that don’t think about it. But yeah, a lot of the things that we talk about, you know, really fundamentals are, are where you need to be when it comes to OT, right, segmenting your networks, so that you’re not on the same vulnerable networks as your corporate devices, you know, making sure if they do have software, that they’re actually patched, and doing some type of monitoring, right, just having some visibility into what it is that’s out there. How do you get to it, and what do you do if it is compromised? What’s kind of, what’s the risk there?

Steven Maresca 08:05
Well, a lot of what we would normally think of as re-architecting is a planning exercise for OT, pre-deployment. Air gap is a term adjacent to network segmentation that Matt just mentioned, in anything that’s OT specific, because you expect to have a discrete severance of any sort of connectivity between the rest of your environment and OT, it’s a deliberate choice. Therefore, you know, if you build it out of the gate, out of the gates being correct, it tends to be secure, it tends to stay secure, even if it suffers from some future vulnerability.

Matt Fusaro 08:42
Right, and I think a lot of this started being an issue when you started seeing things like Smart grid come up. Right, that’s one of the big worries about a lot of electrical infrastructure, water infrastructure, you know, the utilities, coming into a more connected, connected world now. They at first had issues because they were not segmenting those things off, right, and, yeah, I mean, you end up with problems because of what, you’re trying to get automation and get efficiency out of something, which is going to require you connecting these things together. And now you’ve created a point of entry.

Steven Maresca 09:22
So example of them, two examples, far more domestic. If it’s a switch, electric switch, that’s connected to a socket, just to turn on and off a light, you can maliciously flip that on and off so many times that you cause the relay inside to fail. And maybe short out, you’ve destroyed something, at worst caused a fire if there’s some protection lacking in the device. Then the other example is something more, let’s call it real world espionage oriented with Stuxnet, which was a, you know, Microsoft Windows worm that propagated across networks that happened to have industrial controls accessible from them. It was a SCADA, a SCADA vulnerability that targeted Siemens controllers. I mean that, we don’t go into that here, but it’s exactly the same sort of thing, it was used to destroy attached industrial systems.

Jason Pufahl 10:20
Certainly, it seems like most companies probably have a good handle on their information technology resources. You know, they may have an inventory, or at least they have a passable understanding of what’s on their network. I suspect a lot of organizations don’t really even fully understand what they might have in their environment for OT devices, right, either the vendors directly providing services, so they’ve got something hooked up there, or it could be in place for 20 years and have had multiple IT teams even sort of pass through an organization and not know it’s there. So I think inventorying is probably a key initial thing, so you know what you have, and you know, kind of where your risks are, because you brought up Matt, you know, this is really a risk exercise as much as anything, because there may or may not be, you know, patches, or some specific technical controls or compensating controls you could put in place, but you really need to know where your risks are and make some decisions around that.

Matt Fusaro 11:13
Yeah, I mean, a good example is that we we would walk into, you know, back when I did a lot of project deployment for, for IT departments, we would walk onto a shop floor, and they would have tons of CNC machines. And they were all just connected to the same network as the rest of the computers, because they had engineers that were sitting, you know, designing things sending them to those machines. If that place got compromised, you now have access to all of that, right. So if you want to break those machines, if you want, maybe even introduce a small flaw in a part, right, it would be very difficult to detect those things, because they had all these things on the floor, they had no idea how it was connected, right? So the minute we leave, they have a huge problem, if someone were to compromise that network.

Jason Pufahl 12:00
Yeah, and we’re talking about all this as if it’s, say security vulnerabilities, but, you know, it makes me think of a story in the past where there was a unit attached to a phone switch, essentially, that simply collected call records, and those call records were utilized for billing purposes. So when it broke, people cared because they couldn’t bill their customers. But there was only one person in the company that understood that that that call record collector even existed, and that one person had built a custom cable to be able to connect to it, because there was only one way to actually get any visibility. So you literally had this pretty critical, old, call box collector, that only one person could support. And they and that person needs to make sure they had their, you know, uniquely crafted cable to be able to do it. Like these aren’t straightforward things necessarily to interact with always. But but but it’s not always some major security risk necessarily, right, it’s just, there’s a whole variety of things these things control.

Steven Maresca 12:56
Yeah, it’s operational risk as much as anything else, absolutely. And there are lots of surprises beyond the maintenance and, you know, accessibility with our laptops, which no longer have serial ports.

Jason Pufahl 13:07
Yeah, there’s that too.

Steven Maresca 13:09
Some sensors, like temperature sensors, are way more capable than they need to be. We’ve seen them running mail servers, like that makes no possible, reasonable sense. But they exist.

Matt Fusaro 13:22
Solar charge controllers were, those are notorious for having a bunch of services on them, and we found one in an incident once that was being contacted by some questionable IP’s.

Jason Pufahl 13:35
Yeah, so not a lot of care building them necessarily. They get them to perform the function they need, it might be a lower end piece of just traditional computing technology that they repurpose, or something else.

Steven Maresca 13:46
Absolutely.

Jason Pufahl 13:48
So in many ways, I feel like we’ve sort of said in this in this episode, things that we’ve said in the past around, you know, take care of those fundamentals of collected inventory, think about the infrastructure in terms of risk and be judicious about the way that you actually protect it, and then you know, for those things that you can do network segmentation and monitoring, make sure you put those things in place. But I don’t know that we’ve identified anything incredibly unique about these other than they may be old and perhaps a little bit more difficult to, to maintain. As always, if anybody has any questions about operational technology, feel free to let us know, we’re happy to talk. It’s not a it’s not easy sometimes even to get the inventory that you might need. You might not be familiar with it. But there’s a variety of tricks and tips that we can provide you to make that job easier. So we’re happy to chat. And as always, we hope you got value out of the podcast and thanks for listening.

14:45
We’d love to hear your feedback. Feel free to get in touch at Vancord on LinkedIn or on Twitter at Vancordsecurity. And remember, stay vigilant, stay resilient. This has been CyberSound.