Untold Stories of a Cybersecurity Engineer

Jason Pufahl 1:21
So, yeah, I don’t know that, they’re not all crazy. So it’s not like it’s every one of these is pure mayhem, necessarily. But, you know, because as we were talking through it, I was trying to think of one of my favorite ones and I think, it wasn’t, you know, hey, what major systems did I take down in the past which certainly happened, right? Or, you know, what hardware did I buy that just didn’t work out as expected and we had all kinds of problems, it really wasn’t one of those. It was really more, I guess in looking back at it hindsight, you know, how much trust did somebody place on us, which in a way is what this is, but it’s such a reaction. And it’s such a, I think it demonstrates the urgency that people feel when they’re in the middle of an incident, and in many ways, especially when you’re not really prepared, I think it demonstrates just how willing people are to sort of cede control over an incident, right, we’ve seen it time and time again. But in this particular instance, we got a phone call from a company that we’d never worked with before. And they said, hey, we’re having an incident. Is there anybody available for us to have a quick conversation with? And it was really, yeah, it was as benign as that. And so I said, yeah, sure I actually have a couple of folks here, we can jump on and give you just sort of a sense of what this might look like. And they sent us a Zoom link, we got on the Zoom link, it was 3 of us and probably 20 engineers, system engineers, security engineers, straight up through the CIO, the organization on a call, actively containing.

Matt Fusaro 3:10
All with their hair on fire.

Jason Pufahl 3:11
All of them. I mean, they were, it was you just got plopped straight in the middle of a boardroom basically, everybody working on this incident, it was mayhem, and you don’t know any of them. And all of a sudden, you’re just in there, right? And Zoom in particular, boom you’re there, right?

Steven Maresca 3:14
There’s no sales cycle.

Jason Pufahl 3:29
Zero, not only is there no sales cycle.

Steven Maresca 3:32
That’s usually the case here.

Jason Pufahl 3:34
Yeah, I mean, to give people context for that, you know, if we’re just doing a normal project, it can be three months easily, sort of discussions and laying out a project and scoping, right. And in this case, I truly joined thinking, I’m just gonna give them a quick idea of what incident response look like, and we couldn’t have been there 15 seconds before, they’re basically like, tell us what to do. And so we kind of went through the normal yet, because the reality is, right, there’s a bit of a formula to incident response, right. So we kind of went through a little bit of a, you know, have you have you protected your backups? Have you done a couple of things that we’d say you do early on in the incident? And pretty much the answers were largely, you know, no, so we got them. And then it was probably 10 minutes. And I said, well, what do you expect the next steps here from us? And the response was, we thought you were already running the incident. So there was nothing. So the interesting part was, I think the part that I was proud of, frankly, was, you know, we really engaged in a manner that sort of demonstrated, I think, expertise and competence, and they felt comfortable, but it also just showed just the level of panic, that a pretty large institution would simply say, here’s these three people that we randomly were able to get and throw into a call, we’re just going to trust him, and we’re going to do what they tell us. And I think that incident went on for at least a good few weeks, it was pretty large. They had been working on it, I think for at least a week or two before they called us and then kind of realized they had gotten nowhere. And I’ve never been in a situation quite like that, I’ve been brought into emergencies before where maybe you have a passing familiarity with the client, but never one where you just jumped in totally cold, and that was an incident report.

Matt Fusaro 5:23
Yeah, I think a lot of times, we’ll get pulled into an incident and they’re, you know, they’re willing to take the life raft from anyone who’s throwing one. So yeah, like you said, they were working on it for a week and a half or so or something like that, you know, they’re already burnt out. Somebody, please help me.

Steven Maresca 5:40
That aspect is very common. We usually walk in the door and have to play this, you know, counselor to some degree to say no, really, please sleep.

Jason Pufahl 5:43
We’ll get through this.

Steven Maresca 5:48
It’s OK, we’ll work it out.

Jason Pufahl 5:53
Yeah. So I mean, that was you know, I’ve had my share of big issues that I’ve caused, I’d say this is what was a nice one where at least I feel like we rolled in and solved the problem. But, never have I simply been just sort of cold called directly into an incident, especially like that where there was no preamble, right, there was no date ahead of time, there was no, hey, let’s have a quick conversation with the CIO to make sure we have some idea what we’re doing. It was just, you’re in a meeting room.

Steven Maresca 5:54
So I want to share something that’s sort of, the inverse, an incident with a relatively large company, multiple places, multiple sites, multiple facilities. Nice first couple of calls, just, you know, slight warm up into an incident, not quite full bore. And you know, it’s the end of the week, incidents always arrive on Fridays, just so we’re clear. Friday, 4:30, 5:00pm something like that.

Jason Pufahl 6:51
Actually, let’s back that up a little bit. Incidents don’t arrive on Friday, they get worked by a company through the entire work week. and then they realize on Friday, they can’t possibly resolve.

Steven Maresca 7:03
Fair. In this case, that’s exactly what happened. We were called in by, you know, someone on the engineering team. Things seem to be going OK, we indicate what type of access we need, the type of tools that we’re about to deploy, that sort of thing. So it’s like Friday, at eight o’clock at night, we confirm we have access through VPN, or whatever they happen to have available. And we never hear from that person again. At all, literally. No documentation, just thrown to the wolves. And you know, we’re very good at that. To be very, very clear. We have our own toolset to collect information quickly and learn an environment. But we never talked to that guy again. And that’s a real problem, at least in the incident that Jason’s referring to, there were staff and there was support. Domain expertise, institutional knowledge, that’s not often the case. And yeah, it’s quite challenging. But we, you know, we made it through recovered just fine at the end of the day, but, boy, it can be quite a challenge.

Matt Fusaro 8:18
Yeah, it’s tough. I mean, I had one very similar situation, and the guy was fired, I think two days into it. And we had no passwords, no documentation, kind of like your situation there. It’s very difficult. And the business always seems to expect the same type of outcome.

Jason Pufahl 8:35
I was just going to say that, right, the expectations don’t change.

Matt Fusaro 8:37
Hey, are you guys done yet? No, I don’t even know where I am right now.

Steven Maresca 8:43
And oh, by the way, I now need you to authorize us to break into these systems. Are you OK with that?

Matt Fusaro 8:47
Yeah, so there is a lot of that. When an IT person leaves or if they’re the only one there, you’re now contracted to hack into systems and yeah, it’s an awkward situation to be in and they always suspect the IT guy of being an insider that did the deed, which is, that’s always tough to deal with.

Steven Maresca 9:10
You know, let’s be clear, we’ve had those. In, you know, actual truth. But it’s never the IT guy. It’s just the first thought of people who don’t like the way their ticket was handled previously. It’s unfortunate.

Jason Pufahl 9:26
So, when we were talking about some of the stories we wanted to tell, and you brought that scenario up, I didn’t remember until just now. And now I absolutely remember exactly that, having to break into systems because we couldn’t get the usernames and passwords we needed because of the unfriendly or adversarial relationship like, that was brutal.

Matt Fusaro 9:42
Yeah, and the type of business they were doing there, too, was not a insignificant one. There was a lot of money being transferred, hands that needed to happen, then nothing could go on there until all of this stuff was fixed.

Jason Pufahl 9:42
Right, yeah, that wasn’t the scenario where you feel like, what do we often say? No lives were at stake. I think in that one, potentially that was the case.

Matt Fusaro 10:05
Yeah, exactly. So it was unfortunate. They ended up okay. But yeah, those are always a fun week and a half, two weeks of not knowing what the heck’s going.

Jason Pufahl 10:15
So I think we probably though, we just did to sort of semi Incident Response related things. In neither case, did Steve or I actually break anything. So I’m really looking forward to your story, I think, because yours actually is a fun one.

Matt Fusaro 10:30
Yeah, for those of you that don’t know, I have a very long history of doing some infrastructure deployment. My stories range anywhere from me taking down movies for the entire country, to me working with a guy that is so notoriously clumsy that they, some people don’t even want to work with him on projects. I’ve had him trip over power cables before in the middle of deployments, that’s always fun, when you take down systems like that, my worst one was the movie times. I had gone into a facility that does a lot of that stuff. They give things out to aggregators. This was, I want to say, almost 10 years ago now, but all of their servers were in a rack in a data center, that one side of it had, you know, your standard 120 volt power strip, and the other one was a 240 volt power strip. And there was, for some reason, a fan at the top of the rack, which, if you’ve never been in the data center before, that’s kind of odd, kind of old school. Most of the air handling is already done for you there. But yeah, I’m getting everything all finished up, I go to plug the fan in, and I went to the wrong side of the rack. And I tripped the circuit for it for the whole rack. Not only did that happened, but it also caused some smoke to come with the fan. And again, yeah, if you’ve never been in a data center, any type of smoke or sparks get you very, very nervous. Being trapped in those places, especially if they have got the older fire suppression where basically, you can’t breathe if that stuff goes off. But yeah, I think for half the day movie times were down across the United States. So that was me. Sorry, if you missed your movie.

Jason Pufahl 12:22
So how did you resolve that though? You know, because you had some, you had work after the fact that you needed to do.

Matt Fusaro 12:26
Yeah, I had to get everything back up as fast as humanly possible and work with the IT guys to get a ton of databases back online. Yeah, it was a full 42 U rack of servers that were all shut off because of that. And it was supposed to be a nice smooth transition of firewalls, but yeah I had a little bit of extra work to do that day.

Jason Pufahl 12:46
So much for that. Does anybody go to movie times anymore? Movie phone? I think that used to be a site.

Steven Maresca 12:46
You just ask Siri now.

Jason Pufahl 12:47
Yeah, I don’t even know. Does anybody go to the movies?

Matt Fusaro 12:59
I brought my my daughters to the movies for the first time. And I actually had trouble trying to find where the movie times. Yeah, they just, they’re listed in weird spots now, sometimes Google has it, sometimes it’s wrong.

Steven Maresca 13:13
It’s punishment for the past.

Jason Pufahl 13:14
Yeah, it is, it’s punishment for the past. Any other good stories anybody wants to share?

Steven Maresca 13:22
I have one that I don’t know if I should share. One of the very first incidents I ever worked, this is a very long time ago, was something that has garnered me a bit of a reputation. I’m the guy who says, don’t forget your printers, when talking about fragile devices or vulnerable devices, and there’s a reason for it. In the early 2000’s, bandwidth was in short supply when used by a lot of people who were interested in streaming movies or illicitly downloading movies, you name it. So you pay attention to systems that are consuming a lot of data or show flows that are to and fro in excess of the norm. There’s a printer doing a lot of traffic. Now printers aren’t, of course, supposed to do that. Maybe if they’re scanning right, maybe they have big documents. No, this was hundreds of gigabytes of data coming out of a printer. And for those who are unaware, printers are often bristling with services that don’t really seem to belong on a printer like FTP servers, file shares. This was a scenario where some material that should not be discussed in polite company was located on a printer that was serving it to the rest of the world and greedily being downloaded.

Jason Pufahl 14:50
They’re good for that for a while.

Steven Maresca 14:52
Yes, so let’s call it a foundational incident for me., that has made me always say, well hold on, don’t forget your printers, because they’re always 8 to 10 years out of date or something obscene.

Matt Fusaro 15:05
Yeah, it’s funny how those types of things kind of, like you said, route you in paying attention to certain things a little bit more than you usually would. We had a situation a long time ago where the backup unit got wiped in the middle of us trying to fix it. And this, this was for an organization that had a lot of documents, a lot of them. And this was back in the day where it’s a problem that these things are gone. Our company ended up paying for the incident by being in that place for I think almost a month, we were sending people back and forth, scanning in documents, to replace all the data that got lost. Ultimately, it was definitely an accident on our part when it happens. And we were working with a vendor who didn’t know what they were doing, either, unfortunately. Yeah, it was a month of us scanning documents back and get them whole again, that was miserable,

Jason Pufahl 16:07
That’s miserable. There’s nothing worse than scanning paper. It would never go smoothly to begin with.

Matt Fusaro 16:13
Yeah, it was such an unfortunate situation where we had a hardware failure, and then a backup overwrote like, an unprocessed backup, or overwriting all the data on us. Never again.

Steven Maresca 16:27
Sounds like fun.

Matt Fusaro 16:28
Yeah, it was not.

Steven Maresca 16:29
So I have another incident that isn’t quite so painful, per se. But it sort of underscores the fog of war nature of security incidents, this is very recent. This has happened multiple times. So if this sounds familiar to you, it’s probably not you. Mid-incident, you know, when everyone’s still very, very sensitive to user driven reports, hey, my system is behaving oddly, or my emails, I can’t access it, that sort of thing. People are on edge. Suddenly, there’s a large influx of reports about suspicious emails. And you know, by sheer coincidence, the phase of this incident was focused at that time in terms of restoring email infrastructure. Turned out that there was a previously scheduled, planned, fully authorized phishing campaign for security awareness that, you know, happened to be scheduled at that time. And I said a moment ago, this is not the first time, it will not be the last time, right. But in that moment, you know, there’s a lot of panic, and it requires some quick action to calm people down. So, you know, a lot of security incidents are the art of wading through lots of conflicting information that has nothing to do with the incident itself, but comes to light in that moment.

Matt Fusaro 17:58
I think the funny thing about all that was the person that reported that all to us was the one that scheduled the phishing campaign.

Jason Pufahl 18:05
They didn’t even put two and two together, right.

Steven Maresca 18:08
Yeah. I think that’s happened in every such event, where that was the case.

Matt Fusaro 18:12
Yeah, it’s easy to forget those things. You’d usually have them on a schedule and they just happen.

Steven Maresca 18:17
Yeah, it was scheduled with another cybersecurity provider months previously up just on the calendar.

Jason Pufahl 18:26
Well, and that’s that age old, you know, don’t have a system outage, also then result in well, let me make a configuration change. So when I bring this up, I take advantage of the fact that something was out so it comes up with a fresh config, like we, how many times have we seen that, right? Oh, there’s a power outage, I’m gonna make this change because I finally have this opportunity. Of course, things never come up the way they’re supposed to when you’re done, right? It seems like a great idea from the start. Just poor decision making, I guess in that case, so what’s the point of all this? Is it just, hey, everybody, crazy things happen, if they happen to you, don’t feel bad about it.

Matt Fusaro 19:06
I mean, I guess.

Jason Pufahl 19:08
Just a little bit of storytime and something different.

Matt Fusaro 19:12
Yeah, I guess, in this industry, there’s a lot of unknowns that happen. You have to be able to be flexible, what goes on, sometimes you’re gonna have a bad day.

Steven Maresca 19:24
It’s the art of coping with how things work or fail when the rules are broken, or when all the toys in the attic are broken or something like that. Inherently, it hurts. But, you know, you get over it. And you know what to look for again in the future.

Matt Fusaro 19:41
If you’re ever on site, and you think something is in the attic, it’s probably just the person that owns the building.

Jason Pufahl 19:46
Yeah, you had that. Yeah, right.

Matt Fusaro 19:49
Almost ended up in a fistfight with someone in a building while we were working, as we thought it was empty.

Jason Pufahl 19:57
Okay, you really can’t make that stuff up, people think you’re an intruder.

Matt Fusaro 20:01
Yeah, no, we legitimately thought the place was being robbed while we were trying to just do some extra work.

Jason Pufahl 20:10
Well, I think on that note, then,

Steven Maresca 20:12
The light note.

Jason Pufahl 20:15
I don’t know, I guess we could say, if anybody wants to share some of their favorite stories with us, feel free. You know, we’re always open to hearing interesting things or events that occur to other folks. Maybe we can share it in an upcoming untold story we got we have plenty of people in the company to choose from to come in and actually talk about some crazy stuff. So if you found this a little bit more, maybe a little more lighthearted than some of our other ones that are more entertaining. As always, we appreciate you listening, Steve and Matt, thanks for joining.

Unknown Speaker 20:46
Stay vigilant, stay resilient. This has been CyberSound.