Threat Hunting with Binary Defense

Jason Pufahl 00:28
So, I think, today, we want to spend a little bit of time on how sort of Binary Defense really, frankly, differentiates themselves in their in their SOC, by sort of evaluating and testing and triaging emerging threats, right? So all the work that you do, kind of ahead of time to make sure that the data that you’re feeding into your platform really catches the most sort of Zero-Day or emerging type threats that are out there, right? So, I think let’s dive directly into that then. So obviously, running a SOC is a lot more than having a malware detection tool, you know, spit out some information and acting on it, right? You need to piece a lot of disparate information together and frankly, you need to be paying attention to new activities by threat actors all the time. So, if you could, you know, jump in, a little bit into, you know, what does it take to put that data together? And sort of what, you know, maybe, what training do you do, right? Tool training and personnel training?

Randy Pargman 02:27
Yeah. First, let me start with an analogy because I think this is a good way to kind of ground the discussion. Before I worked at Binary Defense, I worked for the FBI for 15 years, I did Computer Crime Investigation, but, one of the side duties that I also performed was on the Evidence Response Team, responding to different types of crime scenes, and one of the types of crime scenes that the FBI is primary on investigating is post-blast scene that might involve some terrorist activity. And part of the training that was involved in being ready to handle that was actually very interesting, especially for a computer geek, like me. We went out to a fire training academy where we were a long way from civilization, lots of safe space around. I had a little bit of PowerPoint presentation in the morning, but then the bomb technicians, who were FBI Special Agents, assembled some IEDs and explosive devices, and detonated them out away from people in a safe manner, but it was, it was a real boom. So, in cybersecurity, you know, we talked about prevention versus detection, left of boom, right of boom, this was a very loud audible boom that you can kind of feel in your inner core. And as soon as that was done, we went out and then started collecting the pieces and reassembling what was important. And the point of that exercise was when you’re in a nice classroom, and you’re kind of you know, learning from PowerPoint, and you’re learning the theory, it sounds really easy to put together little tiny pieces, and, you know, figure out what the explosive device did, the point is to gather the relevant pieces of evidence. But when you’re actually doing it, hands on, it’s a different matter. And with some repetition, though, you get good at doing that job. The same thing is true in cybersecurity. You can read blogs about new malware and new threat actor techniques and you can you can say that’s really interesting and you know, mean it, but unless you have some hands on repetitions with the actual technology and seeing how it works, it is difficult to say that you really understand it or that you will recognize it when you see it in real life. So we have, just like the FBI training on a real range, made use of that simulated environment, we also have a range, a cyber range, that our threat researchers and our SOC analysts who are a little bit more advanced have access to. And pretty much anybody within Binary Defense has access to look at the data that we’re producing in that range. And the way that we go about it, is we use real threat actor tools. So, you know, when open-source, new command and control framework came out recently called Havoc, it was really interesting, saw that this was something that was freely available to anybody who wanted to download it. And anybody who’s worked in threat intelligence, for a minute, knows that that’s a clear sign that threat actors are going to pick it up, right, there’s no export restrictions, there’s no control over it, it’s, it’s basically just a free for all. And whatever you might think about, you know, people releasing those tools, it turns out, it is actually really useful for defenders to have access to those things, too. So, at the same time, the threat actors are getting it and this is, you know, a very capable tool, but it’s still in development. We were also downloading it, setting it up on a server so that we had a command and control server that we controlled, experimenting with the different payloads. And the way that we went about it was a couple of different ways. One is kind of the most obvious, which is, let’s just take these payloads and execute them in a few different ways in our, our Cyber Range and our Test Lab. The Test Lab is totally disconnected from anything else. It’s its own standalone infrastructure that uses VPN, to kind of bring together different computers, but it’s, it’s really just set up like a small business. It’s got some domain controllers, it’s got some other Linux and Windows servers, it’s got a bunch of workstations. We found it’s actually useful to have physical workstations, we have mini PCs that are not too expensive to run, but you know, they they accurately simulate kind of a small business environment. And then we use the tools, in this case, Havoc C2, to gain remote control over one of those workstations. We actually have email setup with real email filtering, so we can test you know, what does it like to try to, you know, get a payload through email. And unfortunately, the bad news is as good as all of the email filters are, it’s still always possible if, you’re clever to get around them, to kind of think of a different way to deliver a payload. So, we used a free online service to deliver the payload, we send it through email, simulated, somebody falling for it by just, you know, opening up the email and double clicking on it. Sometimes it’s kind of fun to come up with the ruses. So we’ll even go as far as like, you know, coming up with a reason why somebody might want to click on this thing, run it, and then see what happens. And in our test range, the nice thing is we’ve got it fully instrumented. So we have network monitoring, which is really useful, we’ve got endpoint monitoring through EDRs and we can deploy multiple technologies, sometimes what you might call overkill for a production environment, but perfect for a research environment. And then we can put the malware through its paces. See, you know, what happens when we exercise some of the functionality that threat actors might do, pick up on some of the signals that we would be able to see in a real life environment. And then we can write new detections, queries that will let us see if this happens in one of our clients environments later on.

Matt Fusaro 08:46
Yeah, there’s so much value that comes out of that type of work, especially for a company like ours. Not only does it help people on our side that are doing the defending or just you know, working with Binary Defense to understand the detections that come out of what, what you’ve created out of your intelligence. But even when we do incident response, a lot of times this will help us with things like attribution and things like developing attacks timelines. If we already know how this attack is going to happen because of intelligence that comes out of doing the malware analysis or threat hunting, that helps us so much. We can get to the the endpoint of an incident much quicker.

Jason Pufahl 09:24
Right, yeah. And I think, you know, what’s interesting about what you outlined is sort of the difference, what my opinion is, the difference between an analyst and a practitioner. You know, somebody, a practitioner, can look at a firewall log and read what’s there and say, well, you know, this is what the firewall tells me. Where an analyst can look at data from a variety of different sources, put it all back together, reassemble it, right? Similar to your bomb analogy, and then make sense of it, which I think paints that bigger picture, gives you clarity about the beginning of the incident, and I think, frankly, gives you a lot greater success at either preventing it or getting to the bottom what may have happened.

Randy Pargman 10:01
Yeah, I agree you’re both spot on there. A key outcome from doing this kind of research is the realization that people matter a lot. Technology is super important, you have to have the technology that supports the people in being able to do anything, because we can’t just, you know, natively see ones and zeroes across the wire, we need tools. But, the tools themselves all on their own are not really sufficient. You really need people on your side who understand what they’re looking for, have the experience, the skills, and the repetitions on handling incidents to be able to respond appropriately.

Jason Pufahl 10:38
I think it also, and you can, you can correct me if you think I’m off base here, but I think it also makes it clear that, you know, technology and preventative controls that we have aren’t purely reliant on AI, right? Yeah, it’s a component, but you need that that analyst and that sort of human review to feed that engine and make it more reliable, which is really what you’re talking about.

Randy Pargman 11:01
Yeah, you’re spot on, I totally agree. And I think, I think the right way to see it, it’s not an either or, it’s not AI or a human analyst. The purpose of technology solutions, whether it uses machine learning or it uses, you know, other sort of heuristics, is to filter out all the obvious stuff, take care of all the low hanging fruit, the things that computers can take care of. If you’ve ever stood up a web server on the internet, and started looking at the log files, you’ll notice that before you’ve even had a chance to post your first web page, it’s already under attack, right? Like there’s alway, there’s already people poking, trying to find ways in, right? So, it’s really not a volume that human beings can deal with. The point of the technology is to filter out all of that easy stuff. And then give the human analyst the things that human beings are good at, so that they have the time and the energy to to really dive into it and figure out what’s going on.

Matt Fusaro 12:04
As as you grow as a company, how hard has it been to actually grow out a team that can execute on these things? Because, you know, you can you talk about your SOC analysts and you know, positions that I don’t want to say are easier to fill, but there’s more people that have those capabilities. When you start getting into things like malware analysis, or threat hunting, or even incident response, you’re looking at people with seasoned skills, they’ve done this before, have worked with software, possibly been a developer in their past. I mean it must be hard to find people like that, right?

Randy Pargman 12:36
Um, I actually want to dispel that myth. So,

Matt Fusaro 12:39
Yeah, sure, I’d love to hear that, please.

Randy Pargman 12:40
I weighed in deeply to this debate on you know, is there a cybersecurity skills gap? Is there a shortage of personnel? At the same time, I’m, you know, working with cyber clubs at universities, and people trying to get internships and really smart people, you know, trying to break into the industry that say, it’s not a skills or, you know, knowledge gap. It’s crazy, difficult hiring procedures that are very difficult. So, the way that we’ve approached this is actually using the same technique that we’re using to do the research. So we’ve got this, you know, research lab with all this great data from, you know, real life, very similar to real life attacks in it. Sometimes we actually do let real threat actors play in the lab, they don’t know they’re in a lab, but we get some really great telemetry off of those. And we will give our candidates access to the lab, because it’s remote, they can they can just, you know, log into it in a cloud based service. It’s super easy. We tell them, look, there was something that happened on this date. Here’s kind of the starting thread to pull on, go investigate and tell us what happened. And we give them a reasonable amount of time to do it. And there’s online resources if they need to kind of figure out how to use the tools. And what we found is that just opening, opening it up and trusting that people might be able to do this, whether they’re the typical person that you might think that you’re looking for or not, despite whatever certifications they have, or don’t have behind their name, that doesn’t always indicate the skill that they have, or the ingenuity that they have.

Matt Fusaro 14:24
Yeah, absolutely. Yeah, I think that’s a really great way to handle it. I mean, we’ve always kind of aspired to having some type of type of range like that available for for candidates that we’re evaluating for kind of a lot of the same types of skills. So time has always been our our enemy with that one to get that environment up and running. But you know, I’m sure we’ll talk.

Randy Pargman 14:25
I mean the way that we look at it is if they do a great job, if they actually can engage with the work and they do a great job, it’s worth taking a shot on them. We might hire them as an intern, we might hire them full time. But it’s, it becomes very obvious really quickly during the training process if if they’ve got what it takes or not. And we’ve had some absolute rock stars who came from less traditional or, you know, roles that they might not have gotten a second look if there was a stiff requirement for you know, having these certifications or having the right degree or you know, having 25 years of experience in cybersecurity before you can apply, right? So I think just getting it down to what is actually needed, can they do this work? Do they have the skills? That’s the best way to evaluate and then you’ve got a lot more people suddenly that you wouldn’t have considered otherwise.

Jason Pufahl 15:38
But, I do think it’s interesting to say, to look at candidates and say that they don’t have to fit some specific rigid rubric, right? I mean, we’ve got, we’ve got people who come from, you know, an English background, or English Literature background, who, frankly, are probably some of the best thinkers that we have, right? And don’t have that traditional computer science and engineering, you know, sort of education. So, I mean, I think it’s important to look broadly at the at the, at the candidate pool and look for qualities, right, rather than purely education.

Randy Pargman 16:08
Yeah, absolutely. And of course, experience is the easy one, right? If somebody’s got lots of experience, they’ve done a great job somewhere else, they’re probably going to do a great job for you. But, if you’re willing to take some chances on people, and you know, giving them opportunity to succeed, I think a lot of people rise to that challenge.

Jason Pufahl 16:22
So, so we’ve moved away a little bit here from the the idea of emerging threats, and I want to pull it back. Because one of the reasons that we have been so happy working with Binary Defense is the quality of the SOC, the Security Operation Center, the folks that man that, I think the the way that the data is generated, that you know sort of fed that’s pulled out of your tool, the vision tool, and then maybe into the into the SIEM for review. If you can spend a minute and just outline you know, how does some of the research that you do inform, you know, how your SOC behaves, or maybe even how your your MDR tool, you know, sort of collects and alerts the analysts?

Randy Pargman 17:05
Yeah, absolutely. I think it’s critical that we have free flowing information between all the different teams that are working on research. So, on a regular basis, we actually publish the results of the research that we’ve been doing. And that’s available then to everybody else, not just in the company, but also our clients as well, we send that out on a regular basis, we have a threat intel update every week, and then a major threat hunting review, that talks about all of the threat hunting research, once a month, delves into the hypotheses, you know, the source of the information where the ideas for the hunt came from, whether that was internal or external, the process that we went through, the things that we found along the way that were relevant, and then some specific queries for a number of different sims or other EDRs that can be used. And all of that can go into the technology and also into the SOC analysts quiver of arrows that they can draw on to detect and combat the threats. So, having all of that, I think publishing it is really important, you can talk about it verbally. But unless you’ve got it written down and easily searchable in electronic format, it’s a lot harder to make use of that. So having, even though it’s a lot of work, every single time we publish one of these things, we’re like, wow, that was a lot that went into this. But then we get, then we get feedback from people saying, you know, I put that to work, and it was actually really valuable for me. So that makes it all worthwhile.

Matt Fusaro 18:44
So what’s, so many threats out there. How do you prioritize what you actually look at in your labs?

Randy Pargman 18:52
Oh, that’s an excellent question because the most valuable resource that any of us has is our time, right? It’s not unlimited. We can’t, we can’t just, you know, forego sleep and family time, we actually have to live too. So, there is never enough time to address all of the threats, which is why prioritization is really important. The way that we look at it is kind of the way that a lot of businesses approach risk, which is what’s the probability that we’re going to see this and then what would be the impact if we did? So, for example, one of the really common threats are crypto minor malware, right? And so that’s something that you have to address, but it’s not a super high priority because in most cases, not in all cases, but in most cases, the end result is you’ve wasted some computer resources, wasted some electricity, maybe some, you know, cloud bill, which is problematic, but not the end of the world. In some cases, I do have to point this out, in some cases, the crypto minor, after it’s been running for a while, will be used to install some other malware that could result in, you know, data theft or ransomware. So it’s not a threat to be, you know, ignored. But it’s still not a super high impact. However, the things that we see, like, we understand from our research, that a lot of the big ransomware incidents, the ones that are hugely impactful, start with a tool that the threat actors will use to gain, you know, escalate their privileges, and move from computer to computer getting to the domain controller, and that’s usually Cobalt Strike. So taking Cobalt Strike as the tool that we always need to be aware of, and, you know, look to see if threat actors are doing new things with Cobalt Strike, that is both extremely, highly prevalent. And it’s also a really big impact, so the risk is really high. So we try to just go based on risk, the probability and the relative impact and pick on those things that are going to be the highest impact and the most likelihood, or the biggest likelihood of happening. We also have some clients that specifically, you know, ask for things and we’ve got a threat hunting service, that they’ve got some dedicated analyst hours just working with them. So if those clients who are paying us say, you know, this is top of our mind, this is something that is the highest risk for us, then that becomes our highest priority as well.

Jason Pufahl 21:26
That makes sense. One of the things that I really just want to highlight, and you’ve said it a couple of times, I just want to make sure it’s really clear. I feel like we often have people who feel that these threat actors are doing something unique for every attack, right, or for every potential victim. And what you’ve mentioned more than once is they’re just using tools that we all have access to. And I think that’s really important, right? Because it’s not this black hole, where we’re trying to imagine what the attackers might be doing. We have really good clarity, and it’s much more about how are they leveraging it? Or how are we seeing things develop in that space? But there is some consistency and continuity over time on the on the way attacks are executed.

Randy Pargman 22:10
There absolutely is. And let me draw back again, to my experience with the FBI hunting cyber criminals. When you view it from the other point of view, rather than the victim network point of view, but you’re looking at, what were the threat actors doing, you’ll find that the way they treat their work is like a job. They they work, you know, 40, 60, sometimes 80 hours a week, but they’re doing the same things over and over again, they can’t help it, they’re human beings they’ve got, they’ve got to have some kind of efficiency to their work. It’s kind of scary and a little gut wrenching to think about it, but usually, each threat actor has more victims on the hook than they can deal with. So they have to prioritize like, you know, which which companies have the biggest revenue and are going to be the biggest payday for them. But at the end of the day, they’ve got their habits that they’ve subconsciously formed. And then in many cases, there’s actually written out procedures. So some of the, you know, very prolific cyber criminal threats that I got to investigate and eventually caught up with and got to, you know, sit across the table from the people that got caught, ask them questions and look at their computers, I can see like, they’ve got a standard operating procedure of you know, this is what you do when you get in, these are the things that you run, these are the the tools that you deploy, you know, here’s kind of the way to go about things. So the better you have that information, and the more you’re sharing with other practitioners, other analysts about what they’re seeing, I think the more it can help you recognize these tactics, techniques and procedures of the threat actors, how they operate.

Jason Pufahl 23:52
And make sense. So here’s the here’s the challenge, always with with talking with you, is I feel like we could we could spend an hour on here, but we have a 20ish minute podcast. So, I think in the spirit of keeping to our goal, I want to try to wrap up. I think what you’ve sort of spoken to today is a disciplined approach for sort of understanding the emerging threat landscape and then integrating that data into the way your sort of your SOC analysts and maybe your threat hunters actually approach their work on a day to day basis. And again, going back to the reason that we find working with Binary Defense, you know, often so satisfying is that you are doing, you are spending the time and committing the resources to doing that, you know, the early recon type work that you need to make sure that, you know, you’re well positioned when an actual emergency happens. So, we certainly appreciate it. I think it’s come loud and clear to everybody listening. And as always, Randy, I truly appreciate your time and your willingness to jump on and I hope that we have you again in the future.

Randy Pargman 24:46
Oh absolutely, and I’ll say the feeling’s mutual. We really like working with you, it’s been a pleasure and just having people that really appreciate the threat landscape, understand what’s important, and what is most impactful. That is very refreshing.

Jason Pufahl 25:05
Well, thanks. Thanks for joining. And as always, we hope people got value out of the podcast and feel free to let us know if you’ve got any questions or if you want any follow up. Thanks, Randy.

Randy Pargman 25:18
Buh-bye.

25:19
We’d love to hear your feedback. Feel free to get in touch at Vancord on LinkedIn or on Twitter at Vancordsecurity. And remember, stay vigilant, stay resilient. This has been CyberSound.