The Impact of Supply Chain Issues in IT

Jason Pufahl 00:25
So we’re going to talk a little bit about an issue, I think that, it certainly existed for maybe 18 to 24 months, maybe a little bit longer, right, the pandemic kind of kicked it into gear, you know, all of the issues relative to supply chain. I think a lot of people have certainly experienced it individually, you know, as Amazon has slowed down, and sort of all the other things that have delayed, you know, getting your packages at home, but it’s been a huge impact to businesses across the country, certainly been a business, an impact to small businesses like ours, as we, you know, have your traditional network upgrades and sort of hardware acquisitions that we frankly can’t fulfill. So, Dan, I think it’d be great if you spent a minute or two on, you know, how, really how the the supply chain issues have impacted you. And we’ll have a conversation today around ways that you’ve looked to mitigate that, and maybe what the future holds. Hey, Matt.Matt Fusaro 00:26
Hey.Daniel Kaupp 01:24
Yeah, yeah, no problem. So when we, when we look at the supply chain issues, a lot of a lot of the impact is focused on our project services, specifically in the infrastructure side of the house. We’re looking at six to eight months, and we’ve even more recently seen longer, like 12, 14 months lead time, on everything from networking equipment, switches to servers, you know, so that’s actually what, what, what cued me up to write, to write the article, you know, about those supply chain challenges, because I heard it enough times from our project managers, you know, that just simply couldn’t execute on projects. And as because of that, what we’ve also found is that, you know, some some pieces of infrastructure projects we’re able to get in a timely fashion, but, you know, can’t complete the install. So, it’s all kinds of challenges, from a billing perspective, talking to the finance people, our ability to to recognize revenue, that sort of thing.

Jason Pufahl 02:40
Has it been specific gear that you find most difficult to acquire? Or is it specific components that then impact, you know, different product lines? I’m just curious what you think the sort of the main issue is here?

Daniel Kaupp 02:54
Yeah, I always put it akin to the automobile industry. You know, for the longest time you’d drive by and see, you know, your car dealerships with three or four cars in the parking lot to sell, right, so they’re relying on microprocessors, different makes and models and suppliers. And that’s sort of the same thing that we’re seeing, or at least how I’ve made sense of it in my mind. To get more specific, we see it a lot with certain builds, certain models of switches, and APs, wireless access points. And what we’ve seen now, and hopefully, we’re getting to the tail end of this supply chain issue, but what we’ve seen is a lot of the manufacturers have adjusted to the types of microprocessors manufacturers that they’re using, actually rewritten code to accommodate it so that they could get around the supply chain gap.

Matt Fusaro 04:00
So a lot of that’s been happening, and from a security standpoint, that that always kind of frightens us, right? When you start making choices to, I guess, increase delivery, a lot of times, you’re gonna be doing things like sacrificing performance, or maybe we don’t even know that we trust the chip sources, right? A lot of the stuff is being acquired from overseas, right? Because that’s mostly where chips are developed up to this point, right? You know, I know Intel and a couple other companies are making huge investments in creating factories inside of the United States at this point, but I mean, we still have a big lead time before we get product from them, that’s actually made in the US. So especially when manufacturers are changing those components out, we now have a problem of, hey, are we trusting the silicone that’s being used? Are we trusting even the code, now you have freshly written code, we’re probably introducing bugs inadvertently or advertently, whatever it might be. I’m sure it’s gonna create some challenges even even past today, right? So you’ll put, you’ll get some of this equipment in, you’ll put it into place, and it’ll have problems and it might just take longer to fix things now.

Daniel Kaupp 05:12
Yeah, it’s sort of like trying to get past the marketing fluff, right, because all of our manufacturers that we work with primarily, they’re touting this as a as a great opportunity to sort of rewrite pieces of the code to optimize against the chipset. You know, just wondering how much of that is real, especially when you look at like, and this happened a couple years ago, and is, you know, maybe light on relevancy, but the supply chain attack from a code perspective, right, SolarWinds, Orion, that jumps to mind whenever I think about about that sort of stuff.

Jason Pufahl 05:53
It’s a good point that you do bring up Matt around, sort of the risk management. So, you know, NIST has their supply chain risk management framework, helping to, I think it’s really keen in manufacturing and some other industries, where you’re, you’re, you’re trying to validate the source of your of your products, the security qualities of your vendors. As you know, we’re experiencing challenging challenges getting equipment, the reason we’re experiencing those challenges, the manufacturers can’t manufacture. So they’re looking to do what they can, clearly to put product on shelves, and that might be sourcing parts from places that maybe they aren’t even as comfortable as they used to be, right? So, you know, understanding your risk is, is probably increasingly important. But frankly, also increasingly difficult now.

Matt Fusaro 06:39
Yeah. On top of it, you know, companies like ours have had to do things like repurpose equipment, right? Dan, I know that you guys have worked with a lot of companies to either take old equipment that we were able to recover for inventory, or even just redeploy some things that customers have owned. That was probably a big challenge there, right?

Daniel Kaupp 06:59
Yeah, absolutely. There has been a huge run on refurbished equipment, I think that’s probably considered gray market, but we’ve, we’ve specifically done that on purpose and a couple in a couple of key areas, or a couple key customers that, you know, just had a had a need that had to be filled, and, you know, no way around it.

Jason Pufahl 07:25
So what are the, what are the conversations, Dan, that you’re having with customers look like? So, you have somebody come, they say, we need to make some upgrades in our network or swap some gear out, etcetera? And your answer might be, well, it’s six months or more before we can get the equipment. Are they then, are they shifted their thinking to well, maybe we should move to the cloud, rather than on premise gear, are they thinking, hey, it’s fine to wait six months, it’s okay. Like, you know, give us a sense on how those conversations go.

Daniel Kaupp 07:56
You know, the conversations, they really run the gamut. And I’ll step back a little bit, I think, you know, the hardware shortage, if there ever was an inopportune time, it’s kind of, you know, right now, it’s further exacerbated by the focus on security, and the prevalence of, you know, maybe it’s COVID related, you know, there’s all kinds of theories and all kinds of people talking about that out there. But there’s a huge uptick in the number of attacks right now, so everyone’s security conscious. Plus, there’s just certain things you can’t address unless you get new hardware, right, the latest and greatest hardware that runs latest and greatest software. You know, so there’s a ton of discussion and a lot of discussion around refreshing an environment is security focused and you know, frankly, talking about it only because the security implications of running that aging equipment. And so, you know, we always start the conversation now, you know, thankfully, everyone, including the sales reps have learned, you know, that we can’t promise, over promise and under deliver again, right. So we start the conversation with hey, you know, you can absolutely get this, you know, new shiny, cool thing if you’re willing to wait, you know, eight, nine months, you know, and, and that has driven a lot more conversation about leveraging the cloud, especially like, that’s an easy conversation, if you’re talking about, you know, swapping out your VMware, like Seven for the new VMware running on, you know, whatever, from Dell or HP, they all have the same issues, and it’s, it’s challenging to get that stuff. So, yeah, a lot of cloud service conversation and then, like I allude to in, in the blog post, you know, once you make the decision to move even portions of your environment to the cloud, it opens things up a lot more from a, from a on sort of what’s left on your on prem footprint, and things you can do to make that stuff last longer, make it more secure, or at least secure enough depends on a whole bunch of things, but.

Jason Pufahl 10:21
So the, probably the one area, you know, server infrastructure makes sense, I think for that cloud-based discussion. There’s a lot of software providers, you know, Office 365 for mail replacement, you know, things that that are just natural, and pretty much everybody’s moving toward. The one place we really can’t address though, is that on prem network infrastructure stack, right, the firewall, well, firewalls can be in the cloud, perhaps, but you know, the onsite networking gear we really can’t address and that’s been a huge issue for us.

Daniel Kaupp 10:49
Yeah, it has, couldn’t agree more, it does make some things simpler. And maybe when you when you cut out some complication on the network, it gives you it, frankly, opens up the number of options you have with refreshing your network, right? What I mean by that is if you move all of the, say, production applications and you know, end user needs to the cloud, you know, you don’t need data center switching anymore as an example, right? You also, if you do have a work from home policy, you don’t need everyone VPNing into your data center, you can sort of use the cloud natively from your house. So, it does open things up quite a bit.

Jason Pufahl 11:47
Yeah, the move to work from home must have made, probably has made this a little bit less of an issue than perhaps it could have been otherwise. Right, without everybody being physically in the office. But I’m sure it doesn’t go away, but have you found that it has bought companies time maybe to replace some of this stuff, until the until the issues have subsided a bit? Or, you know, has there has there been a big move over the last couple of years?

Daniel Kaupp 12:12
I think in general, you know, a couple, couple of customers, a couple of situations, actually jump to mind where, you know, the first requests that we got in, and these were managed customers that, you know, we are their outsourced IT, you know, the first slew of requests was, hey, you know, we need VPN clients on all of our on all of these laptops, right, we need to buy a bunch of laptops, we need VPN. And then, you know, fast forward six months, 12 months, hey, you know, the, the internet connection isn’t great into our office, what can I do? Right, and that, you know, when, then once you go cloud, once you start having those conversations, it’s really difficult, you know, we haven’t hit that part of the paradigm where we’ve looped and we start, you know, we’re taking everything out of the cloud and bring it back on prem. You know, it’s, I’ve been in the industry long enough to think that that might happen. At some point, maybe I’m just old, but.

Matt Fusaro 13:13
Yeah, one of our people here call it the great precipitation.

Jason Pufahl 13:19
It’s just a big cycle, right?

Matt Fusaro 13:20
Yeah. I mean, we are seeing some of that now, where some things are being pulled back down, just from service delivery issues, you know, probably start there.

Jason Pufahl 13:29
Yeah, yeah. I mean, everybody, every vendor wanted to get their applications into the cloud, right? No matter what, but not necessarily everything belongs there, for sure. But in this in the space that we’re in, you know, mail and a lot of the server infrastructure, like there’s a lot of, there’s a lot of really easy services to migrate up if people haven’t thought about doing that already. And if the difference is, you know, we can do a project in a month versus six months, you know, that’s really compelling, clearly.

Matt Fusaro 13:58
Yeah, I think a lot of a lot of businesses needed to make those decisions, whether they were ready or not, because they, most of them are not in the business of IT, they just need these things to work, right. So they don’t care where it is, as long as it’s working, secured, and their people can do whatever it is they have to do.

Jason Pufahl 14:16
Yeah, and I think, frankly, the clients that allow us to help them make those decisions for them, right, I mean, they’re the best ones who, hey, we have a challenge, we’re not going to tell you that we have to replace everything with exactly the same type of hardware or on prem hardware, like, tell us how to fix it and tell us how to address the risks that are in the market, and one of them is supply chain. And those are the clients I think who, you know, who rely on us and enable us that sort of as business partners and that’s really what we’re looking for.

Matt Fusaro 14:46
And I think building that into your your risk profile, your strategy from now on, is probably a good idea.

Jason Pufahl 14:53
For sure.

Matt Fusaro 14:54
Hopefully, something like this never happens again, but it could. So being prepared for, you know, what do I do when I can’t get replacements for the things I’ve got in place right now, do I have a second option? Am I able to send all of my people home for extended periods of time, is that something I can do? Just having those things in your plan from now, probably is required.

Jason Pufahl 15:15
So I think, Dan, as we’re looking to wrap up, what does it look like going forward? Are we seeing lead times today, you know, substantively shorter than they were six months ago? Are they about the same? Do you see times coming down? What do you what do you think the future holds?

Daniel Kaupp 15:32
Yeah, you know, that sort of brings me back to the point I alluded to earlier where, you know, I can’t think of any manufacturers that aren’t experiencing the challenge. And I, you know, flipside of that is I can’t think of any manufacturers that haven’t taken pretty drastic steps to address them. So we are seeing certain products from certain manufacturers. And these are products that are direct replacements for, you know, the products we would have been selling, had this not happened, maybe. And those are down to the reasonable lead times, you know, back to a couple of weeks, at most, you know, certainly still impacting because, you know, there’s still that long tail of equipment that we’re waiting on, frankly, and, you know, and still, so we’re still working through that. Overall, I think they’re still key types of equipment that there’s a shortage on and, and it’s still going to be impacting that.

Jason Pufahl 16:41
In sort of that same timeframe that you’ve experienced over the last year, that shortage is still six months, nine months in some cases?

Daniel Kaupp 16:48
It is yeah, we’re seeing, you know, orders placed late last year, June, July timeframes.

Jason Pufahl 16:56
Okay. So fair amount of client expectation management goes, goes along with this more probably more than in the past, even.

Daniel Kaupp 17:05
It does, it does.

Jason Pufahl 17:09
So, I think, you know, maybe the moral of this is, if you’ve got infrastructure you need to replace, start thinking about it now. Because the lead times might be, might be longer than, than you actually think. You know, any any vendor you’re working with should be able to help you with you making decisions that might mitigate that it could be, you know, shifting where your workloads are, making some other sort of adjustments internally. But at the end of the day, in some cases, you may just be waiting, you may be waiting a while for equipment. So think about those upgrades now, and try not to do it at the 11th hour.

Daniel Kaupp 17:45
Yeah, that’s sort of the the closing thoughts, right, plan early and often, that hasn’t changed. You know, maybe now we’re just adding 12 months earlier. Excellent. Thanks for having me.

Jason Pufahl 17:54
Earlier and oftener. Alright, well, Dan, thanks for joining. If anybody, of course, has questions about, you know, potential lead times, from specific manufacturers or specific questions or specific product lines, you know, let us know, we’re happy to try to answer that. And, you know, Dan, thanks for joining. We appreciate it as always, and we hope everybody enjoyed and got value from the podcast today. Thanks for listening.

18:26
We’d love to hear your feedback. Feel free to get in touch at Vancord on LinkedIn or on Twitter at Vancordsecurity. And remember, stay vigilant, stay resilient. This has been CyberSound.