In this episode of CyberSound, Jason Pufahl, Steve Maresca, and Michael Grande cut through the AI hype to explore how businesses can use tools like large language models for real-world impact. From model selection and prompt engineering to integrating with existing systems and avoiding inaccurate or misleading outputs, they offer a practical look at applying AI effectively—well beyond chatbots.
Narrator 00:01
This is CyberSound, your simplified and fundamentals-focused source for all things cybersecurity.
Jason Pufahl 00:11
Welcome to CyberSound. I’m your host, Jason Pufahl, joined today by Steve Maresca and Michael Grande, remotely today, hello. So we’re gonna, we’re gonna try something. I’m not saying a little different. A lot of people are certainly talking AI. But what we really wanted to do here was maybe AI beyond the chat bot, right? The, everybody’s familiar with that piece. There’s, there’s so much, so much discussion around that. And, we just came out of a conference where I think you had somebody gave a comment to you, where they just felt inundated.
Steve Maresca 00:45
Yeah, they’re burnt out.
Jason Pufahl 00:46
yeah, but, but then, but weren’t walking away necessarily with more of an understanding of what to do with it. And that’s kind of our intention today. I don’t know that we’re going to give you know, here’s step A, Step B, to do something, but more. How do you think about this from a business practice a little bit, and how do you get value from it?
Steve Maresca 01:04
Yeah, and I talked to a lot of customers that are in a similar position to the people at at the conference. This is not really a build it and they will come type of thing. This is a come with a problem, solve a problem, type of discussion. You know, we’ve thrown a lot of cold water on AI and machine learning in the past, on CyberSound and appropriately, there’s been a lot of breathless marketing. It’s
Jason Pufahl 01:27
It’s in every security product.
Steve Maresca 01:28
Absolutely.
Jason Pufahl 01:29
Everyone.
Steve Maresca 01:30
There’s a gap between delivery and expectations, and nevertheless, there’s a lot of promise, and that’s what we want to talk about at the moment. So strategy for using large language models, llms, generative, AI in business, that’s kind of what the our goal is today, talk through and ultimately, businesses have a lot of opportunity. There are huge pieces of infrastructure and services available to them, whether we’re talking about Gemini from Google, Cloud from Anthropic, GPT from OpenAI, Copilot from Microsoft. There’s a very large landscape where folks can explore. But what do you do then and next? That’s the big issue.
Jason Pufahl 02:17
Beyond exploration, right?
Steve Maresca 02:18
And I have to say, just as sort of a base introduction, chat bots are not the answer. They’re appropriate in some narrow situations. But just as a basic premise, let’s start, let’s start there.
Jason Pufahl 02:32
So I know that you wanted to spend a little bit of time on the key terms get thrown around and trying to make sure people had that baseline understanding of what is a model? What is, you know, what’s inference, right?
Steve Maresca 02:46
So, what is the model? I mean, at the end of the day, it’s data that has been the output of a training set, input into a very complex set of code that reduces it and distills it so it can be computationally useful to a computer, to an application. They’re very large. They’re basically data that enables scalar algebra, linear algebra, if you remember that from class, it’s the magic behind these things. It’s how I’ve tried, I’ve tried really hard to forget. I was just thinking. I don’t know that. I remember that but this is not something that users of these tools need to think about. But that’s what a model is. It’s the data that enables the math. Okay, when you hear about the size of a model, really, that means how much training data set it received. Big models have a huge amount of input. Small models, smaller. And bigger is not always better. It’s also how they’re structured. So that’s the basic thing to know about. There are lots of types of models, and we can talk a little bit about that, maybe in passing, but a model is just the basis upon which data is ingested, transformed and generated. The next thing to cover is inference and training, and those are terms thrown around that are foreign to a lot of people. Training, we kind of alluded to a minute ago. It’s how you produce a model. It’s, hey, we have a million images of animals. We’re going to train a model to identify what pet it is. Yeah, that’s a training data set. Training is the process of consuming those pieces of input, normalizing them and generating something that can be extracted for features that a computer can understand and summarize in a distilled way. Inference is actually where most businesses and users are spending their time, even if they don’t know it. But if you’re trying to solve problems in business, inference is essentially the use of a model to produce an output to a prompt.
Jason Pufahl 04:50
A trained model.
Steve Maresca 04:50
A trained model, yes, and candidly, most businesses have no need whatsoever to train use an off the shelf model. It’s going to be good enough. I think it’s, it’s critically important to understand that, because many people you know try to read up on data science and machine learning and so forth, and think, oh, we have to build on our own data.
Jason Pufahl 05:13
We need all our own data to not the train this thing right scratch. Yeah.
Michael Grande 05:17
Is, Is it possible though, when you know, utilizing, you know something where you want a specific set of answers to be available, or within the confines of something constructing guardrails around that? And is that, is that training? Is that, where does that come in? And I don’t want to, I know we’ll talk about prompts and things like that. But if you’re really trying to set up guardrails, whereas you really don’t need it to, you know, opine on things that aren’t relevant.
Steve Maresca 05:47
So honestly, I think that’s a reasonable segue into prompts. Guardrails for large language models and generative AI are at the input, the data it’s trained on, the output, the data that it emits upon generating and in some way the prompt itself.
Jason Pufahl 06:07
And the prompt feels almost the most important part, because that’s where you give it the parameters to answer questions from or to do something from. Yes,
Steve Maresca 06:14
Yes, absolutely. And there are multiple levels of prompt. It’s you know, what the user asks a generative AI platform to do, add up the columns in this spreadsheet and give me a summary across these 15 other spreadsheets. That’s a prompt. Additionally, there could be something under the hood that says, only consume files that are Excel spreadsheets. The two of them are combined, and they’re the aggregate product. Bottom line is, prompts are how you interact with these tools. Prompt engineering, which is a term bandied about as a related subject, is the art and practice of coming up with a good prompt. You get garbage in, garbage out.
Jason Pufahl 06:55
Like anything that hasn’t changed, right? But,
Steve Maresca 06:57
But, but this is the specific skill, and it is something you need to practice. But in terms of guardrails, you know, if you have an input data set and you’re trying to get a large language model to reason about it or summarize documents, it’s an organization’s responsibility to say only include these documents and exclude those. That’s that’s something you can do in almost any of the tools were that are on the market today.
Michael Grande 07:27
I was going to say the only other sort of comment around prompts. You know, I we had a prior guest on one of our episodes, and he introduced me to what, you know, they referred to as the Auto Map framework. And I had never heard that before, but it’s, it’s very interesting because it’s, it’s about how you frame, exactly how you said, sort of what your query is or what your request is in a context that tells specifically what to the model, what it is that you want to retrieve, and in what voice, right? So you know, what are you acting as? What’s the persona, who, you know, who’s the user, what’s the targeted action, what’s the output definition, what’s the mode or tonality, right? You know, what’s an atypical case. And then, you know, sort of topic, white listing, as we said, sort of that the guard rails.
Steve Maresca 08:14
What these do, just for context, is to basically set the audience and tone as you said, it will generate different words if you say the audience is a technical person versus someone who is in a non technical role. That is the reason for doing that type of thing. And I said the word context casually a second ago, I want to address that very specifically. Context is the aggregate constellation of information in the session of prompt processing. That might be documents consumed in order to fulfill the prompt. That might be the guard rails in a system prompt, it might be the prompt itself or any subsequent conversation in a chat bot. All of that together is carried along the way while something is processed, and that enables a model to produce an output. Large contexts enable complex workflows. Small contexts are fast, but maybe not necessarily all that intelligent. That’s the main thing to take away. Now we’ve talked sort of basics about documents being possible to use retrieval, augmented generation or rag is the other term of note to know about here. It’s not the same thing as training a model. It’s just telling it, hey, you can get data over there if you need it. To answer my question, this is one of the best features for a business trying to make sense of their data, because you can say, hey, go summarize our earnings reports for the last five years running. What are the common topics? What have we never covered? Give you some inspiration for the next time around, we could do the same thing with spreadsheets, any other content that the platform you’re using intrinsically supports, but rag is how you get to it without having to train anything brand new. Now, in this in the prompt context, in the document processing context, hallucination is probably one of the other things that people should know about, and it’s been discussed as a terrible thing in a lot of corners. Basically, it just means what a large language model does when it doesn’t have enough information, and it will try to satisfy the prompt because you told it to and that’s not necessarily going to produce an outcome you want.
Michael Grande 10:40
I do recall hearing in certain legal matters that there has been dockets or submissions to the court, and ultimately they were proven to be AI generated, and it was through a hallucination, citing made up case information and historical data.
Jason Pufahl 10:57
And it’s very thorough.
Steve Maresca 10:58
And that’s because the prompt said, produce what I want. It didn’t have guidance around what to do when the information wasn’t present. That’s the necessity and how you avoid that type of thing. Important to know about. It’s just a fine tuning of the systems of this variety. The people using them need to be domain experts in what they’re asking systems to output. If they’re not, they can’t evaluate the output. They can’t find the gaps like this. Base expectation overall. So basically, next, what’s the strategy for building some sort of a reasonable tool? Honestly? First step is define the actual problem. Most of the people who are tired about this subject are exhausted because there’s a lot of enthusiasm, but not a lot of here’s how we solve the problem, or here are good problems that are solvable with these tools. And honestly, they’re all business specific. I would encourage people to go out and look for problems that other businesses have solved with generative AI. But examples include clear goals like automated document processing, summarizing, extracting data, automated document construction, for example, consuming some sort of internalmarketing material and emitting an email introducing it to protect prospective customers. That is a very clearly defined goal, and insisting internal teams, you know, meeting notes, coding assistance, things of that nature. Those are discrete things, and they can be put into a box and defined in a prompt and a data set to support them. Choosing a clearly defined model, or, excuse me, a clearly defined problem, means that it’s attainable, and that enables choosing the right model for the job. Some of them are vision models. They’re only good at imagery. Others are fantastic at producing code. They’re not going to help you write a document. They might produce a program, you have to choose something appropriate for the task. For example, GPT-4. It’s general knowledge. It’s not intended to be subject matter expert in medical terminology. It’s not necessarily a translator, though it is multilingual. You don’t want it to do that work, because it’s a general purpose tool, and some models are better suited to other tasks. If we’re working in an orchestration framework or building a workflow, you can use multiple models to accomplish an outcome. Use something from chatgpt, use something that’s an open source model. Combine the two of them, you get a net output that’s better. That’s kind of the the thought process behind model selection, and that requires experimentation. There’s a lot of information out in the world about what is problem and domain specific. You can do the research and find those things.
Michael Grande 13:58
I have a question as it relates to data access, and, you know, there’s, there’s certain questions, I think it’s an overwhelming feeling, I would say, right? So I’m not the most technical person in the world. You know, I know enough to be dangerous, and then as soon as I want to sort of dive into something, I feel slightly overwhelmed about, right? The unlimited scope of what it is I’m actually trying to get to. And on the sort of converse side, it’s, well, what access am I giving this? I’ve had peer groups that I’ve met with and talked about, you know, it’s not something where you want to be uploading, you know, 10 years of your financial documents to try to spit out some analysis for you, rather than going to speak to a financial consultant. So, you know, where is the line, or, you know, it seems like it’s a fine line, and what access we’re granting to hook into some of this data? And what are things people need to be cognizant of and thoughtful about?
Steve Maresca 14:56
So when subscribing to a particular open source or an open market model, you have to look at the contract, what the license actually allows. Does the vendor protect your data if it’s uploaded or not? Sometimes it’s free or cheap because they don’t give you that protection. Google, OpenAI and Microsoft are good examples of entities with contracts that protect that data so you can safely transmit something sensitive, whether that’s permissible or not, right? Corporate policy is a separate question. You can still use those models with workflows that are housed and run in your own systems on prem, perhaps that reach out to those APIs for integration purposes without disclosing the sensitive data. That boundary can be established with some of the workflow tools that exist. And that’s how you sort of straddle the line. You have to choose what you’re willing to allow out the door and what you need to keep closer to the vest. And honestly, that’s a recent reasonable segue. You have to leverage APIs for scalability and integration purposes. Salesforce, HubSpot, SAP, Oracle, Slack, Teams, Microsoft, Google, they all have APIs for grabbing data from different repositories, for emailing individuals, for generating documents, for extracting meaning and the best integrations for, you know, AI, or the tools that we’re talking about have lots of libraries for integrating with those platforms. So what you’re after next is a tool suite, low code, no code that allows building a flow chart to say, here’s the problem I’m solving, here’s step one step two, here’s what it flows through. Here are the things that it reaches out and grabs, and here’s the output. There are lots of example tools of that variety, just in passing. Nadem is a great one that has had a lot of adoption. Some are local applications. Developers are using tools, for example, that enable them to run, run their own code generation, and those build all of that stuff locally. That’s a that’s an option as well. I think identifying problem specific tools are really the most important outcome of the next steps here, and it’s basically how you wire your tools together. If you have a CRM and it has an API, has it been already wired up with a market, an element, lmm integration tool, that’s where you go next. So I think at the end of the day, defining the problem, seeking integration opportunities, and then selecting a workflow tool that is built for LLMs is what needs to be done for solving a problem. Only after that can you really understand what it’s going to cost from an API perspective, how you can fine tune that in terms of what models you choose, what you don’t, and what next steps you take.
Jason Pufahl 18:07
Understanding the tools is one of the more it feels like, one of the more complicated parts.
Steve Maresca 18:10
It is.
Jason Pufahl 18:11
And there are so many on the market.
Steve Maresca 18:12
Right, and it’s honestly the area of this conversation where we’re not going to be very specific, because it’s moving so fast, looking for LLM agentic workflow tools, specifically low code, no code, will get you a whole heap of opportunities. Pick one that seems appropriate for your business, because there are some in different verticals, and choose them. Choose an example vendor that seems to integrate with the tools that you already have. That’s how you select the tool. Try not to build any code, and you’re, frankly, better off. Yeah?
Jason Pufahl 18:51
Yeah? I mean, ultimately, the you so you outlined the definitions, and the major premise was, don’t experiment as much. Actually identify a business problem. This is, this is problem solving 101, and trying to move away from the idea of, I know AI exists. What can I do with it? It’s more I’ve got a business problem, and is AI appropriate to actually help me solve it?
Michael Grande 19:15
The other, the other aspect, seems like it would be a little bit like, caveat epto, or like, be aware of what you’re receiving in some cases. We talked about hallucinations. We talked about, you know, potential copyright issues in some cases. And like, Be cognizant of the fact that, you know, it’s not something that you generated on your own, and the practical application of where it’s going to go right? So just as that attorney right to be a little bit more cognizant of it, yeah, a human needs to be in the workflow from start to finish, evaluating the output, evaluating the input, making sure all steps appear appropriate, otherwise the outcomes will not be what are expected and not supporting the actual problem to be solved, just to help with what we do, what options that can be automated. Go to nan.io and look at the workflow template library not endorsing the platform. There are lots of others like it. It’s just, it’s very helpful from a visual perspective. It’ll show you where you start certain operations, exactly. And I think that that will really open the door for where you go next.
Steve Maresca 19:15
Right.
Jason Pufahl 20:27
And, you know, that’s probably one of the most useful things that you can suggest. Because I think people just, they know OpenAI, they know Copilot, and they struggle to know what the other alternatives are. The other options are for doing this, right? Because you’re you’re really limited in those platforms, right? You’re only one piece of the puzzle, right?
Steve Maresca 20:44
Exactly.
Jason Pufahl 20:47
It’s a big topic.
Steve Maresca 20:48
It’s an enormous topic.
Jason Pufahl 20:50
And, you know, I think we sort of always end this way, but I think this is particularly appropriate, which is, if you have ideas that you want to explore with us, we’re happy to do it. If you have tools that you think are great, frankly, we may not know about them, you know, let us know. We’ll probably have a discussion again in the future around more specific business cases, maybe that were that we brought into or ways to tie data together. But it’s a really, it’s certainly an emergent topic everybody’s trying to wrestle with. What value can they get from it. And I don’t think most people actually know kind of what they even want to do with it.
Steve Maresca 21:24
And certainly in the video posting and so forth, we can add some content for others to take a look at here that’s intrinsically sort of proving the point here. I think it will. I think it’s necessary for anyone interested in going this direction. Yeah, well, we’ll collect that.
Michael Grande 21:39
And, you know, make sure you change the batteries in your smoke detectors. That was the other big takeaway from this.
Jason Pufahl 21:44
Big big takeaway. Big takeaway.
Michael Grande 21:46
Yeah, public service, yeah.
Jason Pufahl 21:49
All right. And as always, everybody, thanks for listening. Hope you got a takeaway from this and look forward to future episodes. Thank you.
Narrator 21:57
We’d love to hear your feedback. Feel free to get in touch at Vancord on LinkedIn and remember, stay vigilant, stay resilient. This has been CyberSound.
