View Recording: Transforming AI with Semantic Kernel

Brian Haydin 0:03 Thanks for joining us today with concurrency as we talk about semantic Colonel and today’s topic is really about transforming your AI journey with semantic kernel. Amy Cousland started transcription Brian Haydin 0:16 And if you haven’t heard of what that is, stick around. You’ll hear a little bit about what it is and how it can help your business, but before we get started, quick introduction my name is Brian Hayden. I’m a solution architect that concurrency here’s the QR code for my LinkedIn. If you wanna reach out to me and connect at some point, love to hear from you. Uh, and talk about uh, some of the technology problems that you have, I like to say that I have like, the greatest job at concurrency I get to play around with some of these cool tools and talk with a lot of customers out there that you know are trying to figure out how to use use AI. So today what we’ll be talking about is Samantha. Kernel. What’s on the agenda? Let me explain it to you. This is a. This is definitely A-100 level course. What it is? Why you should be thinking about using semantic kernel in your development cycle. How do you go ahead and get started with it? And then I’ve got a couple of practical use cases and maybe a fun one to uh, like, near and dear to my heart as well. But before we start on that agenda, let’s just talk a little bit about technology. We we talk about this in terms of framing the AI journey and where technology has been really accelerating that growth. You know, over time here you can see this chart technology is has generally been thought of to, you know, really increase productivity and productivity growth. Umm, people talk about the printing press as being, you know, something that was great that really brought knowledge to a lot of individuals, but you can see it really didn’t have a lot of effect on on productivity growth and neither it’s steam engines for that matter. But as we started to get into the 2000s, we had more things like the Internet came around. Smartphones came around. AI is now the new technology, and here’s where you’re starting to see some of the massive productivity green gains. Uh, you know, for the country and for the world. Really ohm, but within this AI journey there’s a couple of different contexts. On the one hand, you’ve got commodity based AI, so this would be like your Microsoft copilots uh, things that are enabling individuals to to work with AI in the tools that they they use. And then you’ve got more mission driven and mission driven are typically custom built applications or leveraging tools that can do something that’s meaningful and developed specifically for your business, things like supply chain forecasting, elements like that ohm. And to illustrate this a little bit further, you can see kind of like this commodity versus mission driven as being how accurate and precise at the very top you’ve got like your Microsoft copilots really it’s about just worker productivity your your that’s like the general web copilots and then you’ve got your Microsoft 365. So how am I going to write a faster email? What about generating PowerPoint content? I’ve got some really great stories about how I’ve been able to put together a full presentation, not this one. Took me longer than 30 minutes, but I’ve been able to put together a full presentation and gone out and talked in front of 100 hundred people, you know, and just took me 30 minutes, 45 minutes to get a presentation done. Then you start getting into like the semi custom space and the custom ML and this is where the mission driven. You know, elements of AI are really start to manifest themselves. Sometimes you’re using pre built AI models. Sometimes you’re creating your own custom models, but you also have to have a way to be able to interact with these on the more commodity side of it. You’ve got copilot studio which is power platform enablement that allows you to interact with alms and other models to get work done, but in a low code fashion. What we’re going to be talking about today is is similar to that, uh, we can use custom uh models we can use out of the box models with semantic kernel. So what is Samantha, Colonel? Semantic Kernel really is just a a lightweight framework. It’s open source provided to by Microsoft, and it’s designed to help developers create applications and agents leveraging AI components without having to do all the fussy, messy work of marshalling all this, all the reasoning and actions you know together, so it allows you to it’s really an orchestration component that brings all the different components of this together. And what does it really look like? So, umm, with Semantic Colonel, you’re going to have your application leveraging semantic kernel semantic. Kernel’s gonna have plugins or other agents that it interacts with, and those plugins and agents can all interact with the other AI applications like Azure Open AI service or you know the yeah open AI, as you know, chat, GPT itself. You can use all these different components together as part of semantic kernel to build a smart, intelligent application. Uh, you know, for your company, your organization. And why would you need it so? Ohm. Really, when we start talking about Semantic Kernel, we’re talking about some of the more advanced functions of AI, and in this context, we’re gonna think about agents as the foundation of like, complex AI. So I’ve got kind of like a wilderness, theme for those that know me or those that you don’t. Big outdoorsman, I spent a lot of time out in the woods. Umm, so uh, we’re going to be talking a lot about this wilderness, theme. So I’ve got a task and I wanna plan a two week back country trip out to Idaho. Never been to Idaho. My niece lives there. She says it’s beautiful, but I wanna go on the remote wilderness. Ohm, what are some of the things that that I would have to do in order to plan this out? I probably wanna take a look at the weather. What the conditions are gonna be like? What am I gonna need to bring along with me? You know what kind of tools do we need? What am I gonna survive out there for two weeks? How much food do I need? What? What kind of routes am I gonna take to get from point A to point B? What are my options? Am I gonna have access to water? You know, does it have, you know, what happens if something goes wrong and they need to get out of there? Evacuated. Is there an easy the exit strategy for me and this is what agents can do so when we with AI agents what you’re doing is you’re asking the agent to plan this trip for me and you’re relying on it to figure out all these different components in order to have a successful trip. And uh, we’ve also evolved, you know, agents over time. In the early days of computing, you know, we were doing basic automation. We’d write, you know, jobs or scripts that could, you know, get certain tasks done. And then we started to get a little bit more automated with this and a little bit more intelligent. We started using RPA, you know like UI path or other elements, but now we’re looking at this like, you know, new AI world that allows us to do many other things. LMS like chat assistants where where the thing that kind of sparked all this creativity about two years ago, ohm and now we’re harnessing that energy of the L I’m chats and building agents out of them. So that’s where we are today. Agents can be strictly language like chat bots or they could be multimodal where they’re interacting with images, and you can even have multi agents and all this is to prep us for more autonomous agents and replicating agents in the future of AI that we still don’t know what it looks like today. O what are these agentic patterns? They’re really acting as like personal assistants for you to accomplish some sort of larger, more complex task. We talked about trying to plan a trip out to Idaho. Uh, that’s something that I would probably spend weeks, you know, planning out. And I’m not going to build a chat agent or a AI agent today. That’s gonna, you know, reduce that from 2 weeks and probably get pretty nervous about it, but they can use a bunch of knowledge from a bunch of different sources and help you actually come up with something that’s more well reasoned and even just give you ideas of the things that you should be thinking about and how you’re going to formulate your plan. So. They you know, more formally, they they take actions. They have memories that you can respond to. Some of the suggestions that has they can do planning agents orchestrate different elements or different sub agents and they’re like I mentioned, they’re very event driven and they’re interactive that allows you to be able to say plan this trip for me and then it says you need, you know, I don’t know, 6 bags of tapioca pudding in order to survive. And I’m gonna question it and say I don’t think I, I think I probably need some chili cause it’s got more more protein, umm, and it’ll adjust to that and it’ll give you new recommendations based on that you and you know most of you have already been using chat, GPT, you understand I didn’t like that response. Can you try again, but do this other thing with it. When you develop agents, you’re gonna get those kinds of feedback elements and you’re going to be able to respond to those in kind. So how does it all work? You know, how am I gonna actually use it? Uh, so there’s several main components that I want that we’re gonna talk about. We’re gonna dive a little bit deeper into as part of this conversation, umm. And at the very center of it, you’ve got the Kernel itself in semantic kernel ohm, and then you’ve got all these different components that are orchestrated together. So we talked about the agent framework. There’s a plugin framework you can adapt to a bunch of different AI services. Uh, there’s ways to store the data. There’s ways to help the agent do planning and then finally, what popped a little bit about personas. This is something that not personas specifically is something that not a lot of people are talking about, and we’ve actually been driving a lot of conversations around that here at concurrency being in the software development field for most of my career UX and the way that computers interact with individuals is really super important to me. Umm and personas, you know, really kind of follow what UX? Uh, you know type of discipline. So let’s start at the top. What is semantic kernel? So I said it’s basically a lightweight framework and the Kernel itself is really. If you’re thinking about it from a developer’s perspective, it’s it’s really just a dependency injection container that you can interact to be able to bring all the different components together as part of your application or as part of your agent. It’s the heart, you know, semantic kernel ohm. And you know, if you want to think of it as kind of like a trusted compass, uh, you know, or your backpack, you know that you’re gonna load up a bunch of gear into it helps you manage all the different essential tools for working with different AI agents. So it’s going to select the optimal path it is going to build your plan, you know, as you make request to it, it’s going to initiate the journey and help you interpret like, you know the results. What is the feedback from the user? What are some of the changes that are going on as part of this interaction? And finally, you know, you get the treasures that are going to be returned to you. So it gives you the your application all of its findings and it’s reasoning to make sure that you’re successful on this expedition. So AI services are a definitely a big component of this, because without those services you’re just gonna have, uh, generic interactions. So one of the kind of the big things about Samantha Kernel is that it’s not Microsoft centric per se. UM, it now is. Is currently supporting Mr role. It’s supporting open AI chat GPT. You can incorporate hugging face models and even Google AI is all part of what it has to offer, but the different AI services are gonna do things like text completion, chat completion. So think of like how you would interact with but chat GPT, but it also has a bunch of different, more experimental things that you can play with. So text to image, image to text, text to audio, audio to text. These are all things that you can incorporate into your agent as you include different services from a code standpoint, it’s as simple. Like I said, this is a dependency injection framework, so all I’m gonna do is I’m just gonna add a new hugging face model using the kernel to keep track of the different services that it that it has. Umm. Now you know from a future perspective this is great because you know, while I might be using chat GPT 40 today, what about when chat GPT 5 comes out or a different, you know, a new service that comes out. So in this kind of model, all you’re really doing is registering services that you’re going to be using. You could switch them out. You know you want to develop your own custom models. You get a new iteration of it. This framework and dependency injection is going to allow you to go ahead and remove or or replace that AI service with something that’s more impactful and meaningful or performs better given the use cases that you have. However, not without some caveats here in terms of how you can use it. You can see on the chart here that I provided for you that C’s kind of the major supported framework in semantic kernel. You certainly can use it with Python And Java, but you can see that as Microsoft makes investments into the framework. Umm, that they’re doing that in C sharp 1st and they’re bringing along Python And Java. You know for the ride as they as they mature so. Next, let’s talk about the agent framework itself. You know the agent is the interaction aspect of it. This is where the questions kind of come through. So they’re kind of like seasoned adventures. Uh, you know that? Have their own unique set of being able to respond to things. Umm, you know, you’re high level agents are, you know, probably going to interact with lower level agents. I’ve got some examples of that, but agents can come on a couple of different flavors, so you’ve got your typical chat bots ohm. You know that help. Uh, you know, answer questions or share information. It’s kind of like, you know, a way for you to interact with all the different maps and journals that you’ve collected over the year over the years for your company. It’d be interacting with your company documents, asking questions, and finding information about it. And then there’s more like copilots. So these are assistants. These are like the explorer that’s walking next to you, assisting you with, you know, different tasks like helping build. Help me build a fire. You know things along those nature. So they give you suggestions that you can, you know, accept and modify and work with. And then there’s the fully autonomous agents. So autonomous agents are starting to come to fruition, especially in the manufacturing space as it responds to events in the real world. These are essentially tires explorers that are just waiting for something to happen so that they can be helpful on your journey. You know, in the wilderness ohm and they don’t require any input. They can make decisions and perform actions on their own. Uh, plugins, you know, we talked about plugins, you know as a maybe sub agents or you know more specialized tools, but I like to think of them as the tools that you put in your backpack as you’re going on this journey, right? If I’m gonna be out in the Backcountry for two weeks, you know what are some of the critical things that I’m gonna need? Definitely need the compass. Probably gonna need something to chop up some wood. Maybe clear some brush or clear trail. Don’t wanna leave home without you know, a first aid kit. In case something happens out there, it can take care of myself, but they really are the specialized tools. You know the plugins you know are are there to actually perform very specific tasks, and we’re getting into some of the examples you’ll sort of understand that. But you know the Kernel is going to have knowledge about your plugins. Umm, you know with the dependency injection you just register your your plugin and you let the semantic kernel engine figure that out how to use it, when to use it and it keeps your hands free. You know to be able to do the more meaningful tasks. How does this work? Umm is yeah, a concept of planning or a concept of planners. So, you know, imagine you’re on this like, you know, really long adventure and you’ve got to pass that needs to get done. I wanna build a campfire. Well, how would you go about doing that? First off, you’re probably gonna want to assess these situation like is it dry outside? Is it wet outside? Is it safe for me to build a fire? You know, what about the location and my restricted to be able to access that then you’re going to have to gather materials. You’re gonna have to choose the location. Prepare the fire pit. All this requires planning, and if you think about each one of those steps as being an agent, the planner and as part of semantic kernel is what actually communicates between those different steps. So semantic kernel introduced the idea of planners. You know, probably was like the first to to really introduce that concept and it’s now been, uh, you know adopted as part of some of the other more popular frameworks open AI introduced a new technology called function calling and now Gemini, Claude Mr they’re all using some sort of you know planning or function calling that allows information to be shared between the different agents so that it can accomplish the task. We did talk about personas and when you think about personas, there’s a couple different ways that you can you can think about it. I like to start at the UX part first and for me how a user’s gonna interact with your agent is very important so that you give them the information that’s most relevant to the type of question that they’re going to ask. And often that starts with proper UX and identifying the different types of users. So sticking with the Explorer theme, I’ve got a theme, people that are marching out, you know, into the wilderness. Umm, who are the different people? I’m gonna have along with me, right? So I’ve got a navigator and when I define that persona for my agent, the navigator is gonna want very precise and detailed oriented answers, right? He’s trying to ensure that the expedition stays on the right path, but the scout, you know, he’s got a different set of requirements. He’s alert, he’s cautious, he’s responsible for jumping ahead of everybody, making sure that the path is safe. Uh, that there’s no Dragons. You know, on the other side of the mountain that are ready to spit fire. Definitely wanna have a medic, you know? Medic. I want them to be, you know, very calm and supportive. And I want them to get freaked out. If you know somebody has an accident and your leader, you’re explorer like he’s going to be confident in decisive. So all these are different ways that would manifest itself and how you direct the agent through prompt management ohm and by thinking about this, you’re gonna have a well, well defined agent that’s gonna be able to, you know, give people the answers that it wants in a way that, you know, makes it feel comfortable for them. Ohm. Another concept that we talked about was the vector stores. So in the world of of AI, we hear about vectors and vector databases. Essentially, you know what we’re talking about is storing information or gathering information for our AI agent to be able to use as part of its response. So, you know, imagine that you’re, you know, going on this big trip. What are some of the information pieces that you’d wanna grab with you? You’re probably gonna have a topographical map. You’re probably gonna have a trail map. You may have, you know, collected some weather information so that you have an idea of what to expect over the next few days and use your compass to sort of navigate between, you know, to supplement these maps and help you figure it out. So in the world of semantic kernel, vector stores are kind of like your map rooms. Umm. And they’re really essential for, you know, you to navigate the complex, you know, terrains such as an LLP or custom vision, computer vision and recommendation systems. So just as you would like store maps and notes and coordinates to help fund your way, the vector store holds that information about different topics, different keywords, facts, opinions and sources that would be related to your journeys goals. And then we have memory and complementing some of the vector stores. Uh, you know, vector indexes. Uh, you know, talking about the Microsoft offerings, we’ve got Azure Cosmos DB, Postgres. Uh, Cassandra and certainly AI search. It’s important to, you know, think about which ones are better for your specific use cases. There’s a variety of reasons why you would choose one over the other. Some of them were simple to use, some are more expensive to run and having a proper place to store your memory is important for this journey. So let’s think about some of the use cases I’m gonna start with a fun one to illustrate exactly how all those kind of comes together. So let’s say that I wanna go fishing on Lake MI Lake MI’s a very Big Lake. Lake for those of you that might not be in the Wisconsin area, it gets dangerous really fast. And sometimes you’re going to be forward 5 or even 10 miles offshore, so not similar to like being out in the open water in the ocean, right? So I’m gonna create a phishing agent and what are some of the things that I need to do in order to make this work? So my agent is gonna, you know, be defined. I’m gonna have personas. That that are gonna define how I wanna interact with it. Are you the captain? Are you a passenger? You know what are the different people that you would interact with, you know, with your agent ohm. The agent’s gonna be responded to responsible for determining what is the journey, gonna look like for you. What’s the plan for this? This fishing trip and then it helps to orchestrate that plan, you know, through all the different plugins and agents that you’re gonna be using. And finally, it’s an engagement mechanism for the user. So when you think about how to build your agents, one of the things that you can do is just think about what are the different steps that you would want to do in order to accomplish this skull, building a campfire? Going on a fishing trip. So for me, being an avid fisherman, what are the things that I do when I go on a fishing trip? Well, first thing I’m gonna do is check the weather is the is the lake gonna be safe? Are there storms coming into the area and you create a weather agent that’s gonna be able to collect that information? Check the forecast and you know help give that information back to the phishing agent so they can determine what are some of the the correct things to be to be planning for. I need to know about the fishing regulations in that area. Are there any changes to the regulations? How many fish can I catch? What kind of equipment should I can I be using and that I can share with the gear agent which is gonna tell me what is the checklist of different gear options that I do have? What about safety equipment? What about other supplies that I might need when I’m out on the lake? Like, you know, flares or light vests and things along those nature. And then obviously I have to have a route. I’m going to get from point A to point B. Point B is gonna be wherever the fish are, you know, and you know, how do I, how do I actually navigate the waters to be able to get there, I have to go through a Marina. There’s channels for us, for me, to go to, there might be restricted zones that I can’t go through that I have to be aware of. And then finally, like once you’ve built these these plugins, you have to have the knowledge you know for the agent to be able to answer that. So some of the things that I thought about in this particular use case would be like fishing reports. I’d want to collect those and be able to. You know, you leverage other people’s more recent experience. Ohh, what are the different fishing techniques? You know what kind of fish can I expect to get out there? And how would I actually present, you know, lures to them that would that would make sense given the current lake conditions and all that other, that other information. So these are the major components of how you would build an agent and semantic kernel, and these map directly to some of the concepts that we went through before. So let’s use something a little less fun and more business oriented for the people here that that joined, you know, for that, that aspect of it. Umm, but I’m going to stay with the camping thing. Help me find a new tent. I’ve got an ecommerce store and how am I gonna go ahead and, you know, pick a key, you know, pick a 10th for a customer. So here we’ve got an orchestration agent again, we’ve got a persona. We’ve defined that agents persona to help this particular consumer, umm, and it’s going to do a product recommendation, so it’s gonna figure out all the different things that it needs to do to figure out which tent that this user is gonna want. And so if I was, you know, a customer service person sitting at the outdoor store and a person came to me and said, I wanna 10, what are some of the things that I would need? What are some of the information pieces that I would need in order to serve that customer correctly? I’m I wanna ask him. What experiences do you have? What kind of history have you, you know, bought from us before? Uh, do you like higher end or lower end gear? How often do you go? You know, go, you know, out in the wilderness and and experience these things. I also need to know what’s available. Do I have this in stock? Are you going this week? You going next week? Can you wait? And I’m gonna make decisions based on that. So I’ve got an availability agent and then I’ve got a bunch of customers reviews. What have other people with similar experiences felt about the product that they bought and that would help give me advice to that customer? Uh, and uh, so that they, you know, would feel good about this purchase. And then what other other people saying about it from a social media standpoint? So all these are different components are gonna feed into the recommendations that I’m gonna serve back to the customer and then I need to have the knowledge behind that. So how am I gonna, you know, pull this information together? A product you know, product knowledge, database sales history, you know, inventory data, equipment, data, all these are different components that are gonna feed into that, so. That is what an agent looks like and uh, in terms of next steps. Umm, you know, First off, I know it was kind of a brief, you know sort of webinar to introduced this you know to everybody hopefully you found this really helpful. If you have any questions, I’m gonna stick around for a little bit, so feel free to throw some questions in the chat, but if you wanna talk a little bit more about Semantic Kernel, I’d love to meet with you and your organization. We can interact in a couple of different ways, but we are offering a complimentary AI readiness workshop. These are a lot of fun. Uh, we we talked to customers about what are some of their objectives and goals around AI. And then also you know if you wanna learn a bit a little bit more about AI agents and see how they actually work. You know, we’ve got a couple of demos that we can do, maybe walk through some use cases, you know, for your organization working with me or some of the other architects on my team. Umm hope you guys found this informational and let me know if you have any questions. We have a question. How many agents are available out of the box and AI studio or the like and how many do you have to construct? Ah, so umm, agents are are essentially mostly custom. Uh, in terms of the interactions they do, there’s AI services I think maybe would be, you know, I could answer this in two different ways. The difference between I services are the models themselves, but the agents themselves are doing things that interpret the information you know and and provide a reasoned result. So they typically have a little bit more to it. Think of think of as. Think of an agent as being like, you know, having a prompt that’s associated with it, and I am trying to get this thing out of this AI model in terms of the number of services, AI services that are out there. What if we go back to the slide that I had that had a list of the different services that are supported? There’s Google hug and Phase chat, opening eye, and Azure’s opening I services. So anything that’s hosted there would be available to you. So I mean, like literally thousands models, when you start thinking about hugging face in terms of the agents, those are mostly going to be things that are going to be important to your special use case. So I think that, you know, think of it that way. I hope that answers your question. So Semantic Kernel absolutely helps with the creation of agents. So we let’s go back and take a look at one of the the slides that has some of the code on here. Umm, so here’s what. Here’s what a typical agent would look like. You know very small, lightweight agent. I’m essentially using the Kernel to to interact with it. I’ve registered my services and all I’m really gonna do is essentially collect the requests and process it through the different components of my agent. And so you register, you know, as part of like your main, if you wanted to give it that way, you’re gonna register all the different components that were part of this, like register the agents, register the plugins, register the AI services. All those are gonna be relatively single lines of code and that you can use to basically say these are the tools that you are putting in the backpack. And then I can ask the backpack to help build me the fire. Umm. And so how it does that is through the planning mechanism, which is way more complicated than I can explain in a couple of minutes or a couple sentences. But think of it as they’re sharing information and it’ll help you figure out. Like what are the things that you need to do? So when I ask my agent to build me a campfire, it knows or it’s aware that I have access to a kindling agent and a spark agent and it’s gonna figure out how. How those pieces need to interact with each other in order to give you the correct answer so it Semantic Kernel in a lot of ways is an orchestration engine. Great questions. Ah, I’m going to do a follow up. Maybe in another month on this one, where or maybe one of the the delivery somebody, one of our architects here too where we actually do like a live demo actually build one in real time, but that’s something that we happy to do Pete for you if you wanna reach out as a follow up we can do a session for you and your organization as well. And as far as the developer interface, it’s, you know this is all built into Visual Studio, is available as Nugent packages and Visual Studio, Visual Studio Visual Studio code. All right. Well, once again, thanks a lot for your time. Hope you found this helpful and we’ll talk to you soon.