View Recording: How to Combine Copilot Studio, Microsoft Agent Framework & Azure AI for Enterprise-Ready Agents

Brian Haydin So thanks everybody for joining. Today I am going to be talking about how agents don’t fail because of tools, but really because they fail because of skipping like a control model. And I know that’s gonna be kind of a nebulous thing, but we’ll get to that. 0:0:47.813 –> 0:1:6.373 Brian Haydin My career was built developing software, enterprise software and net and whatnot, but I have been building agents lately in Copilot Studio, so I wanna take a look at building an agent but looking at it from a lens of. 0:1:6.413 –> 0:1:25.693 Brian Haydin Lens of like what’s the best way to do it? Should I be gravitating towards Copilot Studio where the promise of low code and building agents really quickly or should I be building it in procode? I’m really gonna help you figure out like How to make that decision process. 0:1:26.93 –> 0:1:44.653 Brian Haydin And so for the next 45 minutes we’ll kind of talk through a specific example and how I frame up whether to use procode, low code or hybrid. So our agenda. First I’m gonna talk about like the problem and the problem is. 0:1:45.973 –> 0:2:1.413 Brian Haydin Because I have a lot of meetings and I can’t attend them all and I need an agent that’s gonna help me figure it out. And but we’re gonna do this like the same agent kind of looking at it through like 3 different lenses and I’ve got a scorecard that you can use. 0:2:1.453 –> 0:2:18.293 Brian Haydin Help you decide which way you you know, which way is the best way, like what the trade-offs are. And so you can take that decision matrix, you know, and maybe modify it a little bit to to what you want to do, but you know you could also use it as is. 0:2:18.693 –> 0:2:38.413 Brian Haydin So I mean, first and foremost, like why do agents stall even when like a demo or like I saw somebody do this and it looked really great. So I think that there’s really kind of four key areas where agents, you know, tend to stall. They don’t, they die the moment. 0:2:38.613 –> 0:2:58.53 Brian Haydin Security and OPS show up, you know, it’s kind of like the thing, right? So we start off typically with a pilot and you know the tool first decisions, you know how does this actually work? And pilots are great for demonstrating like short term value, but they don’t actually like get into the nitty gritty. 0:2:58.333 –> 0:3:16.693 Brian Haydin Yesterday I was talking with a customer, we were talking about the 80 percent, 20% rule and a lot of times out-of-the-box we can get 80%, you know, of things to like work correctly like on the first try. But really the value like nobody’s spending time on those 80% things, they’re spending time on the 20% that that are really hard. 0:3:18.53 –> 0:3:33.213 Brian Haydin So it’s not really saving time if it solves the thing that people don’t really spend, you know, effort and energy on. So that’s one area where we just have like mismatched expectations. Security, you know, definitely is an area where it stalls. 0:3:33.333 –> 0:3:51.933 Brian Haydin You know, when people have this vision that this agent is going to do these five things for me and then, you know, somebody comes in and says, wait a second, I have to have a human in the loop on this one. We can’t just pass this through. That causes a lot of conversations within the organization. 0:3:52.133 –> 0:4:8.533 Brian Haydin And so the autonomy is really kind of unclear. Am I supposed to act on this or am I supposed to give you advice on what you’re supposed to do? And sometimes we start with, you know, the advise and never get past it and get to the act. And that can be for a variety of reasons, another area where we run into. 0:4:9.573 –> 0:4:28.773 Brian Haydin A stall point is around OPS. Data access and governance come way, way too late. I do a lot of Fabric projects and I talk about Microsoft Fabric a lot, and I always bring governance into the data layer. Like at the very beginning, it’s so hard to come back after the fact. 0:4:28.973 –> 0:4:44.933 Brian Haydin And try to like put guardrails around it. So it’s going to cause a lot of rework and people and organizations don’t necessarily have the energy to put into something like that. And then scale, that’s the last like kind of area where it where these kind of efforts fail. 0:4:45.373 –> 0:5:4.693 Brian Haydin How do I take this pilot that I built for two or three users to get and gather some feedback and transition this into something that’s going to help an entire department or even more importantly, like help the organization holistically? Should I be, you know, using MCP? Should we expose this through our A to A? 0:5:5.53 –> 0:5:24.813 Brian Haydin Framework. These are all things that people don’t necessarily think about when they’re building it. And so don’t get hung up on not thinking of these four things. They need to be a part of the conversation when you start. But before we pick Copilot Studio or. 0:5:24.893 –> 0:5:44.653 Brian Haydin We want to pick one thing. How autonomous is this agent that we want to build? How autonomous are we gonna allow it to be? So I’ve got this control model set up and what I mean by this is, am I going to have an agent that controls and executes work or am I gonna have an agent that? 0:5:44.813 –> 0:6:4.533 Brian Haydin That is just recommending things that I need to do. And a good example of this is Copilot Studio in the recommend mode, right? So here we’re talking about summarize my emails, tell me what I missed on this call that was recorded, summarize the transcription of this, give me some information. 0:6:4.813 –> 0:6:23.693 Brian Haydin Information, right. Those are just like the recommend type of thing. What should I think of next? What’s the next action I can take? Or there’s like I want to. I’ve gotten this e-mail coming in. I want you to tell me what the sentiment analysis and what we, you know, the classification says we should be doing and then go ahead and perform that action for me. 0:6:24.413 –> 0:6:43.453 Brian Haydin By doing that, I’m giving an agent quite a bit of like a blast radius around what can go wrong. And I want to make sure that I’m not, you know, I’m not overexposing, you know, the the kind of problems that people tend to see with with AI agents giving. 0:6:44.93 –> 0:6:59.973 Brian Haydin Incorrect information, hallucinations, et cetera. So you know that control mode is really is this control model is really important to kind of think of like how far do you want to go in the execution mode and how far do you want to just stay this in the recommendation mode. 0:7:0.493 –> 0:7:16.453 Brian Haydin I actually with a lot of our projects start off in the recommend and human in the loop sort of approval before we go into an execute mode, especially when the impact is significant in terms of financial impact of getting the getting something wrong. 0:7:18.133 –> 0:7:36.53 Brian Haydin So let’s pick a use case that we can, you know that we can walk through today. And so we’re gonna start with like a a very basic MVP. You know this is not like this, this we’re gonna be kind of in a recommended mode versus like you know the the execute mode. 0:7:36.693 –> 0:7:53.733 Brian Haydin And that’s where I think most people should start. But let’s take a look at what that use case looks like. So for me, this is my actual calendar. It goes back a couple of months, but it’s very busy. I have a lot of meetings on a daily basis and I can’t attend all of them and so. 0:7:54.173 –> 0:8:14.13 Brian Haydin I’ve come up with a pattern that allows me to like, you know, sift through the noise and the mess and you know, figure out what I missed, you know, you know, by not actually attending the meetings. So my practice, my policy is if I’m not required to be in a meeting and I know I’m not going to attend in it, I use the nice new feature. 0:8:14.93 –> 0:8:33.613 Brian Haydin In Outlook to say I’m gonna follow this meeting and what that does is it gives me access to the AI insights that are being generated, but I don’t have to actually attend the meeting and it doesn’t show up on my calendar. I’ve got filters and people can book over the top of it. Super effective, but now I’ve got all these. 0:8:33.613 –> 0:8:52.693 Brian Haydin Meetings that are on my calendar that I’m following and I’m not actually paying any attention to it. And the theory is that I’m going to go back and listen to these recordings or I’m going to go back and ask copilot what I missed in these meetings. But I’m I’m not actually going to do that. There’s too many of them. And like many, many of you that are probably in this audience, you’re not doing that as well. 0:8:53.133 –> 0:9:12.733 Brian Haydin So what am I going to do? I’m going to build a missed meeting agent. I don’t need more meetings, I need what did I miss and what do I owe list as a digest that I can work through. So how’s this agent going to work? I’m going to take certain inputs like my calendar and my. 0:9:12.813 –> 0:9:32.573 Brian Haydin Transcripts and the AI insights, and I’m going to generate some sort of a daily report or transcription report that’s gonna get sent to me in an automated fashion that says, hey, this is what happened today that you missed. These are the meetings that you followed, that you didn’t attend. Could even be like they were meetings that I said I was gonna. 0:9:32.653 –> 0:9:52.373 Brian Haydin Attend and wasn’t able to get to, but not just a meeting summary because in a lot of times I don’t even, I don’t even really care what the meeting like that, what the contact was, was I called out? Did somebody ask like we should talk to Brian about this? That’s the important thing that I really want to have reported back to me, not. 0:9:52.493 –> 0:10:11.173 Brian Haydin A detailed summary of the 15 meetings that I missed today. So I want to keep this kind of like an MVP tight though, so we can like look at the frameworks and the methodology here and most importantly to figure out are we going to use low code or are we going to use pro code, right. So we’ll stay, we’ll stay with that MVP tight. 0:10:11.853 –> 0:10:31.493 Brian Haydin And we’re going to look at it in three different ways to figure out what what’s the best way to do this without slipping into like a science project, right. So useful, not perfect. Find the meetings that I missed. And I like I already kind of mentioned that there’s two different sort of criterias I want to use on this one. So I would. 0:10:31.693 –> 0:10:51.533 Brian Haydin I would look at it as was it just was I not in the attendee list is the most important way to look at it. Pull the AI recaps when it’s available. Do I have access to it? I should have access to the AI recaps if I was invited and followed the meeting. If I decline the meeting, I’m not gonna have access to it. It shouldn’t. 0:10:51.573 –> 0:11:10.573 Brian Haydin Even show up in my list and then extract mentions, get the action items. This is something that AI is actually pretty good at, is doing those kind of summaries and figuring it out. And then just once a day I don’t need like I don’t need noise. I don’t need at the end of the meeting to send me a recap. That’s not what I’m looking for. 0:11:11.253 –> 0:11:30.973 Brian Haydin Looking at a digest that’s at the end of the day and and only send it to me if there’s something important. I don’t know if anybody else has worked in like, you know, site reliability or you know, just basically been support, you know, in their career. But you wind up getting these emails on a daily basis. Nothing happened, nothing happened, nothing happened, something happened. 0:11:31.93 –> 0:11:51.13 Brian Haydin Nothing happened. Nothing happened. What happens is it just becomes signal noise and you don’t actually pay attention to it. So I only want this e-mail to come out to me when something actually happened. We’re not going to do any fancy UI. We’re not going to build any tasks out of it. We’re just trying to get a digest, right? And just for me, let’s stay. 0:11:51.93 –> 0:12:10.333 Brian Haydin Away from the multi tenant, we’re not going to sell this thing to anybody. So that’s MVP definition. What goes on in the agent though? So as I’m going to be building this, you know, as an architect, I try to think to myself, all right, how do I get something like this to work? I’ve got all these different tools, I’ve got these different patterns. 0:12:10.613 –> 0:12:25.453 Brian Haydin What are the things that are available to me? So we look at the sources first, like the knowledge sources. I’m going to be looking at my calendar. So what events you know did I participate in? I’m going to be looking at any kind of e-mail exchanges, maybe somebody had. 0:12:25.693 –> 0:12:44.693 Brian Haydin Reply to all to the group. After that I might look at the meeting chats to figure out like what happened in the meeting chat and then obviously I wanna get access to the meeting insights. So those are my knowledge sources that I’m gonna be using. In the center I’ve got this AI layer. This is like the black box that we’re gonna. 0:12:45.253 –> 0:13:4.933 Brian Haydin How to figure out how to implement it. Now sources don’t really change based on whether we’re doing procode or low code. This is just what we need in order to make our decisions. But that AI layer is what we’re really gonna spend time trying to figure it out. This is a hopefully we’re gonna find some reusable patterns here, things that we can lean. 0:13:5.13 –> 0:13:23.293 Brian Haydin Into on a regular basis and then the delivery, that’s just what we need. I can do e-mail, I can do teams, doesn’t really matter to me, but at the end of the day it’s that’s that’s not gonna change how we do it. So how are we gonna make a decision on what goes into that middle box? Like how are we gonna score? 0:13:23.813 –> 0:13:43.333 Brian Haydin You know each one of these different approaches. So I mentioned on the low code that’s Copilot Studio. That is some sort of an agent builder type of thing. Open claw maybe where it doesn’t require me to actually write any code more configuring how the agent needs to work. I might write a little bit of code, but usually pretty. 0:13:43.333 –> 0:14:2.653 Brian Haydin Pretty lightweight procode would be like full net. I’m gonna build this, maybe it’s Python etc. And then hybrid is I’m gonna figure out are there some low code things that I can use and some procode things that I can use to deliver the most value in the right amount of time. So the. 0:14:2.933 –> 0:14:21.413 Brian Haydin The six criteria that we’re gonna walk through time to value. How long is this going to take me to build and get actual value? So like these metrics are changing a little bit as we get into vibe coding, but let’s just let’s ground on the fact that we know that low code is gonna be quicker. 0:14:21.893 –> 0:14:41.253 Brian Haydin Because we don’t have to like build deployment models, we just, you know, build it in a in a platform. Then we have like governance and compliance. So what are like some of the security guardrails and stuff that we need to put in place? Are there, you know, compliance aspects that we need to pay attention to and does the tooling in low code? 0:14:41.333 –> 0:14:58.733 Brian Haydin Or procode support those kinds of those kind of initiatives. Can I extend this? How am I gonna reuse this if I build this in low code? Sometimes it’s not as easy for you to build it in a reusable fashion. Sure, you can probably put some MCP layers over the top of things. 0:14:59.13 –> 0:15:15.613 Brian Haydin But a lot of times I want to like reuse this specific function and and if I do a custom inside of it low code, I may or may not have that ability to reuse it. What’s the cost in the overhead? How many tokens am I going to burn? How do I actually figure out that? 0:15:16.813 –> 0:15:35.853 Brian Haydin What if something changes? How do I track it? Does my team have the ability to actually do this workload? That’s an important one. So you might hire a company like Concurrency to build something like this, but at the end of the day, it’s yours and you’re gonna have to be able to support this. So do I have software engineers that? 0:15:36.93 –> 0:15:55.413 Brian Haydin Can actually support a procode solution? Or am I better suited for low code? Do I not have either right? And I’m gonna have to use a partner, so you wanna think about that as well. And then do I have access to the data and can these tools actually get access to the data and in Copilot Studio? 0:15:55.973 –> 0:16:15.573 Brian Haydin Just the Power Platform in general, there’s a lot of connectors to data sources that your organization’s probably already using, and there’s workarounds for the ones that they don’t, right? So maybe we have API access and et cetera, but like we have to understand like what and how we’re gonna connect to the data and you’re gonna see why that comes into play. 0:16:15.573 –> 0:16:33.773 Brian Haydin In our decision matrix and what we’re gonna build. So before we get into like the evaluation and there’s someone on like just sort of be transparent with you on that this, these approaches need to be evaluated from time to time because things are changing this ecosystem really, really quick. 0:16:34.213 –> 0:16:53.973 Brian Haydin So here some of the latest things in Copilot Studio that have made it a little bit more of an attractive solution lately are evals being built into into Copilot Studio, MCPS being available, being able to use different AI models that. 0:16:54.133 –> 0:17:11.613 Brian Haydin That weren’t possible before. So we’ve got more tool integrations, more connectors to some native things that would have made it harder for us to do in a pro code solution. Also kind of similarly in that Power Platform, Power Automate continues to get better and better. 0:17:12.133 –> 0:17:31.613 Brian Haydin If anybody’s been using any of the Copilot to be able to build workflows in Power Automate, it’s fantastic and you it’s even becoming less less code cuz you can just say hey connect to this Excel file and extract this information and then you know I can move on from it. 0:17:32.53 –> 0:17:49.173 Brian Haydin Microsoft rebranded, I don’t want to say rewrote is a better word. Their agent framework from Semantic Kernel and Autogen into more of like a reusable Microsoft agent framework that’s accelerated. Excuse me, that’s accelerated a lot of the development efforts. 0:17:50.413 –> 0:18:5.373 Brian Haydin And made it more reusable, more durable and also it’s built more for like the best practices in how we’re developing agents. And then the graph AI insights are being kind of elevated to a first class citizen. 0:18:6.653 –> 0:18:22.453 Brian Haydin So we were having a little bit of a experiment trying to get access to AI assist the A I like notes from a meeting, but now that’s becoming a little bit easier. This makes the procode solution a little bit more. 0:18:22.533 –> 0:18:41.253 Brian Haydin You know, a little less attractive, I should say. So that’s how things changed. And so now we can kind of get into like how I broke this down. Let’s look at like a low code. So low code is how you prove value before you prove your architecture and and sometimes that’s good enough and you can leave it there. 0:18:41.813 –> 0:18:58.733 Brian Haydin But like, it’s best for like getting the value demonstrated very, very quickly. And it’s really, really good at these things like recommended mode, right? I like, I just want the information. I want you to summarize some things for me. I don’t need you to, you know, go and perform a lot of actions. It can do that. 0:18:59.333 –> 0:19:14.653 Brian Haydin But simple actions like sending an e-mail, you know, etcetera are are more like the best, the best fit for it. So in Copilot Studio and using Power Automate, what we’re going to do is set up some sort of a trigger. In our case, we talked about this agent being like a once a day thing. 0:19:15.13 –> 0:19:34.813 Brian Haydin So we’ll set it up on like a cron job or some sort of a, you know, a timer. And you know what if the first thing it’s going to do is go out and get yesterday’s meeting, get the whole list of yesterday’s meeting. And then from there we have to filter down to like the meetings that we care about because I don’t, remember I don’t. 0:19:35.13 –> 0:19:51.653 Brian Haydin Want that digest of every of every meeting that it was a part of. I just want the ones that were important. And so this is an area where we kind of got a little bit tripped up building this. And so from a technical standpoint, I’m gonna get into the weeds a little bit, but. 0:19:51.733 –> 0:20:10.133 Brian Haydin In Copilot Studio and Power Automate, you can get a list of your calendar meetings pretty easily. It takes about two minutes to set up the workflow and you’re going to get that list. And then I want to figure out not just if I was, you know, if I accepted the meeting or what the status was. That’s actually fairly easy. 0:20:11.253 –> 0:20:31.173 Brian Haydin I want to understand was I actually in the meeting or not? And that becomes a little bit more problematic because the object inside of the calendar event that you need to have access to is the meeting attendees and only the person that set up the meeting can actually see that. So I hit a blocker right away in terms of like can I? 0:20:31.373 –> 0:20:50.693 Brian Haydin I access this and so I had to kind of pivot a little bit and figure out what I’m gonna do to get around that. Fortunately, like the insights had a little bit more information that I could use. So after I get the meetings and I filter out the ones that I really don’t care about, maybe the ones that I declined, maybe the ones that like I accepted, there could be some criteria. 0:20:51.253 –> 0:21:9.933 Brian Haydin To whittle that down, I can then fetch the insights and in the AI insights I can typically see who actually was there, who actually spoke. So that gave me a little bit more ability to kind of filter from there as well. Then I can do the AI work, you know, hey, here’s a pile of data, pile of information. 0:21:9.973 –> 0:21:29.573 Brian Haydin Can you create a digest to this? And prompt engineering allows me to kind of filter that out pretty easily and then deliver it to deliver that out, fire off an e-mail with that summary. That part’s also pretty easy in the low code, but it’s kind of like that middle area where we ran into a little bit of a. 0:21:29.573 –> 0:21:49.253 Brian Haydin A little bit of a snag. And So what were some of the reasons why I talked about like the actual object? It’s kind of painful to go through these JSON fields inside a Copilot Studio or a Power Automate inside of the flow. It’s all JSON representation. 0:21:49.333 –> 0:22:6.733 Brian Haydin It’s got some brittleness. You got to really wiggle into it, and then you may or may not have access to it. Sometimes you have access to it, but not through a native connector. You’re doing it through an HTTP request, and that stuff is hard, and it’s kind of diverging away from the low code. 0:22:7.573 –> 0:22:25.693 Brian Haydin You know, it’s getting more into there’s actually code in there. And so that graph, being able to traverse some of the JSON that’s not well documented was kind of an impediment. And it was kind of a place where I said, you know what, I’m gonna back out of this right now. I’m not gonna be able to meet the objective of this. 0:22:25.893 –> 0:22:40.613 Brian Haydin So, you know, it didn’t, it didn’t really kind of meet the the needs that I had. And so how did it score, you know, based on this experience in the rubric, our time to value, you know, I could get. 0:22:41.173 –> 0:23:0.573 Brian Haydin I could get the list of all my meetings, I could get the AI insights, I could create a summary and I could do that and all that kind of work in a day. So I feel like the time to value was pretty good. Governance was a little bit flaky. That was harder for me to control. I didn’t have access to. 0:23:0.773 –> 0:23:16.333 Brian Haydin The types of information that I needed and I wasn’t gonna be able to overcome that definitely was not extensible. There was a lot of like, you know, hard coded things that I had to put in there to get access to some of the decision things that I was making for the filters. 0:23:16.613 –> 0:23:36.93 Brian Haydin Cost OPS, great skill alignment. You know, we do a lot of software engineering here. That stuff was good. So that’s kind of where this scored out. That data complexity was definitely two stars. Didn’t feel like there were a lot of struggles in trying to figure out how to. 0:23:36.133 –> 0:23:54.573 Brian Haydin Access some of the data. So to remove some of these blockers, let’s take a look at procode and how and what this would look like. So the pattern’s pretty much the same. We’re gonna fetch data, but we have a lot more ability to grab that data in a clean fashion. 0:23:55.13 –> 0:24:14.333 Brian Haydin We are using native.net tools that connect to the data sources. Programmers love it. I get all the fine granularity and then we can normalize those into clean data structures. We can actually pattern that to what we expect and grab the information that we need in a really. 0:24:14.973 –> 0:24:34.213 Brian Haydin Really efficient manner before we send it off to the reasoning engine and then using the agent framework, running it through some AI and doing that processing is pretty trivial, but we have to write a lot of code and every one of these if you’ve done software engineering. 0:24:34.693 –> 0:24:53.813 Brian Haydin Is is a lot of step, a lot of testing, a lot of business logic, a lot of AI processing. How do I measure the results? I don’t have these like built-in evals like I have in in Copilot Studio. So I’m putting more effort and work into this and I’m building these contracts so that they’re really tight. 0:24:53.813 –> 0:25:8.853 Brian Haydin And but it takes a lot of effort to get to that point. So this is like gives me a lot more power to be able to do all the things. I didn’t run into any of these roadblocks I could like natively train. 0:25:8.853 –> 0:25:26.93 Brian Haydin You know, traverse the different, the different elements inside of the the Graph API. But I wasn’t, you know, but I was spending a lot of time doing it. This was several days of me tinkering around the graph with the Graph API. The AI stuff was. 0:25:26.373 –> 0:25:46.293 Brian Haydin In kind of an experimental phase. And so it definitely took some time. So rather than getting it done in a day, it’s more like, you know, a long weekend, a couple of days, a couple of weeks. You know, for some developers, time to value definitely shrinks down like it wasn’t green, you know, gold. It wasn’t green stars in that one, but I had a lot more. 0:25:46.373 –> 0:26:2.493 Brian Haydin Control, right? And I could really fine tune this and understand what I was grabbing and who had access to it. So that governance was a lot better. I write good code, or at least copilot writes good code, so. 0:26:2.773 –> 0:26:19.613 Brian Haydin You know, extensibility wasn’t an issue, but now that I’m actually in charge of managing tokens and picking models and you know how many times am I making requests, my costs are probably gonna go up as I’m using AI Foundry rather than. 0:26:19.973 –> 0:26:38.293 Brian Haydin The simple math of a penny, a request you know that would be in Copilot Studio skills. You and the organization might not have the depth of development, so I think that for most organizations is going to be somewhat of an impediment working with agents. 0:26:38.293 –> 0:26:53.133 Brian Haydin And but handling data was not an issue. Like it’s super easy to connect to an API, you know in procode more so than it is in like a low code environment. So so. 0:26:53.413 –> 0:27:12.213 Brian Haydin Kind of like the summary here is like it’s really good for scale, but it’s a lot of complexity and a lot of time that goes into it. So in the first case we hit like low code, we hit low code walls that we couldn’t, that we couldn’t get through without, you know, breaking the wall down on the procode side though it was a lot. 0:27:12.933 –> 0:27:29.613 Brian Haydin Heavier and I’m gonna be like perfectly honest with you. The kind of value that that that I would get out of something like this is not something that I want to spend weeks or months trying to develop something. Low code is like great because like I’m saving myself a couple minutes a day getting this digest. 0:27:30.333 –> 0:27:45.533 Brian Haydin Or maybe 1520 minutes a day, but I’m not saving a ton of time by getting this digest, so I don’t want to spend like, you know, an entire weekend trying to develop this. So what about hybrid? What are some of the things that I can use, you know, hybrid for? 0:27:47.53 –> 0:28:2.973 Brian Haydin So on the hybrid side I can create pro code to go and gather the data from the knowledge sources that I was really struggling with doing some of that filtering right? And then on the low code I can use like the orchestration in the UX. 0:28:3.333 –> 0:28:20.893 Brian Haydin So I I don’t want to think about this more as like a hybrid being a compromise and like your functionality. I want to think of it more as it’s a separation of concerns. What are the things that Copilot Studio is really, really good at right now, the user experience of being able to build agents quickly? 0:28:21.453 –> 0:28:38.533 Brian Haydin And handle topics and really manage the conversation and the workflow. It’s fantastic. That orchestration it’s really, really good at. I can make it a single. I could write my code in a place where it’s just a single Azure function. 0:28:39.173 –> 0:28:58.53 Brian Haydin That I have to go and grab information from and that’s where that procode is going to excel. Complex integration, resilience, traceability, governance, security. Those are the things that procode is going to be is going to be excellent at. It’s more durable and I don’t have to worry about things crashing or. 0:28:58.253 –> 0:29:13.973 Brian Haydin Or like security parameters changing and have to go back and figure it out more secure from the standpoint of I can use things like, you know, keys and secrets that you know that I don’t really have access to in Copilot Studio as well. 0:29:14.333 –> 0:29:33.493 Brian Haydin So, so finding that like a balance between high between procode and low code is really kind of where the sweet spot is. So all right, let’s make this a little bit more concrete. What did this final architecture look like? So on the Copilot Studio side, I’m using like. 0:29:33.893 –> 0:29:53.373 Brian Haydin The intent recognition, conversation flow, orchestration flow, like how do I get from point A to point BI can do the triggering, you know, in that environment. On the back end API though, I’m actually connecting to the Knowledge Graph. I’m actually connecting to the AI components to do some of the processing and the logic that I want to do. 0:29:54.93 –> 0:30:13.373 Brian Haydin And then I can package that all up as a hybrid container. So Copilot Studio is kind of the face, the back end is really the brain and the contract’s really in like how I do that handshake between the two. So any anybody have any questions about this, you know, approach as well? 0:30:14.93 –> 0:30:33.333 Brian Haydin Feel free to drop some stuff in the chat. I know that you know we’re moving moving through this a little bit quick, but all this output now is being delivered back through like Copilot Studio and through the Teams ecosystem, whether that was through Power Automate and sending off an e-mail or I could like. 0:30:33.653 –> 0:30:52.173 Brian Haydin Like I mentioned earlier, we could do a Teams, you know, build this up in like a a Teams bot that sends me a message on a daily basis. And so now we looked at a little bit of the architecture, what belongs in each one of these these components. So low code conversations and prompts, I think those are really, really good places. 0:30:52.533 –> 0:31:12.173 Brian Haydin For the low code scheduling and delivery mechanisms, absolutely. It’s, you know, single click, it’s drag a thing on a screen or even just using the copilot inside a power platform to like schedule something. Approvals and workflows are fantastic for that, like we can set some of those mechanisms up. 0:31:12.213 –> 0:31:31.733 Brian Haydin Using the low code tools and then formatting the output is a great way to do to do that as well. On the procode side though, graph calls, paging, retries, all that kind of heavy lift you know type of software engineering is better to do in the procode. I’d also say like some of the. 0:31:31.733 –> 0:31:46.533 Brian Haydin Experimental things are gonna be easier for you to do on the procode side. Getting the insights like actually working through different models. I found that I had more tools at my disposal. 0:31:46.933 –> 0:32:3.493 Brian Haydin To be able to do classification, intent recognition, you know, etc. That whereas on Copilot Studio I’m sort of limited to the different models that I want to use and then normalization and ranking, getting that stuff to fit into a more compressed. 0:32:3.773 –> 0:32:22.813 Brian Haydin Digital footprint that I can send out the AI. That’s some of the things that that were better in procode. You get some testing in telemetry out of the low code solutions, evals definitely, but telemetry and like trying to figure out what went right and what went wrong a little bit harder. 0:32:23.93 –> 0:32:39.253 Brian Haydin In the Copilot studio. So I was able to use things like App Insights to do a little bit more telemetry on the Procode side to give me an understanding of where those failure points were and where I need to pay a little bit more attention. So hybrid, how did that score? 0:32:41.173 –> 0:33:0.573 Brian Haydin For me, clear winner hybrid, you know is gonna win in a situation like this agent because it scales pretty easily. And if I wanna go from the recommend mode to the execute mode, I don’t have to like really rewrite anything I’ve. 0:33:1.93 –> 0:33:18.253 Brian Haydin Got the ability to like send out a meeting request or send out a task, you know to you know for things for me to follow up. I can do that without having to rewrite it. Those are things that those are like the end point actions that are really easy for me to do inside of Copilot Studio. 0:33:18.693 –> 0:33:33.973 Brian Haydin Time to value. It’s, you know, I’m writing a little bit of code in some like very siloed areas, but I’m not spending a lot of time writing code. I’m doing it for very purpose fit functions, so I felt like the time to value was pretty good governance. 0:33:34.613 –> 0:33:51.373 Brian Haydin Like blew it out of the water. I had control over everything that I needed to have control over. So I would say it’s excellent in that governance and that extensibility, you know, area because you’re getting kind of the best of both worlds cost, you know, a little bit. 0:33:53.53 –> 0:34:9.93 Brian Haydin Mileage is gonna vary, I would say for for most people, but you know we can, we can get some of the request costs, you know, pushed off into a simpler model in Copilot Studio. But we did actually have like consumption costs in both environments. 0:34:9.493 –> 0:34:29.93 Brian Haydin So I would say like that’s the one where like, you know, sort of like didn’t go one way or the other and then the data and the complexity, obviously the pro code’s able to handle that, you know, very easily. So all this really meant that I was able to get, you know, working prototype up, you know, still within like a couple of days. 0:34:29.173 –> 0:34:39.453 Brian Haydin But not within a couple of weeks. Like a real like solid procode solution would would take you. So let’s close the loop a little bit on this one. 0:34:40.933 –> 0:34:57.453 Brian Haydin If that control model determines your build model, the moment that your agent acts, you need to start looking at going into a different class of systems. So on the recommend mode, we’re doing minimal permissions, just has access to things that you have access to. 0:34:58.293 –> 0:35:13.573 Brian Haydin The auditing, you know, can be pretty simple because it’s just giving you information. We don’t have to track like where and what it did and and then we don’t really have a lot of potential impacts downstream. Maybe I gave you some information, but. 0:35:14.773 –> 0:35:29.173 Brian Haydin The whole point of having a human in the loop is that somebody’s looking at the information and saying does this make sense? A lot of AI tools are gonna give you the source documents, reference documents so that you can understand where it came from. 0:35:30.293 –> 0:35:48.973 Brian Haydin And you can, you know, audit, you can kinda like make sure that it’s it’s not hallucinating on the execution side though I would say like you know the approvals and like human in a loop confirmation like that is gonna be in the execute. Like if you’ve got a human in a loop in order to perform those actions, now you’re transitioning into the execute node. 0:35:49.493 –> 0:36:9.173 Brian Haydin If you have really tight, limited scope on the actions that you want to manage like you’re going into the execute and a lot of times it’s what we’re doing. I we we do agents where we’re performing financial transactions like those are least privileged like we’re we’re being really careful about that. 0:36:9.173 –> 0:36:26.613 Brian Haydin That and then monitoring and having like a really good solid rollback plan that’s definitely in the execute and like that procode that proc phase. So if you are interested in taking a look at that decision framework that I had. 0:36:27.413 –> 0:36:42.453 Brian Haydin You know, please reach out to us. I believe that Amy dropped a a form or a brief survey here in the in the chat. So would love to hear from you like what you thought of the information that we shared today. 0:36:43.893 –> 0:36:59.933 Brian Haydin And and if you have any questions, you know, drop those in the chat or reach out to me on LinkedIn. Happy to talk about any of these types of topics. So key takeaways for today if you’re going to, you know, venture into the agent world. 0:37:0.373 –> 0:37:19.333 Brian Haydin I would say start in the recommended mode. Try to prove that value fast, like what am I going to learn out of this idea that I can say I need to go to procode for this and low code for that and then keep a, you know, keep an open mind that you’re probably going to land in that hybrid. 0:37:19.973 –> 0:37:36.893 Brian Haydin You know, area for most use cases. I just don’t find that one falls, you know, exclusively in low code or pro code on a regular basis. We’re typically landing somewhere in the middle. And that being said, I will stick around for a minute for any questions. 0:37:38.373 –> 0:37:40.973 Brian Haydin And thanks for joining today. Appreciate it.