View Recording: Taming Agent Sprawl: A Control Plane for Your Copilot Portfolio

Brian Haydin I just want to start off and say thanks for spending a little bit of time with us today. If you don’t know me, my name is Brian Hayden. I’m a solution architect. I am joined by Jim Brown, otherwise known as JB. He’s also a solution architect here at Concurrency. I want to just give you an idea of what we’re going to be talking about for the next half hour, 45 minutes. 0:0:33.609 –> 0:0:53.529 Brian Haydin You know, most organizations don’t end up with Agent Sprawl problem because their maker problem failed. They’re ending up with one because more or less like the agents are actually working, right? So a lot of times just like 3 or more people built their own little agents, you know, 2 departments started to, you know, build more agents. 0:0:54.9 –> 0:1:9.849 Brian Haydin Someone wired one system into another system, and next thing you know, like IT is like, what’s going on, right? So today we’re going to kind of answer the question, how can we build agents responsibly? And that’s what we’re going to start off today. 0:1:12.89 –> 0:1:29.529 Brian Haydin Before I do start in a little bit, I just want to like invite everybody to connect with, you know, with Concurrency on LinkedIn. A lot of companies are asking, how do we get actual value from AI? And Concurrency has been helping customers figure that out. 0:1:29.929 –> 0:1:49.529 Brian Haydin So we do these webinars. If you want to get notice, get notifications, the best way to do that is just to follow us on LinkedIn. And then we’ll see you at the next one. But to get started, let’s talk, let’s define the problem, I would say. 0:1:49.849 –> 0:1:55.769 Brian Haydin And I think that most people are right now are still are still defining the problem with the wrong lens. 0:2:0.169 –> 0:2:20.489 Brian Haydin The problem isn’t that you have a lot of agents, 100 agents that are known to the organization that are owned by individuals and that are doing you like actual useful things. That’s a healthy problem. That’s a program and it’s working if you have something like that. But if you have 20 agents where nobody remembers who built the 0:2:20.969 –> 0:2:38.809 Brian Haydin and half of them do roughly the same thing, and three of them are reading from a SharePoint site that got decommissioned last year, or maybe that’s over permissioned. That’s the kind of Sprawl that we’re talking about. So Sprawl is not a count, it’s a ratio between what you have and what you can account for. 0:2:40.9 –> 0:2:59.209 Brian Haydin So here’s how that happened, and none of these steps is really a mistake. You typically go from a few agents to many, maybe just a few makers to distributed makers across every one of the departments. And then, you know, also graduating from simple and obvious use cases to overlapping capabilities. 0:2:59.569 –> 0:3:18.489 Brian Haydin And that causes like 2 teams solving the same problem twice without really talking to each other and knowing what each other one, what each other’s doing. And you know, so then that problem is basically you have owners which turn into orphans as people change roles or leave the organization. 0:3:18.889 –> 0:3:43.729 Brian Haydin and they don’t have like any way of handing off those agents. So that’s kind of my definition of Sprawl. And I’m going to lean into Microsoft Center of Excellence guidance, and it puts it as bluntly as I can. And it says the first five agents work fine, but by Agent 50, you’re probably going to have a governance crisis. And now one more thing, because the Sprawl that 0:3:43.809 –> 0:4:4.409 Brian Haydin you can see is the manageable half. Everything that I described lives in Copilot Studio where at least there’s an admin center analyst. But a lot of times your developers are running on Cloud Code and running stuff on their laptops. Those are agents too. They have file system access. They have credentials. 0:4:4.889 –> 0:4:23.849 Brian Haydin They’re not in anybody’s inventory. And Microsoft shipped Shadow AI Discovery into Agent 365 back in May. And so you can use some of those tools to find some of those things, but not all of them. What we’re going to do today is spend most of our time talking about the governable half. 0:4:23.929 –> 0:4:42.649 Brian Haydin the stuff that you can actually see and discover. But don’t make the mistake that this is the entire picture of what we’re going to be talking about, because there still are going to be shadow AI agents that you won’t be able to track. JB, this, you know, this doesn’t usually show up as a technology complaint, does it? 0:4:43.529 –> 0:5:5.689 Jim “JB” Brown No, it really doesn’t. A lot of the conversations we have, it’s not an agent problem. It comes with, you know, it might come with a question from the CFO, how did this happen? Or maybe it shows up as an audit finding or a security review, stalling a project. Or maybe somebody finally asking, hey, we got all this AI stuff. What’s our ROI? You know, 0:5:5.929 –> 0:5:27.449 Jim “JB” Brown And the honest answer is that a lot of people can’t say because they don’t see half of it. They just don’t have their fingers on it. So by the time you’re getting all those questions, you know, it becomes a blocker to the credibility of the adoption of AI. It just can become a mess quickly. And so, you know, that’s why you want to focus on it now before it gets out of hand. 0:5:28.89 –> 0:5:31.529 Jim “JB” Brown so that you’ve got an answer when somebody comes knocking at your door. 0:5:32.329 –> 0:5:33.929 Brian Haydin Yeah, very good point. 0:5:36.89 –> 0:5:55.209 Brian Haydin Too many clicks. All right, so I think the reflex is when leadership typically sees this problem, the first time that they see it, they try to lock it down. I hope they’re not on the call right now, but literally was on a phone call yesterday where it was like, we can’t have this, just lock it all down. 0:5:55.929 –> 0:6:17.609 Brian Haydin And that that was, you know, that was the solution to the problem. We got to understand the problem first. You know, we’re going to lock everything down. So, but another way that this can kind of transpire is that, like, you know, people have been setting up things like review boards and whatnot. So they like, you could lock it down by saying, we’re going to send every agent possible through the review board. 0:6:18.169 –> 0:6:41.449 Brian Haydin Now, I want to argue that that’s probably one of the most expensive mistakes that you can make. And let me, you know, let me lean into Microsoft again to let them make the argument for me. So their guidance basically says that a single governance checklist applied to every possible solution is going to fail. And it’s going to fail in a couple of different directions. 0:6:42.9 –> 0:7:0.329 Brian Haydin If, you know, if you over-govern the simple stuff, that’s going to add friction and it’s going to slow adoption and you don’t want to do that. But on the flip side of that, if it under-governs things that are complex, which leaves real gaps in security and it creates liability, 0:7:0.969 –> 0:7:20.89 Brian Haydin Like, that’s also a problem. And so somewhere there’s this line, which I think is the best sentence in their document, is that a uniform process feels fair, but it treats a meetings notes summarization type of agent and a system that moves money as if they’re saying the same thing. And we want to 0:7:20.249 –> 0:7:38.649 Brian Haydin We want to avoid that type of scenario. So look at the spectrum here. On the left, I have an agent that summarizes my meeting notes, and it reads what I already have access to. So if it’s wrong, I’m going to be the one that notices, and I’ll probably be able to fix it in just a few seconds. 0:7:39.129 –> 0:8:3.49 Brian Haydin But next one over, I have a team knowledge agent, and now that other people are trusting its answers and a wrong answer there, it could mislead somebody who doesn’t know enough about it or is maybe a new member of my team. And then you have an agent that updates a record in the system. Now I’ve crossed another significant line. It’s not just giving me information that I’m trusting or not trusting. 0:8:3.409 –> 0:8:24.169 Brian Haydin It’s performing actions on my behalf. And that mistake persists, like it does something wrong. And then on the far right, like you go not only to those actions, but you start impacting customers. And customer facing agents operating on their own, where a failure could turn into revenue shifts, it could turn into… 0:8:25.289 –> 0:8:45.369 Brian Haydin you know, compliance events, it could turn into a lot of bad things. So really at the end of the day, I want you to understand that those four things genuinely need different amounts of governance. They’re not the same problem. And Microsoft and their COE has helped to formalize that into three different risk tiers. We’ll come back a little bit later into the specifics of that. 0:8:45.929 –> 0:9:4.409 Brian Haydin But I do want to say that there’s a practical reason that this matters just beyond like the whole fairness question. Microsoft names the failure mode pretty directly in that when a governance body reviews everything and enables nothing, then teams are going to start to row around that whole problem. Like they’re just not going to listen anymore. 0:9:4.849 –> 0:9:24.169 Brian Haydin So don’t make the mistake of over governing everything, you know, the visible half of your estate. And because all that you’re going to really do is shift everything into that like shadow IT zone, you know, on its own. So that’s the caution I would be, you know, that I would start to think about how you frame that up in your mind. So. 0:9:24.289 –> 0:9:45.929 Brian Haydin The question then is, if we’re not governing creation, then what are we governing? And I would like to, I’m trying to make the point that we’re governing the journey at this point. And it’s roughly the same journey that every successful agent is going to take. For most of them, it starts at the exploration. Somebody builds something, it usually is narrow, it helps them. 0:9:46.329 –> 0:10:7.769 Brian Haydin And those types of things are very low stakes. But at some point, it’s going to extend to other people. They’re going to share that agent, maybe through Microsoft 365 or, you know, enable it on Teams. You know, it could do things like picking up a connector or second now outsource, you know, it starts to grow a little bit. 0:10:8.289 –> 0:10:26.569 Brian Haydin And that’s where you graduate. That’s the moment that we really care the most about today. It’s where things stop becoming somebody’s clever little tool or clever little agent that they use and starts becoming capability that the business is actually starting to depend on. So if that agent goes down, we have a problem, right? 0:10:27.529 –> 0:10:48.249 Brian Haydin So that’s going to help us move into a production state that’s operate, where it has to be watched, where we have to do things like maintaining it, and we have to be able to answer for it when it’s not available. And then the final stage is evolve, which is, I think, the one that people haven’t really thought of yet, or not thought of, but haven’t. 0:10:48.289 –> 0:11:7.49 Brian Haydin thought through yet. And that’s where you ask whether it’s still worth having at all. In some of my other talks, I made the comment that the most mature organizations are the ones that are actually retiring the most agents. And it’s a little bit counterintuitive, but it’s actually getting to the point where you can realize what works. 0:11:7.249 –> 0:11:28.289 Brian Haydin and have the maturity to say this one’s not providing enough value anymore. So, all right, so that those are the two lines of the experiment. The agent’s going to travel along that top lane going from, you know, experiment, you know, shared capability, manage capability, observe service, and finally to a portfolio decision. 0:11:28.809 –> 0:11:49.209 Brian Haydin But notice also that the human is traveling at the bottom lane at the same time. Maker, sponsor, accountable owner, a steward, and a decision maker. So that relationship starts to change at every single one of these stages as well. And the maker who was hands on keyboard at the beginning of the exploration process, 0:11:49.609 –> 0:12:9.769 Brian Haydin He’s not the same person or not doing the same things when they get to the operate. And that’s what scaling really looks like. So autonomy, you know, autonomy can increase, you know, but accountability though is the thing that we have to make sure doesn’t even operate when we get these. 0:12:10.9 –> 0:12:28.969 Brian Haydin Agency to evolve. JB, you and I were talking about things earlier this week and, you know, trying to frame things up, and I like the way that you framed you what what how you framed it on a day-to-day basis, and it was better than me, and I think you’re going to do a better job of explaining it, so… 0:12:13.329 –> 0:12:13.689 Jim “JB” Brown Thank you. 0:12:29.369 –> 0:12:30.649 Brian Haydin Why don’t you talk about that? 0:12:31.209 –> 0:12:50.129 Jim “JB” Brown Yeah, I mean, this is just, you know, something we found when we were doing research and such, but there’s really two loops. There’s not just a human in the loop, because we’ve talked a lot about that, but there’s also this other loop, which is really what we’re talking about today, and that’s that governance loop. And those two loops, they’re focused on two different questions. 0:12:50.209 –> 0:13:10.89 Jim “JB” Brown You know, the inner loop, that’s the one that belongs to whoever owns the agent. It runs all the time. Is it doing the right thing? Is it providing the right value? Has, you know, the quality stayed the same as cost being managed? Then there’s that outer loop, which becomes our governance cadence, and, you know, it belongs to the COE. 0:13:10.289 –> 0:13:31.289 Jim “JB” Brown which is kind of how we’re framing that. It runs periodically, maybe once a quarter. It’s not asking whether the agent is right. It’s asking whether the agent should exist at all, every time. Does it still earn a place? Is the owner still want it? Should it move tiers? Is it become less critical? And we can. 0:13:31.569 –> 0:13:50.649 Jim “JB” Brown kind of change the way we manage it? Or, you know, should it be merged with something else because something else came up and now it makes sense as we’ve matured to have just one system or one agent. You know, the owner asks what’s right, the COE asks whether it should exist. And, you know. 0:13:50.809 –> 0:14:9.529 Jim “JB” Brown the ownership you could think of the should exist, that’s portfolio governance because you have a whole portfolio of agents working on your behalf. So I want you to hold on to that because we’re going to come back to that concept towards the end. And I think it’s a big difference in having governance that works versus just having 0:14:9.609 –> 0:14:12.249 Jim “JB” Brown governance as a checkbox. 0:14:13.289 –> 0:14:32.729 Brian Haydin Yeah, I would agree with that. And like I said, I love that concept. And I’m glad that you brought it up. But let’s walk this journey together a little bit. And I want to start with the explorer. So remember, the agent at this stage is usually pretty narrow. It’s got a small audience. Maybe it’s just the person who built it. 0:14:33.929 –> 0:14:50.249 Brian Haydin And these agents typically have limited access. It does things like assist things, but it doesn’t necessarily execute. Want to make sure that we like have that as a as a gate, right? Like that, that’s a distinguishment or a feature that we want to want to exclude at this point. 0:14:51.689 –> 0:15:10.889 Brian Haydin And I would think one of the more critical things to think about is that it’s pretty easy to replace. So if it vanished tomorrow, the one person that was using it is probably going to be inconvenienced, but like it’s not going to like stop them from doing their job. And it’s definitely not going to impact like an entire department or an operation. 0:15:11.369 –> 0:15:32.249 Brian Haydin So the instinct at this stage is really to require a business case, an intake form, and a review. And I want to just say, don’t do that. Microsoft, in our guidance, you know, as well, is that the safest path here also has to be like an easy path. And every hour that you 0:15:32.569 –> 0:15:53.449 Brian Haydin make somebody wait or every day that you make them wait, trying to make a decision and whether that’s an, you know, appropriate or not, is going to turn into like them doing loops around the whole process. So what I want to say is like, instead of building like a barrier, build the lane, give them environments where the experimentation that they want to do 0:15:53.769 –> 0:16:15.369 Brian Haydin the productivity that they want to enable is allowed by default. That means that you’ll set data policies that decide which of the connectors that you want them to have access to. So that the maker themselves isn’t going to be forced to make a decision that really at the end of the day is an organizational decision, it’s not theirs to make. 0:16:16.329 –> 0:16:33.529 Brian Haydin So set up authentication defaults, channel restrictions, you know, things along those natures so that people don’t accidentally do things that they aren’t aware that their actions are going to transpire to. Publishing your Coca-Cola ingredient list, you know, on the internet. 0:16:35.689 –> 0:16:57.609 Brian Haydin and have templates, right? Like give them some of the templates they can use to build some of these things that will help them do it the safest, you know, in the fastest way. And so again, governance at this stage, I think starts with the safe path, you know, not a big committee path. And the one thing that I would want you not to skip is some of the tier one controls. 0:16:58.129 –> 0:17:19.329 Brian Haydin So the very lightest tier that you can have, what you define, is still requiring something, and that’s a named accountable owner for the agent. And we’ll get some of the basic usage and errors and stuff like that once we get this agent registered. But I, you know, I want you to like focus on that. The most important thing at this stage is just that you have an owner. 0:17:19.849 –> 0:17:31.769 Brian Haydin So that’s the day one ask, not a business case, not a review board, just the name. And then we’re going to start to build on that for the rest of this, for the rest of these as well. So. 0:17:33.689 –> 0:17:52.409 Brian Haydin Four dials and notice the consequence, you know, that that what comes out of them when you turn, you know, turn each one of these. You’ve got reach, who depends on this? You know, we’ve got access, who can see this? We’ve got the authority, you know, 0:17:53.369 –> 0:18:14.9 Brian Haydin you know, who, what can it do? And so these are the four, the four dials that we’re going to look at when we start talking about, when we talk about the extend aspect of this. Jim, you say that consequence is what comes out when you turn these dials. So you want to walk this through with us? 0:18:12.729 –> 0:18:13.49 Jim “JB” Brown Yeah. 0:18:14.729 –> 0:18:34.409 Jim “JB” Brown Yeah, so let’s focus on agents. You know, we started out talking about tier one, but they don’t usually stay there. Somebody shows it to their team, lights a fire, somebody looks over a shoulder and says, I could use that. You know, but they also grow in their complexity. So somebody might add a knowledge source or a connector to a critical system. 0:18:34.969 –> 0:18:52.569 Jim “JB” Brown And while I, your agent, isn’t just informing people, it’s actually doing things or it’s capable of doing things. And then, you know, you know, without usually anyone making a formal decision, it’s now our agent. It’s not my agent anymore. And that’s one of those thresholds that you should really care about. 0:18:53.529 –> 0:19:13.369 Jim “JB” Brown And so the question I want you to be able to answer for every agent in your environment is, has my agent become our agent? A very common growth pattern. It usually happens because you’re successfully adopting AI. So it’s not a bad thing. It’s just that’s what happens. So let’s make it a concrete talking about this dial. 0:19:14.89 –> 0:19:32.569 Jim “JB” Brown analogy or symbolism here. You know, notice something about these dials. As you mentioned, Brian, consequence is what comes out. You turn the dials and it’s the consequence of the agent and what it can do. Microsoft has this line down there, you can see in blue to red. And on one side of the line, 0:19:32.729 –> 0:19:52.489 Jim “JB” Brown there’s these agents that just do the assistance. On the other, you know, side of the line is agents that do things. And so as you cross across that, you know, something changes. You know, on the left side, you’ve got the human in the loop. On the right side, you need to have oversight and structure and make sure that it’s operating as it should be. 0:19:53.849 –> 0:20:11.529 Jim “JB” Brown So when you look at the dials, as you turn these things, your reach, is it a team, is it a person, the type of things that you access, the ability it has to make its own decisions, and how much of it does it on itself, kind of turns into 0:20:11.609 –> 0:20:12.249 Jim “JB” Brown Um… 0:20:14.329 –> 0:20:32.249 Jim “JB” Brown you know, your structure that you need to have to manage these things. Microsoft published, you know, some interesting way to think about it. They say, don’t categorize an agent based on, you know, how important it feels or whatever, but really focus on what it does. 0:20:32.569 –> 0:20:51.689 Jim “JB” Brown And so if you focus on what it does, you can govern the consequence from what the agent can do, not the creating of it. You know, focus your governance on what agents do and how they do it. Which kind of brings us to the next step in our journey here, which is the next slide, Brian. 0:20:55.209 –> 0:20:59.49 Brian Haydin All right, so graduate, right? That’s… 0:21:1.289 –> 0:21:5.689 Brian Haydin That’s A threshold that we have. Let’s see, Jim. 0:21:9.209 –> 0:21:27.609 Brian Haydin So on this side, safe experimentation when we get to the graduation, right? That’s the left side. And then on the other side, we have more managed, you know, enterprise capability. So I want to talk a little bit about what changes technically when an agent crosses over that line. 0:21:28.369 –> 0:21:48.329 Brian Haydin It starts to get a risk tier is, you know, is what the big shift is for us. And Microsoft has three kind of default tiers in their approach. Tier one is the individual productivity, the named owner, the basic monitoring, some sort of a release checklist, you know, self-service deployment. 0:21:49.129 –> 0:22:7.769 Brian Haydin And it typically is going to get published inside of, you know, the guardrails. That’s what we were talking about, you know, up until this point. Now, tier 2 is a little bit different. We get to expert knowledge and we get internal service agents. So it’s everything in tier one, but we also get the domain expert who validates, you know, and then, you know, answers. 0:22:7.889 –> 0:22:32.89 Brian Haydin you know, things that are correct. It does knowledge quality monitoring so that it can do things like catch stale content, you know, before the users, you know, get, you know, get those responses. It typically is going to have something like a formal release gate. Accuracy is going to be measured over time, you know, and then we get to tier 3, so these are business critical and customer facing type of agents. 0:22:32.569 –> 0:22:54.809 Brian Haydin Now you’re adding a formal process owner who’s accountable for the business process itself, not just for the agent, but for the entire business process. And then we have to move and graduate into a production level SLA monitoring. A security review happens and a responsible AI assessment needs to take place. And that happens before you actually do the release, not after you get it into production. 0:22:55.369 –> 0:23:14.289 Brian Haydin We’re going to build out decision rights frameworks, you know, things that state what the agent may decide on its own and when it has to escalate, you know, or also having an incident response plan so that when it does start to behave badly, and it will, they’re not going to be perfect. 0:23:15.49 –> 0:23:34.449 Brian Haydin that you can, you know, you have that response like built up right out of the gate. And then a quarterly maturity review. So we talked about like this idea of retiring agents, but you know, you need to be able to monitor those, you know, on a regular basis. So I would also argue that it also picks up a registry and 0:23:34.569 –> 0:23:54.969 Brian Haydin entry. It’s got an identity. It has documented knowledge sources, documented actions, and defined escalation rules. So I do want to clear one thing, clarify one thing. And graduation’s not like this one-time ceremony that you perform. It’s, you know, to start an age and at the 0:23:55.49 –> 0:24:14.329 Brian Haydin tier that fits its behavior now, and then move it up in scope or as the autonomy itself actually starts to grow. So it’s more like a trigger rather than like a step, you know? And you can fire that trigger more than once as capabilities happen. And I would also say like, 0:24:14.729 –> 0:24:34.89 Brian Haydin There are times when we build agents that we start to pull back a little bit, and that trigger can push it downward as well, where the autonomy actually starts to decrease, or maybe a business process change, but the agent still got some value. So think a little bit about both of those kind of use cases as well. It’s more of a dial. 0:24:34.169 –> 0:24:43.769 Brian Haydin not like a door that you go in and out. The technical half is a little bit more, you know, is, you know, probably a little bit more on the easier side. JB, what do you say? 0:24:44.409 –> 0:25:3.569 Jim “JB” Brown Yeah, well, it’s a lot more familiar to us. I mean, we do compliance, we do releases, we do a lot of these things already. But organizationally, things have to change. And this might be harder because it requires a specific human being to be accountable. And that might be a new role for a lot of people. 0:25:3.649 –> 0:25:22.889 Jim “JB” Brown we’re putting these, you know, AI automations into workflows and things. And so want to talk a little bit about, you know, reinforce what those roles are. You know, we talked about a business owner and a business owner becoming explicit, actually named, you know, right on the agent. That’s not the person who builds it. 0:25:23.209 –> 0:25:41.929 Jim “JB” Brown But it’s the person who owns why it exists and they’re responsible for it. And they probably have knowledge of what it’s supposed to do and whether it’s actually doing those things. So owner and the person building it is typically different. And you want to make sure because sometimes, you know, builders move on and nobody 0:25:43.209 –> 0:26:1.289 Jim “JB” Brown owns the agent because the business owner was there. So 100% own, somebody has to own that agent. The solution owner being identified, that’s the easy part. We’re familiar with kind of how that works. They support it, they implement it, there’s a relationship there. They’re responsible for the technical quality of it. 0:26:2.449 –> 0:26:21.129 Jim “JB” Brown An outcome is defined. What is the thing supposed to be worth? What does it do? And how do you know that it’s doing it? And on graduation day, somebody should be able to answer that and document it because 18 months from now, people have moved on or things may have changed. And you want to make sure that 0:26:21.209 –> 0:26:45.609 Jim “JB” Brown you’re not paying for something you’re not even using anymore. But so you want to make sure that that outcome is measured in that inner loop while the agent is with us. Decision rights are documented. So what should the agent be able to do? Who approved that? What has to come to a person? When does a person in that inner loop have to intervene or approve things? 0:26:45.729 –> 0:27:8.649 Jim “JB” Brown what thresholds. And then finally, a review cadence is established. So that’s part of the outer loop driving that. But you should have a schedule for when it gets reviewed, and you should definitely review it, just like a digital employee having its performance review, if you will. Brian, you said that the technical, on the technical side, it’s a dial, not a door. 0:27:8.969 –> 0:27:28.769 Jim “JB” Brown Well, I’ll carry on that and say that the organizational piece, it’s a signature. Somebody has to sign for the outcome of this process. And, you know, I think it’s a very important point to take away from the session that an agent should never graduate into production without its accountability. 0:27:29.49 –> 0:27:31.449 Jim “JB” Brown graduating with it that needs to be defined. 0:27:35.129 –> 0:27:53.289 Jim “JB” Brown So who runs all of this? This is the point where somebody says the COE and half the room quietly nods and people think, yep, somebody should be doing this. Let’s start with what it isn’t. And I sort of like this kind of thinking because it helps us guide us. And Brian mentioned these before. 0:27:53.689 –> 0:28:13.249 Jim “JB” Brown But, you know, the Microsoft describes this in their framework document as, you know, there’s three failure modes that are common. There’s the ghost. That’s where you set up the COE, hey, we’re going to do this, but it’s not really doing much. You know, it’s an org chart, it’s a Teams channel, but it doesn’t have authority and it’s not. 0:28:13.609 –> 0:28:32.329 Jim “JB” Brown it’s not really doing much. There’s the gatekeeper. That’s where everything is stopped until further notice and it doesn’t enable anything and teams just go right by it. And then the perfectionist won’t let anything through. You don’t have any agents in production, job well done. So. 0:28:32.649 –> 0:28:52.9 Jim “JB” Brown This is a bit of an exaggeration, but I’m sure you’ve all felt this before with governance activities. And it shouldn’t be this way. So what should it be? Well, every organization already has a pattern that you follow, but consider this. The COE owns management of, you know, owns that outer loop management of the system. 0:28:52.409 –> 0:29:13.929 Jim “JB” Brown The business owns the agents. That’s the inner loop. Think about how finance works in your organization. A good analogy. Finance doesn’t approve every purchase, although some organizations maybe it does, but finance builds the system inside which purchase can happen. Managers have signing authority, the routing, the reporting, how it gets approved, making sure there’s multiple people viewing it and so forth. 0:29:14.209 –> 0:29:17.929 Jim “JB” Brown But nobody thinks that Finance owns the laptop when they’re done. 0:29:18.809 –> 0:29:23.449 Jim “JB” Brown So the COE owns that outer loop, but the business owner owns the inner loop. 0:29:25.129 –> 0:29:43.729 Jim “JB” Brown You know, the COE can own the guardrails, the happy path, the release gates, the reporting, and kind of put some structure to help the people in the inner loop do their ownership and just keep a heartbeat going. The business owns the purpose, the outcome. 0:29:43.849 –> 0:30:5.769 Jim “JB” Brown and the value. And so, and one other thing with compliance, most companies are not going to create a separate compliance or COE organization. This should be integrated into your compliance, or if you have some type of governance activities already happening, this should be integrated into what you already do. There’s a lot of 0:30:5.849 –> 0:30:11.529 Jim “JB” Brown A lot of overlap and I guess, you know, structure that you can take advantage of there. 0:30:12.889 –> 0:30:19.289 Jim “JB” Brown So the COE exists to answer 4 questions that should be asked about every agent that’s created. Who owns it? 0:30:20.449 –> 0:30:21.609 Jim “JB” Brown What’s the life cycle? 0:30:22.529 –> 0:30:29.49 Jim “JB” Brown What are the decision rights and how do we monitor it? That should be, in a nutshell, the focus of the COE. 0:30:33.449 –> 0:30:53.129 Jim “JB” Brown All right, well, let’s summarize the roles we’ve talked about, just to just to make this, you know, actionable. You know, there’s kind of four roles. There’s the business owner owns the agent, what it’s supposed to produce, supposed to review it, and this person is probably not who built it. They’re the in they’re the person in the loop. 0:30:53.489 –> 0:31:12.449 Jim “JB” Brown The solution owner implements it, configures it, maintains it, probably on speed dial, so if it breaks at 4 P.m. on a Friday, you’re ready to go. The COE and the platform that owns the management, the outer loop, the system, the standards, the release and kind of process and drives that. 0:31:13.209 –> 0:31:31.529 Jim “JB” Brown that cadence. And then finally, security and compliance, of course, owns identity, access, sensitive data rules, and threat response. Those are all things that organizations have in place and AI maps into those things. The tools are there. 0:31:32.409 –> 0:31:51.609 Jim “JB” Brown But that guides a lot of the risk management associated with the process here. So some good news, I think it’s actually kind of, it was new to me, but Microsoft is building that accountability right into the system, into the identity layer. So there’s a product set called Entra. 0:31:51.929 –> 0:32:11.129 Jim “JB” Brown Agent ID, which is identity for agents, just like we have enter ID for people. And within the records, as you create agents, there’s a field right there for the name of the owner that should not be blank, right? Right there. And they even went so far as to build in some documentation in that. 0:32:11.209 –> 0:32:30.809 Jim “JB” Brown process for succession planning. What happens when the owner person walks out the door? It’s a sort of a pre-plan or pre-succession planning to make sure that ownership is not forgotten, ownership transfers to where it needs to go. And it’s a field in every agent. So that’s kind of cool. I thought that was pretty neat. 0:32:31.409 –> 0:32:32.89 Jim “JB” Brown Next slide. 0:32:35.449 –> 0:32:36.329 Brian Haydin Oop, too many. 0:32:38.489 –> 0:33:0.9 Brian Haydin Yeah, all right. So we’ve been talking a lot about the organization and the process, and I’ve deliberately held out on this deck like the product conversation until now. And you started to like, you know, talk a little bit about the features, you know, inside of Agent 365. And, you know, the first half of this was about governance, and we’re now starting to get into like 0:33:0.49 –> 0:33:22.89 Brian Haydin the control plane. And the control plane that you know you need to have deployed before you know what you’re trying to control is if you do it that way, you just have a dashboard. And that’s not what people want right now. What they want is a little bit of control. So the Agent 365 framework, that is the enterprise control plane for agents. It went 0:33:23.289 –> 0:33:45.369 Brian Haydin It went GA, you know, a little bit earlier this year in May. It came out pretty much right around the build time frame. And, you know, it’s $15 per user per month as a standalone. You can get it bundled as well into your minimum E7 license. But, you know, that’s the licensing aspect of it. So it’s a SKU that you have to purchase. 0:33:45.929 –> 0:34:4.489 Brian Haydin But what does it actually give you? It organizes in three different ways. It allows you to observe, govern, and secure. And observe is this registry that they’ve created. So every agent across whatever platform, it’s not just the ones built in Copilot Studio, it’s, you know, other platforms are supported as well. 0:34:4.969 –> 0:34:24.649 Brian Haydin who owns it, what it’s connected to, how much it’s running, and what it’s costing you. So the overview dashboard in the 365 admin center, that gives you like the registered agent counts of 493,688. Anybody else feeling like they’re a little bit like behind right now? 0:34:26.169 –> 0:34:43.529 Brian Haydin But it tells you like active users, growth trends, connected platforms, total runtime hours, which is important. And then slowly we’re starting to see it like bring up like the risk signals as well. Governance, you know, governing if there’s a policy, what’s allowed, what’s blocked. 0:34:44.9 –> 0:35:2.329 Brian Haydin what needs review, you know, before you can ship an agent. And then securing is the connection to the systems that you already run. So that’s that entre for identity, mapping it to Purview for data protection, and having the ability really to audit like the agent actions. 0:35:3.129 –> 0:35:22.809 Brian Haydin using things like Defender for threat detection. So there’s a few things that have shipped lately too, because this is a really emerging field. People are starting to get used to it. Microsoft is, you know, is taking the feedback. So we’re starting to get signals from the third party security tooling, kind of 0:35:22.849 –> 0:35:45.449 Brian Haydin bubble up into Microsoft’s own as well. They’ve added things like organizational-wide adoption insights, oversight of usage-based AI spend, and lately they had a multi-tenant management pain as well. And I think there’s some things to watch a little bit carefully. So registry sync that pulls AWS 0:35:45.929 –> 0:36:7.369 Brian Haydin and Gemini Enterprise agents into the same inventory. We’re going to see evolution in there. That’s public preview, not quite GA yet. So if you’ve got a genuinely multi-cloud agent strategy or agent estate, that’s a feature that you’re going to want to pay a little bit of attention to. And because it’s preview, I wouldn’t build a 2026 plan on it. 0:36:7.409 –> 0:36:28.89 Brian Haydin but I would start to think about it for 2027 and how that is going to mature. And I’ve, you know, put this slide together after the graduation for a couple of different reasons. Agent 365, it didn’t wait for graduation. It’s inventorying and governing right now. It’s doing it from day one. 0:36:28.969 –> 0:36:48.809 Brian Haydin The discovery is pretty much how you find the agents that graduated without telling you. So it’s having that discovery aspect and allowing you to create that registry and inventory. And that’s where, like, I think this is going to be, you know, moving from like a nice to have to just something that you’re going to start to use on a regular basis. 0:36:49.449 –> 0:37:11.129 Brian Haydin So let me put that whole stack into one picture, you know, and give you a little bit more holistic view, because the product landscape is kind of confusing right now. And it’s not your fault, mostly because it changes like every minute, every blog announcement, every, you know, every hour. At the bottom layer, you’ve got your creation layer. Right now, that’s Copilot Studio. That’s 0:37:12.169 –> 0:37:31.529 Brian Haydin You can use Microsoft Foundry in that bottom layer as well. But that’s where the makers are building. Everything that we said about the templates and the golden paths and, you know, how you help them that’s going to live at this like governance layer. And second, you’ve got the platform governance itself, though. 0:37:32.249 –> 0:37:50.889 Brian Haydin So, this is, you know, right baked into the Power Platform Admin Center. It’s a familiar console that everybody’s, you know, pretty much been using in this environment. You’ve got environments, environment groups, you’ve got policies and data policies that apply to each one of those environments and can be attached to the connectors. 0:37:51.169 –> 0:38:9.849 Brian Haydin so you can control what the makers have access to, what they can see, and then, and also inventory things across like your entire tenant, you know, including taking a look at who has the ownerless resources. And I’d say like two things about this layer. First, if 0:38:10.9 –> 0:38:31.689 Brian Haydin Anyone on your team is planning to install the COE starter kit, I probably would advise not to do that. They’ve deprecated that. I think it was maybe two months ago that I saw the first kind of soft announcements that they’re not going to be actively maintaining it. And so if you’ve been, people have recommended doing that or… 0:38:31.889 –> 0:38:53.249 Brian Haydin Like I would just say, I would skip those steps. A lot of those capabilities have that that brought to the table actually started to ship natively into the admin console. So you’re not going to need it anymore. And then I would also like give you just a little bit of a naming warning. So Microsoft now ships something in the Power Platform called the Agent Center of Enablement. 0:38:53.609 –> 0:39:13.369 Brian Haydin And that’s abbreviated the Agent COE. And it’s really 3 governing agents. Highlights like daily snapshot of the tenant activity, insights continuously scan for governance issues, ownerless resources, things like that. 0:39:14.409 –> 0:39:33.929 Brian Haydin activity in the default environment, and it will help you like look at those things that are happening and then prioritize them by the impact. It has an action plan that turns a lot of those findings into recommendations or remediation plans, things that you can actually review and approve before it starts to hit. 0:39:34.729 –> 0:39:53.929 Brian Haydin do any of those actions on your behalf. And it’s enabled pretty much out-of-the-box if you have at least one managed environment in the tenant. So it’s the key word is the managed environment. So a lot of these are useful features. It’s not the center of excellence, it’s the center of enablement. And 0:39:55.209 –> 0:40:10.169 Brian Haydin Gotta give Microsoft credit where credit’s due to give us all confusing names, right? But then there’s this third layer that I want to talk about. That’s the enterprise control, and that’s the Agent 365 things that we’re talking about. That’s got… 0:40:12.329 –> 0:40:32.649 Brian Haydin you know, really that control plane, that visibility on observing and governing the actual identities. And then the last layer up on top, we’ve got trust. And that’s where the entre for identity access is going to live, where per view policies for DLP, auditing of agents is going to sit at that top layer. 0:40:33.449 –> 0:40:57.609 Brian Haydin And then, you know, I want you to like think back a little bit and notice how the four dials that we talked about map pretty much straight into the stack. So reach is your channels. We had access, that’s per view, and that’s your data policies. We had the authority dial, that’s for the connectors and the actions. And then we had the autonomy, you know, dial for triggers and scheduling. 0:40:58.89 –> 0:41:11.369 Brian Haydin So, we kind of talked about them before as more metaphors, but in reality, they map the things that you can actually that you can actually work with, the dials that they really are dials. 0:41:16.89 –> 0:41:16.409 Brian Haydin All right. 0:41:18.9 –> 0:41:18.569 Brian Haydin Jim. 0:41:21.289 –> 0:41:21.689 Jim “JB” Brown Kate. 0:41:25.769 –> 0:41:44.249 Jim “JB” Brown So, okay, so we’ve talked about a lot here and we’re working our way out now, but here’s the trap. Let’s say you do a great job at the COE. You’ve got everything checked, every agent’s registered, every agent has an owner, policies are applied, you’ve got all the layers for security, DLP, security review. 0:41:44.569 –> 0:42:5.249 Jim “JB” Brown dashboards green across the board, but the agent is useless. You know, so this is an emphasis on that people part about making sure that the inner loop and the outer loop, that people are involved. You know, if the agent is answering out of a knowledge base that hasn’t been updated since March, 0:42:6.249 –> 0:42:24.809 Jim “JB” Brown or 11 people used it last quarter, and it’s the project team that’s developing it. You know, it costs more in consumption than you were expecting. Control isn’t the finish line. You can have Agent 365 telling you all the details, and if people aren’t using that information or 0:42:24.849 –> 0:42:48.489 Jim “JB” Brown aren’t actively being a human in the loop, you’re not going to get the squeeze, the juice out of the squeeze that you’re hoping for. So don’t forget the human, you know, from a COE perspective outside the loop. Think of this as, you know, dashboard can’t make a decision. Someone does have to make a decision. Dashboard can give you the criterion. 0:42:49.369 –> 0:43:8.209 Jim “JB” Brown But the tool isn’t going to do anything without you. Microsoft, I think they’ve hired poets now to write some of their documentation, but it says the tools surface the signals, but the rhythm is the discipline that acts on them. I’m not, that might be the start of a haiku. But their point is, is that people need to stay in the loop. 0:43:4.169 –> 0:43:4.249 Brian Haydin If. 0:43:8.809 –> 0:43:18.889 Jim “JB” Brown And that’s a big gap if we focus only on the tools and doing everything right, but people need to adopt this and to be trained in it. Next slide. 0:43:21.609 –> 0:43:40.9 Jim “JB” Brown Okay, so just this is a bit of a summary here, but now let’s talk about the stage that everybody should now plan for. Every agent in your portfolio should have to answer one question on a schedule. Does this still serve to exist outer loop? Here’s the signals that you can look at. 0:43:40.169 –> 0:43:55.689 Jim “JB” Brown Right across the top here, you’ve got usage. Is anybody actually using it? Or is it just the project team? Quality? Is it accurate? Human in the loop, in the inner loop, should be able to show that. The value of it, the cost of it. 0:43:56.649 –> 0:44:15.729 Jim “JB” Brown And, you know, dependency, because you are putting these things into the enterprise and the businesses depending on it. So what happens if it breaks and goes away? Then, you know, from that, think about four outcomes. Hey, we’re going to keep it, it’s working, leave it alone. That’s a decision and it should be an explicit decision, but 0:44:16.569 –> 0:44:35.569 Jim “JB” Brown That is one of the outcomes. If there’s something going on and it needs improvement, but you still want it, make sure you fund the fix. Make sure that you’re actually making corrections in it. And that is a part of the life cycle of any agent. 0:44:35.889 –> 0:44:41.929 Jim “JB” Brown is to measure the value and see if it’s drifted. There’s a bunch of different drifts that an agent might do, but fund the fix. 0:44:43.49 –> 0:45:5.289 Jim “JB” Brown Consolidate happens very often. People are being creative in different places and all of a sudden you have three agents that do the same thing. Notice that. And as part of the review, look for that because there’s a good chance you could consolidate that into one better agent if it’s really doing the same thing. And then you have all the controls in place so it can operate. 0:45:5.329 –> 0:45:25.289 Jim “JB” Brown the way it should be. And then finally, retire it. It’s done. Shut it off. You know, we’re sort of in this earlier phase of AI adoption where there’s a lot of pilots going on, a lot of things happening. Wouldn’t be surprised at all if there’s a bunch that should, you know, stop the noise, be retired now. So, you know, a cultural challenge here. 0:45:26.809 –> 0:45:45.129 Jim “JB” Brown And we started out talking about this, but the reason we’re talking about this is because AI adoption is being successful. There’s lots of agents now. People are really excited about it. So retiring an agent isn’t an admission of failure. It’s just discipline and 0:45:45.769 –> 0:46:5.249 Jim “JB” Brown and focusing people on the right things and keeping a clean house so that you’re doing what you thought you were doing and people aren’t wasting resources. You know, I thought about this is if you keep keeping these things around, it’s not a it becomes an attic where you stick all your all your stuff. Don’t 0:46:5.329 –> 0:46:7.769 Jim “JB” Brown Don’t let that happen. Retire them. 0:46:9.289 –> 0:46:29.129 Jim “JB” Brown So, you know, thought about the Microsoft COE document. Microsoft does a quarterly loop. That might work for you, but maybe it’s more faster, maybe it’s less, but what you should be doing is you should be deciding what to scale, what to sustain, what to retire. 0:46:30.329 –> 0:46:40.329 Jim “JB” Brown so that the investment that you’re putting in it is doing value. So put it on a calendar, embed it into your existing compliance, risk management, whatever that is. 0:46:41.289 –> 0:46:59.129 Brian Haydin Exactly. So getting to the end here, I want to pull it all together. And you know, what you actually, I think what you want to do is run day-to-day on a loop, right? So discover what exists, including the things that showed up without 0:47:0.329 –> 0:47:20.849 Brian Haydin without people coming to the governing board. Classify which of the four dials that we talked about are moving and what tier does that make that agent belong to? And graduate, did it cross a threshold with accountability attached with the named owner? Observe the signals from the control plane. 0:47:21.929 –> 0:47:39.969 Brian Haydin and evaluate if it’s the right and if it’s still valuable. And then finally, improve or retire and, you know, come back around to the top. That’s the loop. That’s the machinery right there. That’s the signals, the tiers, the controls, registry, identity, policy. 0:47:40.89 –> 0:48:1.689 Brian Haydin All of it’s buildable right now, and most of it you either already have or, you know, or have licenses, or you can get a license for something like this. But I would say the caution is like the licenses and that machinery, that’s not the answer. This is where those two loops kind of come back to play, right JB? 0:48:2.89 –> 0:48:13.609 Jim “JB” Brown Yep, keep the humans in the loop, and there’s more than one loop. You need that governance. That’s what really helps people be successful and focus on their creativity with, you know, being productive. 0:48:16.729 –> 0:48:26.569 Jim “JB” Brown So yeah, this is the diagram to take home today. That’s Agent Ops, you know, the recurring discipline, running those loops underneath it. 0:48:28.409 –> 0:48:48.329 Brian Haydin Yeah, I would agree. All right, so what can you do, right? Five things that you can do starting Monday, you know, or tomorrow if you want to work on Friday, if you want to do this on Friday. But define your thresholds, write it down on a page. What makes the agent consequential enough in your organization 0:48:48.729 –> 0:49:7.769 Brian Haydin that elevates it to be managed as a business capability, right? Not a personal assistant, but a business capability. And then use those four dials that we talked about, reach, access, authority, you know, and autonomy. Number 2, create that safe path, you know, define the environments, the data policies, the templates. 0:49:8.289 –> 0:49:27.769 Brian Haydin Make sure that you enable people to build things the right way, but also the fast way. And then third, let’s require a name. Every agent from day one has somebody that you can hold accountable. That’s the owner of the agent. I think that that’s probably the easiest thing that you could do as an organization. 0:49:28.489 –> 0:49:49.49 Brian Haydin And it’s probably more than 90% of organizations are doing today. Then get portfolio visibility. Whatever tooling you’ve got, whether you’re using Agent 365, some other admin center, maybe you build something, a spreadsheet that you’re maintaining by hand, do something because that’s better than not having nothing at all. 0:49:49.209 –> 0:50:9.369 Brian Haydin And then put a portfolio review on the calendar. We recommend quarterly, monthly is a little bit too much. People don’t really have time for that. You know, yearly is definitely not enough. This is a fast changing, you know, ecosystem. So quarterly, you know, and look at these. Are we going to keep them? Are we going to improve them? 0:49:57.689 –> 0:49:57.849 Jim “JB” Brown Yeah. 0:50:9.689 –> 0:50:19.449 Brian Haydin Have I got some redundancy where I need to consolidate them, or is it just time to retire it? So, Jim, any last thoughts? 0:50:20.9 –> 0:50:39.849 Jim “JB” Brown Yeah, you know, just before we open it up for questions, there’s a QR code on this slide. We love to talk about this and other topics that definitely were available. So, you know, we could call it a short COE education workshop, but it’s really just a conversation. You bring your actual pain points. We love to hear what’s going on in the market, and we certainly 0:50:39.929 –> 0:50:58.369 Jim “JB” Brown bring to the table what we hear from our customers. This isn’t a sales thing, this is a conversation, and Brian and I would be glad to have the conversation. And, you know, hopefully you’d leave that with a clear sense of what your next steps are going to be. So, you know, I guess the 0:50:58.729 –> 0:51:19.289 Jim “JB” Brown concept here is the goal isn’t really fewer agents in control. The goal is to create an agent to state of digital employees that you understand well enough to keep it growing confidently and in confidence in your agent to state is the ending message here. So with that, Amy, we’re available for taking some questions. 0:51:19.369 –> 0:51:21.129 Jim “JB” Brown if there are some in the chat. 0:51:21.769 –> 0:51:40.969 Brian Haydin Yeah, there’s just one question from Michelle so far. She asked if we’re going to share the slides. Absolutely, absolutely. So fill out the QR code. Amy’s got a link in the chat as well. Click that link, fill it out. And we’ll follow up with the slides. They usually get published on the website as well. 0:51:26.249 –> 0:51:26.569 Jim “JB” Brown Ohh. 0:51:42.9 –> 0:51:49.609 Brian Haydin And if there’s no other questions, I want to just say thanks very much for spending some time with us today, and I hope you found this valuable. 0:51:53.369 –> 0:51:54.329 Jim “JB” Brown Great. Thank you.