View Recording:Frontier Firm Part 5: Governance & Security for AI – Zero Trust Approach to AI

Brian Haydin 0:13 First, thank you so much for joining. Second, I was having a little bit of technical difficulties getting the presentation to show. And 3rd, really, really good timing on this, but somebody is adding their AI notes and I hit deny and I thought it was really cool. I might even drop a screenshot in here. because it’s really relevant to what we’re going to be talking about today. And sorry for killing your AI notes, apologize. But again, welcome to everybody. I want to say thanks for being here. This is the final session of our Frontier Firm series that we’ve been doing since the beginning of the year. If you’ve been with us across the first four sessions, you probably already know that we’ve been building a little bit towards this moment. And if this is your first one, don’t worry about it. You’re not really behind. You’re really just kind of walking in where this is going to get to like a practical application. A little quick housekeeping before I jump in. You know, we’re going to record this and we’ll get the replay back, you know, probably by tomorrow or early next week. But if you have any questions, drop them in the chat as we go along. I’d rather that I answer some of the specific questions that you might be dealing with and get it in here so I can, you know, talk about it throughout the talk. But I do have a little bit of time carved out for Q&A at the end if you want to, you know, ask any questions. So if you have something like, you know, hey, what about my plant? Or like, what about my regulated environment? Go ahead and get it in early. I’m going to try to weave it in as, you know, as we talk about, you know, some of the other topics. But in today’s session, we had the word security. you know, like front and center on it. And I want to be a little bit clear about this right out of the gate. This isn’t like a security scare talk. We’re not here to walk through a wall of like breach statistics or a giant list of like vendor add-ons or products. I’m really just here to talk about like agentic workflows and what, you know, how you actually operate inside of a real organization. It’s about the control systems that are going to let you scale agents without the chaos that typically goes with it. And we’ll be talking about why this operating model, not like policy PDFs, is really what separates organizations that capture AI productivity from organizations that merely buy a bunch of AI tools. So, let’s get into it. Let’s put today into context. This series has had a pretty clear arc, and we did this deliberately at the beginning. Part one was about the culture and the organizational readiness. Before we ever got into the tools, we started talking about the foundations, values, accountability, clean data, and the standard ways of working. Because if you skip all this culture work, AI transformations are going to fail. And that’s really, you know, what we see happening in the marketplace. Not failing, but like if you skip that. Part 2 was, you know, intelligence on tap. This was about how Copilot lands at the individual level. So more like a digital intern that gives every employee a productivity lift. For most people, this is where AI shows up first in your daily work. And then part three, we reframe the organization itself from an org chart to a work chart. Once humans and agents started working in the same team, you had to stop thinking about like reporting lines and start thinking about how the workflow, you know, how the work actually flows throughout the day. And then part 4 was becoming an agent boss and managing a work group that includes like digital labor. We talked about concepts like ownership and life cycle observability and the muscle that’s needed to lead agents the way that you would lead like normal people in your organization. So today. Capstone. First 4 sessions, we talked about how AI shows up in the business. Today, it’s about how to make sure that it shows up safely, responsibly, and in a way that leaders can actually manage this at scale. So governance isn’t really new in part 5. It’s just, you know, we planted it all the way back in one. What we’re doing is kind of bringing the whole picture together. So here’s the question. How do we scale AI without scaling risk? Most of the executives that I’ve talked to in the last six months had some version of this question, you know, right out of the gate. The board’s asking about AI. The CFO is asking about AI. Line of business leaders are asking when they can have what all their peers down the street are already using. And somebody usually reporting to the CIO, the CISO, or the chief risk officer has to figure out how to say yes without setting the building on fire. Now, here’s the answer. And this is going to be part of what we’re talking about the next 5 minutes. 45 minutes. The Frontier Firm cannot scale on trust alone. It needs more verifiable trust. That means that every user, every agent, every prompt, and every data source, every tool, every action, they all need the right identity, the right access, the right policy. monitoring that goes along with it in the right life cycle. I know that’s a lot, and we’re going to dig into it, but let’s start by looking at why right now. Why May of 2026 is a different moment than six months ago? This deck really looks different than it would have if I had written this six months ago, and that matters because it’s changing the conversations that we should be having internally. Just May 1st, the beginning of this month, Microsoft Agent 365 went generally available. Microsoft Entra Agent ID went GA. you know, the same day and purview protections for agent 365 also hit the documentation as GA. So these aren’t really separate, three different product launches. What it is is Microsoft saying in a single day that the agent governance story is no longer just a slide deck. Funny that I’m saying that as they put it into a slide deck. but it’s an actual product. And now, if we back up, you know, six months from there, let’s look at this a little bit more historically. In December, Purview Network Data Security started catching unmanaged AI apps the way it catches like unmanaged file shares. In February, Defender for Cloud added AI threat. threat protection in preview. In March, we had Microsoft publishing explicit 0 trust guidance for autonomous agentic risk. And in April, we had the Foundry control patrol plane and observability documentation that expanded things a little bit more significantly. Every one of those was building block leading up to May, really, if you look at this reflectively. The building really hasn’t stopped either, though. Look at what’s coming up in the future. May 19th, we’ve got the European Commission opening high-risk AI classification that, you know, if you operate in Europe or if you sell in Europe, that, you know, you probably want to put something like this on your radar. May 21st, we’ve got Agent 365 registry documentation that was rolled out. You know, that covers Bedrock, Vertex AI, Agent Force, Databricks, July 1st, just a few weeks from now, Agent 365 subscriptions. are going to become a requirement for some of the defender agent protection features. And so what does all this mean? It means that, like, if you walk away from today thinking that AI governance is something that you’re going to put on a roadmap for 2027, you’re already falling a little bit behind. The platform reality is really changing right now, and that means that these kinds of conversations need to change with it. So here’s the qualitative shift and that kind of matters. Co-pilots, if you think about what co-pilots are, they mostly answer. You ask a question, you get a response. The primary risk is just bad output. You know, you’ve heard of hallucinations, oversharing, maybe it drafted an embarrassing email. But agents are different. They remember things. They have tool calls. They access actual systems. Agents take action with delegated authority and sometimes with very, very little human involvement. And that’s where you give software the ability to act. And when you do that, the risk surface moves from what it says to what it does next. And so look at some of these flows that I put up here on the screen. A user gives a prompt, the model interprets it, the agent calls a tool, and maybe that tool is an API, maybe it’s an MCP server, maybe it’s a connector into your ERP system or your CRM. The tools that it’s touching their external systems. And that with that comes side effects. So an order gets placed, a record gets updated, the email actually gets sent out, maybe a payment gets released. Every one of those things is a place where something can actually go wrong. Now, you know, I promised this wasn’t going to be like a bunch of scare stuff. but prompt injection happens from a document that maybe an agent retrieved, tool misuse, you know, because the permissions were too broad. We have data leakage because the agent’s context includes things that maybe it shouldn’t. And something that, you know, people are running into a lot just in their daily use is memory poisoning, where. like past interactions can corrupt like the context window and do things to the future behavior. And when we get into like these multi-agent systems that we’re starting to develop now, that these failures begin to cascade. One bad call starts to propagate all the way down, you know, down the line through the other ones. And, you know, so what I would say is that AI does not create new governance problems. All it really does is expose the ones that you already had, but it gives them a turbocharger. So that bad SharePoint permission, you know, things you’ve been meaning to clean up for the last five years, those are all now AI surfaced permission problems. And the broken process that you tolerated because maybe it only ran like once a month or once a quarter, that broken process now runs, you know, all the time at agent speed. So same problems, faster, broader, and they’re definitely going to be harder for you to ignore. So that’s really what the threat model looks like today. So risk in the abstract is hard to act on though. So let me give you 4 shapes that this problem is starting to take. The 4 failure modes that I see in most of the conversations that I have. The first one, shadow AI. Employees, they’re already using these tools, frankly, and even if you didn’t authorize them, people are using ChatGPT, they’re using Claude, they’ve got the free image generators, whatever is the easiest thing for them to get. But they’re not being malicious people. They’re not, they’re not like. They’re not trying to break the rules. What they’re just craving to do is be productive. And they went around you because the sanction path that you created for them was too painful. Or, you know, even with a lot of organizations, especially in the Midwest, it’s just too slow. Second problem I see is oversharing. SharePoint sites, your shared drives. Most of those permissions have been set up over the course of the last decade, multiple decades, drifted for years. Nobody noticed it because nobody could find the files. And the thing that’s different right now is that Copilot, it can find all of the files. Everyone can find the files because everyone can use Copilot. And that’s not really a Copilot problem when you think about it. It’s a permissions problem that Copilot exposed. Speaking of permissions, over permissioning agents. Somebody builds an agent for a very, very specific task and gives it the credentials of a service account. That’s what’s been happening for the most part. It works great until it does something that it’s not supposed to do. And then finally, unobserved actions. So agents do things you don’t necessarily know what, and you don’t know why, and you don’t know whether it was the right thing because there was no audit trail, because there was no telemetry. And so that’s another area that. that people should start thinking about. And these are the four failure modes. Shadow AI, oversharing, over-permissioned, and unobservable actions. If you don’t remember anything else from today, try to remember those four things, because this is really a talk about how Zero Trust is going to help you overcome those. And that’s the thesis of this. of our webinar here today. Verify every agent, constrain every tool, protect every data path, observe every run, approve every high impact action, verify, constrain, protect, observe, and approve. Those 5 verbs, they map directly to the failure modes that I just talked about. And they map directly into Microsoft Zero Trust principles. Verify explicitly, use least privilege, and assume data breach. Verify each agent means that every actor in your environment, whether it’s a human, an agent, or even just a workload, it has an identity that you can verify. You’re not trusting vibes, you’re trusting an identity that has been authenticated, governed, and reviewed. Constrain every tool means that the agent only has access to what the task requires. Not a service account, not a service account’s whole life, not every API, not every system it might need someday, just the tools using the scopes and actions required to do the job that’s in front of it. Protect every data path means that the… data the agent can see what it’s grounded in or act on is governed by the labels, sensitivity, permissions, and DLP. Copilot only sees what other, what Copilot only sees what you as a user is allowed to see. And agents, when they’re built, should only be able to reach what they’re allowed to reach. So observe really means that every, observe every run means like every action leaves evidence. So we’re talking about traces, telemetry, evals, monitoring, all those like little components, not just like the question, does it run? Did it run? What did it do? What did it touch? you know, or, you know, those types of questions. And every, you know, every one of these is going to have some sort of a risk assessment assigned to it. So when we have low risk, all we’re really doing is like monitoring it. We have medium risk. We want to do something like review it. exceptions. High-risk activities are going to require explicit approval before the action happens. And so try to remember these, you know, three, 5 verbs that I laid out on the screen. And we’re going to dive deep into each one of these. So most government, most governance. conversations start with the model. A lot of people start asking questions like, what model are we using? Has that one been approved? Is it safe? And I would say that that’s the wrong starting point. The better question is, what is this agent going to do? And we’ve talked about these risk tiers before. We want to risk tier those actions. not the model. So Microsoft’s own maturity guidance is explicit on all of this. Treating the agents the same regardless of their criticality, that’s a mistake. And here’s the tiering that concurrency has been recommending to clients. Tier one is like the retrieval and the assist patterns, the Q&A kind of agents. The agent answers questions, summarize documents, it drafts emails or helps with the research. Side effects on something like this is just the text on the screen so we can score the risk lower. Minimum control, we’re looking at monitoring and auditing. Pretty much it. You want to know what is being asked and how it was answered. But you probably don’t need approval gates for someone that’s just going to be summarizing some content like a maintenance log. In tier 2, we’re talking about like task agent, like task agents. So an agent here is going to execute scoped actions with bounded tools. Important words there, scoped and bounded. Maybe it updates a ticket, maybe it creates a draft purchase order, and maybe something like that would have a human approval. Or it could even be doing something like, you know, adjusting a setting in one of your systems. But the side effect really still exists here, you know, because we use those scopes and bound it, they’re pretty contained. So what we’re recommending on the minimum control here is scope permissions, maybe an exception review, and then the full traces of what’s going on. You don’t have to approve every action, but you are reviewing and approving any of the exceptions. And then tier 3 is when we get into these autonomous workflows, and we’re seeing a lot more of these today. This is where agent triggers real external side effects. It’s actually placing the order or it’s doing the release on the credit hold. It’s signaling, you know, things in the production line or might even coordinate with other agents that are, you know, part of like a multi-step agent workflow. So what that means is that these are going to be activities that are very high risk. And the minimum control here is going to be a lot more. We need a named human owner, approval gates on the consequential steps, full observability, and most importantly, a kill switch that an operator can hit when something goes sideways. That’s the reason, you know, the reason that this matters is pretty simple. When you risk hear the action, you stop having one huge giant governance debate. And you can start having like 3 specific operational conversations. In tier one, what are we monitoring? Tier 2, who reviews the exceptions? Tier 3, who approves, and how fast can we stop it? What that means is that we have three, we can have three very specific conversations instead of 1 paralyzing one. Here’s the map that we can use. Over the next couple of slides, I’m going to walk through each one of these layers in the Microsoft control stack. But I want you to look at the shape of this, like shape of the model first. There are five control planes in the Microsoft stack around AI governance. Agent 365 is really, think of it as a registry in an admin facing control plane. It’s where you’re going to have the inventory, you’re going to have your agent’s inventory, where the ownership lives, and where the cross-platform sync is starting to happen. Microsoft Entra with agent ID, that’s the identity plane. Every agent is going to get a first-class identity, just like every user has. like every one of your user has an ID today. Microsoft Purview is the data and compliance plane. We’re talking labeling, DLP, audit, postures. Then we move to Microsoft Defender. That’s the real-time detection and hunting, you know, for things plane, real-time blocking. alerts and AI threat detection. Finally, the Microsoft Foundry control plane, relatively new. This is the build to production operations plane. And honestly, it’s primarily for like your Azure centric agents with cross project, you know, inventories, evaluations. There’s policy enforcement in there as well. So let me get like, I’m gonna like really move past some of the marketing slicks and you know, tell you what people, you know, what most of the product teams aren’t really going to tell you. Microsoft has been converging on this as an experience and It’s not just a single pane of glass today. What you’re going to hear from like other people talking about other organizations, other vendors is that there’s an easy button for the entire agent estate. And that’s just not the reality today. And pretending that it is is probably going to get you and the organization in trouble. So here’s what I want you to hold like the model in your head. The registry is going to tell you what exists. Identity is going to tell you who it is. Data controls will tell you what it might touch. Runtime controls should tell you what it’s allowed to do. And then the observability is going to tell you what actually happened. Those 5 planes, those five questions, those five operating disciplines, That’s what you need to start thinking about. So let’s take a look at each one of those right now. Layer one is that identity. So your organization already, you know, you already understand like identity for individuals, for people, every employee that you know, the organization has a has a badge, you know, whether it’s physical or some sort of digital, There’s credentials, they have role-based access. Somebody joins your organization, you reach out to IT help and you have them provision. And when they leave, you de-provision them. So when their role changes, you know, throughout like the years that they worked there, you adjust the permissions. This is the stuff you’ve been doing for decades. And now when we bring agents into the equation, we got to do this. They need to have the same treatment. Humans have badges, agents need badges too. So Microsoft Entra agent ID, which just went GA, that’s where it’s going to make this real. Every agent, whether it’s in Copilot Studio, whether it’s a Foundry agent or something that you had custom built internally or maybe with a partner. They all get first class identity, you know, within the Microsoft tenant. It’s not a service account hack. It’s not a shared credential. It’s a real identity that you can trace back and can be governed. Entre also introduces this concept of a blueprint. And a blueprint is going to what I talked about, those 3 conversations. It’s kind of a policy class. You define the governance rules once and then every agent can fit that class, that blueprint class and inherit them. So instead of chasing individual bots, you know, and trying to keep their permissions current, you can just associate them within a class and you can govern them that way. a new agent gets created, you just give it a blueprint and it inherits those policies. When the policy changes, all of the agents that have that can change as well. So conditional access also applies and it extends to agents as well. We’ve got OAuth 2, MCP, we’ve got agent to agent protocols, these are all supported as part of that conditional access. We’ve got lifecycle management, onboarding, role changes, retirement. And if your organization’s identity and governance is already in a pretty mature state, you’re going to find this to be pretty easy. It just extends what you’re already doing. But if you’re not, and honestly, a lot of organizations are a little slower on this. You should probably take this as an opportunity to force this function and make you and let it make you fix the identity gaps that you already have. So I want to take a couple of seconds just to look, you know, take a look at what this actually looks like in practice. You know, take a look at the shape of the inventory that I’ve got laid out here. First you see an agent, then you see the owner, and every agent has a named human that’s going to be accountable for it. And that could be an actual person or it could be a group. Then you see that the blueprint that it’s associated with, every agent’s been mapped to that policy class. You see conditional access that’s there. You know, every agent’s going to inherit the same policy, you know, that your team uses for human identity. That’s kind of the point. I’m not trying to teach you the UI. I actually have a screenshot of like our UI, which since this is all pretty brand new, you know, not all the details have been filled out. These things are, you know, we have to go back and apply them. But I want you to notice something really important. The shape of this is pretty familiar. This looks just like the IT consoles that your team is already operating in. So the mental model is going to be familiar. And that means that the discipline is going to be familiar and their skills are probably going to transfer. Your IT operations team should already be able to do this. And the important point is that they don’t have to learn a new skill set. They don’t have to suddenly become AI researchers. they can take this discipline that they already have and put it into practice. And that’s really what the point is, making it simple for the users. Then we have layer 2 is the data. And here’s the line that gets the most nods when I’m like talking to customers. Copilot does not break, Copilot does not break into the file cabinet. it opens the drawers that you already left unlocked. So Microsoft’s Copilot guidance is pretty explicit. Copilot surfaces, it surfaces the data that users already have permission to access. So it sounds kind of like a safety feature if you think about it, and it really is. But it also means that everybody that overshared A SharePoint site or over permissioned a document library or like. got lazy and just like shared it with everybody. Those everyone accesses now become touchable by every AI system that you put into your put into your ecosystem. And those agents are much better at finding information because it’s like their job than people are. So think about what I’m actually saying for you manufacturers out there. You might have things like engineering drawings, maybe there’s some supplier contracts, an Excel spreadsheet with pricing. Maybe you had some M&A diligence files that didn’t get cleaned up. All those are data points that can get found. And it wasn’t really supposed to be found, like you didn’t intend for it to be wildly accessible. But security by obscurity, you know, was maybe a discipline for a while. Copilot’s making that discoverable. And so there’s been a lot of maturity in Microsoft Purview, especially this year. And that product right now is shifting from the older DSPM for AI experience into like a broader, more. data security posture management model. That matters because it brings the AI data story back into the government’s discipline that you have a need for with the sensitive information. Purview gives you sensitivity labels. DLP for prompts and outputs are included as well. And that ability to exclude labeled… files from Copilot grounding, meaning that you like can say like these, I don’t want this to be included into the semantic model. I can have AI specific audit logs. I can posture workflows that tie remediation to specific data security guidances. But it’s also going to extend past. the Microsoft ecosystem. One of the things that I really liked was that I could use ChatGPT, you know, extended into that as well. But it also includes things like Google Cloud, Snowflake, Databricks, and it’s got partner integrations that go beyond that as well. So Sierra and BigID and OneTrust are all like. Partners that they can leverage this as well, so that’s why I think of Purview is a little bit more like a data nervous system, rather than, you know, rather than anything else. And if you are gonna roll out Copilot broadly, I would strongly encourage you to use Purview. You know, if you don’t, you’re probably just gonna be rolling out problems. rather than rolling out some of the people are going to find useful. So. Data governance, quick little visual here. I don’t want to take any screenshots in my system because it’s got a lot of like pretty interesting things. But here’s a quick visual. On the left side, I’ve got Purview, like a kind of a mock-up of what Purview’s DSPM view would look like. And look at the shape of it, what you would see here. I see my AI data state. like my AI is a state data posture basically in one single pane of glass. I can see the oversharing indicators, sensitive data, exposure scores, event like audit events for co-pilot and agent interactions. But if we shift our, you know, view over to the left side and the DLP intercept, Here’s, you know, here’s kind of a snapshot of what can happen. A user tried to include a sensitivity label in the engineering drawing in one of their co-pilot prompts. And so what happened is that I had the policy fire and the prompt was blocked. And rather than just like a big red scary X, you know, the user got like a coachable message. And then the audit log was used to capture what the attempt was. And so this is the kind of features and functionalities that we’re seeing implemented as of the beginning of the month. And so it’s not like a theoretical thing anymore. This is the governance and practice that we can use today. So… Let’s move on to the next layer. Let’s talk about runtimes. And this is where agents get a little bit weird. Up until now, we’ve been talking about who an agent is and what data it can see. But the real danger is what an agent can do next, the side effect. Microsoft Defender is that runtime plane. And it does three things that are going to matter. You’ve got real-time blocking. So if a tool call is about to do something that’s risky, Defender can stop it before the API call actually completes. You’ve got near real-time detection. It can see patterns, raise alerts. And that will hopefully give a human time to go ahead and intervene. You’ve got AI-specific threat protection, Defender for the Cloud, that covers jailbreak attempts, data leak patterns, credential theft. You get more of that OWASP style risk categorization. and those are starting to show up a little bit more. And like anybody that knows me knows that I like analogies. So here’s the way that you can rethink this, hopefully to land with people in the Midwest. Think of like a bird dog. So you’ve got a bird dog that points. you know, to your pheasant or whatever that you might be hunting. And that’s its job. It doesn’t actually pull the trigger. It’s there to do the point. And that’s the whole job of a good agent, you know, especially when we think about like the tier 3 agents, that autonomous workflow tier. It’s to the point to the thing that should happen, surface it. recommend it, prepare it. But the trigger pull, that consequential action, the irreversible step, we’re still leaving that as a human decision. Or at a minimum, it should be a checkpoint that a human has to sign off on at the policy level. So for some of you, This is the same discipline that you already have in practice today. You’ve got interlocks, you might have lockout, tagout. The systems, you know, that you’re using today are pretty analogous here as well. And Defender is how you, Defender as a product is how you actually make that real. And in July, we’re going to get like a little bit more, we’re going to get a little bit more agent protection experiences, and they’re going to be gated in the Agent 365 subscription. So as we get a little bit closer to that date, like reach out to us and love to have some of those licensing conversations. Make sure that you’re set up for success. How do you decide when a human pulls a trigger? So that’s, you know, think of this back from slide 8 that we had. Low risk action, monitor it. Maintenance log summary, meeting minutes, drafting the email, your audit trail. That’s enough for you. You don’t need to have a pro over somebody that’s just summarizing some some some email thread or some maintenance log. On the medium risk though, review a set, we have to review the exceptions. The agent processes a, maybe the agent’s going to go and process 100 purchase orders and you sample like. just a small percentage of that. And then you have an exception queue for those cases that fall outside of the policy. So you’re not, you’re setting up an agent for a system that the human isn’t bottlenecking every action. They’re just sort of like overseeing, you know, what the overall patterns are. And then you’ve got like a high risk. scenario, high risk scenarios. That’s where you want to have the approval required. You’ve got synchronous, named approvers. This would be scenarios where you want to release a credit hole, the triggering an outbound payment. You know, those are the kind of scenarios that you want to consider here. The important thing is that a human’s going to sign off before the side effect actually happens. You don’t want to get into that state. And whether it’s regulated or sensitive, you know, have an explicit policy, a full audit trail. That’s where the compliance team and your legal teams, to meet, you need to have them at the table, you know, all the way from the beginning at the design phase. And I’ve, you know, been using a little bit more of this Microsoft agent framework. That’s the old semantic kernel, you know, as like some of our, like, that’s the new platform that we use to develop the agents and from a like a pro code perspective. And what’s cool about it is that we’ve got all these like. built in features as like first class control planes that are built into it. It’s workflow pauses are baked into the platform and the platform SDK. And we can use Defender, you know, you know, natively within there too, to do explicit recommendations for human approval when we have like MCP actions that. you know, that happen, or maybe when there’s like an elevated prompt injection risk or something like that. So that’s a good way to use some of these tools, you know, with the pro code environment as well. And then layer 4 observability. And I’ve been doing a lot of talking about this. It’s just been a really hot topic at a lot of the conferences that I’ve been speaking at. So, you know, I think this is a layer that separates organizations that want to scale from organizations that sort of hit this like ceiling and, you know, and never really, you know, have a way to climb past it. And it’s a pretty simple principle. Dashboards for agents should look like, you know, like less. I can’t think how to say this. They should look more like an operational command center, right? You don’t need to have a huge dashboard that looks like novelty telemetry that has like a bunch of like, you know, squiggly charts on it that nobody’s really ever going to pay attention to. You really want to look at it as like a control plane, how you’re going to manage work. So Foundry, Microsoft Foundry combines, you know, some of these things in. All in one place, it’s a good example of what I’m talking about. You’ve got evaluations, evals, you’ve got monitoring and tracing, and tracing is really that gives you the waterfall effect, like what happened, what each one of the steps were taken in what order, with what latency, and what. token cost, you know, happened with it as well. Evaluation gives you the quality signal. What was the, was it the correct response? Was it complete? You know, I also do things like did the tool call that I wanted to happen, did it actually occur? And then monitoring gives you that operational metrics, like what was the full token consumption? How is that latency distribution rolling across the system? Are my accuracy tool call accuracy trends, you know, going the right way? So, and like, here’s an industry shift, you know, that’s happening today is we’ve got this new. This new open telemetry for agent observability, it’s kind of replacing the telemetry components that we were using in the past, and what’s cool about it is that Microsoft and Datadog… ChatGPT, they’re all using the open telemetry, you know, semantics in part of it. And that’s where Microsoft Agent Framework has been jumping on board as well. So we’ve been converting a lot of our telemetry stuff into open telemetry as well. And that helps you keep that instrumentation for your agents consistent across whatever platforms that you’re using. So, and then Microsoft Foundry is using it natively inside there too. You can even do things like prompt agent tracing, you know, in there as well. The new workflow features that all has the tracing and open telemetry built into it. So you’re going to see that, you know, start to mature quite a bit. I just did a talk about Agent Ops webinar, like I think it was last week or the week before. So you can go back and take a look at that one too. But observability, I think this is a backbone and that people need to start thinking about a little bit more. Quick little visual. I’m keeping an eye on time here and seeing that I’m running a little bit behind, so I’m gonna try to speed it up a little bit. This is kind of what you would expect to see from an agent trace. You can see the model call all the way up at the top, but you can also see things like latency, token cost, and accounting. confidence score like, you know, for the interaction. And then you see like a retriever step, which documents it would pull, you know, you start to get all these different components like baked into it. And the reason why I keep coming back to this is that it’s important to know why your agent is doing this. what your agent is doing and how it’s actually making the decisions. And without that, like you just, you know, without that observability, you’re not really going to be able to uncover the root cause of why things were going off track. Lastly, the final layer, quick architectural clarification. The Foundry control plane and Agent 365, a lot of people get these, they start talking about them because the control plane applies, or they use the words control plane in both of them, but they’re not the same product. They do have a little bit of an overlap. They do complement each other. But the Foundry control plane is really the Azure operator layer. And that’s where your engineering team is building, testing, deploying, and operating the agents. You’ve got cross-project, you know, inventories that you can set up different projects there, have policy enforcement and, you know, custom agent registrations. That’s all managed there. But what it isn’t doing is not a replacement for Agent 365. Because that’s at the enterprise governance level. It’s like how to exclude, you know, it’s, I’m trying to think, simple way to hold, the simple way is Foundry is the operating layer and Agent 365 is like the enterprise registry. So you need both of them to exist. but they cover different audiences with a different set of data. So I would look into both of those as your governance, inside of your governance model as well. So co-pilot governance. Copilot Studio now. This doesn’t always get said out loud. Copilot Studio governance is really distributed across three different planes today. In Copilot Studio systems, you’ve got the executive framework, tenant level controls, you’ve got Copilot governance overview. You can use Purview, and that gives you some of the label semantics, like… DOP and audits. And then you got Power Platform data policies that govern the actual control, the Copilot Studio mechanics. That’s where the connectors can be used, which knowledge sources, what HTTP calls, you’ve got skills, channels, and even like some of the event triggers. So you’ve got three different consoles in three different policy languages and real. Real coordination between these three is going to be is going to be required. So, the pragmatic answer to all this, and this is what I, you know, what I’d put in the operating model, RACI chart if we ever had one, and we’ll talk about it a little bit later, is assign one team as the through line. Someone owns the integrated view across all of these three. Otherwise, you’re going to wind up getting policy drift between the consoles, and really no one’s going to notice until something goes sideways in production. Microsoft has been converging a little bit on this, but, you know, we need to plan for this, you know, happening accordingly over the next, you know, couple of months. I… So this one I wanted to talk a little bit about multi-cloud, you know, and some of the differences between Microsoft, AWS, Google, you know, Salesforce and Databricks. So Microsoft Agent 365, this is the Microsoft-centric control plane. for the Agentic workforce. If your shop is already anchored in M365, and most of the people that are on this call, because we’re a Microsoft shop and you’ve probably heard about us, you know, you probably are as well. Agent 365 plus Foundry plus Entra plus Purview plus Defender, that’s probably the operating system that you’re looking at. But if you don’t operate, you know, in those areas, AWS Agent Core, that’s the packaging identity, policy, observability, and runtime. You’ve got Google, you’ve got Google’s Gemini Enterprise Agent Platform. That’s packaging things up like agent identity, registry, agent gateway. Okta is going vendor neutral. They’ve got agent identity governance. Palo Alto is going vendor neutral on some of the runtime protections. You get Datadog, Dynatrace. These are all like basing, they’re converging on that open telemetry that I was talking about for observability. But You know what the table on the screen shows right now is that the industry has converged on some of the same primitives. Identity, registry, gateway, observability, runtime defenses, and that’s pretty much across all the different platforms. The names might be a little bit different, but the semantics and the things that they’re actually dealing with, those are all converging. And what that means for you, really. is pick your anchor and plan where you want, plan where you want those overlays. If you’re Microsoft anchored, then Agent 365 is going to be where you want to put your anchor. But if you have agents that are like living in AWS or in Salesforce, Agentforce, you should, you know, you should expect to need to. build in some of these other overlays as well. Agent 365 registry sync is starting to be able to cover some of those, like incorporating Bedrock and Vertex AI and Agent Force. But preview, these are preview features that are baked in right now. And so preview means preview. And you need to assume that there’s going to be some sort of changes between now and the next like. you know, 6 to 12 or even 18 months. So take that with a grain of salt, experiment with some of these things, bring these ecosystems together. Trying to think what Jim Savage always says that Microsoft is the marketplace for AI. So, you know, not necessarily going to own everything. they’re being pretty supportive of all the things that you can do outside in the ecosystem. So where are some of the gaps? And I was thinking about this, and I’ve got five that I want to kind of call out, and maybe even a little bit about what you can do about each one of them. So. One, Agent 365, the registry sync, that’s still on a preview feature. The platform list is still pretty limited. You’ve got Bedrock, Vertex, Agent 4s, they’re covered in preview, but everything else that you might want to bring into your system, that’s going to be, that’s going to need some sort of a custom registration process. Mitigation for the platforms that haven’t been covered yet is customization and using the, there’s a registry API that you can use for a third party, for third party tools. Number 2, observability. You know, it’s, I think that this agent ops story that I’ve been telling is resonating because people are finally starting to think about it. And, but not everything is completely rolled out yet. You do have prompt agent tracing. You’ve got workflow agents, hosted agents, you know, and custom agents. Those are in preview though. So I would, to mitigate that, what I would recommend that you do is you standardize on open telemetry across pretty much everything that you build, because everybody else is starting to natively incorporate that. And that’ll give you like vendor neutral observability that you can aggregate later. The 3rd is the runtime protections. You know, right now, there are some integration boundaries. So for non-Microsoft runtimes, you’re probably going to get a little less than what Defender actually offers you. And for mitigation strategy, maybe use an AI gateway as a central enforcement and observation point. regardless of what runtime that you’re running on. That way, you know, the gateway gives you like a little bit of a choke point to protect yourself. Another one is Copilot Studio Governance is distributed really across the three different consoles that we saw before. So try to figure out how you can get those. a single line, a single through line owner, you know, baked into it. Build out some of the RACI charts so that you don’t let any one of those consoles go ungoverned. And then five, I would say the last one is like the cross-platform lifecycle actions are pretty limited right now. You can register in inventory agents in other clouds. But you can’t like necessarily start, stop, and version them all from like a single central place right now. So what you’re going to want to do is bake into your processes some explicit playbooks for each one of the platforms until there’s some sort of cross-platform lifecycle management tool that’s baked in there. And then, let’s see, cost money is money is always a funny little topic, and this is really something that is changing on a daily basis right now. Everything we’ve covered, you know, up until now has you know, has some sort of financial impact to the organization, whether it’s licensing or whether it’s, you know, token cost. But let’s break down, let’s break down the actions that you can take into three different things. So, One thing you can do is you could do nothing. That’s the easy way to do it, you know, but look at what you’re going to wind up carrying as like a cost. So I mentioned the shadow AI. That’s real and it’s going to continue to happen. Employees, they’re using unsanctioned tools. you know, whether you like it or not. And that’s going to cause, you know, a lot of like problems like oversharing, you know, or audit failures when you get audited or regulatory inquiries. So doing nothing is, you know, is an option. but it’s probably not going to end really well for you and the organization. You could do, you can go the other way, you could overgovern. And that’s where a lot of organizations fall into this trap. They see the risk and they start to overreact. They start banning all the tools. I’m going to give a shout out. in a really negative way to here in Wisconsin, our own DNR. And I was at a DNR facility several weeks ago and was trying to get into Chad GPT to prepare for a meeting that I had with them. And it was completely blocked on their networks. You know, they have a very hard rule. Nobody’s allowed to use AI at the organization. and their guest Wi-Fi just completely has the tools blocked. So you can go completely the opposite way, right, and over-govern. And, you know, that’s going to cause, you know, people to not be able to use these tools and learn how to use and adapt to them. And so I think that you’re not, you’re going to wind up not getting any return on AOI because AI, because you. don’t have any that’s being used in the system and you’re going to fall behind. So I don’t think that’s the right way to go either. But then in the middle path, you’ve got, you know, you got what we were talking about today with the control planes and with Agent 365. You can start to inventory your. your agents and give them named owners. You can start to implement some of these risk tiered, you know, approval gates, you know, set up those blueprints that we talked about that allow you to manage them at, you know, kind of like a holistic level without micromanaging each one of the agents and really overwhelming your team. So I would spend, you know, this is where I think like you should spend some of your time is finding that right size governance for you and the organization. It’s going to help you reduce, you know, some of the exposure that you have. You’ll be able to implement some of these, some of these features pretty quickly. So you get quicker ROI on some of the investments that you might be making in some of the the licensing as well. So find that balance and we can definitely help you help you figure out where the where the sweet spot is. So. Like, I would say, like, you know, from a plan perspective, you and the organization should think about what to do over the next 30, 60, 90 days. And I’ve laid out kind of like where most organizations are starting and what they’re able to actually accomplish, you know, through that 90 days. So start off with like an intake, right? Agents get registered when they proposed, you know, when they’re proposed, not after they’re already been, you know, put out into production. Start to identify some of the risk tiers. You know, every agent, when they get provision or when they get registered, assign a tier to them. do design reviews, security and data folks, bring them to the table, you know, at the beginning of it. And then learn how to do controlled pilots, controlled deployments, build in some of that open telemetry, observability so that you can track, you know, what’s actually happening to the agents that you’re piloting. before you roll them into production. So day one through 30, I would say you can focus on the inventory and ownership. Find out what you have, start to assign the owners, baseline what you want to do around purview and what your posture should be. Maybe stand up, defenders, AI, you know, agent inventory to help you with some of this. And then define your intake model, like how you’re going to actually accomplish it. Who’s going to be accountable, who’s going to be responsible. Build out the racing model. Don’t try to fix everything in the first 30 days. You’re not going to. I just showed you a screenshot. a little while ago from our ecosystem, we haven’t done it, you know, you’re not going to get everything, you know, right out of the gate. And then 30 days, 30 through 60, start to build in that instrumentation. Get onto your agent ID, you know, out with the blueprints, you know, the first set of some of your blueprints. roll out a couple of sensitivity labels and start and start building out data loss policies for AI. You should maybe turn on Defender and get like one or two real-time blocking definitions in scope and then pilot some of this, you know, so you can see how it works under a real. any, you know, under a real workload. And then from 60 to 90, automate and build up some evidence. So start, you could start looking at implementing things like conditional access enforcement, or maybe some of the automated like remediation workflows. But at the end of the day, like. you know, start to figure out how you can operationalize this, what those agent ops dashboards look like, how they live, and who’s actually going to be watching them. And if you follow some of those three steps, you’re starting to build a program, you know, that you can start to, that you can mature over the, over the, over the rest of the year. So don’t boil the ocean. Start with the 30-60-90 plan. And, you know, then we can reevaluate where you’re at in 90 days and figure what the path forward is. So if what we’ve covered today resonates with any of you and you’re looking to talk a little bit more deep you know deeply about it. I’ve got three different ways that concurrency could, you know, can work with you. If you’re looking for an AI governance and security readiness assessment, I would say this is a good place to start if you don’t know what agents already exist in your environment. Maybe you don’t know who owns them. There’s, you know, a lot of people are building agents with the co-pilot tools, you know, and we’re finding that some organizations that say they don’t have any agents, you know, you go under the covers, you actually see hundreds of them. So we could take a look at that with you. Second one is maybe one that I’m like really passionate about is getting into these agent ops. you know, conversations. And if you’ve already got some agents that you’ve got up and running and you’ve been experimenting with, this would be a good place for you to start with that agent ops enablement. And, you know, we can, you know, help you figure out how to apply open telemetry, how to instrument it, and then serve that surface that up for people to review. And then lastly, we can do a Frontier Firm operating model kind of workshop. And if you’re starting to scale beyond like just a handful of agents and you need to figure out how that whole operating system looks like and not just the tooling, we can help you figure that out as well. So reach, you know, I think that we dropped a link into the chat, you know, for you to figure to reach out to us and if you want to do one of those three things. So I’m going to wrap it up here. I want to leave you with like just a little bit of a message that hopefully ties all five of these different workshop series together. So the Frontier Firm, You know, it’s not the company with the most agents. It’s the company that can actually safely, directly measure, govern, and improve digital labor faster than your competitors. That’s the whole point here. Not the model that you picked. It’s not the number of pilots that you’ve launched. It’s do you have the operational maturity that lets… your digital labor scale without any of the mistakes and the chaos that is typically happen. And so with that, I’d say thanks for spending the last 60 minutes with me. And is there any questions that I might have missed in the chat? Amy Cousland 57:40 No, I think we’re all good. We can go ahead and end this and let us, you can reach out if you have any other questions. Thank you so much. Thank you, Brian. Brian Haydin 57:46 Yep, thanks.