View Recording: Fresh From Foundry: What the Latest Announcements Mean for You

Brian Haydin All right, hello everybody. Thank you for joining us today. I’m joined today with Nick Miller. 0:0:17.15 –> 0:0:36.15 Brian Haydin haven’t seen or met me before, Brian Hayden. I’m a solution architect at Concurrency. And what we’re going to be talking about today is just, you know, some of the latest announcements coming out of Foundry. This is an ever-changing ecosystem, and we put some time on the calendar just to 0:0:36.735 –> 0:0:56.255 Brian Haydin talk about what’s new, and Microsoft didn’t disappoint. But before we get started, I want to just introduce those of you that don’t know Concurrency. We’ve been in business for 35 plus years. We are a Microsoft aligned solution integration company. 0:0:56.975 –> 0:1:15.775 Brian Haydin dealing a lot with AI. We do a lot of these webinars pretty much every week. So if you want to follow us on LinkedIn, there’s a QR code or just search for Concurrency Inc. You’ll find us pretty quickly. And not to spend a lot of time on the sales aspect of it. We’ll just, you know, start with some of the big news. 0:1:16.335 –> 0:1:38.415 Brian Haydin So yesterday was kind of a big day. I was not expecting this. And we had this on the calendar for probably 6 weeks or something. And I just knew like this timing, there’d be things happening. And we wound up having three relatively decent sized, you know, announcements yesterday. One is that we got Grok 4.6, you know, deployed in into our model libraries. So now that’s in preview, we can use it. 0:1:39.695 –> 0:2:0.655 Brian Haydin But then in addition to like that new thing, we had two things that kind of got removed. So for those of you, just a public service announcement, two things got retired yesterday. The Azure OpenAI Assistance API was retired, and then there was the Azure AI inferences that were retired as well. We’re going to get some of this stuff a little bit later in the talk. 0:2:1.455 –> 0:2:23.295 Brian Haydin And the last thing I thought was kind of interesting was this blog that I came across yesterday where Microsoft talked about how to measure success. And rather than measuring the tokens, which is kind of an interesting thing that they like, said, don’t pay attention to these tokens as they release their new model. Don’t pay attention to tokens. Look at the cost, the overall cost per outcome. 0:2:23.695 –> 0:2:37.375 Brian Haydin And that’s something that I’ve been having a lot of conversations with my customers about is like really focusing on that outcome. It’s part of a series. There’s going to be more, they’re going to be talking about a little bit more. But Nick, which one of these kind of landed for you? 0:2:38.575 –> 0:2:40.95 Nick Miller Well, for me… 0:2:41.775 –> 0:3:1.135 Nick Miller The last one is the day in and day out of what I always do. So it’s not really news to me, but it’s really good information. And I think, you know, us saying that is important, but you know, Microsoft is saying it is another kind of data point that, yes, that’s the right way to go. But I think, you know, the first one, the new Foundry models 0:3:1.215 –> 0:3:8.335 Nick Miller kind of interesting. There’s a lot of good stuff there. But for our group, I’m going to guess maybe the third might be the most interesting. 0:3:9.935 –> 0:3:10.495 Nick Miller We’ll see. 0:3:10.95 –> 0:3:30.95 Brian Haydin Yeah, I think so too. We’re going to get into like a new concept, a new model that also got released besides just Grok. And the question I kind of want to argue or talk about a little bit is Microsoft actually changing the game in Foundry. So what am I talking about? It now, Microsoft 0:3:30.415 –> 0:3:49.375 Brian Haydin At Build, they announced their own version, their own models. And we got to play with a couple of them here and there. But as of the last couple of weeks, they released their AI thinking model in Foundry. And it’s priced pretty reasonably and something that might actually be useful. So 0:3:49.855 –> 0:3:57.55 Brian Haydin Yeah, I think let’s talk a little bit about like what that actually is. What is the Microsoft AI model? 0:3:58.575 –> 0:4:0.415 Nick Miller So it’s MAI thinking one. 0:4:1.935 –> 0:4:23.295 Nick Miller Yeah, basically it’s a thinking model that is similar to GPT-4, GPT-5. Maybe, I don’t know if I directly compare it to GPT-56, but the idea, like the thinking models where you can set a thinking or a level of effort. So same model, but level of effort is low, or maybe smaller tasks, medium or high. 0:4:24.95 –> 0:4:33.935 Nick Miller So it basically spends more cycles evaluating the problem. It takes a longer response, but in general, it gets a better, it gets a better outcome for you. 0:4:35.55 –> 0:4:54.695 Brian Haydin Yeah, and so some of the value propositions I think that that kind of stood out to me is that the cost of it is quite a bit less than other similarly reasoning models. I mean, you’re right to call out that it’s definitely not at a sol, you know, high reasoning. 0:4:54.775 –> 0:5:17.735 Brian Haydin you know, kind of adds that just not, it’s not the same thing as you’re not going to same kind of responses out of it. But from a cost comparison, it’s, you know, certainly much, much more affordable. And how does this actually like, like talk a little bit more about this, how this affects the catalog and what we would be using it for. And I know we got a couple of slides, we’ll dive a little bit deeper into it, but. 0:5:16.975 –> 0:5:21.415 Nick Miller Yeah, sure. I think, in general, the. 0:5:23.295 –> 0:5:41.855 Nick Miller The way that the catalog advertises models, and actually the only way that I know in general how to do it ubiquitously, is that they have these performance metrics, these benchmarks, and so they have performance evaluations. So this model scores a 0.75 out of 0:5:41.935 –> 0:6:0.575 Nick Miller one on this specific benchmark. And then you can take that, you can compare model to model against that benchmark. And they’re useful, but even just that one benchmark itself, it doesn’t necessarily tell you how it’s going to perform on a specific use case. 0:6:1.135 –> 0:6:25.735 Nick Miller And we’ll talk about this later, but you know, this is really where model evaluation comes in, and we have this a little bit more detailed, right? So, there’s that, but then there’s also, you know, kind of perform like cost per performance, if you could think about that. So, if you have that one metric, like just a generic metric we talked about, performance benchmark, and you know that benchmark is a is a fifty-five, and then for MAI is a fifty-five and GPT-5-6 is a… 0:6:5.695 –> 0:6:6.175 Brian Haydin Yeah. 0:6:25.735 –> 0:6:46.175 Nick Miller the 67. When you talk about what’s the score for performance, like what’s the price per performance? So if there are two models that have basically same performance on these benchmarks, well, then you have, you can actually compare equivalent, like, hey, this one has a certain dollar per performance on this benchmark. And that’s a way to. 0:6:46.735 –> 0:7:9.695 Nick Miller get a high level understanding of how the models were performed. So you can maybe get similar standards for a better price, right? That’s one high level way. It’s a little bit naive, but it still is. I think it’s, if you’re going to look at rough cuts, maybe that’s a good way to take 100 options and narrow it down to maybe 10 options that you might want to do some actual evaluation on. 0:6:58.255 –> 0:6:58.735 Brian Haydin Yeah. 0:7:10.95 –> 0:7:31.535 Nick Miller And the other thing we have here is about the deployments. So right now, in AI, I think is only global standard. For most people, that’s great. Actually, the token rates are typically higher, the token costs are typically lower. But if you do have requirements to keep your, what is global standard? We’ll talk about that first, I guess. 0:7:30.615 –> 0:7:33.935 Brian Haydin I was gonna, I was gonna say, like, for the audience, what is global standard? 0:7:32.95 –> 0:7:32.495 Nick Miller Yeah. 0:7:33.815 –> 0:7:52.975 Nick Miller What is global standard? Well, basically it means that even though your model is deployed in East US 2, if East US 2 is busy, it can send overflow requests for that model to a different region that has availability. So it may go to North Central US or somewhere else. And if you have… 0:7:54.655 –> 0:8:17.575 Nick Miller If you have data residency requirements, because it’s global standard, right? Not just US standard, global. So it could send it overseas even. Now for the US, probably not as much, but if you’re in Europe and you need to keep it in your country or something like that, then maybe global standard doesn’t work for you. But if you don’t have residency requirements for your data, meaning like it may not be like PHI or HIPAA or 0:7:59.935 –> 0:8:0.415 Brian Haydin Yeah. 0:8:17.655 –> 0:8:37.855 Nick Miller or some other sort of protected information requirement, then no big deal. You can use it. Global standard. And the other thing is provisioned throughput, so TPUs. If you ever go to Sam’s Club and you walk through Sam’s Club, you’ve checked out, done self-checkout, which I love. I like people, I just don’t like going through cashier lines. 0:8:38.135 –> 0:8:58.415 Nick Miller So I check out, I get my groceries, and I’m walking through the line, any of these pillars, they have cameras, and they take pictures of the side of my grocery basket and from above. And it’s actually using Azure. And it’s actually using Azure models and AI on the line. It takes pictures of what’s in there and compares it to what’s on my receipt. And you know what? Sam’s has a bunch of PTU. They have a bunch of… 0:8:58.535 –> 0:9:20.255 Nick Miller purchase throughput, they get a much lower price on its reserve capacity. So if you have a large enterprise project, then yeah, you need PTU. So what this really tells us is that it’s kind of, I wouldn’t say preview now, but it’s for lower scale, lower reliability workflows at the moment. And then also you can’t have those data residency issues. So. 0:9:20.655 –> 0:9:21.695 Nick Miller Global standard only. 0:9:22.495 –> 0:9:41.775 Brian Haydin So I think that’s where the preview aspect of it comes, right? It’s not, this is their first roll out there. They’re going to be working through the kinks a little bit. And then, yeah, so interesting model. We’re going to talk quite a bit about it. I think one of the things that I like about Microsoft 0:9:42.575 –> 0:10:4.255 Brian Haydin Joining the fray is it gives us more choices, right? And Microsoft has done, this is something I’ve said over the last few years. I respect a lot of what Microsoft is doing in terms of supporting other models, allowing Claudio to be, you know, to live inside of the Foundry, allowing open AI, well, I mean, open AI, it’s kind of obvious, but 0:10:5.55 –> 0:10:23.855 Brian Haydin You know, Grok, you know, it’s an open ecosystem. So they’re entering the fray here to compete a little bit, but compete on the merits, not to make you just, not to force you to use just Microsoft’s model, right? And then they’re also making their model, not just Foundry, they’re making it available on other platforms as well. 0:10:24.255 –> 0:10:33.295 Brian Haydin So they want to make, they want to give people the choices, right? You know, with a decent model. Anything else you’d kind of add to that? Like what are your… 0:10:28.815 –> 0:10:29.135 Nick Miller Yeah. 0:10:32.735 –> 0:10:51.535 Nick Miller No, I love it as well. I don’t want there to be, you know, one to two model overlords that control the entire economy, like Anthropic and OpenAI, right? I want there to be an ecosystem, right? So we can all have freedom and choice in the models we want, because different perspective, different use cases. 0:10:51.935 –> 0:11:11.55 Nick Miller Right, so you can have Grok, you can have open source models even, right? Of course, some of these open source models, if you’re going to get big with that, you have to host your own hardware. So that’s a whole other scale. But I love more choice, right? No one has ever said, you know what, there’s too many ice cream flavors. 0:10:57.55 –> 0:10:57.455 Brian Haydin Yeah. 0:11:11.615 –> 0:11:20.15 Nick Miller We should only have vanilla and chocolate, anthropic, open AI. Nope. We need strawberry and that. And this is Microsoft getting in there and all of that. 0:11:14.575 –> 0:11:14.975 Brian Haydin You’ve never. 0:11:20.615 –> 0:11:39.975 Brian Haydin You’ve never met a Latvian woman who would disagree with you completely. So, all right, so probably the more interesting conversation is like, what is it going to be good for? Like, how should we be thinking about if MAI is something that we want? 0:11:23.495 –> 0:11:23.615 Nick Miller Yeah. 0:11:24.895 –> 0:11:25.455 Nick Miller Okay. 0:11:27.55 –> 0:11:27.535 Nick Miller There. 0:11:35.535 –> 0:11:35.935 Nick Miller Right. 0:11:40.95 –> 0:11:44.15 Brian Haydin To use, you wanna you wanna talk through this a little bit? 0:11:43.135 –> 0:11:58.175 Nick Miller Yeah, yeah, absolutely. So this is where we talked about evaluations. A couple slides back, we were looking at the model token costs, you know, dollars per million. And it’s partly like, it’s partly that, but also, 0:11:59.295 –> 0:12:18.815 Nick Miller How does it work? Like, how well does it perform? And that is really what requires your specific business use cases. And if you’ve ever heard any of our other presentations, we did one on model evaluation. And the point is, is that everything your model does, you should have a golden test data set. 0:12:18.895 –> 0:12:39.55 Nick Miller with a prompt or whatnot, and then you should have some expected output of that. And there should be lots of different use cases. So if you’re going to deploy this model to handle a business case, you need to have evaluations that score that. There’s different types of evaluations, and I won’t bore you with my statistics background with things like recall and precision. 0:12:39.455 –> 0:12:59.775 Nick Miller But the point of it is to say that if this model is supposed to identify an issue, let’s say you have a model that’s supposed to identify types of errors in a document, and you know that there’s 100 errors in that document, you need to run evals on that. And how many errors does it find? Does it find 80 out of the 100? Does it say there’s 120 errors when there’s actually only 100? 0:13:0.15 –> 0:13:18.575 Nick Miller And actually, the 120, I only got 80 of them right. That’s recall and precision. But the point of that is that they’re all evals. And so how do you know how good MAI is? Well, it depends. It depends on your use case, what you’re trying to do. So you have to apply it. You can look at high level, you know, what are the types of use cases that could be applicable. 0:13:18.695 –> 0:13:38.15 Nick Miller but then you really just need to run it through the evaluations. And hopefully this is something you’re doing anyway, right? So you have a project or a use case that you’ve deployed that’s agent related. You should be running that agent through these evals before it even gets to your development branch, right? It’s A CICD gated pre-merge thing. 0:13:38.495 –> 0:13:57.935 Nick Miller And so, yes, performance. So what you do is you make sure that the models perform, like all your different array, like models, no, one, like 10 models, like a lineup, you know, American Idol, but modeled, you know, version. So which, how well do they sing? What’s that? So that’s the quality. How well does the model perform? 0:13:58.335 –> 0:14:17.935 Nick Miller But then, okay, what’s the quality performance? And if they’re above that, which model does it maybe the fastest, maybe the cheapest? It could be that, you know, this model isn’t as expensive as the others, but you have a latency requirement where it has to be less than 10 milliseconds. Okay, great. Well, then which ones do that? 0:14:18.335 –> 0:14:32.95 Nick Miller And then once you’ve met those, that could be considered quality as well. So once you’ve met the performance and then the latency requirements, then pick the cheapest model. And maybe MAI is the one, right? Because it is cheaper than the other. So that’s really the high level way to do it. Now, 0:14:33.215 –> 0:14:38.735 Nick Miller Every use case is different and the way you do the emails is different, but in general, that’s the way you do it. 0:14:39.775 –> 0:14:43.135 Brian Haydin So, um, alright, so I uh… 0:14:45.615 –> 0:15:8.95 Brian Haydin I’ve got two different models, and MAI is one of them, but let’s say that I currently have an agent that’s running using 56 Luna. One way to look at this is that those are just different. Those models are built in a way that they think through the problem differently. And even though the token cost might be similar between them, let’s just say that they are. 0:14:55.175 –> 0:14:55.455 Nick Miller Okay. 0:15:8.335 –> 0:15:33.15 Brian Haydin for arguments purposes, I’m probably going to have different cost per outcome, right? Because Luna might think through this problem faster than MAI, or maybe MAI is thinking through the problem faster. So the actual token consumption is going to be different, right? So, all right, and you brought up like this whole basis of evals and stuff like that. Going back to that Microsoft. 0:15:8.815 –> 0:15:9.215 Nick Miller Right. 0:15:10.295 –> 0:15:10.495 Nick Miller Shared. 0:15:25.295 –> 0:15:25.775 Nick Miller Right. 0:15:33.295 –> 0:15:52.655 Brian Haydin blog and something that I’ve been preaching pretty much non-stop this year is, you know, what makes up the eval like metrics? What are you going to measure? So I think task success is certainly one of them, right? Did it actually work or not? But also did it actually do the things the way that you’re supposed to? 0:15:47.135 –> 0:15:47.455 Nick Miller Yeah. 0:15:53.135 –> 0:16:11.695 Brian Haydin So coming back and telling me what my account balance is, just because it got it right, doesn’t matter if I can’t verify that it actually reached out and asked the bank. You know, it actually has to call the right tool. And then did it give the information that was grounded in reality? Did it cite a source? Did it actually look up the data? 0:15:53.695 –> 0:15:54.95 Nick Miller Yeah. 0:16:2.975 –> 0:16:3.215 Nick Miller Derek. 0:16:12.255 –> 0:16:31.215 Brian Haydin You talked about latency. Those are metrics that we’d want to understand too. And then the most important one that Microsoft and I have been talking about in terms of that cost metric is like, I don’t really care what the token count is. I just need to know how much did it cost me to get that successful outcome. And, you know, because 0:16:31.415 –> 0:16:42.95 Brian Haydin The tokens don’t matter if it chews through 5 million tokens at every request, if the other model is able to get it done in like, you know, 10,000 tokens or something like that. So. 0:16:35.535 –> 0:16:36.15 Nick Miller Right. 0:16:41.295 –> 0:17:2.415 Nick Miller Absolutely. And that’s why you build your evals to be like a specific outcome. So this eval is judging the model’s ability to, you know, perform this task. And there’s different ways to do them, right? But yeah, if you have that eval set up and when you run the eval, it does look at the things like part of that eval suite, per se, like maybe it’s not a single eval, but that suite of evals. 0:17:3.215 –> 0:17:22.175 Nick Miller is, yeah, did it call the right tools? Did it call them in a weird way and just got lucky? It happens depending on your use case. And turns per completed task, that could be important, but maybe it’s not quite as important because it takes 5 turns, 4 turns, whatever. It’s really about total, in my opinion, it’s really about total latency. 0:17:20.735 –> 0:17:21.55 Brian Haydin Yeah. 0:17:22.655 –> 0:17:42.735 Nick Miller right? So how long does it take in total? Then also it uses a certain amount of tokens to do the entire thing. And then what’s the token cost? And so that eval suite is 1 use case, and it costs you 25 cents, right? It doesn’t matter how many tokens it was. It was 25 cents. And what was its performance and latency? And so that’s what you really compare. 0:17:24.495 –> 0:17:24.895 Brian Haydin Yeah. 0:17:27.855 –> 0:17:28.255 Brian Haydin Yeah. 0:17:42.815 –> 0:17:58.735 Nick Miller Right, so if, if, if, if MAI cost $0.20 and the other one cost twenty-five cents, and the performance, like the accuracy, the recall, the is all the same and the latency is the same, we’ll choose the 20 cents going over the twenty-five cents. It’s 20, it’s 20% cheaper. 0:17:43.695 –> 0:17:44.575 Brian Haydin Yeah, exactly. 0:17:59.535 –> 0:18:20.255 Brian Haydin Yeah, and you got to have the harnesses to be able to do that. So, you know, we also talked a little bit about tools too. And, you know, I want to recognize that things are shifting inside of Foundry in terms of its capabilities. It’s not just a model catalog anymore. And, you know, a lot of the early part of this year, the shift was, you know, adding governance controls. 0:18:2.175 –> 0:18:2.735 Nick Miller Yes. 0:18:20.535 –> 0:18:41.295 Brian Haydin and having that control plane inside of Foundry to be able to manage things. But I think, you know, recently what we’re seeing this month is Agenta capabilities, you know, being added to it. So one of the big announcements, you know, a couple of weeks ago or 10 days ago was around Claude and, you know, Claude and Microsoft Foundry. 0:18:41.655 –> 0:19:3.455 Brian Haydin It wasn’t just the models being available, which is, you know, the model catalog, but it was the tooling around it that became available as well. So I think that’s some really exciting news, you know, being able to get access to some of these capabilities. Like, I don’t know, I have these arguments all the time, like I want to use ChatGPT’s models, I want to use Claude’s models. 0:19:3.775 –> 0:19:12.815 Brian Haydin you know, and a lot of it comes down to the tooling and what the capabilities are. And now we’re going to see some of that available as well. What do you, you know, what are your thoughts on this too, Nick? 0:19:13.375 –> 0:19:32.815 Nick Miller Absolutely, it’s not just the single thing, and it’s the whole platform. And that really is Microsoft’s whole premise and Azure’s whole premise. In some cases, no, I hope they’re not listening, but in some cases, their one single offering, this one thing they do, may not be the best. Like it may not be the best of breed of that thing. 0:19:33.375 –> 0:19:51.535 Nick Miller But because that thing is part of this larger ecosystem, the entire value is higher, right? And so that’s where the value is. And this is the same thing here. When you’re talking about agents, it’s not just a model. The model is just once, it’s not a small part. You have to validate it, of course. 0:19:38.335 –> 0:19:38.735 Brian Haydin Yeah. 0:19:51.935 –> 0:20:14.135 Nick Miller But it’s all the other things, the way it integrates, the way you can authenticate, the conversations API, the responses API, and what that allows you to do. And in here, these new tools that you talked about here, host on Azure, these are just extending that capability. So it’s less things you have to create yourself, right? You could write all your code in compile language. 0:20:8.495 –> 0:20:8.815 Brian Haydin Yeah. 0:20:14.415 –> 0:20:32.15 Nick Miller you know, but no one does that because we have things like Python and.NET, right? And if there’s a Python package that exists to do something you want to do, you don’t write the package from scratch, right? You use what’s already there and tested. This is kind of, this is an analogous thing, right? Like for example, tool search. 0:20:33.135 –> 0:20:51.695 Nick Miller If you’re not familiar with how agents use tools, let me break it down. What you do is every single turn that the agent has, so you ask the agent a question, it injects in the background, it injects these tool metadata. So it says, here’s all the tools. 0:20:51.815 –> 0:21:11.455 Nick Miller Here’s the description, here’s how to use it, you know, here’s when you want to use it, et cetera, into every single turn, right? That’s just tokens that are chewed up, and you may not need to use a tool, right? If you have one tool, no big deal. What if the agent has 10 tools? It’s all of that context for every single tool, for every single turn, right? And that’s wasted tokens. 0:21:11.775 –> 0:21:13.935 Nick Miller Now you might say, Nick, what about caching? 0:21:12.15 –> 0:21:20.175 Brian Haydin Every time it every time that it does a turn. Yeah, exactly. It’s asking like, hey, wait, what what tools do I have access to? What is what do you do? What do you do? What do you do like? 0:21:19.535 –> 0:21:38.895 Nick Miller Yeah, yeah, exactly. It’s a waste. It’s a waste. And so, but then you could, there’s a kind of a counter argument maybe to that, or maybe a mitigation rather, is that you can have caching of prompts. And then in general, caching reduces the token costs to about 10%, right? So if the token costs are… 0:21:21.495 –> 0:21:21.775 Brian Haydin Yeah. 0:21:39.455 –> 0:21:57.855 Nick Miller $2 per million, then the caching is, you know, 20 cents per million. So it saves you 90% on those cached. So yeah, okay, that helps, right? It does help. But what about this? You know, with tool search, it basically, instead of injecting all 0:21:57.975 –> 0:22:18.15 Nick Miller of the tools into the prompt, it injects kind of like a higher level thing. It doesn’t load them up front. Basically what it does is it just says, it tells tool search, I need to do this thing, right? And then tool search will say, okay, you need to do this thing. Let me find. 0:22:18.135 –> 0:22:40.95 Nick Miller The most likely tools, it’ll return a subset of your tools, the top like three to five matching tools, and it will return those tools as prompt as context for the agent, so that it can say, “Hey, here’s probably the list of tools you want to use,” so it basically makes that process, it reduces the context for using tools, so… 0:22:40.335 –> 0:23:0.175 Nick Miller It can basically reduce the token usage by eighty-five percent, and the cool thing is, if you have, you know, and then you could also layer on top of that cached prompts, right? So, cached prompts to save you money on the token price, but also it’s going to be faster. 0:23:0.735 –> 0:23:16.935 Nick Miller this way as well, because it doesn’t have to load as much context in to use the tools. So the tool search really handles that for you. And basically, you define all your tools in this tool catalog, make them available to tool search, and then you make tool search a tool in your agent. 0:23:18.215 –> 0:23:20.575 Nick Miller Is that like tool in section, the movie? 0:23:20.655 –> 0:23:22.575 Brian Haydin I feel like I’m a tool. 0:23:22.15 –> 0:23:23.775 Nick Miller Entering the tool matrix, I don’t know. 0:23:23.855 –> 0:23:45.215 Brian Haydin I am in the tool matrix. So one like kind of call out here is that this is beta. This is, you know, like it’s got some caveats. We’re learning a little bit about it. And if you’ve got like, you know, data protection, data retention, you know, kind of things that you need to like take a dive a little bit deeper into the documentation to make sure that it fits your scenario. 0:23:25.295 –> 0:23:25.615 Nick Miller Yeah. 0:23:45.615 –> 0:23:46.15 Nick Miller Right. 0:23:46.15 –> 0:24:5.55 Brian Haydin So, you know, it’s eventually probably going to get to the point where you need it, but, you know, just be careful about it because it doesn’t have all the same policies applied to it as well. So if you just, you know, read the documentation a little bit more. So, you know, but in addition to tools, 0:24:5.215 –> 0:24:24.735 Brian Haydin Like, there’s this, like, this data stuff, right? And Microsoft has been confusing everybody by changing IQ into, like, first we had this, like, this madness of everything was a copilot, and we still do, but, like, there was copilot for this, copilot for that, copilot. 0:24:24.575 –> 0:24:24.815 Nick Miller Yeah. 0:24:24.895 –> 0:24:46.335 Brian Haydin Now we have these IQs, like I have this queue for here, IQ for this, IQ for that. And I get people ask me like a lot of times, like what is the difference between them? So thinking about it like as like the work IQ, fabric IQ, Foundry IQ, work IQ is really just asking questions about where and how people do their work. 0:24:46.775 –> 0:25:6.415 Brian Haydin So like the Microsoft 365 kind of signals that you would ingest, SharePoint, chat, emails, you know, stuff like that. And then on the fabric side, this is like your data that sits in a more structured, well, it could be unstructured, but the fabric ecosystem with ontology defined between it. 0:25:6.495 –> 0:25:29.135 Brian Haydin so that, you know, when you ask questions, it understands what a sales order is, what tables that actually means, and how they’re interacting with each other, as well as like the governance structure that sits over the top of it. And then the Foundry IQ is more intentional built stuff. So Foundry IQ doesn’t just magically appear like work IQ does. Somebody sends you an e-mail, 0:25:29.375 –> 0:25:51.775 Brian Haydin boom, you know, work IQ understands it. Foundry IQ is us creating knowledge bases, knowledge sources that is purpose built for an agent to be able to retrieve information out of that. And it sits, you know, within that ecosystem. So just giving a definition so that when we talk about this a little bit more, that people understand what we’re talking about with those different IQs. 0:25:52.95 –> 0:26:0.415 Brian Haydin This diagram I think gives you a better reference architecture around it and how we would actually use it. So how are we using this stuff now today? 0:26:1.455 –> 0:26:19.215 Nick Miller Well, double click on, we’ll just go left to right, but double clicking on Fabric, you know, Fabric IQ, you said ontology and some other really big words, which are correct, by the way. But what I would say maybe a more simple way to explain it is, 0:26:20.255 –> 0:26:40.255 Nick Miller your data warehouse, right? So the unstructured data, but also the data you have in your data warehouse, it talks about schemas and understanding of semantics. So when you have tables defined, what are they, what is like, so a column says revenue or net income. What does that actually mean? 0:26:40.495 –> 0:27:0.335 Nick Miller You know, because there can be different definitions. Maybe there’s a part of that metadata is the formula for net income or revenue or whatnot. And why does that matter? Well, if an agent’s trying to get information from your data warehouse, the agent doesn’t necessarily understand the tables, the relationships between the tables. 0:27:0.575 –> 0:27:20.975 Nick Miller the semantic meaning of the different columns within a table. And Fabric IQ is what exposes that and makes that available to your agent, right? So if you’re thinking of, I want a sort of a data analyst agent that can give my executive team access to our data in a structured way. 0:27:21.615 –> 0:27:42.495 Nick Miller Well, you give it access to the specific tables that have the semantic meaning, and they can ask questions about, like, hey, what was our net revenue over, you know, by a customer group for the last three months? Divided, you know, shown on a weekly basis. Yeah, whatever, right? It can do that, and it, because it understands, it interfaces with Fabric IQ. 0:27:42.655 –> 0:28:2.735 Nick Miller Fabric IQ has the concepts and the understanding of that. Now that Fabric data agent is how it is exposed via Fabric IQ. So that’s kind of Fabric left. Now middle, 365 Copilot. This one’s a little bit funny, this graph, and I understand why they did it, but Work IQ doesn’t belong to Copilot. 0:28:3.55 –> 0:28:22.335 Nick Miller Work IQ can be used by a copilot agent. Work IQ can also be used by a Foundry agent. So it doesn’t have to be one or the other. But that work IQ, like you said, is just information about emails or conversations in Teams, files in Teams, which behind the scenes is SharePoint, as we know. 0:28:23.55 –> 0:28:41.775 Nick Miller So if you’re having an agent that’s wanting to give you information about what was said, and then it can harness work IQ, and that’s that context. And one thing to note on work IQ, it’s on behalf of, always, which means that if you are interacting with the agent, 0:28:42.735 –> 0:29:1.55 Nick Miller The agent knows that I’m Nick, and here’s my access. And so it knows what I have access to. And so when it sends that request over to Work IQ, it’s going to say, hey, Nick is asking this question, and Work IQ has access to Graph, which is the behind the scenes 0:29:1.375 –> 0:29:19.775 Nick Miller you know, infrastructure of all of these things like emails and messages, et cetera. And so it only looks at the graph. It filters the graph down to what I, Nick, have access to. And so then it can ask those questions. So when you’re building an agent in Foundry, that Foundry has to have that on behalf of. So you’re going to have to make sure you do it that way. That’s one. 0:29:20.15 –> 0:29:38.655 Nick Miller kind of constraint, if you will, but that’s baked into work IQ. It doesn’t allow you to have, like, say, for example, an agent that has an identity that’s generic, and everyone that uses that agent uses that same agent’s identity, and that identity has access to, you know, certain tables in Fabric, for example, right? 0:29:39.55 –> 0:30:0.495 Nick Miller You can’t do that with work IQ. So it has to be on behalf of. So that’s kind of interesting. Yeah. And so, but if you think about that, what’s the use case for that? I mean, if you have to understand what’s going on, hey, remind me of all the conversation, like you’re in a project team and you want to say, what were the, you know, what were the latest comings and goings of this project? 0:29:43.55 –> 0:29:43.455 Brian Haydin Right. 0:30:0.735 –> 0:30:19.575 Nick Miller I remember that we talked about some sort of issue with a project and it was related to this. Can you find that? And it’ll find the conversations, it’ll find the chats, the notes, etc. And that’s really what work IQ is. It’s an agent way. It’s an Ave. into that information that your agents can then access. 0:30:19.695 –> 0:30:27.855 Nick Miller So you can build your agents to do whatever it is you need to do and access work IQ. And then the last one here on the right is Foundry and Foundry IQ. 0:30:28.815 –> 0:30:47.695 Nick Miller So Foundry IQ is only accessible via Foundry, unlike Work IQ. And you have an agent there. And what is Foundry IQ? Those are things that are, like you said, more specific. They’re purpose built for this situation. They don’t exist unless you build them. Work IQ exists if you have 365 products. 0:30:48.335 –> 0:31:6.975 Nick Miller The Foundry IQ could be things like AI search indexes, right? We have projects where we are extracting certain information from our ERP into an index because we want to find information in our ERP, like, hey, give me all the purchase orders from a certain company. Or 0:31:7.215 –> 0:31:28.175 Nick Miller you know, information about finding customers. You extract information from your customers, it lands in AI search. The AI search index is a component of Foundry IQ. It can also, it memory stores, file stores, things like that. So Foundry IQ is the collection of resources that gives you access to 0:31:28.495 –> 0:31:51.455 Nick Miller that supplies that information to your agents in Foundry. And so the cool thing is, you can have an agent that has Foundry IQ and Work IQ, because maybe it needs to do both things. So you can have an agent that does both of these. And they’re really just tools or connectors to your agent. So remember how we talked about tool search? Well, as part of your agent’s tools, it can have tool search. You can define web search in there. 0:31:52.135 –> 0:32:10.455 Nick Miller an AI search index, we have Foundry IQ or Work IQ, and those should become all different sorts of tools that your agent has to use to get context. And I think what’s really cool about these is agents have to have context, and they’re only valuable if they know stuff. Well, that’s not true. 0:32:10.535 –> 0:32:31.415 Nick Miller I mean, agents have built-in knowledge that they have not illegally taken from the people who own the copyrights to the information, because Anthropic won that lawsuit, so therefore it didn’t happen. But anyway, they have embedded knowledge. But if the agent needs additional knowledge, like about your company, 0:32:19.695 –> 0:32:20.335 Brian Haydin Sometimes. 0:32:31.855 –> 0:32:53.495 Nick Miller The cool thing is with this setup here, your agents can have comprehensive understanding of everything that’s going on in your company, right? And the value that agents can bring is based off of the practices, the conversations, the SOPs, the drawings, all that good information. And this really brings it all together. 0:32:53.615 –> 0:33:2.655 Nick Miller right, in a unified way. And one thing that’s not on here, and if I have enough time, we’ll go very lightly into it, is Agents 365. Do we have time for that? 0:33:3.55 –> 0:33:21.55 Brian Haydin We will, but let me cut you off just a little bit because I want to bring the story, like I want to bring this into like a big picture. So I talked about, we started with the definitions of the IQs, right? And so now if we kind of think of this holistically as a story, we’ve got this Microsoft IQ. And I’m going to interact with these 0:33:6.15 –> 0:33:6.735 Nick Miller Okay, sure. 0:33:9.535 –> 0:33:10.15 Nick Miller Okay. 0:33:13.295 –> 0:33:13.695 Nick Miller Yeah. 0:33:21.775 –> 0:33:40.575 Brian Haydin these different intelligence, these different data, you know, information systems in a bunch of different ways. So here’s the story. I have a supplier event that happens, right? We’re in the Midwest, it’s manufacturing kind of thing. Something happens, I’ve got a supplier disruption event. That’s going to happen through, probably through some sort of an e-mail. 0:33:40.935 –> 0:34:0.295 Brian Haydin hey, we have a problem, can’t get this out to you. What are you going to do about it? This can be picked up by a Copilot Studio agent sitting inside of that Microsoft 365, right? It’s going to trigger a notification to you. You ask a question and you go over to your Fabric data agent, which is going to interact with that information through the Fabric IQ ecosystem. 0:34:0.735 –> 0:34:21.215 Brian Haydin But that’s not enough. We got to solve the problem, right? So I have agents that I’ve built out in the Microsoft Foundry that understands what the supplier terms are and how I’m going to move, you know, to a different supplier, et cetera. So that’s how all this is supposed to, you know, is it intended to work? And there are three different problems, three different solutions. 0:34:22.15 –> 0:34:42.415 Brian Haydin you know, and different platforms are going to use it, but all working inside of that Microsoft IQ ecosystem. So, you know, but, you know, I don’t want to say that, you know, these are disconnected systems, that reference architecture is not intended to be the only way that you can solve this. 0:34:43.375 –> 0:34:56.655 Brian Haydin And we’re kind of growing through, you know, through this as well. I know you want to talk a little bit about Agent 365 and I don’t know, we actually have like a slide on it. So let, yeah, why don’t we, yeah, yeah, that’s right. Go for it. Yeah. 0:34:52.735 –> 0:35:13.615 Nick Miller We don’t, we don’t. I could just, it’s very light, but the what where Agent 365 comes into play is you have copilot agents, you have Foundry agents, you have IQ Fabric agents, and maybe you actually have agents in other vendors. You have some Anthropic agents or you have some OpenAI Direct. 0:35:13.775 –> 0:35:21.935 Nick Miller agents. How do you get a view, a global view and a global management of these? How do you control token rate? 0:35:23.415 –> 0:35:41.775 Nick Miller of all these different agents. How do you control who has access to certain agents? Well, Agent 365 is sort of this overlayer that allows you to have that global view of all of your agents at one time. So not just the, like, because in Foundry, there’s an agent management plane, and yeah, you can see all the agents. 0:35:41.935 –> 0:36:6.575 Nick Miller that are in Foundry, and you can see any of their evals. So if someone’s trying to do prompt injection or whatever in your agent, you can see that in Foundry. You can see token costs or if there’s any issues with agents. That’s cool, you can see that. But what about your copilot agents? What about your anthropic agents that are outside of this? You don’t have that. So Agent 365 allows you to bring a view, one, a management perspective and observability of 0:36:6.695 –> 0:36:25.375 Nick Miller all your agents. And that’s really, I can kind of stop there because the exact details of how it works is another conversation. But I think when we talk about holistic or comprehensive, if you’re going to do this, you probably want to ALA Agent 365 over the top of it so you can get a holistic view of what’s going on in your organization. 0:36:7.775 –> 0:36:8.175 Brian Haydin Yeah. 0:36:25.935 –> 0:36:44.655 Brian Haydin Yeah, I would agree. All right, so we started off a little bit talking about some announcements. And I want to circle back to that just to sort of make sure that people are looking at it. Your agent broke yesterday and you got a bunch of alerts. Here’s a couple of things that got retired. 0:36:45.455 –> 0:37:6.815 Brian Haydin that I might go back and take a look at. It like wasn’t, I mean, I haven’t seen a lot of news around it. So if you were using the, you know, the assistance API, which we weren’t using a lot of that stuff, you know, but I know some people were using it. You probably got a couple of alerts today. Go take a look at that. 0:37:7.375 –> 0:37:24.575 Brian Haydin That’s more like the classic portal, kind of like, you know, Azure AI inferences, you know, dependencies. And there’s not a like a subscription level kind of discovery tool. You’re going to have to do a manual sweep of your deployments. So go take a look at that. And then these. 0:37:26.655 –> 0:37:45.535 Brian Haydin New things that are coming out, the Foundry workflows, you know, some of this stuff, this is the good stuff, right? That we’re going to be moving some of these flow workflows into, you know, agent framework, better capabilities, better traceability, you know, better usability. So put that on the radar as well. 0:37:46.335 –> 0:38:3.695 Brian Haydin And then, you know, maybe some point, you know, we’ve got to march around Foundry Agent Service, the classic agents. We’re going to have to do some work on that in a little bit. But those two retirements on the left side, that was yesterday. You know, make sure you take a look at your ecosystem on that. And then, you know, hey, 0:38:3.375 –> 0:38:23.695 Nick Miller Can I cover one thing on that real quick if we go back? So the Foundry workflows, so yes, that’s coming GA, but one thing, the opposite side of that is that the Foundry workflows, the declarative workflows are being retired. So if you do have declarative workflows, so 0:38:5.55 –> 0:38:5.935 Brian Haydin Yeah, yeah, go for it. 0:38:23.935 –> 0:38:43.55 Nick Miller In Foundry, you had agents, you had declarative workflows, and then hosted agents at a high level. And if you have an agent framework workflow, it’ll be part of a hosted agent. And so that’s awesome. Agent framework is the successor to semantic kernel. It’s well built. There’s a lot of great stuff in it. The move makes sense to me. 0:38:29.615 –> 0:38:30.15 Brian Haydin Yeah. 0:38:43.375 –> 0:39:2.895 Nick Miller But I had something that was built in declarative workflow, which is sort of borrows from the Power FX library, is a kind of Power Automate where you can connect things together. And because it was low code, it was a little bit painful to develop in at times. And it actually wasn’t fully baked out. 0:39:3.375 –> 0:39:22.135 Nick Miller The declarative workflows never really had all the capabilities of Power FX, which is part of the problem. Like JSON didn’t work. Imagine not being able to use JSON. That kind of stinks, especially for structured output. So maybe that’s why they’re retiring it. But if you do have a declarative workflow, make sure you retire it, migrate it over to that Microsoft Agent framework. 0:39:23.135 –> 0:39:42.335 Brian Haydin Yeah. All right. So Paige dropped the link in the chat. I would say first off, like feedback is a gift. You can use that to let us know what you thought of the content here today. Also, if you want to talk, you know, about any of the topics that we talked about, 0:39:42.655 –> 0:40:1.935 Brian Haydin Nick and I can hop on a call. That’s the mechanism to go ahead and reach out to us. But I would say like, hey, couple things like you should probably think about doing next week. First, do that sweep, check the dependency list, make sure that you have everything that’s running in the environment. And then if you’re interested in using the Microsoft AI, 0:40:2.15 –> 0:40:23.855 Brian Haydin models, make sure that you have a representative evaluation set. And if you need some help kind of like thinking that through, Nick and I can hop on a call and talk about it. But the idea is be able to compare like what is working today against if you switched to these other models, how does that compare? Is my token cost going down, my latency going up? 0:40:24.15 –> 0:40:36.415 Brian Haydin general improvement there to validate that. And then use that to actually choose the model and your context together, right? So a couple of things you could do. Anything you would add to that, Nick? 0:40:38.55 –> 0:40:39.55 Nick Miller That makes a good summary. 0:40:39.775 –> 0:40:46.335 Brian Haydin Cool. All right, well, I want to say thanks to everybody for joining us, and we’ll see you next time. 0:40:49.295 –> 0:40:49.935 Nick Miller Thanks.