View Recording: AI Compliance Without the Panic: A Practical Governance Primer

Jim “JB” Brown Okay, welcome everybody to a seminar or webinar today about AI compliance without the panic, a practical governance primer. I’ve personally been through these audits and I’m sure you have as well. It’s always fun. And so this is going to have a little bit something old, a little bit something new. 0:0:24.995 –> 0:0:45.475 Jim “JB” Brown and we’ll see if there’s something blue by the time I get done here. Just wanted to remind you, Concurrency, we’re a Microsoft partner. I would say we provide a lot of events and they’re posted on our website, so please check out. You can use the QR code here or you can go to our website to check out some of the things, but lots of thought leadership. 0:0:46.595 –> 0:0:49.235 Jim “JB” Brown and excited to share all of that with you. 0:0:52.355 –> 0:1:12.915 Jim “JB” Brown So today we’re going to talk through sort of an IT parable about AI Compliance Without Panic. And the story is about Alex here. He’s an AI product manager. And AI product manager is a hot role right now that we’re seeing a lot of interest in. You know, in this story, Alex is the person at his company who has to care about AI, who’s 0:1:13.155 –> 0:1:33.315 Jim “JB” Brown looking for adoption, business value, ownership, readiness. He’s worrying about the use cases and risk ownership and scaling the portfolio. Now he typically works with business owners who actually own the workflows and the agents, and he might work with a forward deployed engineer. So that’s another hot topic. 0:1:33.355 –> 0:1:52.955 Jim “JB” Brown topic we’re seeing today. You know, a forward engineer typically embeds with the customer, with the business owner team, and is accountable for taking that AI concept and making an agent that’s useful for the business. They pay attention to things like building, production AI systems, integrating with enterprise data. 0:1:53.715 –> 0:2:13.875 Jim “JB” Brown making it happen. And they may or may not be a data scientist, which is another pop role, which is someone who asks the question, can we model this? Is this something we can predict? If so, can we change it into an agent? And, you know, they might be looking at the data, but statistical analysis, machine learning, predictive models, and so forth. 0:2:14.195 –> 0:2:33.635 Jim “JB” Brown But these are the characters that we have today. Sara is the internal auditor. So she’s not her first rodeo. She’s been doing this for a while, and she has some great advice for Alex as he’s making his journey. And of course, there’s an agent in our story because we’re talking about AI. Now, the thing I want you to take away 0:2:33.795 –> 0:2:50.115 Jim “JB” Brown from Compliance is you want to focus on what’s important. And the three questions every AI initiative must answer is, what does it do? That’s data, access, everything. Who owns it? And then can we prove it? What’s the evidence? 0:2:52.595 –> 0:3:13.395 Jim “JB” Brown So AI product manager, Alex here, he’s been successful, which means things are happening at the company. A lot of adoption of AI, people are excited, and he knows that there’s an external audit that’s coming up for something that AI is now part of because it’s embedded in the workflows and the processes for that business line. 0:3:13.675 –> 0:3:25.635 Jim “JB” Brown whatever it might be. Could be PCI Compliance, could be ISO, could be whatever. And he’s excited, but he’s also a little bit overwhelmed because, you know, he’s been focusing on getting the job done. 0:3:28.195 –> 0:3:52.795 Jim “JB” Brown So Sarah calls and she’s internal audit and she says, hey, we need to talk about this. First thing Alex is thinking, okay, great, I got to work with internal Compliance. But actually, there’s A twist in the story because Sarah definitely knows how to work and enable the business. So she’s going to be a partner and she’s going to help him. And the way to scale and adopt AI is through governance. 0:3:53.75 –> 0:4:1.395 Jim “JB” Brown and reliability and all the enterprise controls to prove its value, but also to make sure that, you know, you’ve got your finger on it. 0:4:3.795 –> 0:4:25.75 Jim “JB” Brown You know, from Sarah’s perspective, she probably has a checklist a mile long, and she probably has 20 different compliance or rules or other things that she integrates. That can be of great value to Alex, as well as we’ll see a little bit later on. So Sarah comes and steps over to his desk and says, okay, hey, we’re here to help you scale this. 0:4:25.635 –> 0:4:39.75 Jim “JB” Brown The earlier you can get us involved, the better. So we’re shifting left, we’re moving up compliance so that Alex can build it in as he expands AI within the organization. And that can be a game changer. 0:4:41.875 –> 0:5:1.515 Jim “JB” Brown So Alex’s assumption, he thinks the conversation is going to be about what framework are we going to do? What are the controls going to be? You know, there’s NIST AI RMF. That’s A mouthful, but people are familiar with the National Institute of Standards and Technology. And then there’s the Artificial Intelligence Risk 0:5:1.635 –> 0:5:19.555 Jim “JB” Brown management framework. There’s a bunch of these though that are out there now and I put some links for a sprinkling of them. And so the frameworks are important. They help you provide, you know, an objective viewpoint on what you’re doing and due diligence and why you’re doing it. But Sarah’s not, that’s not where she wants to start. 0:5:20.835 –> 0:5:28.355 Jim “JB” Brown So when Sara says, hey, let’s just start this with, can you answer these three questions? What does the AI do? 0:5:29.555 –> 0:5:30.435 Jim “JB” Brown Who owns it? 0:5:31.555 –> 0:5:40.435 Jim “JB” Brown then we can talk about can we prove it. And so the reality check is Alex does have some work to do to answer these questions across the enterprise. 0:5:43.315 –> 0:6:5.155 Jim “JB” Brown So what does he do? Of course, he sets up an AI agent to go ahead and do compliance for him. Of course he does. He’s the AI product manager. So he asks AI to discover all of the activities that are going on and what are the characteristics of them. Now, can AI actually do this? Well, it depends on how they’ve set up their systems and their data and everything, because AI could do this. 0:6:5.475 –> 0:6:24.115 Jim “JB” Brown Or maybe Brian has to work with his business owner, Alex has to work with his business owners and compile the list. He might already have a list, but there’s almost always something not on the list. And so this is just an exercise of discovery, inventory. And by the way, if I could get Brian to do this at an enterprise level, 0:6:24.755 –> 0:6:34.115 Jim “JB” Brown I would trademark it, I would patent it, because this would be a billion-dollar idea, and I could go on TV and talk about things. 0:6:36.75 –> 0:6:55.315 Jim “JB” Brown So I’m going to move on here though. So the first discovery is that AI is everywhere because they’ve been successful. But you know, the type of AI they’re doing doesn’t require full compliance on a lot of things. And so there’s a couple different kind of spectrums here to think about. One is, you know, does it impact a lot of users? 0:6:55.515 –> 0:7:14.915 Jim “JB” Brown What kind of data does it use? Is it embedded into business? Is it customer facing? And so depending upon, you know, the type of data it uses or whatever, there’s a few parameters determining if this requires the controls. You know, so not all AI requires the same amount. Everything should have an owner. 0:7:15.475 –> 0:7:29.315 Jim “JB” Brown But, you know, this is something that the business has to determine, not necessarily Alex himself or even Sara, but they do shepherd the organization. You know, they’re part of the organization that helps people think through these things. 0:7:32.435 –> 0:7:41.75 Jim “JB” Brown Sara does take a break from her question asking and kind of talks to Alex to get some consensus around the framework. You know, and I’m… 0:7:42.960 –> 0:8:2.200 Jim “JB” Brown kind of mapping those questions, what does it do? There’s a map phase within NIST’s framework. Who owns it? That’s governance. And do they know that they own it? Because that’s really important to make sure that it’s transparent because a lot of stuff happens. There’s a lot of things that people… 0:8:3.160 –> 0:8:21.480 Jim “JB” Brown you know, are working on. So do they know that they own it? And then can we prove it? And that’s where we get into evidence and controls and other things. You know, so at this point, we’ve made a lot of progress in the story because Alex knows pretty well the answer to what does it do, that first question. And then what data, what access, 0:8:22.440 –> 0:8:27.560 Jim “JB” Brown And then he can help apply the framework accordingly based on, you know, Sarah’s coaching. 0:8:30.40 –> 0:8:54.440 Jim “JB” Brown So question #2 is who owns it? And, you know, an off the cuff answer might be, well, IT owns it, which everybody knows that’s not quite true, but often it falls to IT as the responsibility to perform compliance, if not ownership, because there’s technology involved. You know, Sarah’s coaching him saying, well, IT can enable it, but the business must own it. And really, there’s more than one 0:8:54.840 –> 0:9:8.680 Jim “JB” Brown human in the loop here. There’s an outer loop and there’s an inner loop. Ownership, you know, it includes risk authority, oversight, escalation, the ability to stop or change the use. You know, there’s lots of things going on here. 0:9:9.880 –> 0:9:30.200 Jim “JB” Brown So just want to reintroduce this topic about the humans in the loop. AI is a great tool, but it requires humans to be in the loops, and there’s actually multiple loops. In this example here, just illustrating that, there’s this inner loop, which we think about, right? That’s the use of it, the workflow of it, the business owner. You have to make sure that it’s reviewed. 0:9:30.680 –> 0:9:49.800 Jim “JB” Brown and even do evaluations and testing to make sure after changes, or even just that the model hasn’t, the results haven’t strayed from micro changes in the environment or whatever. You also have to keep it up to date. So if I have knowledge coming in and I carefully curate that knowledge, 0:9:50.520 –> 0:10:9.400 Jim “JB” Brown works great, the more time I spend on that context and that grounding really can create impactful results in AI. But things change, the standard operating procedure might change. And you don’t want both versions contradicting the agent. So it requires maintenance on the way in. 0:10:9.800 –> 0:10:29.0 Jim “JB” Brown adding new data and also requires taking out old data and other things. So there’s maintenance to do being that human in that inner loop. In the outer loop, you know, you could coin this as a COE, but you know, it’s IT, it’s governance, it’s asking a little bit different question than, is it working right? It’s asking, should this exist at all? 0:10:29.80 –> 0:10:48.760 Jim “JB” Brown and who owns it and making sure that it’s sifted through so that the organization has awareness of it and ownership is set and so forth. So you’ve got these loops that you could probably, based on your organization, you might have two or three other kind of loops as this integrates into the enterprise. But the thought is, who owns it? 0:10:49.720 –> 0:11:9.400 Jim “JB” Brown from a business standpoint and is it registered and is it, you know, controlled by the organization? That’s a really important question. Because when it comes to compliance, often you don’t tell the owner exactly what to do, but you say, hey, we’ve got this compliance. These are the controls that are required. 0:11:9.800 –> 0:11:30.560 Jim “JB” Brown how do you want to do this? Because they understand the system better, they are the owner, and they can really direct exactly how compliance is met, either through controls directly or through compensating controls. Didn’t really define controls, but I’m guessing most of you understand that, you know, control is something that gives the organization 0:11:30.840 –> 0:11:49.400 Jim “JB” Brown a way to control, I shouldn’t use the same word, but allows the organization to manage an area according to some standards and produce evidence that the control is actually working in order to satisfy some kind of compliance item. 0:11:49.880 –> 0:12:9.200 Jim “JB” Brown Often, you can’t actually meet the control as it’s written, and you have to put in other things, compensating controls that will achieve it. Whatever those things are, you know, in this story that we’re working with, Alex is working with business owners, for deployed engineers, other people, to have them tell him. 0:12:9.640 –> 0:12:13.960 Jim “JB” Brown And ultimately, Sara, how they’re meeting compliance for what they’re working on. 0:12:15.320 –> 0:12:34.280 Jim “JB” Brown So this brings us to the third question. And third question changes everything. Really? Yeah, it does. And that is, can I prove it? What’s the evidence of it? Because that’s where we’re meeting the auditor. They’ll come in with a big checklist and they’ll be asking for you to go through screen snaps or… 0:12:34.440 –> 0:12:39.480 Jim “JB” Brown Records or logs, and they’ll be looking for evidence that you actually have the controls in place. 0:12:41.80 –> 0:12:56.200 Jim “JB” Brown You know, so Sara comes back and she says, hmm, you know, this is the compliance kicker. Can we prove that we’re doing all the things we need to do to maintain AI? And Sarah’s point, we have to generate, you know, that evidence as part of her oversight. 0:12:57.320 –> 0:13:16.120 Jim “JB” Brown But the good news is we aren’t starting from zero. Every organization has compliance of some degree, and we can map right into that. The traditional controls for network and policy and all sorts of things, including code reviews, still stand. Those things are still there, and hopefully that heartbeat is continuing. But with AI, there’s 0:13:16.360 –> 0:13:36.640 Jim “JB” Brown several other different things we need to map into it. There’s this identity now of an agent, and it has an identity. It’s separate from a person. So it’s like a new class of users in the organization. What permissions, what authority, how is that reviewed? All those types of things. It has data and grounding, and there should be control over that data. 0:13:37.800 –> 0:13:56.600 Jim “JB” Brown as we discussed in maintenance and so forth, and evaluations of its behavior, traceability, all sorts of different things now will be mapped into your controls. Some of our existing controls might satisfy that checkbox. Others, there might be a new category of controls that need to be 0:13:57.160 –> 0:14:13.240 Jim “JB” Brown you know, evaluated and managed and so forth. So good news is we already know how to do compliance. It just requires a little bit of extension, adding on these AI, you know, whatever the framework is telling us, AI needs to be controlled in certain different ways. 0:14:14.840 –> 0:14:33.480 Jim “JB” Brown So from controls to proof, this is where the agent comes in, a billion dollar idea for Brian to go and do all the gathering. And if you’ve participated in these teams, you could have 100 people participating and providing evidence. Typically it boils down to a few people parading through a conference room over a day or two. 0:14:34.400 –> 0:14:45.800 Jim “JB” Brown where you actually show what the auditor asked for and only what the auditor asked for. And in this case, our agent, Brian, is providing that capability. And so… 0:14:48.120 –> 0:15:8.760 Jim “JB” Brown This is an example of just stepping out of the story for a second, but of a customer that we worked with. And from their perspective, they’re successful in deploying AI. They were just missing sort of a registry capability or a digital employee directory, if you like. And so we worked with them to help define the process. 0:15:9.0 –> 0:15:28.360 Jim “JB” Brown define what information was required based on their business requirements, and give them sort of that handle on their AI environment where they could make sure everything has an owner. They could make sure they knew what it did. And then they could make sure they knew what the evidence and the controls and everything associated with it, and that organizing capability. 0:15:28.880 –> 0:15:48.400 Jim “JB” Brown What’s interesting about this is that there’s a lot of development going on, and many of these things, like an agent registry, are being built into systems and other things. So just depends on what platforms you’re using, what your stance is, what you’re protecting, and but, you know, developing these tools and other things. 0:15:48.920 –> 0:15:51.0 Jim “JB” Brown helps you maintain that control. 0:15:54.600 –> 0:16:15.0 Jim “JB” Brown Okay, so, you know, we’ve gone through quite a bit of the process now. So Alex has been educated. Sarah is feeling comfortable about the new technology. And of course, we’ve got Brian working like a charm, like a champ, gathering evidence and being prepared. And so we’ve got one team performing together. 0:16:15.320 –> 0:16:28.600 Jim “JB” Brown like Compliance does, you know, each performing their role. And so we’re feeling, we’re feeling actually okay about this, less panic. And now it’s time for the external auditor to come in. 0:16:29.800 –> 0:16:50.360 Jim “JB” Brown So one of the audits I participated in, there was a very good auditor and she brought her dog to the audit, which was kind of interesting. And you know, you could think maybe that’s good. Maybe every audit should have a dog participate. And they come in with their checklist and they ask, do you know where AI is used? Do you know what data? 0:16:50.440 –> 0:17:11.240 Jim “JB” Brown it’s using? Do you have control over it? Are there any gaps? Show me and provide evidence of what you’re doing. And of course, the team has been working on this and they’re working together as a team. So, you know, they can answer these auditor’s questions. And maybe there’s two or three things that get popped up that can be remediated quickly. 0:17:11.480 –> 0:17:15.80 Jim “JB” Brown Hopefully, that that is the case, and they’ve successfully passed the audit. 0:17:16.200 –> 0:17:34.280 Jim “JB” Brown So AI is a new technology. It fits into your controls and your compliance. You just need to make sure you identify what due diligence you want to do and what resources you want to put those towards based on whatever risk level, value level, 0:17:35.160 –> 0:17:39.560 Jim “JB” Brown data sensitivity level that your compliance requires. 0:17:40.520 –> 0:17:59.400 Jim “JB” Brown So we’re nearing the end of our story here, and you know, wanted to put up what each of these people have just kind of reflect on and learned. Alex is now realizing that adoption can be slowed down if you don’t have compliance. And now he sees that to scale this, he’s got a lot more pillars to stand on. 0:18:0.40 –> 0:18:18.680 Jim “JB” Brown having passed the audit. And so there’s less fear, uncertainty, and doubt, and he’s feeling pretty good about adoption. Sarah’s, this is a new technology. Great. I kind of knew this going into it. She’s thinking the discipline’s the same. I just need to map the controls, make sure people take accountability, and I’ve got everything I need to do that now. 0:18:19.160 –> 0:18:38.760 Jim “JB” Brown Brian, he can find, map, and organize. Humans still have to decide and remain accountable. So that human in the loop, if you use agents to do this particular function, whatever that may be, but Brian, you know, was a very useful part of this. And me, I personally, my little end here is I 0:18:39.80 –> 0:18:52.760 Jim “JB” Brown patented this as Arthur Auditor and I made bank. Because if you could get an AI agent to do compliance for you in an organization or even just help, wow, that’d be really, really cool. So, you know, governance helps to answer the questions. 0:18:53.880 –> 0:18:57.240 Jim “JB” Brown and it’s the framework and Compliance stands on that. 0:18:58.360 –> 0:19:3.480 Jim “JB” Brown And it requires the organization to prove the answers. And so you have that wrapped up in there. 0:19:5.240 –> 0:19:25.560 Jim “JB” Brown So, you know, this was my story about compliance with AI and wanted to leave you with these closing thoughts. It’s really the same game as PCI Compliance or maybe HIPAA. You know, every one of those frameworks are written differently, they’re interpreted and so forth, and you still have to do all of those things. 0:19:25.880 –> 0:19:45.840 Jim “JB” Brown but it maps down to what does it do and what kind of data and access, who owns it, can we prove it? And then you put the controls in place based on your framework of choice, whatever that may be. I wanted to, before I open it up for questions, just wanted to include a couple big pictures in the video here, but 0:19:46.120 –> 0:20:5.0 Jim “JB” Brown A lot of these controls are being built in to the system now in terms of a Microsoft kind of focus, but I know other companies are working on the same thing. You know, evaluations, the ability to have an agent registry, the ability to identify owners by identity within 0:20:5.160 –> 0:20:24.520 Jim “JB” Brown the enterprise. All of that has been built or is being built, and I think it’s come a long way. And I do think it’s ready for, you know, utilization within the enterprise. You know, another way to look at a mapping of what Microsoft’s been doing in terms of what products and what layers 0:20:25.480 –> 0:20:44.320 Jim “JB” Brown You know, Microsoft has been providing a lot of this for a very long time, especially with Enter ID, but it’s just expanded to be so much more for the agent control plane that Microsoft is working on. You know, culminating kind of in this latest phase with E7, moving from the past where you just had people, 0:20:44.440 –> 0:21:5.440 Jim “JB” Brown to, you know, today where you have assistance and every, you know, people are interacting with AI to everyone having agents. And how do you secure that as a control plane across the organization with all those identities? And then this last picture, just a little bit different view, similar of the same thing. So, Amy, you know, at this point, we can open it up. 0:21:5.560 –> 0:21:10.120 Jim “JB” Brown if there’s any questions out there before we close. 0:21:14.200 –> 0:21:15.720 Amy Cousland I don’t see any questions. 0:21:16.280 –> 0:21:37.40 Jim “JB” Brown Great. All right, well, thanks for story time, sharing story time today with me. Hopefully you have a little bit clearer sense of, you know, it’s really starting with answering those questions and, you know, who owns it is the number one question and keeping track of that and then tying into your compliance program. 0:21:37.400 –> 0:22:0.200 Jim “JB” Brown So thanks for your time today. Really appreciate it. And if you, you know, want to share feedback, please scan in the QR code. That will be available on our insights webpage. And now if you’re looking for a next step, just give us a call. We can have a conversation about your circumstance and see, you know, what we think might help you in the current phase that you’re in. 0:22:0.520 –> 0:22:2.680 Jim “JB” Brown So really appreciate the time and have a great day.