View Recording: Intelligent Document Processing with AI

Hello, welcome everybody. You’re in the Intelligent Document Processing with AI webinar. We’re going to talk about workflows, tools and integrations and validation techniques for AI document processing. My name is Nick Miller. I am an ML architect at Concurrency. And first, we’ll start with a very brief intro to who I am. I started off my career with an MIS degree from UT, and from there I went to the Navy. I was a logistics officer in the Navy, did that for about 10 years, got out and wanted to speak civilian instead of just militaries. And so I got an MBA from UT. And shortly after I got married, you can see my wife, Heather, there on the right. And I predicted that we’d need more money because we’re gonna be having kids and all that sort of stuff. So I went back and got a master’s degree in statistics. Really, the impetus for that was that I really I took a few classes in the MBA program that really exposed my love of data science. And I realized I couldn’t get a job without a at that time, at least as a data scientist without a master’s degree, minimum in statistics, math or econ. So I went back to A&M and got my master’s degree if you’re from Texas. You know that UT and A&M are rivals, and so I’m not really sure if who to root for on game day, but it’s all great. Both schools were awesome and a good experience. There’s my family there. I’ve got two kids, Sadie and Bam, and two dogs, Olive and Layla, and my wife is adamant about matching Halloween costumes. This year we’re gonna be the minions. And then probably the newest aspect of my life, especially one that I can nerd out with numbers on, is golf. And so anyways, that’s a little bit about me. Thank you so much for joining today and I think we have some great stuff to talk about. Before I do, want to let you know that we’re offering a complimentary intelligent document processing session where you’ll meet with us to discover how AI can streamline your document workflows, talk about your specific use cases, any specific questions you have, get some practical insights. Kind of you know this webinar is kind of a high level overview and you may. Have you may be further along or maybe not quite as far along. So you mean we can really kind of tailor that session to wherever you and your company are at at that time and just let us know. We’ll later in the session we’re going to drop some booking links in the chat. So if this is interesting to you, if you’re not sure you’re kind of think it might be useful. Let’s meet up and we’ll find out. It’s no risk meeting. All right. So now that we have the options out of the way, let’s talk about the agenda. So today we’re gonna talk about the challenges in document workflows. If you’re here, you probably have some good ideas already with. Those challenges are then we’re going to talk about some workflows or common scenarios of document workflows that you might see and we’ll talk about some of the commonalities and things that that are that are common between different workflows or scenarios and different and how we can handle that. And then we’ll talk about one of the key components of that workflow, which will be the AI document intelligence tooling from Azure. And then we’ll give some demos of using that tooling. And then we’ll next talk about some tools and integrations and validation techniques and then we’ll demo some of that and then we’ll close and of course anytime. Please ask questions and then if you ask a question and I’m not answering, hopefully Amy will will answer the question, not answer the question, but will let me know there’s a question because it’s kind of hard to see them sometimes in the chat where you’re presenting. But please ask questions if you want to know anything or something does not make sense or you want some clarification. So first, challenges in document workflows. As digital as we are, as advanced as our ERP systems are and SaaS and all this sort of stuff, we still have the siloed systems between companies. Even intra company communication is often with documents. Our customers require compliance documents from us. They send us purchase orders in e-mail. We have contracts in PDF that are not necessarily accessible or searchable. We get payments from our customers. You know, there’s there’s receipts, there’s all sorts of documents that are just being used for communication, for communicating business. This information today, they’re not going anywhere anytime soon. So that’s I guess a good thing and a bad thing. So the bad thing is it’s a challenge and the good thing is we have a solution. The other part about document processing workflows is unless you’ve already got that, unless you’ve already implemented the things we’re talking about in this. Webinar, then the way you’re handling those documents is very likely manual, right? So you have someone looking at the purchase order, you have someone looking at the ACH payment details where your customers have paid their invoices, but you don’t know which invoices the ACH payment even belongs to. So you know, it’s a manual process or maybe you have what you call a expense policy. People submit receipts and you have to look at the receipt and make sure that they follow the company policy. Well, that’s costly, takes time and and. You know the the employees time is not free and also it’s error prone, especially if you’re extracting information from those documents and not just reviewing for for policy validation. The other problem or challenge I like to I like to think opportunities for excellence, right? So instead of a problem, it’s a challenge. An opportunity to excel. So another opportunity to improve is the fact that there’s so many different formats, right of documents. And when I say formats, I mean the actual file format. And then within the file format, let’s say we’re talking about purchase orders. Do your customers all use the exact same format in their purchase order? And the answer is no, right? So it’s the old school document processing was very look at this form, find this thing. And when your forms were very, very uniform, it worked. But that’s not the real world. So that’s another challenge. And then the other problem here is that there is an A I skill shortage and it’s kind of hard to find folks with A I skills. There are a lot of democratized A I toolings, but still it’s a challenge. But the good news is we’re going to give you solutions. We’re going to demonstrate how you can overcome all these challenges and show those opportunities for excellence. So after you get the challenges, what are some of the scenarios or workflows in these? Now these are high level and they’re kind of generic and you can they can be kind of built into lots of things. So in document processing this is very generic, so you there’s a document input. You extract some data from it and you take action. Now an example of that might be a PO process. So you receive a purchase order from a customer. That’s the document input. You extract the data. Right now it’s manual, so you’re looking at it. What’s the PO number? What do they want? Type, type, type, type, type, type. Enter the purchase order in your ERP is taking the action. Right, so that’s an example of document processing. Knowledge base management is fairly similar, except instead of taking that data and doing something in the RP system right away or triggering off some process. You’re actually extracting the data so that you can build a knowledge base so that when you have multiple documents, you can ask questions and get information across the entire knowledge base, right? So you can think of this maybe as. Let’s say you have some company standard operating procedures, right? You have standard operating procedures and you have customers and they ask you questions and they say, well, I want to know what your policies are. They give you a survey. What are your policies for XY and Z? And of course, you probably don’t want to hand over your standard operating procedure because, you know, that’s your secret sauce perhaps. And so, but you do want to let them know that you have policies and procedures in place to mitigate the risk of whatever it is they’re talking about. So they say a survey, tell me how you handle such and such risk. And you would extract the information from all your SOP’s and then you encode them and store them in an astage base. Oftentimes those are vector store vector databases like AI search and then you would use an LLM or some other process that would look through that database and. Find the relevant information related to that and summarize the data in such a way that you can answer the question without sending them your SOPS, right? And the first part of that is knowledge base management. Now what you do with it later, there’s different ways of doing that, so I left that off. These two things are like, in my mind, they’re kind of the most generic foundational things. Now there’s different flavors, there’s different tooling. You may take the action, may be part of an application or a workflow or something else. So there’s a lot of options there, but at its heart, these are the two. Primary workflows or scenarios that we see. Amy, do we have any questions yet? Amy Cousland 9:27 Not yet. Nick Miller 9:29 All right, great. OK. Just wanted to check. All right. So what we’re going to do is we’re going to give a, we’re going to go through AI document intelligence. Now in that previous slide, if you remember, and actually I’ll go back, extract data and extract data, right. So they both have this input and they both have this data extraction piece. Now the primary tool, at least a first step for data extraction is going to be Azure’s AI document intelligence tool. And so we’re going to give an overview of that and the name is changed and AI is really awesome. And so you know, everything that Azure offers is now called AI something or other. And that’s it’s a little bit of a marketing thing, sure, but there is actually these days there is AI embedded in the document intelligence tool, especially in some of their custom models that you can build and also in some of their pre built. So they pre build these models and pre train them and. There’s some AI behind the scenes in those models as well, and we’ll talk about that. So there’s there’s AI built in and it’s not just a marketing gimmick with that AI document intelligence resource. Now in general, there are two types of models. There’s the prebuilt and there’s the custom. The prebuilt, as the name sounds, it’s ready to go out the box. You access it with an API. The custom model, you have to provide training data, you label it, you perform a training session, and then you use the model, right? And so that’s the difference between the prebuilt and the custom. Now once the model is built, you still access it via an API. So they’re both API driven. And so because they’re API driven, you can integrate them into your work processes or into your workflows. The first one we’re going to look at is the layout, all right? And the layout is we’ll just actually look at the website for that if I can. Ull that U. To find my right website, OK. All right, so this first. Uh, when you are in document intelligence. The first thing you see is you see your document analysis. Now it doesn’t call these pre-built explicitly, but they are pre-built. If you look at the documentation, they’re pre-built, but these are kind of generic ones where you have a layout OCR read. This is just OCR data layout. They actually extract structure, general documents in your pre-built, right? So you’ll see that. And. The so I’m trying to resize this. Pardon me here. Hey, there we go. It’s a little bit better. All right. OK, here we go. Sorry about the delay there. So in layout and maybe I need to zoom in a little bit, that might be a little small. So in layout you you add your document and what it does is it gives you it extracts some standard structures. So instead of just, you know, taking the PDF document or whatnot and just extracting a bunch of text, it gives you text, it gives you lines of data, it gives you page information. It gives you tables, et cetera. So let’s say we’re looking at this receipt document here. Now there is a prebuilt receipt. We’ll talk about that, but just for fun, I’ll show you this. So look at this Denny’s receipt. There are different items in here in the content. There’s text and it shows you all the text that’s in there. It says Denny’s. It gives a paragraph, I guess it’s a paragraph. And let’s see what else. It gives you kind of table 72, server, etcetera. So it gives you the text. And there’s selection marks. This one doesn’t have any. And there’s tables and then they can be figures. A figures are like images and it could be charts or whatnot, but basically images in there. And then from the table you have two tables. Your first table is are the items on the receipt and then the second table looks like the total, like the bill total. Tables. You have your subtotal, tax, gratuity, etc. And then the total. So this right here, this is layout. This is a prebuilt model, fairly inexpensive. So you can run this via API and get this text output and that may be all you need. It depends on the document, it depends on. Your solution or what you’re trying to do. You may just take your documents and use layout to extract all that form, and you may take all the text, maybe page by page. You’ll extract that Jason file from. This output and you’ll get for page one, you’ll get some text, you’ll get a table, you’ll get a figure, whatever’s in there, and then that whole thing becomes a document. The text becomes a document inside a vector database, the the table becomes a document inside a vector database, and then the figures become a document. Document inside that vector database. And so then your document is now searchable. You haven’t really enriched it. You haven’t really done much to it except extract the data and put it into a vector database. And maybe you encoded the data where we’re talking about that knowledge management, right? So you used layout and you encoded it and turned those texts into vectors. Vectors, which is basically just a series of numbers and the LLM’s use it to find matching. So when you ask the vector database something, it encodes your request into a vector. It finds other vectors in the database similar to yours and then returns the plain text backs to the model. And it maybe like 34520 of those just depends on what you want. And then it takes that selection of returns and then it summarizes what it finds, right? So that that may be all you need and that’s totally cool. So that’s the layout and so you’re getting the text, the tables and the figures from that. OK, now one step up from layout is general, right? And so we go to general. Document. So what general document does and this can also be used. You can also use this in the same way that we just spoke, right? You can parse this and push this into a. A knowledge is not a knowledge base database. It gives similar things. It gives text, it gives selection marks and it gives tables. The difference though is that what it does with general is it it you can get key value results if you want and so it’s a little bit similar there but. Notice that it kind of gives a little bit of a different of a table. It gives you uh. It guessed with the key value pairs that the server was straw, so it’s it doesn’t always perform really well, right? And that’s OK like it was trying. This is showing you what works and what doesn’t work, right? So there’s no tables in the CVS Pharmacy one. How about here in this California Pizza Kitchen one, do we have a table? No, no table. It’s and it’s supposed to extract tables and it does. However, it just it didn’t. It doesn’t always extract them very well, where if you look at the layout table, it tends to to output those tables a little better so. It’s just kind of up to you what you want to use there. So that’s the general. And then we get into the pre-built specifics, all right. And a specific pre-built and there are a lot of them. And actually I’m going to go back real quick and show you some of them. There are, there are a ton now they don’t have everything you need and and. To be honest, no one has ever asked me to build them a solution using a pre-built model, probably because they feel like if there’s a pre-built model they can do it themselves. But you know, I don’t know. But that doesn’t mean they’re not useful. They are useful. You have invoices, receipts. Identity documents, pay stubs, I guess mortgage information, marriage certificates, contracts, business cards, right? So there’s cool stuff in there for sure. But what if you want to extract ACH payment details? So you want you receive a document. With a ACH payment reference and it has a whole bunch of invoice information on it, and that’s how you’re going to find out which invoice payments have been paid by that ACH payment. Well, it doesn’t exist. There isn’t a prebuilt model for that. You’re going to have to build that yourself. And so while there are a lot of options, it doesn’t have everything, but if it does have it, it’s pretty good. And so let’s go check out this prebuilt receipt model. And so you look at this, the Denny’s. So what it does, instead of just extracting the data, it has a schema in mind, right? So if you’re getting, if you’re if you have a receipt model, you might imagine, well, if you’re getting receipt, what is the schema? You’re probably going to have a merchant, right? And and maybe the address and there’s going to be items you buy and quantity. And quantities and prices and like dates, totals, tax, things like that, right. And so you see that it has that, it has the merchant address. So we didn’t have to tell it it was a merchant because it’s a receipt, it just knows it’s part of the schema. So that’s what that pre-built means. So you get the the merchant address there. It’s in Las Vegas in the city. So you get this, you get. If you extract this and you say give me merchant dot city, you’re gonna get Las Vegas. So it gives you a lot more structure, more reliable data. So if you do want to have a. A business process or workflow where you’re taking this document and you’re extracting the data. It gets pretty good results right now. Is it perfect? It is not perfect. No, nothing’s perfect. So I don’t want to set the standard or the bar of perfection. But you know, there are little things out here on the receipt like this chowder Hut receipt. They got a Coke and a spicy fish. Sounds pretty good. And then it subtotals it for food and beverage, which you know, I guess would have been more impressive if they had more than one item of each. But anyways, subtotals food and beverage, then tax. Maybe the tax is different based off of food versus beverage, I don’t know. And then they add this little thing here, SF, it’s in San Francisco, SF like employee mandated something or other. I don’t know what that means. It’s some sort of government fee after the tax. And so if you put in some validation rules and we’ll talk about. Validation rules later. A reasonable validation rule was for every item in here added up and that’s the subtotal. And the subtotal plus tax and tip should equal the total payment amount. And if it doesn’t extract this information, you’re gonna be off by $0.09. You’re gonna flag this and be like. It’s there’s something wrong with this receipt and 9 cents. Maybe that’s minutiae and you don’t care and that’s totally fine. But the point is just to demonstrate that even though the receipt thing is awesome, like it doesn’t, it doesn’t have like mandated government fees necessary in its schema. Or maybe this 83 was on top of this and it messed it up. I don’t know. But regardless, it’s not in there. And one of the cool things though is you look at this table of items and you expand this table and it says the description of this item was they got a Coke. It was a quantity of one and it was $3.09 USD. That’s pretty good. Accurate. Spicy fish one and 1629. Pretty good, right? So you have this pre-built model of the seats. Pretty nice. Yeah. Oh, sure. Amy Cousland 21:31 Nick, we did get one question. It says is there a reasonably straightforward way to determine whether your document data extraction requirements could use canned out-of-the-box capabilities or will require customization? And also is there an in in in between? Nick Miller 21:48 Yeah, that’s good. Yes, there is. So the way that you know is you, it’s really a business question, right. So there’s nothing to say that you have to do a validation rule that matches your subtotal plus tax and tip with the total. But if you have a business process where the document has information, well, one easy one is when you’re looking at the prebuilds, is your business document, is it one of those prebuilds, right? And if it is, then you’d want to do some experimentation. So like if it is, let’s say an invoice. Or a purchase order or something like that. If you get that, you get invoices. Let’s say you have invoices from you’re on the receiving end of an invoice and you get invoices from 10 suppliers, like maybe your 10 most common suppliers. If you throw those in there and you look and you you analyze them in here in the UI, like just do a little experiment, little test. In data science, there’s this concept called no free lunch, and it means that you don’t get anything for free without a little bit of work. Now, AI makes that less true, but still. So you take those 10 documents that are somewhat representative of the invoices you receive, you throw them in there, you do this analyze just like we’re doing here in the UI, and then you look at it and is it getting? The things that are important to you. Now what’s important to you? It may be things. Everything may be on there, but there may be a thing that you want to extract from that that that it doesn’t work. We’ve had invoice information from clients where. The the table said the table said invoice number and it was the date and then the columns of date was the invoice number. Rich, you know they goofed that up. Now would this handle that? Maybe, but we’ll talk about some other ways to handle that at the end. When I show you kind of my preferred method, although these are really good and I’m not disparaging these at all. So it’s really just a little experimentation. So you have to take 30 minutes or an hour before you move some big business process to this and invest some time and money into it. Do a little experiment. So if it meets your needs, great. If it doesn’t, there’s one more. Option and that other option is custom and we’re gonna talk about that as well, right? And then the reason I showed you layout is that you may have a use case where you just want to take the text out of the document and put it into a database. And that’s totally fine. And if that’s the case, you can use that layout pre-built and just save the page number and all that stuff. So it really is driven by the business use case and the business value you’re trying to provide. We’ll also look at layout a little bit later because at the end of the program I’ll show you how I use layout along with. A large language model to perform similar extraction but add a little bit more. Does that answer the question? Amy Cousland 24:36 We’ll get more. I’ll let you know if we get more questions. Nick Miller 24:39 All right. OK, great. Thank you. All right. So this is the receipt. So we covered pre-built. So we covered layout, we covered general and we covered one of the specific pre-built of receipt. And the next thing we’re going to look at is custom and in custom. You have different models. Let’s see here, label data. So I added some receipts. So I’m focusing on receipts today just so you can see the difference, right? So if I did different documents for each one, you really wouldn’t be able to see performance characteristics and performance differences between the different models. But in custom there are two options. There’s an extraction model which does not use AI, and then there’s a neural model which does use AI and the guidance there. And this is in the documentation of course, but when you’re looking at the two if the. If the if the layout of the document is not a standard pre-built document and and the form and and the layout itself is uniform, right? So most likely it’s an internal document. So if the the the internal document is uniform, then you can use an extraction model, right? Now there’s a little bit of a caveat to that, as in, let’s say that internally you have two flavors of this of this form, but they’re the same purpose, but they’re two different flavors. You can do what’s called a compose model and you build one model for one flavor, extraction model for one flavor, another extraction model for the other flavor of document, and you put them together with a compose model and it uses a classification model to decide. Which of the two flavors of model to apply, right. And so that’s that’s the composed model. And so remember how we talked about AI being built into it. Well, one, the extraction model doesn’t have AI, but when you use the classification to decide in that composed model to decide which flavor of extraction model to use, that is AI. So you just give. You basically apply because they’re the same, right? Let’s let’s say it’s a, I don’t know, a quality report. You have some sort of manufacturing process and you generate a quality report and there’s like a quality report for this type of product and product A and product B. But you just wanted one. You have one process that that that takes all the quality report processes at all. You can have a composed model, send all the documents to that, it’ll decide which flavor to send it to and then it’ll process it that way, right? So that’s the custom. And then the the other option is if your if your form or what is is not available in pre-built. So that’s step one if it’s not available there or pre-built doesn’t have everything you need and the layout. Is not uniform, right? So most likely external documents coming to you, so invoices sent to you by your suppliers. It would be nice if they all sent the same form, but they don’t, right? And so in that case you’re going to want to use a neural model. So you can kind of think high level extraction model custom extraction. Is uniform, uniform, but unique, right? So not prebuilt and neural model is not uniform and unique, so not prebuilt and different types of flavors from your customers, different types of formats. And so the neural model is what we trained here. So because we trained the receipt model, our receipts. Now I’m violating my logic. I know because it is in pre-built, but let’s pretend it’s not in pre-built. But then is the second thing, is it unique? Are the receipts unique? In and of themselves, they are right because they’re external documents. I don’t control the creation of the receipts and so therefore they’re a large different variety of items. So I chose to use a normal model because that’s the one that you should use because it adapts to the changes better. And so on that note. You’ll see here we did similar things. We got, we got the dining, we sorry, we extracted. Let’s look at the maybe the two ones that are similar. We got coffee, Pizza Kitchen. Chatter. Oh, here we go. Our our Coke and spicy fish. All right, so we got our spicy fish. We got our Coke. It got the date and time. It did not get the server. It did get the address and it got the tax and it got the payment, right? It didn’t get the additional fees. Additional government fees or whatever. So OK, I didn’t get that. No big deal. This one a little bit hard to see. So you got your this person is eating a breakfast of champions or maybe they’re just not being nice or they’re just hungry, but they got two. It looked to me, I’m guessing. Little Debbie Oatmeal Cream Pies and then Pepperidge Farms something or other oat raisin cookies. I don’t know if that’s totally right, but just looks out the price like a single Little Debbie’s cookie and an oatmeal raisin cookie package for 5 bucks. Seems seems about right. Right. Anyway, so extracted that and that was thanks to Robert over there in Las Vegas. So he extracted the date, the time, the merchant name, received this pharmacy. So it got a lot of the same stuff. Now arguably, in my opinion, it didn’t get. All the stuff that the pre-built receipt got and some of the things that I mean these pre-built receipts in my opinion are pretty cool. So look at what I’m seeing here. Merchant name receipt type. It’s a meal. Well, that’s kind of cool, right? So if there are different types of receipts. Maybe it’s nice to know that and so based off of the receipt type, maybe you have downstream process that handles them differently. I don’t know. So that I thought that was kind of cool. So it realizes it’s a meal. So this is the custom model and the way that we do that is. You just you have to have data in a storage account and then in that storage account you’re going to load that data and then you’re going to build a new model and I call it customer seat, this one right here and you can build a composed model from this. But we label our data and then we put the model in here and we did training for it and then you can test it and see how well it performs and then you’re good to go. And that’s then that model is available via an API. And then if you have one or more, if you have more than these, you can create a composed model. And so that really right there is the overview of. The overview of AI document intelligence, right. So now if we go back, I want to go back to the presentation. Let’s see. All right. So if we go back here, remember, right, document processing, extract data, that middle piece of extract data is the. AI document intelligence, but there are other steps or other things you can do to extract data, right? And so I wanna talk now about some relevant tools and integrations. So at a high level you have four steps, source, orchestrate, extract and store and. Have you ever used some of these to orchestrate ones like Azure Data Factory, that whole factory looking thing on the bottom that has source and syncs and things like that. And what I mean to say by this is that you can kind of choose. You’ll use one or more in each column or maybe not depending on the workflow, right? This is kind of the. I wouldn’t say the universe, but at a high level, this is the universe of of procode tools for these workflows. So the top one, if you’re not, if you if you don’t live and breathe Azure and Microsoft and you don’t have these icons memorized, which I don’t. I had to, I had no, I had to Google this one. I didn’t know what it was. To find it, but this is Graph API, so Microsoft’s Graph tool. So if you want to. So for example we had a project where we were monitoring an e-mail inbox and whenever a new e-mail showed up, if it had an attachment we would extract the attachments and then do some further processing. So that was a source of the attachment. And actually we saved it because we wanted those for later. We used, we used Graph API, pulled the PDFs off, saved them into a storage account here, and then that would kick off some further processing, right? So do you source your data from here? There are other options for source. This is not the only thing possible. But these are some typical, I would say some typical sources. So this is where the data is coming from. It could be an application, it could be a data. There’s lots of things, but in general this is what we’re seeing. Next up is orchestrate. Now there are different ways to orchestrate that top icon. That’s AI Foundry. You can use either like semantic kernel process, which is a semi structured process that you can embed LLMs in. So if it’s a business process that works the same way every time, you can create that process and embed those LLMs in there. You can use a. You could also use a. Multi agent workflow as an orchestrator and where that multi agent workflow has access to tools and maybe some of those tools call document intelligence API’s or some other things that we’re going to talk about later and maybe those are embedded in Azure function even who knows or it’s an MCP server, there’s lots of options there so. You orchestrate the activity with that AI Foundry agent service, and there’s different ways to do that. Or you use Azure Functions. So maybe the Azure Function is watching a storage account and then based on what shows up, it takes some actions and that’s another way to orchestrate and that’s kind of what we’re going to look at today. And then the next step after you have your document. So you have a source to document and then you have to decide what to do with it. And maybe these orchestration tools use AI to decide what to do. Maybe if it’s to find certain text in the document, maybe if there’s a regex pattern in the document, maybe if the document has images, maybe if the document contains certain text. We’re going to delete the document because it has controlled unclassified information. We’re not allowed to process that in Azure. You know, there’s a lot of things that could happen there, right? So there’s a lot of use cases for that. So we can use those and we can orchestrate. But once we decide we want to do something with that document, we use up top here is document intelligence. It looks like a document with the the four squares. That’s document intelligence. And if we, as we discussed earlier, you kind of have an overview of that. Now you could use layout, right, which is generic, but it gives you structures, which is what I like. Use a prebuilt, right? And in that case, it probably has most of what you need. You could still use other tools on it later, or an LLM on it later, or you could still put it into a knowledge base later. There’s all sorts of stuff you could do with it. It just depends on your business use case, but you’re gonna extract the data and then you might then also apply an LLM. To the extracted text and get a structured output. So think the pre-built model receipt, we’re going to look at that in a minute. We want to get that receipt data, but there’s actually extra stuff we want, right? We have business rules that are written down and when we process our documents, the business rules. As written by humans, not necessarily IT folks, will be pulled out and the LLM will apply those business rules and then enrich or supplement that extracted text. And then when you’re done with that, you can store the output, right? So maybe you just put the the JSON of the extracted document. To commit into a storage account and then something happens with it. Great that storage account and then maybe it gets picked up by AI search cuz you’re building a knowledge base. Or maybe it’s part of an application and it gets placed in a no SQL DB like Mongo or Cosmos or whatever. Or it’s so structured and. We’ve done this as well. The Jason gets turned into multiple tables and you make entries into a standard SQL database. All up to you, all business. Those are all based up of business situations, business scenarios, business values and downstream the bigger process. And so the point is, is that. There isn’t one way to do it. You just have a lot of tools and based off of whatever it is, the tooling you have that maybe the business value, the long term plans you have or the way you want to integrate it or how your end users are going to interact with it. Maybe it needs to, you know, be at this after the store. Is it becomes part of an application. That’s totally cool. It’s just there’s so many options. I didn’t. I feel like I’d have to put every Azure icon on here if we wanted to talk about everything. So. So the next part is we’re going to talk about an AI enhanced extraction demo. I’ll demo that rather. And so we’re going to talk about our source. We’ll have documents in that. Database or sorry, the storage account. We’re going to orchestrate extracting that data and we’re going to then use document intelligent to extract the data. We’re going to apply a structured output and then we’re going to store that the output of that. All right, let me switch back to. Uh. See switch back to. All right, so this is our storage account. Yeah, all right. So in here in our storage account, we have these receipts and they’re the same receipts that we use from the pre-built because again, I’m trying to show some similarities, right? Or differences. We have the Celius. I think that was like the spicy fish and a Coke chowder hut. I don’t know what that was. Maybe I was, yeah, whatever. CVS, our cookie, this is our cookie extravaganza, Denny’s and then Jimmy Buffett’s, right. OK, so you have to play a little bit of make believe with me right now. So we are building and I don’t know why you would do this because. Tools already exist that are SaaS and already open source and available. But let’s say you wanted to build your own employee expense management tool, right? And your policies are that you know no, you will not reimburse for alcohol at lunch. OK, sure. That’s an option. Or that you maximum tip is 20%, right? You realize that servers are really nice, but we don’t want to give them 30% because we got to be good stewards of our company money, right? OK, sure. So you have some things or maybe you have a policy that you do. Not reimburse no matter what unless it’s an itemized receipt. Now if we looked at, we looked at the the prebuilts, they’re not in there. We looked at the custom model. That stuff’s not in there. But what we’ll look at, those things will be in there and I’ll kind of show you how we do that. So first we have our receipts and then the next thing is we will look at VS code and sorry this is a little bit smaller. Resize this real quick. All right, don’t want to show my local settings. All right, so this is our function app, and in here there’s not a whole lot in the function app, right? We’re going to call our extract receipt data. A class, which we’ll look at in a minute, and from that we’re gonna extract the payment. We’re gonna give it a BLOB URL, right? So you could think a BLOB shows up in a store. The URL gets triggered via queue or HTTP trigger or whatever. In this case, I think it was a HTTP trigger, so. One process calls HTTP trigger and passes in the BLOB URL. Lots of different ways to do it, doesn’t really matter. And then you get your response right? And this we can look at this extract receipt data class and see what it does. So we’ve got some. We’ve got some prompting utilities. We’ve got some utilities. We have the Azure Open AI. It’s really not complex either. We’ll take a look at that. We have our document intelligence class and we have our storage utils and that’s basically if we take a look at those things, the Open AI prompting is. We’re basically just connecting to the service, right? So we’re connecting to the open AI service, that’s most of it. And then we take whatever we take the message that was provided, the system message, and then we take the data that was passed in from the document and we give it a structured output. Class and we get a response, right? So that’s it. It’s not super complex for document intelligence. We’re connecting to the document intelligence endpoint and we’re going to extract some text. So we’re going to use the Document Intelligence SDK. This is not custom code. You know, analyze the document. We’re going to use a prebuilt. Remember we talked about layout, so we’re gonna use the prebuilt model dash layout and then we’re gonna pull from the body the body of that we’re gonna pass in the document itself. We’re gonna get our results and we’re going to extract the line. Now this part is the way you handle the output. I find this to be retty reliable. But basically what you do is you generate a JSON document output that has the text of the document and any tables. So all it does it just extracts the text and the tables. There’s no necessarily any knowledge, just structure at this point, and so our document intelligence extracts the structure. And then we pass that into open API, open API to get a response and we expect a specific output. Now the fun part here is the output. Now this may be hard to read too. Let me see if I can zoom in a little bit. But the idea here is that remember the receipt, there’s the merchant name. So what we do is we use this merchant name, we give it a description. And when this is we’re not necessarily training, this doesn’t require any training. It just requires you to know what you want from the document. So we want the merchant name. It’s the name of the merchant and it gives an example. You know a restaurant number. Maybe you don’t care about that. So you delete this, right? We don’t like, we don’t care about restaurant numbers. We’re going to delete that. OK, merchant address, transaction date, transaction time, order number, server name, table number. I mean, these are just. I pulled most of this out. I’m a little happy on parameters, but number of guests items. So items are actually the table where we’re going to get the items list of items purchased in the transaction. Pretty straightforward. Itemized. OK, here’s where we’re starting to put some business rules in. It’s a Boolean and it’s true if the receipt contains itemized table of purchase is false. So if you get a receipt and your process shows that itemized is false, you can the person submits the receipt and it goes analyze it and they say the merchant name. Well he gets the merchant name and prefills all stuff like it goes. This receipt is not itemized. You may not submit this receipt for reimbursement. Now, maybe you’re not so draconian and mean hearted as me and you just give them a warning, right? I don’t know. But that’s a possibility, right? So itemized. Nope. Who knows what they bought? Subtotal tax tip and then I put in tip percentage. So if a tip is given, calculate the percentage and what you could do and I didn’t specify an axe tip, but you could then have some logic. If tip percentage is greater than blah, you have a warning or deny the expense or reduce the reimbursement based. Off of what a 20% tip would have been. You could do things like that. And then the other thing is the additional fees. Some restaurants or governments charge additional fees. The amount is usually is between tax or tip and the final tar, blah blah blah. So that’s cool. So what we’ve done so far. Is we have loaded our prompting utility which we’ve just shot which you just saw right? And the prompting utility is open AI, our document intelligence for extraction, our storage account that we want to get and then save the document. And then I created just this little notebook so you could see it. You can follow along with me, right? All right. And so let’s say let’s clear this, restart, clear outputs, and then we’ll run them all. And so we’re passing it. This this is the CVS receipt and. It says CVS Pharmacy, Las Vegas. It’s a LD oatmeal cream quantity one. It’s a food or beverage. Oh, that’s the other thing I added. Maybe you don’t reimburse for alcohol, right? And so this is a food or beverage and. The extended name that I got this wrong. Lynn and Larry’s Omo Cream Cookie. Well, that’s not totally crazy. This other one, PFFHSC Pepperidge Farm Farmhouse Omo Raising Cookie. That’s pretty darn accurate. Great. So maybe there’s some tweaking we can do, or maybe we can give it some other knowledge. Or maybe you don’t really need that. I don’t know, maybe it’s just for fun. Um, itemized. This one, it says false, although that is definitely not false. That is true. You’re not supposed to see that in a demo. You didn’t see that that was that was true. It said true. You did not see any false. Looking at which one was it? Was it Chowder House? Which one was not itemized? It looks like Jimmy Buffett. Jimmy Buffett one’s kind of fun cause it shows that if you’re Jimmy Buffett’s Margaritaville, you’re gonna be drinking some alcohol, right? And so. We’re gonna look at the Jimmy Buffett. And all right, so it’s in Las Vegas, 6:30 PM. So you could also have a thing like says breakfast, lunch or dinner. You could have that as a field appetizer trio. It’s a food and beverage Landshark. That’s alcohol. Large draft Landshark will. That’s sorry. Oh, this is the extended name Jambalaya. That’s some food. I don’t know how good is Jimmy Buffett in New Orleans. It’s amazing. Then Marg Perfect. That’s alcohol. Bud Light, Jeff Large. That’s alcohol. Fish and chips food. And then Marg Blueberry Pongo. Like I said, these folks. We’re living it up, having fun at Jimmy Buffett, Margaritaville. So there’s a lot of alcohol here. You could even say give me the subtotal for alcohol and food beverage. That’s something you could do. That way you could extract or decrement the alcohol items from the total. It is itemized, so got it right on that one. Thank you for not making me a liar here. AI and then the last one we’ll look at Denny’s cause this one a lot of them don’t have tips um on here. Which is weird. And maybe it’s just like the receipt that didn’t have the tip. So I’m not judging anyone, but you should tip if you have a good server. Anyways, Denny’s Coffee 379 and it says Denny’s Brewed Coffee Orange Jew. Oh, I’m pretty sure that’s orange juice. Yeah. Lumberjack slam. Lumberjack slam. Breakfast platter. Uh, substitute double Berry strawberry topping. Like, it’s pretty cool, honestly. I don’t know. Maybe I’m the only one that nerds out of this. Amy Cousland 48:24 Hey Nick, we have a couple questions here as we’re I just wanted to read them to you. All the examples have been with forms and type data. If the form has uniform fields but is not typed, how do these extraction capabilities work with handwritten information? Nick Miller 48:28 Yeah, perfect. So there’s a way to extract handwritten data as well using document intelligence. We don’t typically see a lot of that, but it’s there is a form, so we use prebuilt layout. You might use like because there may be written things and type things. Makes it fun. You may use both models. It depends on how reliable of a solution you need, right? So if you need like a super, it’s really important business process. You’d probably process through the layout and the handwritten, and you may even if it’s an image, you may even process that image separately. Through an LLN as well. It just depends on what you’re trying to do, but you can use there’s a the document has handwritten text text recognition. Amy Cousland 49:30 OK, one other question. Are these technologies mature enough for companies to ditch the tech debt? Sorry, the tech debt of their legacy document management vendors with their high annual licensing fees? Companies like Hyland, Kofax, Open, Open Text, etcetera. Nick Miller 49:46 That’s a really good question. So we have customers using them with with really good results. So honestly, not trying to plug us too much, but hey, remember that complimentary session we mentioned that is the perfect use case. Let’s get on the phone and talk about, well, how many documents to be processed. Because we have companies that are using us because where they process tons of documents, it’s worth millions in savings, right? And and not only that, you get really good result, like better results in than from what I’ve seen and it depends, right? There’s only I don’t know which one. In which document, but we’ll look, we could talk about the specific use case. So it may be that the document you’re talking about may be ultra complex and we’d have to find out. I maybe I maybe my answer is I don’t know, right? Or maybe you show me some documents and we say, yeah, absolutely we can do that and then we could do like a quick proof of concept. And say, hey, Yep, here we go, this will work. And then great, if we know it works, then you can deploy it. But as far as these document intelligence has been around for a long time, very long time that’s been around for about now it being enhanced with AI capabilities. You know this that’s been probably three or four years, maybe more the doc, the AI, the LLMS extracting structured output, that’s standard, that’s kind of standard process these days, right. And so Azure doesn’t have one built in necessarily we we. We provide that as part of a solution, but you know that’s very common. We’ve been doing that for years and so it’s it’s working with document workflows. It’s very reliable. We have one customer that they were processing ACH payments. They would get an ACH file, find the ACH amount, go look in e-mail, search through e-mail, find the e-mail. OK, there’s a PDF and the PDF has vendor information, but it’s a different vendor in the e-mail than from what’s in their ERP than from what’s in the ACH thing. Oh, and by the way, the invoice number is truncated. And oh, this is in the wrong column. And we went from that person taking three to five hours a day and now it takes them 10 minutes because we put other things in there to help it learn and match and do things like that. So it depends. If you need it to be absolutely robust, we can 100% do that, right? Or you can 100% do that. But it just depends on how much is the business problem, business process. If it’s valuable, we’ll put whatever needs to be in there to make it, make it work. Amy Cousland 52:17 OK, Nick, actually, I’m going to read one more question that I’m going to drop to kind of go on to the next session. You can stay here and answer questions however long people have them for you, but here’s another one for you. Um Does document intelligence include any contract signature capabilities? Nick Miller 52:32 There are, yes, there is the ability to find signatures as well, yes. Amy Cousland 52:38 Yeah, well, sorry, we’ll let you. Well, we’re kind of up in time here, so if you wanna go ahead and wrap up and if anybody else wants to drop any other questions into the into the chat. Nick Miller 52:44 Yeah. Yeah, I I do have to leave pretty soon here as well. But absolutely those 30 minute sessions we mentioned, there’s no streams attached. It’s not a marriage contract. Just you know, if you if you have an interest, you have a question, we love solving problems. We love working with our clients and even potential clients. So just schedule a session and you we. Amy Cousland 52:49 Mhm. Nick Miller 53:07 You may answer some questions. You may get everything you need and you go on your own and do it. Or you may find out that maybe we can help you. So I do have to go in just a few minutes here. But yeah, so on that note, this is the last slide, so we’re pretty much done anyway. Amy Cousland 53:14 OK. OK. Oh, perfect. Nick Miller 53:22 There. And I just want to say thank you for your time. Thank you for hanging out with us and and you know, asking really good questions. And let me see if I can see the chat now so I can answer questions on my own without the Amy. I know you got a job. Amy Cousland 53:33 Yeah, I think, I think you got everything. So I thank you, Nick, so much. This was a great session and we’ll be sharing out this recording and slides after the event. So I think he answered them. So I think we can go ahead and end this session and we’ll see you all in one of the next sessions. Nick Miller 53:50 Thank you, everybody. Amy Cousland 53:50 Thank you.