View Recording: Optimizing Supply Chains with AI Tools

thanks for showing up everybody. My name is Brandon. I’m head of AI and we data scientist at concurrency. I’ve been doing applied math and AI for about 10 years and across use cases across industries, but today we’re going to talk a bit about supply chain optimization and the two big use cases within that demand planning and relatedly, price optimization. We’re going to start with demand planning and then as time allows us to move into price optimization, but there are, there’s a lot to cover in both of these topics. I’m going to focus on really how and when to get started doing demand planning and price optimization in house. When should you do that? In house, when does it make sense to pursue? Not outside SAS vendor. What’s entailed when you get started? Lot to cover. Super excited to dig in with you. Feel free to post your questions in the chat as we go through this. I’d love to stop in real time and address them as we’re on that topic. Otherwise, feel free to wait till the end. Alright, so my goals today are fourfold. First, umm you leave knowing if you need custom in house solutions. You know, the immediate next steps to get started with implementation. Importantly, you know what to expect along that journey, and that’s really where we’re gonna spend the bulk of our time today is what is on that journey and building demand planning and price optimization solutions. And then lastly, you know concurrency can help you with that. Those are my goals. Super excited about it. Let’s move forward. So why are we talking about these two use cases? Well, the first is they’re valuable. They both are demand planning in particular has been around for a very long time. It’s very common. It’s very important, especially for manufacturers and retailers, to optimize their supply chains and despite it being or having been around for a long time, it’s not often well implemented. I see a lot of value being left on the table some of the time and we have sort of hard one advice for how to prevent that on the price optimization front. UM, very high leverage and it requires demand planning forecasts as an input to it. That’s where we’re talking about it in tandem. Alright, so demand planning in particular is very valuable. It’s pretty low hanging fruit, not very hard to get started with this. If you’ve got historical data on the thing you’re trying to forecast, that’s pretty much all you need to get started. It’s very easy to automate. It’s got great synergy with price optimization. As I mentioned, forecasts go into your price optimization system and it’s kind of a win win. Not like intangibly demand planning is a really good way to learn about what drives your business in a causal way, and you can you can really leave working with these systems, understanding and appreciating your business a little bit more. Umm. Because it’s been around for a long time and it’s really easy to get started. It’s a good gateway into higher impact a I use cases. Umm. And that’s important because a lot of folks are trying to. Organize resources to invest in an in House AI capability. You don’t want to bite off too much to chew at once, and so where do you start? You wanna start? I think big start small and demand planning is sufficiently small to get rolling with that. So what is the goal of demand planning? Umm, it’s really basic. It’s really to make more money and you wanna do this by forecasting future demand for offerings. Specifically, you’re making money or you’re trying to make money through two levers. The first is by reducing over by costs and then the second is reducing under by costs. Right. You could be a manufacturer who manufacturers widgets that require raw material. If you buy too much of that raw material, you’ve got high inventory carrying costs. Your inventory turns are are poor. You’ve got now poor working capital is not as healthy as it could be. That’s problematic on the underbody side. If you sort of overcorrect and don’t buy enough, well now you risk stock out. Uh stock outs, right? This is basically not having enough of the thing you’re trying to sell. If you’re a retailer or a grocery store or a furniture manufacturer or something, if you don’t have product your customers can’t buy it, that’s the opportunity cost is missed, revenue and margin. So it’s problematic that is really the only two ways that command planning drives economic return. And just as a quick uh case study, I’ve got 3 examples of of how I’ve helped various customers with demand planning. The first one here is a restaurant chain, the demand planning system that I built saves them about $30 million a year because of their scale. They’ve got over 600 restaurants across 45 states. The big issue that they had was there wasting annual labor on staffing, empty restaurants and on the flip side of that, they had long customer wait times and complaints because they didn’t have enough people on staff. And so the reason for both of those problems is the forecasts that we’re coming from restaurant managers of sort of when and how many customers were going to show up to eat that day. They weren’t very good. And so now you’ve got Labor schedules that are suboptimal. We wanted to help and we helped by essentially building a demand planning system that predicted hourly demand. Uh, per restaurant and product line, right? If you’re eating at the restaurant or take out and we’re. One of the unique things about this project was that we were using local weather and local event data in the predictions. So if it’s raining now, people show up to the restaurant more or less depending. And if there’s a big football game nearby, maybe if you live in Green Bay, WI or something, right? Do people go out to eat more or less accounting for that information? Improve the predictions. That was great. Uh, it saved them a bunch of money basically because the error between the machine forecast and the store manager forecasts was favorable to the machine and so saved quite a bit of money. Next use case. Very similar different industry. This was a furniture retailer, mid-size, they’ve got about 120 stores across 16 states, very similar solution. And the problem? Sort of rhymes, but basically they had a bunch of there wasting money in excess, carrying costs and their inventory. So they’re essentially buying too many couches or too many beds or chairs that they’re trying to sell. And now you’ve got all this money locked up on the balance sheet. And then the flip side, sometimes they don’t. And for certain products, they don’t have enough in the showrooms. And so customers show up can’t buy it. That’s a missed opportunity. Also, what was unique about this? Another issue that they had was the supply chain analysts who are responsible for placing the orders of the couches and the sofas. And so on that need to be in the showroom, they’re they’re inconsistent. They’re not very accurate and you can’t really track like why they’re making the decisions they’re making. So it makes it really hard to manage to and improve over time. And so essentially that’s all driven by their poor forecasts. I understandably it’s a very hard problem. Uh. The another issue though is that the over and under buy costs. They’re not accounted for in those human forecasts. What that leads to is essentially patterns of mistakes not being corrected and you’re constantly bogging down the business with the same issue. Never really learning or improving and so we wanted to help and we did. So we built the demand planning system that predicted essentially 10 to 20 week demand horizons for about 10,000 SKU and distribution center combinations. Umm. And we addressed in that system a very long standing business problems that they experienced and I’ll dive into later in the talk basically what’s called intermittent demand erratic, lumpy and then smooth demand patterns. I’ll get into specifics on what each of those are and why they’re important to address, but essentially we helped the address each of those demand patterns in a fully automated in production system that had dashboards for their analysts to evaluate historic performance and uh place orders more favorably save them a bunch of money, 16 million annually over 120 stores. Our system is about 15 percentage points more accurate than the human forecasts. That’s where all the savings came from. Intangibly and very importantly, just it’s not an economic ROI is that now they have institutional knowledge that’s accumulating because these drivers of the business are learned encoded and then shared in the system. It’s very important aspect of these projects and often why they’re funded. The last one it’s very similar. Not going to spend a bunch of time on this, but essentially large grocery retailer almost 3000 stores similar to the restaurant chain, they had excess labor costs. Umm, but it wasn’t because they didn’t have a demand planning system. They already did. In fact, umm, one of the reasons why I was able to help was because they weren’t accounting for one of the biggest drivers of grocery consumption, which is or in shopping behavior, which is the weather, right. And so they were evaluating if the weather information if included in the demand forecasts, could improve forecasts of how many people are going to show up and buy groceries and if so, let’s use those forecasts in our labor capacity planning and then reduce how much we spend on labor and prevent those stockout situations. So essentially what we did is we figured out the weather patterns that drove demand the most and then we put them into the system. It’s fully automated. Ran in production for every one percentage point reduction in excess scheduled labor saves them over $900,000 a year. Again, because they have almost 3000 grocery stores, they’re very big customer and so lots of opportunity to drive economic ROI with very small effect sizes. So great. Return. With that, I’m going to move forward into sort of the nuts and bolts of why we’re here and not we’ve sort of couched, uh, demand planning in real world use cases. So what are we talking about when we talk about forecasting? There are three different flavors of this I’m talking about. The short term forecast that typically happen over or less than 6 to 8 weeks. Sometimes this goes by different terms. Sometimes it’s called execution forecasting. Sometimes it’s called demand sensing. Sometimes it’s demand planning. It’s all the same thing. It’s forecasting in the near term if you extend beyond that you’re into like operational forecasting. This is between, you know. Basically, after that short term to about a year, the exact timelines differ a little bit, but the point is this is like near term or sort of midterm forecasting and then strategic forecasting that’s much longer time razons umm and both of those last two operational and strategic forecasting, they’re out of scope. We’re just going to be talking about execution forecasting today or demand planning. And again, this is useful because it supports decisions like price optimization, inventory planning and order, promising sorts of applications. Alright, so within demand planning, you built a demand planning model or set of models umm, and they can be used in a number of ways. There are really three ways that demand planning models can be used, and the first is what everybody thinks of when they hear demand planning, and that’s inventory planning, right? How many widgets will we need in six weeks? The second thing that often is not intuitive when people hear demand planning is what’s called market response analysis. This is related to price optimization, but is it unique application? What this answers is what if I charge 5% more for this item? How is my demand going to shift up or down? Umm if it’s, you know, elastic demand, it’s going to shift down. Quantifying that relationship between price and demand. Super important and it’s in market response analysis that you get those answers from a demand planning model. And then the last application from demand planning is price optimization. So essentially, now that you know how our demand, how and why our demand moves the way it does, how do we adjust our pricing, right. So this is about actually making the adjustments to pricing, not just quantifying the relationship between demand and price as you do in weren’t response analysis. All right. So to drill into the tasks of demand planning, there are really two different types. A forecast and there are scenario evaluation and decomposition, and so I’ll unpack both of those. But this first thing is is point forecasting and this is how many. How many things do I need at some point in time later? Umm. So in many practical applications, knowing like the expected sales or profit value isn’t sufficient for making operational decisions. Umm so you need to you need to address that the. The next thing that’s related is added to point forecasting and it’s related, but it’s probabilistic forecasting. This is essentially saying OK you need 7 widgets week from now. Well, now you know we need between 5:00 and 9:00 widgets a week from now, right? So it’s adding some uncertainty so that you can adjust your decision making very related, but they are different. The next thing is scenario evaluation. This is essentially. Evaluating what if this happens or what if that happens? How does that change demand? Right, so a common one is what if, uh, what if price goes up a little bit? What does the man look like? What if? What if it goes down? Or what if it stays the same? Here’s what demand looks like, and then decomposition. This is more technical thing on the back end, but it lets you decompose the the line of forecast into its component parts, right seasonality, trend and and and so on. So here they all aren’t together. Umm, each of these often need to be formed at different levels of granularity and time horizons. Different levels of accuracy to address different demand patterns and downstream use cases. So it’s conceptually possible to build a single demand model that addresses all of these tasks and scenarios at once. It’s way more common to build multiple specialized models for different use cases. Umm. And that’s really what I’m going to talk about moving forward. So what goes into these things? Right. What do you use to predict demand? There are really 4 common inputs and you don’t need all of them, but it often helps. So you’ve got pricing information, product taxonomy information, calendar feature information and then historical sales or historical. You know why that you’re trying to to forecast? Pricing is usually among the most important factors that influence demand on products and services, so pricing features are important. Plus, pricing optimization is among the most common enterprise use cases that requires demand planning or demand forecasting. And really, it necessitates not only incorporating the pricing features, but also. Isolating and then quantifying the impact of pricing variables. Now, that being said, pricing while super important is often the data point that is missing the most because of things. We’ll talk about a little bit, but essentially you need to know for the same product different price points so that you can observe different degrees of demand when the price change for the same product, right? That’s not always true. You don’t know is run campaigns or marketing campaigns or discounts on your whole product catalog. And so sometimes that’s limited. It’s a real-world constraint that can be mitigated, so that’s probably seen historical sales. This is really just the thing you’re trying to forecast. If you have historical patterns of that, that is a very useful source of information to predict the future, right? Let’s use the paths to predict the future, so long as we anticipate the future looking like that past product accounting information. This is essentially where the product or widget or what have you is situated in the product taxonomy right within categories and subcategories and so on. That information is useful. Umm. And then calendar features. This is just, you know, day of the week, hour, the day, that kind of thing. All very helpful. So typically what shakes out as far as which of these is important is on this screen. Now this is just a rough example of what we typically see. Price is basically as the graph where the horizontal axis is showing the degree of importance that that feature or variable is, and then on the vertical axis you have which feature a variable. It is price is the most important, followed by sales or legs of sales. You’ve got calendar information like week and then you’re probably taxonomy information developed by additional calendar features as being important, but at least important. OK so. Now we know what forecasting is, what tasks are involved, how we can apply those use cases, and roughly the inputs to demand planning systems. Let’s talk a little bit about the challenges that we need to address while we’re implementing. There are four kinds of demand plant patterns that guide most of your engineering effort, and those are shown on the screen here. You essentially have smooth, erratic, intermittent or lumpy demand. These are the technical words for. Lumping Arabic being my favorite, just as a personal preference, I think it’s kind of funny. Smooth, understandably, is the easiest, as you can might imagine to predict you’ve got very smooth historical patterns and so pretty easy to kind of continue the line. If the line has been smooth. Umm, but if it’s sort of not smooth and more erratic you’ve you’ve got a bigger problem. You need to address it in some way and then you have different flavors of both of those where you’ve got intermittent demand. This is very problematic and among the hardest of the four to address this is big spikes in demand, followed by big absences of demand and then lumpy sort of a mix of like sometimes you get demand, sometimes you don’t and you never really know the magnitude of it. When you do see it, so these four different demand patterns can be typified. And then identified in the data so that we can figure out what to do about it. This is really how you figure out doing something about it. It’s kind of a technical slide, but the intuition of this is your essentially taking each of those lines, those squiggly lines that represent the historical demand, and then you’re classifying it into one of those four categories, depending on its characteristics. So you can quantify those characteristics through something called the coefficient of variation. Not going to get into the details, but it’s a fancy way of figuring out if it’s one of those four. One of the things to note here on the right side of the screen. Umm is that once you classify your product catalog or whatever you’re trying to the body of things that you’re trying to forecast for. Once you do that, you’re going to notice that there aren’t a lot of items in that. Are smooth. Umm, there’s a lot of lumpy and intermittent and that needs to be addressed. So how do you go about addressing it? So you have a couple different options and this is a little bit technical, but we’ll break it down. So essentially you’ve got a demand model that you pipe all of your information into, right? You’ve got your product taxonomy information. You’ve got information related to historical sales and so on. You create a forecast and then you try to expand. The things that you can forecast for based on how similar they look to very forecastable items, right? So take in one hand the hard to forecast products, try to match them with easy to forecast products and then borrow information from that easy to forecast group in forecasting the hard to forecast group. That’s essentially what’s going on here. There are various limitations and tradeoffs you have to evaluate, but that’s really the intuition. So I mentioned trade off. One of the tradeoffs that happens when you do this is that accuracy isn’t very good, right? So as you can imagine, if you’re trying to find like with like, but it’s imperfect and it’s likeness. Well, you’re gonna have a cost in terms of accuracy and that’s essentially what this graph is showing. The horizontal axis here is the percentage of product catalog that you’re forecasting for. That’s called coverage and then on the vertical axis you have essentially, uh, accuracy forecasting accuracy. So if you are only forecasting for us like a narrow subset of your product catalog, those are presumably going to be the smooth items, and those are pretty easy to forecast for, right? They’re they’re fast moving products. As you expand coverage by doing that like to like thing I talked about, well, you expand coverage, sure, but your forecast accuracy is going to be a little bit less than the smooth group. And then once you do similarity expansion, it’s a very similar idea. You expand coverage, but again you take a hit on accuracy. Whether or not you pursue each of these expansion tasks depends entirely on your use case, so sometimes for retailers it makes sense to pursue attribute and similarity expansion, and for manufacturers, not so much, right? Because the demand is much sparser and the costs are much higher than you would have in a consumer facing environment. OK, so I mentioned that there are a lot of different, let’s say knobs and dials that you need to like, tweak and play with while you’re doing demand forecasting right, you’ve got different levels of granularity. Do I forecast at the weak level or at the month level? Maybe at the Cordy level, do we do it per week per product? Do we do it per product category per week? Do we a lot of different ways to slice that over what time horizon are we trying to forecast for? What’s the expected level of accuracy then? You’ve got thrown to the command plat patterns. That kind of complicate all of that. There’s a lot happening all at once, so this graph is really just an articulation of or visualization of the whole process. It’s not super important, but I’ll walk through it. Just because the terminology I think is important. So essentially you’ve got this historical demand right here, right? And you’ve got a starting point of the lookback horizon. This is the point in time in the past that you go back to, to evaluate or to basically chunk what goes into the model. You’ve then got a point in time. Call it right now, right? This is the the point at which you forecast, and you’re going to forecast over some time period into the future. That’s called your forecasting horizon, and then you have this little orange dot which is your specific demand forecast. And as we pointed out earlier, that can be both a point forecast which is just a single number, but more usefully, it can also be a probabilistic forecast that gives you a prediction interval of could be between this and that number. Umm, OK, so when you’re at this prediction time period, you just pipe this whole you pipe all the data points you have at that point in time into your model in order to get that forecasted value. That’s great, but that big stack of data points that you pipe into your model that comes from a variety of different sources, right? The historical sales information, the pricing, product taxonomy and so on and depending on which source it comes from, it added differently. So you’ve got lagged values which are really dynamic and observed, right? So they’re dynamic and that they change and they’re observed because you have actual data on it. All of that gets sort of added, but then you have things that are known in the in the future, but they might be static, right? And so these are things like, uh, future marketing campaigns that you’re going to run. You might know roughly the product. Uh cattle categories that are gonna run them for the like the discount levels right? 10% or 20% etcetera. And so on. You take all that planned information, which you need all the way up until the forecast horizon, right? The point in time that you’re forecasting for because you need to know how those things influence that forecast, right? Alright. What does this look like? End to end. Here’s a rough flow, starting with the top. It might be a little hard for you guys to see, but I’ll walk you through it. Essentially at the top you start with the items you want to forecast right at the very bottom you have your use cases of inventory planning, market response analysis, scenario evaluation, and price evaluation. But in between is we’re really where the magic happens. And so once you start with that red box of items to forecast, you apply a filter to identify the active items. Wait, these what are the things that are actually being sold? You’re gonna have a whole slew of inactive things you don’t want to account for and train the model on because the model is expected to forecast only active items. So let’s get rid of all the inactive things if it is. Inactive. Let’s just put it off to the side for now. After we have that group of active items, then we model in that blue box the demand type. Which of the four demand patterns do we see in each of these items that we’re trying to forecast for, right? Is it lumpy intermittent or annex smooth? Knowing that information is super important because in the next step we’re going to filter and if it’s lumpy, we’re going to put it off to the left and we’re going to use a specialized model. If it’s not lumpy, then we’re going to move into checking if there is quote unquote enough price variability in the pricing information that we have per item that we’re forecasting to see if we can use it right. If we don’t, then we’re going to put that off to the side like like I mentioned earlier, this is often kind of where the rubber hits the road with being able to use price information, retailers often have price information for products that allow us to observe different amounts of demand for the same product price differently as the result of marketing discounts, right? Manufacturers. Not so much. Sometimes that’s the case, but often not. Umm, once we have identified the items that have passed that right, the enough price variability we move into what’s called feature engineering. This is just converting the data at our disposal into a format that the models can learn the best from. This is a very iterative approach and often a considerable amount of time is spent on this area. Once we have that, we have our forecasting model. That’s terrific. Now we can forecast for the body of products and do inventory planning or market response analysis, scenario evaluation and so on. There’s a lot on this slide and there’s a lot on these topics I’m going to try to summarize really how you know, you need to own this in house, right? Is it really worth all the engineering complexity to develop this in House or should we through a SaaS solution there’s no one size fits all to this? It does kind of depend on a variety of things outlined. This table is a variety of those things and kind of what you get with each. So really the time to value for SAS is a little bit faster, right? You kind of just kind of plug it into the wall and send it your data and hope for the best. You get to realize the output a little bit faster if you build this in house, it’s going to take a little bit longer 6 to 12 weeks is what we typically budget for at my team budgets for. On the integration side, though, in house stuff is much easier to integrate because you’ve got access to, you know, the systems that drive your system already. Much harder from a staff perspective in terms of customization, you can totally customize any of the solution to your data, whereas with a SAS solution, not so much. That’s really important, as you can imagine, when it comes to using price information, right? Everybody’s marketing group is structured differently and maintains their data differently, and not every plug and play SAS provider. You know, can’t accommodate the differences in in that data structure. Ohh the there are differences between the short and long term costs. Obviously short term cost for SAS is lower, but in the long term it’s higher and then that’s flipped for in house, right, short term it might be a little higher up front as you allocate budget to get it built, but then over the long term you get to enjoy a solution that you don’t have to pay. For every month and licensing fees and so on. Umm, maintenance. This is, as you’d expect. Maintenance, maintenance, support cost is a lot higher for SAS solutions. This is captured in the licensing fee and so on, and that in House it’s much lower. Obviously with the in-house stuff you have to account for, you know the team that’s going to own into the teammate, that’s gonna own it. Moving on to the tangible benefits of like what I’m calling cumulative knowledge and decision advantage, you don’t really get a lot of that in SAS solutions because of how they decide to surface information about the causal drivers. If you’re a business, not every SAS solution is the same. Some of them offer that, but in general, if you own it in house, you have total control over the knowledge that accrues to you in the system data security. You don’t have any control over that. If you pursue Assad solution, whereas in house you’re responsible for data security, that can sometimes be, umm, a great comfort or concerned, depending on one’s role. And then lastly, vendor lock, there are differences. You’re locked into a vendor with a SAS company. You build it in house, you’re not. So it’s a very quick summary. There’s a lot of nuance to this. I’d be happy to cool answer any questions that you guys might have in the chat, or if you shoot me an email afterwards to dive into this. I’m going to talk a little bit about the challenges, right? It’s not all high fives and unicorns as you sort of sail off into the sunset. There are some things that could be. Addressed because they’re challenging. So first of all, out of stock events, bias demand data, what do I mean by that? If you don’t have, rather if you have a data point on how many, let’s say tubes of toothpaste were purchased because you’re a big grocery store that sells toothpaste, but you ran out of toothpaste and so didn’t have the opportunity to sell it, well, now you’ve got what’s called sensor data and that kind of screws things up on the technical side. On the statistics side, you can’t address it, but that is a challenge that needs to be mitigated. Umm, another challenge is reconciliation of the hierarchy. This is just a fancy way of saying if you forecast at the category level and at the item level, there are no mathematical guarantees that adding up the items that belong to a category is it going to square up with the totals from the category. It is what it is. There are ways to address that, but it is something to be addressed. Another thing that that I found that most people don’t naturally think about is how you can’t forecast for your whole catalog. Your whole product catalog. Umm, the for the reasons I mentioned before. Sometimes you know you’ve, I mean, almost always you have lumpy and intermittent and erratic demand patterns in your data. And so as a result, you’ll need to apply business decisions to decide which products you’re going to forecast for and which don’t have sufficient accuracy to justify forecasts for. Right. So that does limit things. Another challenge is what’s called the cold start problem. How do you forecast the future of a product that has no past? It has not existed. There are ways to address that, but it is still a notorious thorn in people’s side. Uh. Critical data is often missing. This is especially true for price, for reasons I’ve talked about before. Umm, this happens in B2B context. Quite a bit, and then there can be low adoption of of price optimization for a variety of reasons. It’s kind of complicated. Everybody has different opinions about how the business works and how you should price things. Not insurmountable, but just a challenge to address. Alright, this probably my most click baby slide but some secrets about demand planning that I think you should know. Everyone wants to know catalog coverage first, right? What percentage of my products can I forecast for right? They want to know that in advance, as you might have have learned, you can’t do that, right? You have to back into that while you forecast, right? Because it’s through forecasting that you decide what you can cover, not you decide to cover it and then forecast well for it. You’ve got to do it the other way around. Really easy for the humans who use this demand planning these demand planning systems to veto the forecast. Which hamstrings the value that organizations get from this? This happened in that. Furniture retailer that I was talking about before. The whole reason why they hired me was because they really had supply chain analysts who knew that the in House demand planning system that they had was empirically superior to what their they were producing, but they didn’t trust it because they didn’t know how it worked. It was complicated and so on. And so they just decided not to use it, right? Understandably, I get that nobody talked to them. Talk to them about how it worked. Not gonna use it. And so I came in and rebuilt the system. And then train people on how it worked, trying to encourage people to stop vetoing those forecasts. Umm, because of that tendency for folks to veto it easily, it does require change management and I don’t often say always or never, but this is a point where I will say always requires change management. If you’re building demand planning, because it it can be delicate in rollout, you’ve got to train people the right way. You have to give them enough information, but not too much to help them trust it. Obviously if you give them too much information, their eyes. Mike Glaze over and then they don’t get it and you’re back at square one, but if you give them no information, you’re not getting off a square one, so. Umm, often uh doesn’t get enough attention. That’s because it’s a long standing in use case. You know it’s not generative AI. It’s not an autonomous car, so it’s sort of just like this back office problem that it’s very easy to neglect. And so, because of that, these projects can fail if you don’t have for non technical reasons. If they lack a champion, right. Umm, very important to have a champion to push these things forward and also. Demand planning only works uh for for items and businesses that are relatively stable. So and the reason for that is because you’re trying to use the past to predict the future based on the assumption that the future is going to resemble the past. If your business is rapidly growing, there’s there are ways that you can use information that historical data, but it’s going to be limited value versus using the same information or similar information for a more stable business or product environment. OK, So what are some of the things that affect the scope of doing these things? So the state of the data, these things are driven and powered by data. And so primarily, how complicated is your data processing? What needs to happen from a data processing standpoint in order to make use out of the? Umm. Data and so got a good question here from Stuart. Defining category A category is a collection of like items that you’re trying to forecast for. So if you’re a retailer, a category might be. You know, home improvement and then within home improvement, you’ve got, you know, I don’t know kitchen and other things that intuitively belong to that. I’m kind of blanking out what that might be. Uh, a catalog is just the roll up, then of all of the categories, right? So catalog is the collection of items that you’re trying to forecast for. So again, if you’re a retailer. You know, it would be the collection of everything that you sell. All right, so another thing that impacts the scope of uh, of of what it takes to get one of these things into production is really the degree to which your. Uh demand patterns are lumpy, erratic and intermittent. The smooth stuff. That’s easy. That’s smooth sailing, as it were, but the lumpy, erratic, intermittent stuff less so. And how much of it really drives the complexity involved in creating some of those specialized models that do the either attribute or similarity expansion models? Another thing is, if you’re related to those specialized models, is is how, if, and how you choose to address that cold start problem and slow moving items. So cold start is how do you forecast the future for a product that does not have a pass, right? These are new items. What do you do there? You can use transfer learning as kind of I talked about before, but if you decide to do that a little more, work a little more complexity. Umm, another thing that affects the amount of work that goes into this and therefore your time to value is how much readability, human readability your business requires to trust this right. Everybody’s a little bit different. Everybody has different sensitivities to. Information and trust, just as a general generality and in high trust environments you don’t need much human readability and low trust environments. You need. You need lots of it where you sit on that spectrum drives effort. Tolerance for exploration? Umm what I mean by this is how willing are you to get to know your data and get to know your business by testing and exploring it? This is a critical, non-negotiable part of these projects. As you as you might have seen in some of the earlier slides, all of this is about the. Nuance and personalized characteristics of your data. All of which needs to be rooted out through exploration. So if you have a high tolerance for that, it’s going to extend the timeline. If you have a short tolerance for that, it’s gonna quicken it. OK. Umm, we have 17 minutes left. I’m going to move on to actually the next the price optimization thing and then I’ll get to the next steps. Does anybody have questions so far before I move on to price optimization? Excellent. Alright, so. Why are we talking about price optimization similar to demand planning? Very valuable. Has a direct impact on revenue and margin. Also easy to automate. Plus, it’s got good synergy with demand forecasts, just like demand forecasts had good synergy with price optimization and that’s excellent. And therefore important to talk about. So here is kind of what we’re talking about when we talk about basic price management and the process that goes behind it. This is a huge topic. Umm, whole whole business textbooks are written about this and I’m trying to condense it all into uh. A cute picture, so necessarily cleaving off lots of information, but at a high level you’ve got your revenue model, which includes both your price positioning and pricing structure. Then you move into doing more strategic analysis kind of stuff. Then you move into planning and evaluation. This is sort of scenario planning, right market response modeling, umm. And then you’ve got execution, dynamic pricing and price personalization, which are different things. Dynamic dynamic pricing is adjusting price of a product based on market forces. Price personalization is about customizing price for a specific person. You’re respective of those market forces and then you’ve got measurement which is kind of a classic. A group or classic body of work that uses a lot of econometrics and some of the demand planning concepts that we talked about demand decomposition on constrained, and so on. So we really what we’re talking about here is planning and evaluation that’s that’s kind of where price optimization lives. And then in the execution area measurement. Oops, sorry kind of could do that. So these are the three areas that we’re we’re focused on in this. And I’m gonna drill into more specifics. So what does that flow look like? Right. So this is kind of a top down flow of everything that was on the prior screen, but we are going to start with identifying different items strategies, right? So there are different ways of going about this. You’ve got, you know, high, low strategy. Umm. A key value item strategy and so on. This is these are strategies that govern how you decide to price an item. So a high low strategy is implemented by retailers like Wal-Mart, right? So it’s a price low and high. You’ve got high price point, low price point. That’s excellent. You’ve got a lot of grocery stores that use key value item item strategies. This is essentially cherry picking the very frequently purchased items like bread and milk and eggs and pricing them strategically so that you encourage consumer perception that your favorably priced grocery store. The point here is identify item strategies for your items and then you can move on to the next next area, which is essentially just grouping like items with like items. You’re identifying things like your objectives and your constraints. All these are important because they get piped into this demand planning process. Umm. And again, this is not about demand planning. This is about using the output of demand planning in your price optimization workflow. Umm, so this is your market response model, right? Which is an application we talked about in the earlier slides. You’ve got your demand forecasting model, which is what we talked about as well and you’ve got different decision support tools that might complement those activities. You pipe all that information into. What if solving right? So these are off the shelf solvers that allow you to set different scenarios that you want to figure out how to price, right? What happens if X? What happens if Y you establish then a pricing policy, right, which then goes into the really the output of this workflow which is reporting and then executing on on the price optimization stuff. So it’s a super high level, lots of details hidden in here. And just in the interest of time, I’m gonna kind of scoot ahead to. How you need to or what you need to know to own this in house. This is really similar. In fact, this is exactly similar to the slide for demand planning for the same reasons, right? The in house. The buy versus build tradeoff. That’s essentially the same thing for both these use cases. So I’m not gonna drain this again, but please let me know if you have questions about this afterwards. OK, so how do we go about building these? Most people, when they hear AI solutions or machine learning solutions, they think of a little a little box of machine learning code. And that’s it. Umm, the reality is a little more complicated and includes a lot of other components on the back end. Umm, you know things that are not as interesting and sort of catchy as as AI machine learning, but more prevalent. They take longer and are points of failure in and of themselves, and so they’re worth talking to. Umm, building all of these components at once? It might sound like a lot to bite off at once, and it is, and that’s why we don’t recommend building these solutions, whatever they are, in one fell swoop, we do recommend phasing. Project so that you derisk the investment in it. You first start with a POC where you’re building essentially just the minimum set of components to validate economic ROI. You’re not really testing things technically in this at this point. Presumably that what happened? You’re selecting a use case that you know is technically solvable. This POC is about proving economic viability, and you do that by building these four things. You’ve got your data collection scripts, you’ve got your machine learning code, you’ve got model validation, which is about essentially evaluating how good the model is. And then your feature engineering, those are the four things you start with and then you move into phase two, which is about turning it online. You build a few more of those components, and now you’ve got something that’s online. We call it an MVP. It’s a minimum viable product and then lastly you move into a more full featured solution which includes more automation. This is the same information in a in a table with a little more detail on what each of those components are, why it matters, and then when exactly they’re they’re built should be noted that this isn’t a hard and fast rule. These are customized by. Uh by by our clients and by particulars of their environment. OK, so let’s say you build it. We’ve still have to evaluate like how good it is even before we build it. How do you go about choosing the right problem? You wanna choose problems that are that have high ROI and there are really three different flavors is how we think about ROI. You’ve got the measurable economic stuff, right? That’s sort of the hard numbers. That’s pretty easy. You’ve got two other kinds of Rois, though. One is strategic, the other is capability of strategic ROI. Is really how the project positions machine learning to help your long term business strategy right? That’s a strategic ROI. And then you have capability ROI. This is becoming more and more important as organizations look to develop their in-house capability to build AI solutions. Right. And so this ROI. Is about measuring how the project itself positions you to upskill in this new technology. Alright, but once you, once you kind of have that sense, you still have to take stock of all the opportunities to to apply AI. And they’ve got this big pile of things. How do you go about prioritizing them? We have a pretty particular way of going about this. It’s really the combination of it’s expected impact, it’s cost and then the risk of. Activating the impact and minimizing the cost and so not going to drain this slide, but there’s a lot that goes into this. It’s a very important part of the workflow, I would argue, and in fact we’ll argue that problem selection is way more is the most important thing. The problem? You decide to start with is going to make a huge impact on your experience with AI, so that’s why we spend a good deal of time on this. OK. So relatedly, we estimate ROI, the economic measurable ROI as robustly as we can, and we do that by simulating the business environment and simulating how effective our solution is. Combining both of those things and arriving at a dollar amount that gives decision makers a sense of how impactful the solution will be. And we do this basically by collecting historic information about the business that we can simulate based off of like the average and standard deviation and so on. Umm, each of those metrics. This is for a particular use case that we designed for manufacturer to autoquote things, so this isn’t demand planning, but it’s the same idea. We do this for all of our use cases, those historical or sorry, those simulated business metrics based off of historicals impact certain metrics, right that we call dependent metrics, each of which are not ROI. In of itself, but do drive ROI and so we combine those dependent metrics to arrive at an ROI number for the different ROI drivers. So as you recall, the drivers of ROI for the demand planning were reduce overbuy costs and under buy costs, right? And so those would be the drivers here. OK. And then you have to to maintain these systems. We use a rubric that helps organizations. Invest in their systems in a way that de risks, but also in a way that matures the solution technically. So this is a framework. That essentially scores a system between zero and seven. Zero is the least ready for product going to production. Seven is like this is the this is the best of breed solution and these are scores that are derived based off of the degree of testing. Like automated or manual testing. That’s done in the system and specifically over 4 different components of the system. There’s the data, the model, the infrastructure and monitoring each of those four components has seven different assertions, or really things that you could test and you could test each of those things either. Uh automatically, which is the best manually, which is OK, is better than nothing or not at all, depending on which of those you’re doing, you have a more automated or sorry, a more production ready system. And so as you can see here, the data component of this system, we’re really just automatically testing if all the input feature code is tested, right? If you don’t do this, you’re going to get unwanted negative impact on the business because they’re unidentified bugs. That’s a problem and you have really unreliable feature engineering, right? Uh. Not gonna drain this. Lots of information here. The point here is you have to be robust and we are robust about how you design these systems for the stage and level of maturity they’re expected to perform at. No. And then we land at that overall system score. Umm, you know which you can assign labels to to make it a little more human readable, right? You’ve got a POC and MVP or or the ML OPS. Umm, really this MVP is highlighted because this is where economic ROI starts to be realized. And so if you build A Level 2 system, there has been a basic first pass that productionalization, but additional investment might be needed just to really make it reliable. Alright, so that’s really it for me. Umm, rather than this cartoon, we’ve got some next steps for for the group that I wanna bring your attention to. So, uh, we do have an offer, a complimentary supply chain assessment if you’re interested in what we talked about today with regards to either demand planning or price optimization or something related to supply chain that we didn’t exactly cover today, feel free to reach out would be happy to get an assessment with your team and my team scheduled umm, the very valuable at minimum you might come away learning something about the challenge that you didn’t know before and then have clear actionable next steps. We also offer a free AI executive, what we call envisioning session. This is a chance for our team, myself and Nathan, our CTO, to meet with your executive group and get alignment on some of the opportunities for AI across the board. And then lastly, we offer an AI and copilot company wide envisioning session, right? So this is a much broader group and it would include copilot as well in both of these envisioning sessions, you’ll leave with a list of the most advantageous AI use cases that are the sweet spot of the ROI that we talked about, right? So each one of those use cases will have a rank which is based off of the ones that are worth pursuing or not pursuing. Based on what we’ve seen, so if you’re interested in any of these three out of the assessment or the envisioning assessments, feel free to reach out. We would love to help. This is very exciting stuff and I hope you guys know you’re not alone in this journey. If you’re interested, does reach out.