View Recording: Why Choose Fabric? Fabric’s Unique Advantages Explained

Brian Haydin 0:05 Well, hello everybody, and thanks for joining concurrency is we have a conversation today about Microsoft Fabric and the modern data estate. I am blessed to be able to fill in for Nathan, who is stranded in New Jersey today. He’s on the call and will be the color commentary today, whereas I normally play the color commentary role, I’m a solution architect that concurrency I’ve been here for good eight years. Umm. And work have had many decades of experience in the app dev and data space. Over my career, Nathan, you wanna introduce yourself? Nathan Lasnoski 0:46 Totally love being color commentary today, Brian. Brian, you’re a Rockstar. Gonna do awesome and looking forward to talking through fabric. So this is gonna be great conversation for the next hour, and I will also be covering the chat. So as ash drop him the chat as we go, answering them throughout the session. So just the chat liberal love to talking about things are important to you. Brian Haydin 1:15 Alright, so Nathan’s having connection issues as he’s stranded in New Jersey, so I’ll just go ahead and repeat what he said because it was hard for me to hear. He’s going to be monitoring the chat, which I think is bandwidth will support and so if you have any questions please drop them in the chat and Nathan will be the one that will be answering for you today. So I thought it would be important for us to start the conversation out, you know, around AI. And the data is dated. Sam Altman had a fantastic quote, talking about the importance of AI to organizations as we move move through the 21st century. U and you know years ago when the iPhone, you know App Store became prominent, you know everybody was bragging about their mobile devices and you don’t really hear that anymore. And it’s just not something that people think of. You have a mobile app because you’re a company that sells products and services. At this point we’re we went very quickly in the last two years from people scrambling about AI and how do I start using my data, leveraging my data for AI and you know, bragging about it to the point now where it’s just expected, it’s it’s an obvious thing, it’s expected that that you’re gonna have that. So, but the reality is that most of the companies that we work with, you know, are still, you know, looking at ways and opportunities to incorporate AI into their product and services. And so Microsoft has developed an intelligent data platform, and this is really meant to kind of bring cohesion to the story. Uh, and uh, what’s interesting to me and we’ll we’ll kind of be talking about throughout the course of this. This webinar today is that in order to build out an intelligent data platform, there’s four pillars that you really need to consider. Obviously there’s the data and the how you handle data from different database technologies that are going to perform and be optimized. Umm, there’s the security components like how do I protect my data? Make sure that the information that I’m serving into an AI platform isn’t being used for malicious purposes or doesn’t, you know, in, you know, expose some of my private information. Ohm, the AI components and then finally like you know what you think about fabric. Clearly it’s gonna be in the analytic space, but the most compelling reason to think about fabric is that if you deploy fabric for your environment and start building a data platform just using fabric, you’re actually touching on all four of these pillars right out of the gate. You get essentially an intelligent data platform, not with all the bells and whistles. You’re gonna need some other components, but but with fabric you can actually have them on our data at a state, and so taking aback, taking a look at the history of of data platforms, say advanced my screen, let’s look at like how we’ve kind of evolved, you know over over the course of the last couple of decades, you know at the beginning we were just dealing with Excel spreadsheets and files and file folders. You know, people had pretty cloggy, you know, ways of dealing with the data, and it became really apparent that we needed to have a process that could centralized the data. You know, cloud wasn’t a thing. So we were doing all this on premise using, you know, archaic ETL tools like you know SSRS and SSIS. So generally it was pretty slow. It was single threaded. Umm, you know, you couldn’t have multiple jobs running at the same time on a on an SSIS server. So we started to solve that problem as we moved our workloads into the cloud and data factory allowed us to do things like multithreaded ETL processes. But it also kind of changed the paradigm as more data services became available to us. So we moved from like an ELT to an ETL to an ELT. Uh, where in the next step, which is kind of about a year. Two years ago, we were doing blank houses and that was kind of like the big concept. You know that everybody was jumping to like centralize this data lake houses. We can move these to a medallion architecture and build out data meshes that are aligned to our data domains. What’s interesting for us here it concurrency and with Microsoft is that fabric brought this all software as a service. So now you get your data state as a deployable unit and you can start building on those lake houses. You can manage your, uh, your ELT processes. Uh, and build out like you know, you’re silver, bronze and gold patterns, you know, directly in the platform. So it’s betting his, you know, it’s been a great journey, and Microsoft was kind of the first one to get, you know, to get to that state. So let’s have a little bit of a frank discussion about like why this is such a good move on Microsoft part. And not advancing my slides very, very well. Nathan, you’re the expert at this. There we go. Uh, so, you know, Microsoft has been dominating the consumption layer, whether you’re on snowflake, you know, or other, you know, data warehouse platforms, a lot of the consumption still actually happens in Azure. Many people are deploying those units there and SQL as a product SQL Server as a product or SQL services have always been like a predominant uh uh data platform for most enterprise systems. Umm, you know? But on the analytic space there was, you know, a lot of competition and you know, let’s face it, tableau since Salesforce acquired them, it’s not the same product anymore that it was. And you’re seeing a lot of organizations, whether their data is an AWS or GCP or Azure. People have been using power BI because it’s such a fantastic tool. Umm, so in the cloud infrastructure game though, you know, Microsoft was competing head to head with, you know, everybody else. But by making it a SAS platform, they’re doing something that nobody else is doing at this point. And if you’ve noticed, when you as a power BI user building in Power BI, you’ve noticed that the URL has changed. It’s now Fabric. So everybody who is a power BI user is essentially a fabric user. Now you’re starting to get used to the tooling that’s available to you. Uh. And it makes it simpler than the competition by uh by allowing you to deploy a capacity and inherently get all the tools that you would normally, you know, have to deploy independently. Umm. And finally, with the with copilot coming out, that tool is starting to mature. You know very quickly and features are rolling out for people that have the appropriate SKUs and that’s making the data analytics space even easier to use for the end users, which is the goal for Fabric. So you know, some of the big reasons for us to consider a software as a service data estate. For me, the biggest one is the cost optimization and the efficiency based on the run rate, we’re going to dive a little bit deeper in there, but there’s also the ease of use of getting the value from your data as we set up data domains and have data stewardship including data lineage. Umm, you know, all these are gonna make thing access to the data pretty easy for users that wouldn’t traditionally have direct access to the data points. Also, with the SAS product, things get things happen. You don’t have to upgrade things, you get those benefits in the cloud. But you know, you also have to manage. Like, are we gonna go to this new system over here if I wanna switch over from one data consumption layer to another data consumption layer I have to rewrite tools. Uh, you know, with with the software as a service, you’re getting innovation in real time when Microsoft deploys it, it deploys it some key things to think about right now is like even we’re seeing this right now with Synapse, Synapse is not getting the features as fast as Fabric, which is underlying gut synapse components to it as well. So it’s clear Microsoft is making a lot of investment in this Fabric platform. So you’re getting access to that innovation much, much quicker if you’re building in Fabric at this point. And then the data lineage I I alluded to. Uh. Once you start building in fabric and you can start to tag your data appropriately, that lineage stays within the ecosystem. So regardless of what layer that you’re using, if you’re using your gold or bronze or silver, you can inspect it, interrogate that data, and see exactly where it came from and who signed off on it. What data owners are responsible for it? Umm. And finally, uh, the last point is around the the real time data access. This is something that’s really maturing very, very quickly. In the last three months, if you read the blogs, they’re making improvements on the real time data. I I saw a couple of announcements over the last week that there’s some stuff that’s coming out to increase the amount of sources that they can replicate in real time. So really interesting stuff going on in fabric and the fundamental why of it is it’s software as a service. And I seem to be frozen. Maybe not. Hope I didn’t jump too many screens. Uh, so ohh fabric provides complete analytics. Fabric with the best of breed capabilities across your analytics workflow. At the end of the day, it’s really an open layer, has no proprietary lock INS. You can use it with a variety of different data sources. It’s built on that lake centric and open kind of data experience and it also delivers like AI copilots to accelerate your productivity, help you with discovering insights and enable you to create custom. You know AI solutions directly from this platform, but let’s dive a little bit deeper into the the complete analytics platform. We talked a lot about the SAS. Umm, when you get into the fabric ecosystem as as a user working in your workspace, you have everything available to you with the caveat to what permissions you have. But users can have access to everything within that system. You can build out your data factory components to bring your data in. You can manage the data you have data governance components that you can assign tags to. Uh, and it really becomes a unified, you know, platform. Umm. Uh, you know? So you know the power BI user having access to the real time analytics having access to data Factory, data sent apps. You know, to me, this is like a huge a huge benefit to the platform that you don’t have to jump around between different tools. You don’t have to ask your IT department like, hey, I’m in the Azure portal. Can’t get into data factory. Can’t look at the logs, you’re just doing that all within the single that ecosystem. Umm. But it’s also based on like the one like, you know, uh, aspect of it. So 1 Lake is is a fully functional data lake. Lake House is kind of the architecture that it was built on. Umm. And every time that you provision your your workloads, your workspace, your provisioning, the ability to use a new instance of 1 like in organize your data, the data is is intuitively organized within that ecosystem and within that workspace you’re managing it. Umm, the data is automatically indexed. You know, you get your labeling and lineage. You can enable things like PII scanning to apply automated labelers to it, and then you can manage sharing in the governance right out of right out of the lake house. Yes. The other benefit to this is that you get one copy of the data for all your computes. You don’t have to move the data around once it’s in the one like, uh, you know, this is all handled for you. Uh, and and you do have access to it as well, right? So there’s API access to your data. You can export their work with it and parquet files. Umm, you know? But it is, you know, the source of truth is is in a single point in the lake house. And you, you don’t have the, you know, need to import and export anymore. Umm, the other part about this is that the compute engines are fully optimized to work within this ecosystem. So the levers that you have to that you used to have to pull you know in order to support your workloads or things that you no longer have to manage independently. Umm, you know, as an example, do I need to scale up this? You know this component because I’ve got, you know, a large, you know, you know a large data extract that’s coming in because I had a on boarding customer. Umm, it’s a it’s a managed to pacity across it, so you no longer have to worry about how to optimize your data. Ohm. So one copy is is basically a new feature that came out and you can get shortcuts to your data through one like, so you’re not actually moving this. This is something that’s relatively new, actually very new. We haven’t done a tremendous amount of work other than Pocs with it, but the Nice part about moving this data is that you can support not only your Azure data and your on premise data because it’s typically where people are looking at how do I get my on premise data into fabric? But it also supports, you know, shortcuts to your data in AWS. In GCP I was working with the customer just just last week. That was talking about using Fabric as their their analytics platform, but all of their data sits in AWS and or at least 90 / 90% of it sits in eight of us. So you know, but they felt it was a compelling, a compelling enough reason to have this SAS platform knowing that I don’t actually have to physically move all my data over the wire and I consent basis in order to get the benefits of it. So it really is a multi cloud data estate. Umm, we’re gonna dive a little bit deeper into into the concept. So the data data domains but important to kind of call out before we move through into the deeper dive of some of this is that there’s the concept of domains you know and being able to assign data to the different and align it with the different business units such as your finance department owns the data stewardship of uh of their of the data that comes out of their their department. Umm, you know that the data is going to be secured and governed one place and they control as the data owners you know who has access to the data and what source that you’re how to define the source that you’re. Ohm also you know with this being a SAS product, you don’t really have to worry as much about you know where the data resides. You can reside in different regions without having to manage your different storage accounts and at vastly simplifies the implementation for it. You know, in that respect, uh, so let’s talk a little bit about, you know, the business users as well. So data activator is is a system of detection. It lets your business subject matter experts monitor all your data in the cloud, and they can use the familiar experiences of fabric to model that data and define the changing conditions in the data that they want to look for. Ohm, you know the data actors. The Activator sends notifications, kicks off workflows. Uh, it can even reach directly into operational systems to optimize how your business runs. Uh, so leveraging, you know, connecting data from from your team is to your power BI is is becoming an easier thing. And then, umm, you know the the the big you know sizzle I guess if you wanna say that is you know the AI stuff. So the copilot features that are coming out in that have been released in Power BI and Fabric specifically allow you to do much more than just write SQL queries using a copilot, but also really inspect and interrogate your data and find trends that that would be available. And so you get a lot of data, data, AI driven data insights. Alright, so, uh, universal compute. It’s a big factor in fabric, so it’s all in one and it is hard to plan for. Concurrency has a handy little spreadsheet that can help organizations sort of plan out some of the capacity, but at the end of the day, when you, when you when you provision a Fabric at Microsoft Fabric, your provisioning a specific capacity and that comes in. In in order of magnitudes of two, so you know 2468, who do we appreciate, but you basically deploy a a A capacity unit and then you don’t have to necessarily worry about if I have one job that’s taking up more than the other, all that’s going to get balanced out in the notes, umm, it’s uh, it’s similar to like power BI capacities, but that was you know very specific in those workloads. This takes care of all the other components as well. Nathan, I’ve seen a couple of questions coming up. Is there anything that you need me to stop and touch on? Nathan Lasnoski 21:06 Uh, big question is on Fabric cost scale management like if it’s scales outside of like you’re you’re doing auto scaling on it on Fabric it it scales to some level that you weren’t expecting. Brian Haydin 21:07 Or yeah. Nathan Lasnoski 21:20 How does that fit into capacity management? Brian Haydin 21:23 Yeah. So I think, uh, you know that that calculator that I talked to talked about it’s got, you know, 30 or 40 different inputs that help you kind of plan for your Max capacity unit umm and uh and would assist the organizations and you know figuring out what the the correct sizing you know for the organization is umm you’re not gonna run into uh. You know the same kind of situations you would run in with SQL Server. You know, as an example of data contentions, you might get some things slowed down a little bit as the capacity is throttling some of the transactions, but at the end of the day, you’re limited on that capacity skew ohm and and it it is kind of hard to calculate. You know, it’s not it. It’s. It’s an art. Definitely not a science. But that’s something that we can definitely help with. Umm. So next question here really quick. Can we speak to the security of the proprietary data copilot offers right? You know it’s secure within the company. Sure. I’ll go ahead before we move on and talk about that very, very succinctly. Your data you know in in terms of copilot is your data, and Microsoft is the is very adamant about that. They do not use any of your information outside of the service boundary, you know, period and the story. So that data security they they draw a very strict line around your tenant and your data is your data. It’s not used to train any other models. Uh, you know, unless you’ve explicitly give explicitly given them access to it or person to do something to do so. Uh, so hope that helps. Umm. Nathan Lasnoski 23:19 That’s the same basic thing that you think about with even Office 365. Like they’re not training some broad model on your option 65 emails that go back and forth. Brian Haydin 23:23 Yep. Nathan Lasnoski 23:30 There’s a SLA guarantee. There’s a contract that you signed that that enforces that. Brian Haydin 23:38 Yeah. Absolutely. Umm, we are doing relatively OK on time, but I do wanna wanna keep this moving along. Keep the good questions coming. Those are those are some fantastic questions, by the way. Uh. So you know there, there’s been a lot of talk about what we just had a couple of questions about security here. So let’s go ahead and dive a little bit into that too, right? So you know I’m the Fabric site. You get the data ownership you get, you get the ability to manage who’s going to have access to your data. Are you in the bility to tag your data and you know have that level of scrutiny, but it works seamlessly with Microsoft purview to take that to the next level, right? So you can keep your data safe and governed, you know, like you would have done with, with purview and have a more integrated you know in in more integrated experience around that ohm you know to maybe dovetail off of this a little bit. We’ve been doing a lot of smaller fabric adoption engagements that concurrency and this is exactly where we start start peoples fabric journey is paying a lot of attention to. Uh, you know the data governance before we actually start getting into a position and load the data. It’s like literally step one. Umm. So how do we do that on? There’s some more slides that that, that I have a little bit deeper into this, but again, you’ve got that centralized one leg data hub where all your data is gonna be flowing through. You have data domains and data owners that are gonna create endorsements on that data before. Umm, you know, before it’s accessible to other business units or other users? Uh, we make sure that we have data, data lineage defined and where necessary we’re doing metadata scanning on the data to extract key, PII or Phi information and make sure that we add sensitivity labels to it before it’s before it’s going through its consumption process. Umm, so the one like data hub. Uh, you know, is a place where you can not only like move your data, but it’s also a place for you to discover your data. Uh, so other users that you have enabled or given permission to discover that data, we’ll be able to go to the data hub and find a version of the information that they’re looking for. It’s indexed, it’s searchable and the data owners are the ones that have controlled who has access to it. So that’s a fantastic tool, but before we, before we talk about uh, you know the data before we go too much further on the data ownership, I I definitely wanted to like spend a little bit of time. Nathan and I were talking about this this morning. I’m really kind of taking a step back and making sure that everybody understood what data domains are, so you know the concept of data domains is that the data like the data usually in organizations lives within the operating units within the business. So obviously there’s gonna be some Executive or financial kind of oversight for the health of the company, but there’s gonna be other areas where, like manufacturing or sales or didn’t know other digital assets are gonna have their own their own data sets that really don’t necessarily relate to each other very easily. Umm. And so we help organizations kind of define like what, you know what those business units, how are we gonna organize that data into into individual pillars for consumption ohm when we do that, you know, we can have, we can actually assign owners to that and doesn’t have to necessarily be a technical person or business person. It could be a blend of both, but at the end of the day there needs to be a throat to choke about. Is this data correct and are we satisfied with that data being shared with other business units and within the organization? So this really addresses like. Ohm, you know value driven use cases about about how to use the data. And then once you have the owners, they control all these other components on, you know like the data set setting up the policies, who’s got the visibility to it. And they might do that differently for business consumers. And as they would for people who need access to more of the raw data. And that’s the beauty of the fabric ecosystem is because the lake house has all that data with the lineage attached to it. So now we can talk a little bit about the endorsements. So data owners on you know their primary responsibility, you know, is to manage this data. But at the end of the day, they certify the data set and endorse it at the highest level of value and quality. Umm, you know, for consumption for business users? Umm, only the authorized users can certify that data set. So that means that like you know that that data domains quality standards are aligned to that individual or individuals. I I quality metrics and when you do endorse them, they get higher visibility in the fabric ecosystem. So they’ll be prioritized as people start to search for things. So if you’re searching for new customer and so new customer as defined by Finance is different than sales and sales has endorsed theirs, Finance hasn’t, that’s probably what you’re gonna wind up getting is the one that’s been endorsed. Umm so the data lineage be, you know talked to and here you can see on the screen umm directly from Fabric what that actually looks like. So as you start to look at the data, you can see you can see where it if you’re looking at your. Platinum level. This is the best no, the best, the best data. Uh, sometimes that’s rolled up in, you know, multiple different data sets, and you can actually dive into it and see where did this data come from? What transformations did it go through and how did? How did it get to the to the state that’s had? This is allows you to kind of troubleshoot where you know some of the mistakes might have happened, and I switched banks six years ago, but now I went back to my that the bank that I was at, you know, on the last decade and they treated me as a new customer. But is that really the case? And how do I fix that data that’s broken? Ohh and then the metadata scanning so there’s API available that help to scan the data and that will help with assigning tags for sensitive data as well. Uh, and then this can all be done with, you know, integrated into your AD account. I’m using service principles and and it also has a lot of support for third party tools. We talked a little bit about uh, about the Fabric capacity, got a couple slides here that I give you a little bit more detail, ohm so you can you can take a couple of notes, but at the end of the day, like the purpose here is to kind of simplify the purchasing, you deploy a capacity unit, you can scale that up when you need to. Most people start at a pretty small SKU and then ramp up as they need it. But sometimes you walk into the situation knowing that, like you, you’re gonna need a higher level skew, but it can be used for any of the workloads. Once you’ve purchased the capacities, so you’ve got multiple workspaces, but it’s all tied to a single Fabric, uh, capacity unit and and that’s shared across the different workloads that you might have created within your organization. You still have the ability to create different Fabric capacities. There’s nothing that would limit you from doing that, but you don’t necessarily have to do that. You can actually share that cost across multiple different lines of business, umm. And then it’s, you know, transparent, you can see where the consumption is coming. Like you know what? Which services are using more? Do we need to split that workspace out into its own Fabric capacity unit? And you have that those monitoring tools to be able to do that. So I mentioned like it’s in the scale of twos. Uh, you know the the capacity units, you know, this takes into, you know, into consideration the number of transactions, the number of you know. You know, transformations that you’re making, what kind of analytics you might actually be performing against it, you know, all those metrics kind of go into it and it gives you a kick out like here’s the Fabric capacity unit. There’s a big savings with the reservations. I mean, it’s 40%. So absolutely, if you know that you’re gonna be going Fabric long term. Definitely do the reservation pricing. You’ll save, you know, quite a bit of money. It does get a little bit pricey. We’re gonna talk about copilot here, but just for reference, if you’re thinking about copilot as you look at this screen, the minimum capacity unit to turn to turn copilot on is the F64 SKU, so it’s, you know, on on the reservation price, it’s it’s roughly around 5 grand a month, but there’s a lot of other benefits that come with a F64 SKU and higher. So a good example is that once you get the F64 sharing those power BI with external users becomes a little less complicated from a licensing perspective. Yeah, Nathan’s calling me up for. Uh for rushing through this, I’ll talk a little slower so that we will have less time for questions. I was just joking. Just a joke. Uh. So what about the other costs? Right. There’s some storage costs that might come, you know, come out and play here as well. They’re actually really small. Yeah. We haven’t seen organizations have a big bill for having a big data footprint. I mean, we’re talking about, you know, a couple of pennies per month per month per GB per month, you know, for storage. It just is. It just isn’t something that happens. There is some base. You get the maybe 500 gigs of data storage, maybe a TB of data storage with your one uh with your Fabric capacity unit that you bought Kenneth based on which you you go with. But anything above and beyond that, you know, these are the charges for it. So there are not currently charging for any ingress or egress, but there could be data transfer charges through other appliances that you might have deployed to support the ecosystem, but those wouldn’t be directly in Fabric at this point. So. We didn’t talk a lot about the the AI part about it, but I I would kind of wanna go all the way back to the beginning and just, you know, call out that, you know, as you as an organization are starting to look at this, this data in AI journey that you can consider Fabric as a platform because of the breadth of tooling that is already enabled and the ease of deployment for the organization so. Do we have any other questions that I didn’t cover and I went way faster than Nathan does? Alright, so next steps we would love to have a one on one conversation with your organization if you want to find out a little bit more. So we’re offering a free data platform strategy session to anybody that wants it. Uh, you know, and we can talk about the benefits in detail as it applies to your organization of Fabric or if there’s other things that you want to consider as well. And and also as people start to think about the data in AI journey, we would love to have in AI and visioning session. These are a lot of fun for the organization, and it’s a lot of fun for concurrency to take you through that journey. Also, I think Amy is going to drop in a survey here in the chat and so if you have an opportunity to fill that survey out, we’ve really love to get your feedback. If you, you know, really love this talk, you know, let us know if you really hated this talk. You know, that’s fine. I’ve got thick skin. Let me know how I can do a little bit better next time. Umm and I did see one question ohm. Nathan Lasnoski 37:51 Yeah. The question is, are there any push down optimization options options that would provide cost savings by minimizing compute on the platform? Brian Haydin 38:01 What I would what I’ve been recommending with our customers is to plan out your capacity effectively and so that you don’t have to, you know, do a lot of push up or push down on a regular basis and then you can maximize some of those, minimize the costs on your deployment. That’s the strategy that we’ve been taking because of the virtues of how the capacity units work at scale. Ohm, you can balance out some of those workloads you know. So you typically would try to, you know, find a more balanced seat capacity unit and then scale up if you need to. That’s the approach that most of our customers are taking right now. But that’s an excellent question. Alright, well, uh, if there aren’t any other questions, please fill out that survey before you leave. We would love to hear from you and absolutely would love to, to meet with your organization if you’re just a thanks everybody.