Making your systems work with Microsoft Fabric

With a swathe of recent announcements from Microsoft at Build and FabCon, it is clear that Microsoft’s vision for Fabric has moved far beyond business intelligence and analytics, and they are firmly positioning it as the foundation for all and any data driven workloads throughout an organisation - business intelligence, decision intelligence, artificial intelligence, data science, agentic automation, and with perhaps one of the most eye-catching recent announcements, Fabric Apps, prompt based app development with a secure data layer baked right in.

What is really changing here, is the range of applications and workloads that we can build directly from Fabric, and an ever-expanding toolbox to play with in building those solutions, many of which do similar things in slightly different ways.

What’s not changing, rather is getting even more important, is the fundamental requirement to integrate the data from our business systems into Fabric and serve it up in a way that will serve all those different workloads in a timely, performant, reliable, and cost-effective manner.

The challenge for solution architects and data engineers working with Microsoft Fabric, lies in understanding differences and nuances between all the different tools available, embracing that flexibility and range of different tools that each suit different use cases, whilst still retaining some semblance of control, consistency, and predictability to the way things are built so it doesn’t become a support nightmare of endless reverse engineering of different individual solutions.

Now, going back to that house and its old wiring - (get our article on working with legacy systems here if you didn’t already read it) Just as there are a million and one influencers out there showing you the perfect way to wire a house that only exists on the internet, there are a plethora of blogs, tutorials, and YouTube videos online telling you how to design the perfect medallion architecture lakehouse environment in Fabric, with beautifully consistent orchestration patterns, based on a data estate that you will never actually find out there in the wild.

This is where the “pragmatic mindset” comes into its own and is actually an area where Fabric really shines. When we start learning about Fabric, one of the first things we hear about is OneLake. Often described as a “OneDrive for Fabric”, that’s a useful comparison to simplify its role as a shared storage layer for Fabric but, for one thing it sells it a little short in terms of its extended functionality, it also creates an impression that we need to move all of our data in there before we are able to do anything with it. That could make a potential migration project look daunting, especially if you’ve already got established enterprise data components in place, like Databricks, or Synapse, etc.

Crucially though, it’s not really true. Fabric is genuinely good at interoperability, not just with operational business systems, but with other data and analytics platforms, and OneLake is not just good at storing and managing data. With features like mirroring and shortcuts, it’s also good at connecting to and providing a conduit to data that sits elsewhere. We can leverage these tools to make the move towards Fabric a gentle slope rather than a steep cliff.

So how do we approach this? Well, everyone is different of course, but ultimately what we try to do is understand the current position and the legacy of how you got there, understand the aspirations and where you would like to get to, and map out a clear set of priorities along the way. Who is the primary audience? What experience do we want them to have? What can be reused? What can be recycled? What needs to be rebuilt? What doesn’t exist at all?

None of this is to say that we are ignoring architecture, we are just approaching it in a phased and modular way that serves the business objectives rather than the technology itself. We don’t need to shoehorn everything into the “best” medallion pattern that we’ve read from a blog, we need to identify what the “best” pattern is for you (which might not be a medallion at all). We need to understand what the primary benefits you actually want to gain from Fabric are, and where the quick wins are to help you get people up and running early, gain momentum, and maximise your ROI.

To refer (stretch!) once more back to that old house’s wiring, maybe we don’t need to rip it all out and replace right away. Maybe, replace the consumer unit and deal with the fire hazards so we can turn some lights on, then we’ll be able to see what the next priorities look like more clearly from within.

A guiding principle that we like to try and use along the way - as the audience widens the technology choices should narrow. That’s not intended to stifle, it’s intended to simplify decision making and encourage consistent usage.

How does that apply here? Well, let’s look at the primary audience we are trying to serve with a move to Fabric and focus on how we simplify their experience first. If these are our report consumers for instance, that’s straight forward. We prioritise getting all their reporting into Power BI so they only have one place to go to access their data, even if the data estate behind those reports may still be a bit more fragmented. We can use OneLake to link it all up, before we’re ready to actually start moving it into a Fabric first model. Then we can start working our way back through the stack, getting that funnel more under control as we work our way back through the layers.

If you’re trying to work out how to make Fabric work with your data estate, rather than making the estate work with Fabric, we’d love to help you think it through and find the best fit for you. Drop us a line at info@ptr.co.uk or follow us on LinkedIn.