Microsoft Fabric Updates Blog

Microsoft’s vision of an open data lake ecosystem: Open lakes, not walled gardens

In today’s data-driven world, enterprise data estates contain many data sources for a variety of reasons, including differences in type of usage (operational vs. analytic), differences in ownership, and the presence of legacy infrastructure that is part of a corporate merger or acquisition. In addition, enterprises constantly acquire and refresh data from external sources. For analytics to be effective, we require a unified view across the entire data estate. However, creation and maintenance of data pipelines to aggregate data have consistently posed a significant hurdle.

With the maturation of cloud-native big data platforms and the exciting revolution in generative AI, the potential for data-driven decisions and operational optimizations has never been greater, raising the urgency of solving the longstanding problem of how to enable organizations to bring together estate-wide data for analytics.

Optimizing processes by simplifying data

We believe that the emergence of open, updatable table formats presents us with a unique opportunity to solve this problem by standardizing these formats across all analytic engines, and by simplifying data replication. In fact, as an increasing number of engines adopt open data formats, we can minimize data replication by instead using references to data sources.

Further, as the value of data is recognized, we are seeing corresponding emphasis on right-use and increasing regulation. Thus, it is important that we be able to govern the entire data estate in a compliant manner, and in particular, evolve current best practices for aggregating estate-wide data to reflect the emerging world of cloud-native data lakes that bring together a diverse range of analytic capabilities, from exploratory tools, to AI models, to tools for serving data and rich business reports reliably, securely, and at scale.

Shaping the future of data analytics

This vision of the future of analytics is at the heart of OneLake design in Microsoft Fabric. We have striven to make it the “one place to bring all data for analytics”, making it easy to virtualize and aggregate data from all sources. Fabric itself then democratizes access to the wealth of insights that can be unlocked, thanks to a Microsoft 356-like simplicity in bringing analytic tools to bear on the data through intelligent software as a service, and by infusing AI copilot experiences to assist with complex tasks in-stride. The entire life cycle of analytics, from aggregating data to unlocking rich insights for appropriately authorized users, can be managed using the data governance capabilities of Fabric and the integrated estate-wide governance capabilities of Microsoft Purview.

Read the whitepaper to learn more!

Related blog posts

Microsoft’s vision of an open data lake ecosystem: Open lakes, not walled gardens

July 24, 2024 by Jason Himmelstein

Welcome to the July 2024 update. Here are a few, select highlights of the many we have for Fabric. Creating and managing Git branches & connected workspaces with Git integration just got with the latest enhancements to Fabric Git integration. You now have the capability to perform restore-in-place of a warehouse in Microsoft Fabric through … Continue reading “Microsoft Fabric July 2024 Update”

July 16, 2024 by Miquella de Boer

This guide will show you how to create a OneLake shortcut to a VPC-protected Amazon S3 bucket. Why use the On-premises-data gateway? Today, organizations are protecting data by leveraging network security capabilities like virtual networks, firewalls and virtual protected clouds (VPC). To access data securely and to provide a bridge between protected environments and Microsoft … Continue reading “Creating a shortcut to a VPC-protected Amazon S3 bucket”