Microsoft Fabric Updates Blog

Introducing Mirroring in Microsoft Fabric

As mentioned in Arun’s blog post, we are thrilled to announce that Mirroring in Microsoft Fabric is coming soon.

In today’s AI driven world, analytics platforms are only as good as their data. With the ever-increasing amount of data being collected in various applications, databases, and data warehouses in an enterprise, managing and ingesting data into a central platform for purposes of analytics and AI is a cumbersome and costly process. Databases and data warehouses use proprietary storage formats making the ability to create shortcuts to their data impossible. Data needs to be extracted, transformed, normalized, and made available in a central place for analytics. Even when this is implemented, data is not real-time making any insights stale pretty quicky resulting in users having to query the data in the source.

Mirroring provides a modern way of accessing and ingesting data continuously and seamlessly from any database or data warehouse into the Data Warehousing experience in Microsoft Fabric. This is all in near real-time thus giving users immediate access to changes in the source!

Let’s dive deeper into its capabilities.

Access and manage any database or data warehouse

Any database can be accessed and managed centrally from within Fabric without having to switch database clients. By just providing connection details, your database is instantly available in Fabric as a Mirrored database. Familiar database editors provide the ability to manage the database. Here is an example of the Azure Cosmos DB editor available within the Mirrored database in Fabric. Any edits go directly against the Azure Cosmos DB source.

A screenshot of a computer

Description automatically generated

Real-time data replication

There is no complex setup or ETL for data replication. With the same connection details, data is replicated in a reliable way in real-time. An initial snapshot is created after which data is kept in sync in near real-time with every transaction whether a new table is created, or new data is inserted/updated/deleted. Replication uses the database’s Change Data Capture (CDC ) technology, transforms it into appropriate Delta tables and lands it in OneLake. Intelligent logic determines when the source has changed, ensuring compute isn’t used unnecessarily before replicating the data. Granular controls enable configuring what is mirrored into Fabric. Detailed monitoring is also available to gain insights into mirroring operations and when the replica in Fabric OneLake was last refreshed. From here on, the data is ready for consumption immediately in any Fabric workload.

Data warehousing experiences simplified!

Every Mirrored database comes with default data warehousing experiences via a SQL Analytics Endpoint which houses the metadata of the Delta tables and points to the data in OneLake. Whether a SQL developer or citizen developer, one can query using the T-SQL editor which comes with full Intellisense or the visual query editor.

A screenshot of a computer

Description automatically generated

Cross-joining Mirrored databases, Warehouses, Lakehouses

As all data in Fabric, whether added as a shortcut to a lake in a Lakehouse or is data ingested through Data Factory, Warehouse or Spark is already in Delta format, data can be cross joined. Now with Mirroring, data across any database can be cross joined as well enabling querying across any database, warehouse or lakehouse whether that be data in Azure Cosmos DB, Azure SQL DB, Snowflake, MongoDB etc.

A screenshot of a computer

Description automatically generated

Data Science experiences unlocked!

Any data scientist can create a Lakehouse with a shortcut to the Mirrored database and use Notebooks to analyze and create models with the data.

A screenshot of a computer

Description automatically generated

Power BI Direct Lake mode

Power BI reports and semantic models can be built in Direct Lake mode getting the blazing fast performance of import mode but without duplicating the data. As Direct Lake mode also supports reading Delta tables right from OneLake, the Mirrored database is Power BI ready.

A screenshot of a computer

Description automatically generated

Mirroring Availability

Azure Cosmos DB, Azure SQL DB and Snowflake customers will be able to use Mirroring to mirror their data in OneLake and unlock all the capabilities of Fabric Warehouse, Direct Lake Mode, Notebooks and much more. SQL Server, Azure PostgreSQL, Azure MySQL, MongoDB and other databases and data warehouses will be coming in CY24. For participation in our early adopter program, submit your application here.

In the meantime, to try out Fabric, sign up for a free trial.

Related blog posts

Introducing Mirroring in Microsoft Fabric

April 16, 2024 by Ruixin Xu

We are pleased to share a set of key updates regarding the Copilot in Microsoft Fabric experiences. The information in this blog post has also been shared with Fabric tenant administrators. Below are the highlights of the changes. This change is an important milestone to eventually allow Copilot to reach GA within this year. See … Continue reading “Copilot in MS Fabric: Soon available to more users in your organization“

April 15, 2024 by Santhosh Kumar Ravindran

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to … Continue reading “Introducing Job Queueing for Notebook in Microsoft Fabric”