Microsoft Fabric Updates Blog

Announcing Delta Lake support in Real-Time Analytics KQL Database (Preview)

As part of the One logical copy effort, we’re excited to announce that you can now enable availability of KQL Database in Delta Lake format.

Delta Lake  is the unified data lake table format chosen to achieve seamless data access across all compute engines in Microsoft Fabric.

The data streamed into KQL Database is stored in an optimized columnar storage format with full text indexing and supports complex analytical queries at low latency on structured, semi-structured, and free text data.

Enabling data availability of KQL Database in OneLake means that customers can enjoy the best of both worlds: they can query the data with high performance and low latency in their KQL database and query the same data in Delta Lake format via any other Fabric engines such as Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more.

KQL Database offers a robust mechanism to batch the incoming streams of data into one or more Parquet files suitable for analysis. The Delta Lake representation is provided to keep the data open and reusable. This logical copy is managed once, is paid for once and users should consider it a single data set.

Users will only be charged once for the data storage after enabling the KQL Database availability in OneLake.

Enable OneLake availability

  1. To enable data availability in OneLake, browse to the details page of your KQL database or table.
  2. Next to OneLake folder in the Database details pane, select the Edit (pencil) icon.

Screenshot of the Database details pane in Real-Time Analytics showing an overview of the database with the edit OneLake folder option highlighted.

3. Enable the feature by toggling the button to Active, then select Done.

Screenshot of the OneLake folder details window in Real-Time Analytics in Microsoft Fabric. The option to expose data to OneLake is turned on.

You can enable data availability at a KQL database or table level.

Once you enable data availability, you can access all the new data added to your database at the given OneLake path in Delta parquet.

You can also choose to create a OneLake shortcut from Lakehouse, Data warehouse, or query the data directly via Power BI Direct Lake mode.

End-to-end streaming architecture in Fabric

Customers can now leverage data availability in OneLake to build more efficient and performant systems to handle high volume, and low latency streaming data in Microsoft Fabric.

  1. Eventstream can capture streaming data from multiple sources at scale.
  2. Eventstream allows pushing raw data into KQL Database seamlessly.
  3. KQL Database can be used to build a medallion architectural pattern with the help of in-built transformation functions such as update policies, and materialized views. The medallion structure is reflected as follows:–
    1. Bronze layer: raw data as received from Eventstream.
    2. Silver layer: deduplicated and enriched data.
    3. Gold layer: aggregated data suitable for reporting.
  4. Then you can either choose to make all three layers available in OneLake, or only make the aggregated data available for building your reports directly with Power BI.

This data design pattern allows you to scale efficiently for large incoming streams. KQL Database serves real-time analysis with low latency while making the data available in Delta Lake.

For more information on enabling data availability in OneLake, see One logical copy.

Relaterade blogginlägg

Announcing Delta Lake support in Real-Time Analytics KQL Database (Preview)

oktober 31, 2024 från Jovan Popovic

Fabric Data Warehouse is a modern data warehouse optimized for analytical data models, primarily focused on the smaller numeric, datetime, and string types that are suitable for analytics. For the textual data, Fabric DW supports the VARCHAR type that can store up to 8KB of text, which is suitable for most of the textual values … Continue reading “Announcing public preview of VARCHAR(MAX) and VARBINARY(MAX) types in Fabric Data Warehouse”

oktober 29, 2024 från Dandan Zhang

Managed private endpoints allow Fabric experiences to securely access data sources without exposing them to the public network or requiring complex network configurations. We announced General Availability for Managed Private Endpoint in Fabric in May of this year. Learn more here: Announcing General Availability of Fabric Private Links, Trusted Workspace Access, and Managed Private Endpoints. … Continue reading “APIs for Managed Private Endpoint are now available”