Microsoft Fabric Updates Blog

Ingest, transform, and route real-time events with Microsoft Fabric event streams

See Arun Ulagaratchagan’s blog post to read the full Microsoft Fabric preview announcement.
 
You can now ingest, capture, transform and route real-time events to various destinations in Microsoft Fabric with a no-code experience using Microsoft Fabric event streams. It enables customers to ingest real-time event data from external event sources into the data stores in Fabric. The events could be transformed into the native formats required for target destination. For example, Eventstream could transform the events into Delta Lake format for the Lakehouse, into SQL columns based on the table schema, or filter events so that homogenous data can be sent to a KQL table.
 
To create a new instance of Microsoft Fabric event streams, navigate to Real-time Analytics in your Fabric workspace where you will find Eventstream.

After creating a new Eventstream item, you will land into a no code & easy to use experience where you can configure your eventstream from source to destination.
  

What is Fabric event streams

  1. Centralized place for real-time events: It provides the capability to capture, modify, and direct your streaming data in real-time using a fully managed and scalable infrastructure.
  2. Multiple source connectors: It enables you to ingest your real-time streaming data from 3 source types today: Azure Event Hubs, Sample data or Custom application.
  3. Multiple destinations: It enables you to transform/capture/route real-time streaming data to 3 destination types today: KQL database, Lakehouse, or Custom application.
  4. No code experience: It provides an intuitive and easy to use drag & drop experience with end-to-end data visibility and monitoring.


 
Let’s dive into each point stated above in detail:

1. Centralized place for real-time events 

Microsoft Fabric event streams utilize a scalable infrastructure that efficiently manages scaling and resourcing automatically. This enables you to effortlessly capture your real-time events, facilitating streaming ETL (Extract, Transform, Load) operations, as well as transforming and routing the events to your desired destination.
 

2. Multiple source connectors

Microsoft Fabric event streams enable you to ingest your real-time event data from 3 source types today:

    • Azure Event Hubs: You can seamlessly ingest events from an Azure event hub configured using Fabric Data connections.
    • Sample data: You can easily configure your eventstream to continuously ingest sample events from either Yellow Taxi data or Stock Market data.
    • Custom application: With this source, a streaming endpoint is created that allows customers to have their custom streaming application communicate directly with Fabric with either a Kafka client or an AMQP client.

 

3. Multiple destinations

Microsoft Fabric event streams enable you to transform and/or route your real-time event data to 3 destination types today: 

    • Fabric KQL Database: This destination helps with direct ingestion of your real-time events into KQL database. Once the data lands in KQL database, you can then perform additional queries/analysis to gain deeper insights or build Power BI reports.
    • Fabric Lakehouse: With this destination, you can transform (manipulate, filter, aggregate etc.) your real-time events before routing into a Lakehouse table. Real-time events are converted into Delta Lake format and then stored in the designated Lakehouse tables.
    • Custom application: With this destination, you can configure your custom applications to use their Kafka or AMQP client application to pull the events directly from Fabric.

Evenstream sources and destination

 

4. No code experience

There is a super simple & easy to use drag & drop experience with end-to-end data visibility. You can view the shape of your data at every step from source to transformation to destination. You can also monitor each step by looking at meaningful insights.

Drag & drop no-code experience. Transformation logic and data preview.

End-to-end data visibility

 
To see detailed documentation, please see link.  If you have any questions or feedback, please reach out to us at askeventstreams@microsoft.com 
  

Get started with Microsoft Fabric

Microsoft Fabric is currently in preview. Try out everything Fabric has to offer by signing up for the free trial—no credit card information required. Everyone who signs up gets a fixed Fabric trial capacity, which may be used for any feature or capability from integrating data to creating machine learning models. Existing Power BI Premium customers can simply turn on Fabric through the Power BI admin portal. After July 1, 2023, Fabric will be enabled for all Power BI tenants.
 
Sign up for the free trial. For more information read the Fabric trial docs.
  

Other resources

If you want to learn more about Microsoft Fabric, consider:

Related blog posts

Ingest, transform, and route real-time events with Microsoft Fabric event streams

May 23, 2024 by Will Thompson (HE/HIM)

Microsoft Fabric’s new Real-Time hub makes it easier than ever to discover, preview, and use streaming data from around your organization. But the data in these streams is only as useful as the actions and changes it can drive in your business. Data Activator provides a no-code experience for automatically taking actions when patterns or … Continue reading “Alerting and acting on data from the Real-Time hub”

May 23, 2024 by Salil Kanade

We are excited to announce the public preview release of Copilot for Data Warehouse in Microsoft Fabric, a groundbreaking AI assistant designed to transform your data warehousing tasks. Data warehouse development can be daunting for SQL developers, especially under tight timelines where insights are needed “yesterday”.  Developers may spend hours writing code, building schemas, documentation, … Continue reading “Announcing the Public Preview of Copilot for Data Warehouse in Microsoft Fabric”