Microsoft Fabric Updates Blog

Decoding Data with Confluent Schema Registry Support in Eventstream (Preview)

We are pleased to announce that Eventstream’s Confluent Cloud for Apache Kafka streaming connector now supports decoding data from Confluent Cloud for Apache Kafka topics that are associated with a data contract in Confluent Schema Registry. The Challenge with Schema Registry Encoded Data The Confluent Schema Registry serves as a centralized service for managing and … Continue reading “Decoding Data with Confluent Schema Registry Support in Eventstream (Preview)”

Enhancing Data Transformation Flexibility with Multiple-Schema Inferencing in Eventstream (Preview)

Introducing multiple-schema inferencing in Eventstream! This feature empowers you to work seamlessly with data sources that emit varying schemas by inferring and managing multiple schemas simultaneously. It eliminates the limitations of single-schema inferencing by enabling more accurate and flexible transformations, preventing field mismatches when switching between Live and Edit modes, and allowing you to view … Continue reading “Enhancing Data Transformation Flexibility with Multiple-Schema Inferencing in Eventstream (Preview)”

User Data Functions now support async functions and pandas DataFrame, Series types

Microsoft Fabric has introduced new features for its User Data Functions (UDFs), enhancing Python-based data processing capabilities within the platform. These updates include support for asynchronous functions and the use of pandas DataFrame and Series types for input and output, enabling more efficient handling of large-scale data. • Async function support: Developers can now write async functions in Fabric UDFs to improve responsiveness and efficiency, especially for managing high volumes of I/O-bound operations, such as reading files asynchronously from a Lakehouse. • Pandas DataFrame and Series integration: UDFs can accept and return pandas DataFrames and Series, allowing batch processing of rows with improved speed and performance in data analysis tasks. An example function calculates total revenue by driver using pandas groupby operations. • Usage in notebooks: These functions can be invoked directly from notebooks using pandas objects, facilitating efficient aggregation and analysis of large datasets interactively within Microsoft Fabric. • Getting started and benefits: Users can enable these features by updating the fabric-user-data-functions library to version 1.0.0. The enhancements reduce I/O operations, enable concurrent task handling, and improve performance on datasets with millions of rows.

Secure Data Streaming with Private Endpoints in Eventstream (Generally Available)

We’re excited to announce the General Availability of Managed Private Endpoints (MPE) in Fabric Eventstream. This network security feature allows you to stream data from Azure resources to Fabric over a private and secure network without the complexity of manual network configurations. Why Network Security Matters for Streaming As organizations increasingly adopt real-time data streaming … Continue reading “Secure Data Streaming with Private Endpoints in Eventstream (Generally Available)”

Permission model improvements for Azure and Fabric Events

Azure and Fabric Events offer a powerful capability within Real-Time Intelligence that enables you to ingest system events that are generated in Microsoft Fabric and Azure to deliver them to consumers in Microsoft Fabric like Activator for setting event-based triggers or Eventstream to stream and process events to other destinations. Permission model To subscribe to Azure and Fabric events … Continue reading “Permission model improvements for Azure and Fabric Events”