Schema Registry: Creating type-safe pipelines using Schemas and Eventstreams (Preview)
Additional Authors: Anasheh Boisvert
Real-Time Intelligence is an end-to-end solution in Microsoft Fabric that powers event-driven and streaming applications. It enables the extraction of insights, visualization, and action on data in motion by handling data ingestion, transformation, storage, analytics, visualization, tracking, AI, and real-time actions. Fabric Eventstreams enables ingestion, transformation, and routing of real-time data across Microsoft Fabric RTI and external systems. Organizations that rely on real-time event processing to power analytics, automation, and decision-making, require reliable delivery of data. When creating new pipelines, it is important to easily discover the structure of data you are ingesting into your Eventstream. Without it, you risk missing data, or ingesting data you didn’t expect creating failures, and ultimately, unreliable insights and outcomes.
Schema Registry in Fabric Real-time Intelligence is a centralized way to discover, manage, and evolve data schemas in Fabric Eventstreams and beyond.
Why use Schema Registry with Eventstreams?
When connecting data sources to Eventstreams, you rely on inferred schemas. If the first few events in a stream don’t fully represent the structure of your data, you could end up with incomplete or incorrect field detection. Additionally, if you have ‘dry streams’ and don’t have sample data or events, you can’t determine the shape of the data until the flows start. Furthermore, once you have learned the schema, and configure downstream sinks such as Eventhouse, a change in the event structure could result in new tables being created. In some cases, this might be desirable. However, if you want to have predictable pipelines, with a pre-determined structure, you can use schemas to control what data is allowed into your Eventstream.
With Schema Registry in Microsoft Fabric RTI, a registered schema is your guarantee of what’s in the stream you can:
- Explore and understand the data structure instantly.
- Start applying pre-processing configurations or write queries without guesswork.
- Implement governance by using shared data contracts between publishers and consumers.
In addition to building type-safe, reliable Eventstream processing, any downstream components you add to your Eventstream also benefit from schema awareness.
Destinations like Eventhouse can now use registered schemas to:
- Automatically create a table per schema.
- Or store all schemas in a raw format within a single table.
Your downstream systems can now start from trusted, explicit, and well-structured definitions.
What Is the Event Schema Registry?
Schemas act as a contract enforcement layer between producers and consumers of event data. Schemas define the expected structure of events, names, types, using Avro (v1.12) to ensure that the data received by the Eventstream is structured as expected.
The Event Schema Registry is a centralized repository for organizing and managing data schemas within Fabric RTI. It provides a structured framework for defining what your event data should look like—specifying fields, data types, and structure. By registering schemas, you ensure that only events matching your defined structure move through your architecture, catching errors early and keeping your data clean, consistent, and as expected.
Why Use Schema Registry in Fabric Real-Time Intelligence?
- Improved Data Quality: Schema validation is applied at every stage—from when events enter through Eventstream, during preprocessing and transformation, and before delivery to destinations like Eventhouse.
- Control and Consistency: Only events that conform to registered schemas are allowed, preventing bad or unexpected data from disrupting downstream processes.
- Data Integrity and Reuse: Schemas can be reused in different Eventstreams, and access controls are in place to help with permission management
Key Concepts and Terminology
| Concept | Description |
| Event schema sets | Logical containers for related schemas with centralized access control. At the group level, users can manage who can view, edit, or modify the schemas within. |
| Avro Format | Currently the only supported format, chosen for its compact binary encoding. |
| Registration Methods | Create schemas using a visual UI builder, upload a file, or paste your schema directly in Code View. Today, schemas must be added individually. |
| Versioning | Supported, but semantic versioning and compatibility checks are not yet enforced in public preview. |
Modifying and Versioning Schemas
You can modify schemas after adding them to Fabric RTI. However, while in preview, compatibility checks are not enforced. Recommended practice:
- Add new schema versions instead of modifying existing ones.
- Update Eventstream topology to reference the new schema.
- Retire old schemas once validated.
This approach minimizes risk until compatibility support is in place.
Known Limitations (Preview)
We are working to improve these constraints. If any issues affect your use of Schema Registry, please share your feedback.
Schema-based validation: Schema-based validation must be enabled at the time of Eventstream creation; it cannot be applied retroactively.
- Input Sources: Supported sources include Custom Endpoint and Azure SQL Change Data Capture (CDC).
- Destinations: Events validated by schema can be written to Eventhouse in Push Mode or to a Custom App. Additionally, schema-validated events may be streamed to a Derived Stream.
- Compatibility Modes: Enforcement is not supported. Ensure that changes, such as modifying an attribute type from integer to string, do not disrupt existing flows, as this may result in downstream complications.
- Non-Conforming Events: Dropped events are recorded in Fabric Diagnostics but are not routed to alternate streams or storage locations. Future updates will improve visibility and provide reprocessing options.
Getting Started
To register your first schema:
- In the Fabric RTI portal, select ‘+Create’ and choose ‘Event schema set (preview)’.
- Name your schema set and create it.
- Add new event schemas by uploading, pasting, or building them in the UI.
- Assign schemas to input sources and manage them over time
Best Practices
- Monitor Fabric Diagnostics regularly to track validation issues.
- Add new schemas for major changes rather than modifying existing ones.
- Plan for future enhancements as public preview progresses.
Schema Registry in Fabric RTI doesn’t just clean up your data — it removes uncertainty from the entire analytics pipeline. You get clarity, reliability, and a faster path from event to insight.
What’s next
Support for sourcing events from Azure EventHub’s into Schema-Enabled Eventstreams enters private preview on 8/15, and entry is open until 9/15. To sign up complete the form.
Reach out and participate in our previews and influence how we shape schema management. Stay tuned for more features and expanded support as this capability evolves!
