Processing CDC streams using Fabric Eventstreams SQL
Why process database changes in real-time
Modern applications don’t just store data—they react to it. Orders are placed, inventory levels change, prices are updated. Each of these changes can be a signal that something needs to happen now: update a dashboard, trigger a workflow, or feed an operational application.
That is why many organizations are turning to real-time processing of database change data capture (CDC) events instead of waiting for batch pipelines to catch up.
Fabric Eventstreams can ingest CDC data directly from Azure SQL, PostgreSQL, MySQL, and several other databases. For relational sources, the changes are streamed into Eventstreams using Debezium, a standard CDC format.
The challenge is not getting the CDC events—it’s making sense of them. Raw CDC formats, like Debezium, force every downstream consumer to understand low-level semantics and encoding formats.
The challenge of CDC streams
Raw CDC events are rich, but verbose. They include nested before and after images, operation types, transaction metadata, and source details. While powerful, this format often forces every downstream consumer including analytics queries, alerts, operational dashboards, to understand low‑level CDC semantics and write custom JSON parsing logic.
Figure: Example of a raw Debezium CDC event
Shape CDC events once in Eventstream using SQL
Fabric Eventstreams provides SQL operators that let you centralize CDC shaping in one place, instead of pushing that complexity to every consumer.
To see how this works, let’s work through a simple scenario involving grocery store sales and inventory changes.
Scenario: Grocery store sales and inventory changes flowing from Azure SQL DB
Imagine an Azure SQL database backing a statewide grocery system with two main tables that change with every transaction.
- SalesTransactions – records sales and returns (mostly insert)
- ProductInventory – tracks inventory levels as products are sold and replenished (inserts and updates)
A common requirement is to monitor inventory levels in near real-time and alert procurement teams when stocks run low.
If you connect this database to an Eventstream using the Azure SQL CDC connector and route the data directly to an Eventhouse or Lakehouse, all CDC events, across all tables, land in a single destination table as raw JSON.
Figure 2: Raw CDC events in a single Eventhouse ingestion table
At this point, it becomes the user’s responsibility to:
- Filter relevant tables
- Parse nested JSON payloads
- Normalize events into clean, query-able shapes
Shaping CDC event feeds with Eventstreams SQL
With Eventstreams SQL, you can transform raw CDC events into business‑ready streams using a small amount of familiar SQL.
- Filters for data-change (DML) events.
- Selects events from the ‘ProductInventory’ table.
- Flattens the “after” image into separate columns that reflect the source table schema.
- Retains key metadata like operation, source table information, timestamp etc.
SELECT -- EXTRACT THE REQUIRED SOURCE METADATA INFORMATION payload.op AS OperationType, payload.source.db AS SourceDatabase, payload.source.[schema] AS SourceSchema, payload.source.[table] AS SourceTable, payload.source.[table] AS SourceTable, -- EXTRACT ALL THE SOURCE TABLE COLUMNS USING A WILDCARD payload.[after].* INTO [grocerystoreinventory-table] FROM [grocerystoreinventory-es-stream] WHERE (payload.[after] IS NOT NULL OR payload.before IS NOT NULL) AND payload.source.[table] = 'ProductInventory'
The result is a normalized stream where each row reflects the source table schema—no nested JSON, no custom parsing. Instead of working with opaque payloads, downstream systems can now reason about structured, meaningful data.
Figure: Data preview of a normalized flattened CDC feed
Putting it all together
The previous pattern changes how teams work with CDC:
- CDC complexity is handled once in Eventstreams.
- Downstream consumers focus on business logic, not parsing.
- The same normalized stream can power analytics, alerts, and applications.
- Changes to CDC logic don’t require rewriting every consumer.
With just a small amount of SQL, raw CDC events are transformed from low-level database changes into analytics-ready streams that are easy to reason about and easy to use with:
- Eventhouse for real-time analytics.
- Lakehouse for historical analytics & model-training using Fabric Spark.
- Activator to trigger alerts or workflows.
Instead of forcing every consumer to understand CDC semantics, that logic is centralized once — in SQL.
Need to do more?
In a full production system, you may have more complex requirements. You can meet all of those (and more) using simple techniques and capabilities in Eventstreams SQL.
- Need only specific columns from a given source table? Just explicitly specify each column you need in the SELECT statement e.g. “payload.[after].StoreID AS StoreID”.
- Want to reuse the core shaping and metadata extraction logic? Use the WITH statement to do exactly that.
- Want to add your own custom logic when processing CDC events for a specific table? Use the WITH statement in conjunction with a SELECT statement filtered to the events for the specific table.
What’s next?
As systems grow and complexity with more tables, frequently evolving schemas, and higher event volumes, CDC processing must scale without adding operational burden.
Stay tuned for some exciting announcements in the coming weeks!
Get started today
Ready to get started with these ideas? Explore the following resources to help you get started.
Microsoft Fabric Eventstreams Overview
Add Azure SQL Database CDC source to an eventstream
Process Events Using a SQL Operator
We’d love your feedback!
If you find this blog helpful, please give it a thumbs up!
We would love to hear what real-time scenarios you’re exploring and what topics you’d like us to cover in future posts. Drop us a comment or reach out to us at askeventstreams@microsoft.com
Have new feature ideas? Submit them at Fabric Ideas – Microsoft Fabric Community
