Activator as an Orchestrator of the Fabric Event Driven flows
With Fabric Events general availability, the role of Activator expands from setting notifications and acting on your data in real time to becoming an orchestration centerpiece.
Activator acts as a connecting tissue enabling complex event-driven and data-driven flows. Let’s look at a very common architecture we often see our customers implement:
In this architecture we have a Data pipeline executing data transformation and Database ingestion job, storing the data about the executed pipelines in the Eventhouse.
But what we are really interested in is not simply setting up the pipelines and letting them run unattended, but making sure the pipeline jobs are executed properly, no data loss has occurred, and, in case of errors or failures – that they are properly and automatically handled. And in the worst-case scenario, where failures accumulate, we make sure we notify a proper person for manual investigation and intervention.
To achieve the above, we subscribe to Job Events reported by the Fabric platform.
We stream these events of job executions, both successful and failed, into the Eventhouse via Eventstream. We store the data from these job executions in the Eventhouse table for further reporting but also for real-time management orchestrated by Activator.
Once in the Eventhouse, we can set up an Activator Rule based on executing a KQL query that inspects the reason for the failure and reruns the pipeline if needed. We can also set up a Real-Time Dashboard that tracks the pipeline jobs executions over time and if a certain threshold of failures was breached. The activator will trigger the alert to a relevant team or person for further investigation.
The Real-Time Dashboard can also be used for overall reporting and monitoring of our pipelines.
In this flow activator is the instrumental component orchestrating the data processing flow, ensuring the processing meets the SLA and automating the operational control without writing a single line of code.
This recorded demo shows how this flow is set up:
Activator event driven orchestration is available on top of Azure storage events, Workspace events, Job events, OneLake events, and soon Capacity Utilization events, when become available publicly. The orchestration is done in a similar manner by subscribing to the events we are interested in as they are emitted by the Fabric platform.
To learn more about fabric events check out this blog post: Build event-driven workflows with Azure and Fabric Events (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric
