Microsoft Fabric Updates Blog

Introducing Data workflows in Microsoft Fabric

We are thrilled to announce the preview of Data workflows, a transformative capability within Microsoft Fabric that redefines your approach to constructing and managing data pipelines. Data workflows in Microsoft Fabric is powered by the Apache Airflow runtime, and provides an integrated, cloud-based platform for development, scheduling, and monitoring python-based data workflows, articulated as Directed Acyclic Graphs (DAGs). This innovation delivers a Software-as-a-Service (SaaS) experience for data pipeline development and management using Apache Airflow, making Apache Airflow runtime readily accessible for the development and operationalization of your data workflows.

Some key functionalities:

  • Instant Apache Airflow Runtime Provisioning: Initiate a new Data workflow and immediately access an Apache Airflow runtime to run/ debug/ operationalize your DAGs.
Instantly provisioned Apache Airflow runtime when you create a new Data workflow.
  • Versatile Cloud-Based Authoring (IDE): In addition to your existing development tools to craft Apache Airflow DAGs, you can utilize the cloud-based authoring environment provided by Data workflows for a truly cloud-native and SaaS-optimized authoring and debugging experience.
    Screenshot demonstrating the authoring capabilities for DAGs in Data workflows.
  • Dynamic Auto-Scaling: Execute hundreds of Apache Airflow tasks concurrently with our auto-scaling feature, designed to mitigate job queuing and enhance performance.
  • Intelligent Auto-Pause: Achieve cost-effectiveness by automatically pausing the Apache Airflow runtime minutes after inactivity in Data Workflows, optimizing capacity usage, particularly beneficial during development phases where continuous runtime is unnecessary.
  • Enhanced Built-in Security: Integrated within Microsoft Fabric, the Apache Airflow runtime supports Microsoft Entra ID, facilitating Single Sign-On (SSO) experiences when interfacing with Apache Airflow UIs. Additionally, it incorporates Microsoft Fabric workspace roles for robust security measures.
  • Support for Apache Airflow Plugins and Libraries: Since Data workflows is powered by Apache Airflow, it supports all features, plugins, and libraries of Apache Airflow, offering comparable extensibility. If you’re currently using Workflow Orchestration Manager in Azure Data Factory, you have the option to transition to Fabric. This allows you to execute the same Directed Acyclic Graphs (DAGs) within Data workflows. 
  • Custom pools for greater flexibility: When you create a new Data workflow, the default pool used is a starter pool. This pool is instantly available and optimized to provide a server-free Apache Airflow runtime experience. It also turns off when not in use to save costs, making it perfect for development scenarios. However, if you require more control over the pools, you can create a custom pool. This allows you to specify the size, auto-scale configuration, and more. Setting up your data workflows for production in this manner enables unattended operation with an always-on Apache Airflow runtime, supporting the Apache Airflow scheduling capabilities.

    Custom pools can be created using the Workspace settings. This approach ensures your workflows are tailored to your specific needs.

Screenshot of workspace settings to configure custom pools.

To get started:

Prerequisite for enabling the preview on your Microsoft Fabric tenant:

Enable the Data workflow preview using admin portal or reach out to your Fabric admin.

  • Access the Microsoft Fabric Admin Portal.
  • Navigate to Tenant Settings.
  • Under Microsoft Fabric options, locate and expand the ‘Users can create and use Data workflows (preview)’ section. Note: This action is necessary only during the preview phase of Data workflows.
    Screenshot showing the tenant admin portal in Microsoft Fabric using which the preview feature of Data workflows can be turned on for all Fabric users within the tenant.

  1. Create a new Data workflow within an existing or new workspace.
    Screenshot showing how to create a new Data workflow.
  2. Add a new Directed Acyclic Graph (DAG) file via the user data workflow user interface.
    Screenshot showing how to add a new DAG file.
  3. Save your DAG(s).
    Screenshot for saving the doc.
  4. Debug your DAG interactively using the Data workflows user interface.
Screenshot for running the dag

Use Apache Airflow monitoring to observe your DAG executions. In the ribbon, click on Monitor your DAGs in Apache Airflow UI.
A screenshot of a computer

Description automatically generated

Resources

Relaterade blogginlägg

Introducing Data workflows in Microsoft Fabric

oktober 7, 2024 från Alex Lin

Introducing Managed VNet Support for Fabric Eventstream! By creating a Fabric’s Managed Private Endpoint, you can now securely connect Eventstream to your Azure services, such as Azure Event Hubs or IoT Hub, within a private network or behind a firewall. This integration ensures your data is securely transmitted over a private network, enabling you to … Continue reading “Secure Data Streaming with Managed Private Endpoints in Eventstream (Preview)”

oktober 4, 2024 från Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”