Running Apache Airflow jobs seamlessly in Microsoft Fabric
Apache Airflow is a powerful platform to programmatically author, schedule, and monitor workflows. It is widely adopted for its flexibility, scalability, and ability to handle complex workflows with ease. With Apache Airflow, you can orchestrate your data pipelines, ensuring they run smoothly and efficiently.
In May 2024, we launched the preview of Apache Airflow job in Microsoft Fabric, enabling data engineers and scientists to write DAGs without worrying about setup. Key features include:
- Instant Runtime Provisioning: Immediate access to Apache Airflow runtime for running, debugging, and operationalizing DAGs.
- Cloud-Based IDE: Create and edit Apache Airflow DAGs directly within Microsoft Fabric.
- Intelligent Auto-Pause: Automatically pauses runtime after inactivity, optimizing costs during development.
- Enhanced Security: Microsoft Entra ID integration for SSO and workspace role-based security.
- Full Feature Support: Compatible with all Apache Airflow plugins and libraries.
- Flexible Pooling Options: Default starter pools with option to create custom pools with specific configurations.
- Managed Service: Create and manage Python-based DAGs without infrastructure concerns.
Announcing General Availability of Apache Airflow Job in Microsoft Fabric
We are thrilled to announce that Apache Airflow Job in Microsoft Fabric is Generally Available! Following the successful public preview in March 2024, we have been working diligently to bring you a robust and feature-rich experience.
Key Features and Enhancements in Apache Airflow job (GA)
1. Improved Diagnostics Logs: We have significantly enhanced the diagnostics logs for clusters, schedulers, web servers, and workers. These improvements provide deeper insights and better visibility into your workflows, making it easier to troubleshoot and optimize performance. 
2. Fabric REST API Support: Managing your Apache Airflow jobs has never been easier with our new REST API support. You can now programmatically manage your workflows, integrate with other systems, and automate your processes seamlessly.
3. Support for Apache Airflow Version 2.10.5 We are excited to introduce Apache Airflow version 2.10.5. This latest version brings numerous improvements and new features, ensuring you have access to the best tools for your workflow orchestration needs.
4. New Job Versioning Story Our new Apache Airflow job runtime versioning story is designed to bring you the latest updates within six months while allowing enterprises the time they need before deprecating older runtime versions (18 months). This approach ensures you can take advantage of new features and improvements without disrupting your existing workflows.
We are committed to providing you with the best tools and features to manage your workflows efficiently. Thank you for your continued support, and we look forward to seeing what you build with Apache Airflow job in Microsoft Fabric!
