Simplify your Warehouse ALM with DacFx integration in Git and Deployment pipelines for Fabric Warehouse
Managing data warehouse changes and automating deployments is now simpler than ever with the integration of DacFx with Git and Deployment Pipelines for Fabric Warehouse. This integration enables seamless export and import of your data warehouses, efficient schema change management, and deployment through Git-connected workflows. Whether you’re collaborating with a team using tools like VS Code or automating deployments, DacFx allows for consistent management of your warehouse lifecycle. It supports version control, providing a streamlined way to handle schema versions and deploy changes across your environments.
What is DacFx?
DacFx (Data-tier Applications) is a powerful tool from Microsoft that allows users to manage and automate warehouse/database deployments. It provides a unified approach for exporting, importing, and deploying warehouse schemas, making it easy to integrate database management into continuous integration/continuous deployment (CI/CD) pipelines.
With DacFx, you can:
- Export and import warehouse schemas to and from your data warehouse.
- Manage Workspace branch out and update from git workflows
- Incremental deployments pipeline-based workflows.
How DacFx Integrates with Git and Deployment Pipelines in Fabric Warehouse?
The Fabric platform enables efficient export and import workflows for various scenarios. For instance, when branching to a new workspace, you can export a warehouse database project from the source workspace and import it into the target. Similarly, deployment pipelines can export a warehouse from one stage and import it into another, ensuring smooth transitions across different environments.
There are several such scenarios where export and import workflows use DacFx to achieve CI/CD requirements of customers.
Export
In DacFx, “Export” is known as “Extract,” which is used to pull out a data tier application (dacpac). For Fabric users, the export process extracts a database project, offering a more intuitive approach to warehouse management. Additionally, this export functionality now supports security features like custom roles, schemas, and will soon include more complex objects like functions and external tables. The process is powered by DacFx, ensuring consistency across client tools. Future improvements will further enhance capabilities via the “Download SQL Database Project” option in the ribbon and other workflows mentioned above.
When you click on download, the browser will download a zip (database project) file. When you unzip the file, you can see different folders highlighted in the image below.
The export with DacFx comes with the ability to export schemas with or without objects in it, custom roles created by users in the warehouse and more to follow.
Import
Similarly, you can import an existing database project with additional object types support in Fabric Warehouse using “SQL database project” with DacFx integration.
As you see, not only the objects were imported, but the security (custom role) and shared queries are also imported which were exported before.
Problems Solved by DacFx in Fabric Warehouse
- Handling Data Loss During Schema Changes:
Current deployment workflows risk dropping data when new columns are added or existing ones are removed. DacFx solves this by generating incremental or differentialALTERstatements, ensuring no data is lost when schema changes are made. - Exporting and Importing Empty Schemas and Security:
Previously, it was challenging to export and import empty schemas or custom roles between stages. DacFx resolves this by enabling the deployment of empty schemas and custom roles, providing a more complete and secure migration between environments. - Improved Error Handling:
Errors like “DMSImportDatabaseException” were often vague and difficult to address. With DacFx, errors are more descriptive and integrated with existing error codes, making troubleshooting easier. These error messages can be searched online, and you can replicate them in tools like VS Code for quicker resolution.
There are many more features planned to make the life cycle of your warehouse easy, and we think DacFx is a fundamental block that helps achieve the goals.
Conclusion
DacFx integration with Git and Deployment Pipelines for Fabric Warehouse workspaces offers a new level of control, automation and consistency between client tools and web-based experiences for your data warehouse management. By leveraging version control, automation, and streamlined workflows, you can confidently manage and deploy your data warehouse schemas across environments, saving time and reducing errors.
Ready to optimize your data warehouse AML workflows? Take advantage of DacFx integration in Git and deployment pipelines to bring automation, collaboration, and efficiency to your Fabric Warehouse!
Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community. To get into the technical details, head over to the Fabric documentation.
