CI/CD with Warehouses in Microsoft Fabric
We are excited to announce the support of Fabric Git integration and deployment pipelines for data warehouse in Microsoft Fabric! Warehouse in Microsoft Fabric, built on the evolution of SQL server, already offers local CI/CD experiences today. This blog will focus on how you can take advantage of CI/CD capabilities directly in the web, as well as optional integrations with your favorite tools.
Most organizations work in teams to build analytical solutions. Having a team of developers collaborating in the same environment can be challenging. Challenges such as code conflicts, schema drift, inconsistent environments, and deployment errors can arise, impacting the reliability of the data infrastructure. With the introduction of these features, developers can now build and maintain warehousing solutions efficiently from end-to-end with source control in a web-based experience.
End-to-End Warehouse Lifecycle Management in Microsoft Fabric
Let’s explore how to build warehousing solutions – from code integration to code deployment – entirely in the web, in Microsoft Fabric.
Git Integration for Warehouse (Continuous Integration)
Git integration in Microsoft Fabric empowers developers to seamlessly integrate their development processes, tools, and best practices directly into the Microsoft Fabric workspace. Getting started is as easy as a couple of clicks. Configure the Git repository and branch details in the workspace settings to set up a connection.
This enables developers collaborating on a warehouse to:
- Backup and version their work.
- Revert to previous stages as needed.
- Collaborate with others or work alone using Git branches.
After committing a warehouse to a remote Git repository, the warehouse schema will be exported in its native format, represented as a SQL database project. As a SaaS experience, we automatically extract and create a SQL project to ensure your warehouse schema is expressible in a standardized accessible, deployable, code-first format. Each object in the warehouse is represented in Git as a .sql file, shown below.
Deployment Pipelines for Warehouse (Continuous Deployment)
We talked about integrating code, now let’s discuss how easy it is to make this code available to end users in production. After code changes have been finalized in development environments, Fabric’s deployment pipelines enable database developers to automate the deployment of schema changes across different stages, like testing and production. This allows developers to:
- Ensure that the warehouse is always in sync with the source code.
- Verify and deploy code changes quickly and reliably.
- Compare warehouse schemas across different environments.
- Reduce the risk of errors and conflicts that might arise from manual interventions or inconsistent configurations.
Deployment pipelines empower teams to produce updates in short cycles with high speed, frequency, and consistency. Code changes can be promoted at any time with a single click.
Tying it Together…
Git integration and deployment pipelines are two powerful features that work together to deliver continuous integration and continuous delivery (CI/CD) all in a web-based experience, for warehouses in Fabric.
- Git integration allows developers to collaborate freely without stepping on other team members’ work using versioning and branching capabilities.
- Deployment pipelines enable developers to accelerate and standardize the delivery of warehouse code to different stages, like testing and production, with minimal manual intervention
Using these features together enables developers to synchronize and deliver their warehouse code across different environments with ease and confidence. These features extend outside of just the warehouse as well, including support for other items like pipelines, notebooks, reports, and more.
Automating CI/CD with APIs
Taking this one step further, with APIs, developers can create automated workflows and pipelines that ensure quality, reliability, and productivity for data projects. Tying in all the experiences mentioned above, leverage:
- Warehouse CRUD APIs to create, read, and delete warehouses.
- SQLPackage to extract and publish warehouse schemas.
- Git Integration APIs to connect, commit, and update workspaces to/from the remote Git repository.
- Deployment Pipelines APIs to push code changes to development to higher environments like staging or production.
Using Source Control with Tools
We’ve discussed how to go end-to-end – from checking in code to deploying code in production – in a web-based experience, but what about development in client applications?
We know that developers are most productive when empowered with the familiarity and features of their favorite tools. Source control capabilities for the warehouse are not only confined to the Fabric web experience but are integrated across your favorite tools as well. Developers are empowered with the flexibility and features of their tools while working on their Fabric projects. Let’s dive in.
SQL Database Projects
With SQL Projects support for Warehouse in Microsoft Fabric – available inside of Visual Studio Code and Azure Data Studio – developing warehouse schemas in tools has never been easier. Using Source Control with Git in Visual Studio Code, developers can connect to the same Git repository of the Fabric workspace to:
- Access the database project
- Develop in a local environment
- Push code changes
dbt with Fabric Warehouse
dbt is also deeply integrated with warehouses in Microsoft Fabric. dbt is an open-source framework that simplifies data transformation and analytics engineering for your data warehouse. Source control capabilities can be leveraged with dbt using any Git provider as well. To use dbt with source control, simply connect your dbt project to a remote Git repo. This integration enables you to maintain a well-documented and versioned history of your project files.
Learn More and Next Steps
In this blog, we discussed how using source control practices can empower developers to streamline their workflow, collaborate effectively with team members, and ensure the integrity and consistency of their database-related work.
Get started with source control development with warehouses in Microsoft Fabric: Source control with Warehouse (preview) – Microsoft Fabric | Microsoft Learn
Have any feedback or ideas to improve this experience? We’re listening! Post your thoughts to Ideas (microsoft.com). Thanks for reading!
