Microsoft Fabric Updates Blog

CI/CD with Warehouses in Microsoft Fabric

We are excited to announce the support of Fabric Git integration and deployment pipelines for data warehouse in Microsoft Fabric! Warehouse in Microsoft Fabric, built on the evolution of SQL server, already offers local CI/CD experiences today. This blog will focus on how you can take advantage of CI/CD capabilities directly in the web, as well as optional integrations with your favorite tools.

Most organizations work in teams to build analytical solutions. Having a team of developers collaborating in the same environment can be challenging. Challenges such as code conflicts, schema drift, inconsistent environments, and deployment errors can arise, impacting the reliability of the data infrastructure. With the introduction of these features, developers can now build and maintain warehousing solutions efficiently from end-to-end with source control in a web-based experience.  

End-to-End Warehouse Lifecycle Management in Microsoft Fabric

Let’s explore how to build warehousing solutions – from code integration to code deployment – entirely in the web, in Microsoft Fabric.

Git Integration for Warehouse (Continuous Integration)

Git integration in Microsoft Fabric empowers developers to seamlessly integrate their development processes, tools, and best practices directly into the Microsoft Fabric workspace. Getting started is as easy as a couple of clicks. Configure the Git repository and branch details in the workspace settings to set up a connection.

This enables developers collaborating on a warehouse to:

  • Backup and version their work.
  • Revert to previous stages as needed.
  • Collaborate with others or work alone using Git branches.

After committing a warehouse to a remote Git repository, the warehouse schema will be exported in its native format, represented as a SQL database project. As a SaaS experience, we automatically extract and create a SQL project to ensure your warehouse schema is expressible in a standardized accessible, deployable, code-first format. Each object in the warehouse is represented in Git as a .sql file, shown below.

Deployment Pipelines for Warehouse (Continuous Deployment)

We talked about integrating code, now let’s discuss how easy it is to make this code available to end users in production. After code changes have been finalized in development environments, Fabric’s deployment pipelines enable database developers to automate the deployment of schema changes across different stages, like testing and production. This allows developers to:

  • Ensure that the warehouse is always in sync with the source code.
  • Verify and deploy code changes quickly and reliably.
  • Compare warehouse schemas across different environments.
  • Reduce the risk of errors and conflicts that might arise from manual interventions or inconsistent configurations.

Deployment pipelines empower teams to produce updates in short cycles with high speed, frequency, and consistency. Code changes can be promoted at any time with a single click.

Tying it Together…

Git integration and deployment pipelines are two powerful features that work together to deliver continuous integration and continuous delivery (CI/CD) all in a web-based experience, for warehouses in Fabric.

  • Git integration allows developers to collaborate freely without stepping on other team members’ work using versioning and branching capabilities.
  • Deployment pipelines enable developers to accelerate and standardize the delivery of warehouse code to different stages, like testing and production, with minimal manual intervention

Using these features together enables developers to synchronize and deliver their warehouse code across different environments with ease and confidence. These features extend outside of just the warehouse as well, including support for other items like pipelines, notebooks, reports, and more.

Automating CI/CD with APIs

Taking this one step further, with APIs, developers can create automated workflows and pipelines that ensure quality, reliability, and productivity for data projects. Tying in all the experiences mentioned above, leverage:

  1. Warehouse CRUD APIs to create, read, and delete warehouses.
  2. SQLPackage to extract and publish warehouse schemas.
  3. Git Integration APIs to connect, commit, and update workspaces to/from the remote Git repository.
  4. Deployment Pipelines APIs to push code changes to development to higher environments like staging or production.

Using Source Control with Tools

We’ve discussed how to go end-to-end – from checking in code to deploying code in production – in a web-based experience, but what about development in client applications? 

We know that developers are most productive when empowered with the familiarity and features of their favorite tools. Source control capabilities for the warehouse are not only confined to the Fabric web experience but are integrated across your favorite tools as well. Developers are empowered with the flexibility and features of their tools while working on their Fabric projects. Let’s dive in.

SQL Database Projects

With SQL Projects support for Warehouse in Microsoft Fabric – available inside of Visual Studio Code and Azure Data Studio – developing warehouse schemas in tools has never been easier. Using Source Control with Git in Visual Studio Code, developers can connect to the same Git repository of the Fabric workspace to:

  • Access the database project
  • Develop in a local environment
  • Push code changes

dbt with Fabric Warehouse

dbt is also deeply integrated with warehouses in Microsoft Fabric. dbt is an open-source framework that simplifies data transformation and analytics engineering for your data warehouse. Source control capabilities can be leveraged with dbt using any Git provider as well. To use dbt with source control, simply connect your dbt project to a remote Git repo. This integration enables you to maintain a well-documented and versioned history of your project files.

Learn More and Next Steps

In this blog, we discussed how using source control practices can empower developers to streamline their workflow, collaborate effectively with team members, and ensure the integrity and consistency of their database-related work.

Get started with source control development with warehouses in Microsoft Fabric: Source control with Warehouse (preview) – Microsoft Fabric | Microsoft Learn

Have any feedback or ideas to improve this experience? We’re listening! Post your thoughts to Ideas (microsoft.com). Thanks for reading!

Zugehörige Blogbeiträge

CI/CD with Warehouses in Microsoft Fabric

Oktober 4, 2024 von Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”

Oktober 2, 2024 von Miguel Llopis

Last week was such an exciting week for Fabric during the Fabric Community Conference Europe, filled with several product announcements and sneak previews of upcoming new features. Thanks to all of you who participated in the conference, either in person or by being part of the many virtual conversations through blogs, Community forums, social media … Continue reading “Recap of Data Factory Announcements at Fabric Community Conference Europe”