Microsoft Fabric Updates Blog

Data Factory Adds CI/CD to Fabric Data Pipelines

When building successful data analytics projects, it is very important to have source control, continuous integration, continuous deployment, and collaborative development environments. Many Fabric engineers with previous Azure Synapse Analytics and Azure Data Factory experience have utilized the Git integration included in those PaaS offerings for those important capabilities. Now, we’re excited to share that we have added Git Integration and integration with built-in Deployment Pipelines to Data Factory data pipelines in Fabric as a public preview!

CI/CD features to utilize your own Git repo in Azure DevOps, or to use the built-in Deployment Pipelines in Fabric, will eventually light-up and become available to all Fabric items. Now that data pipelines can be used with these features, read more about the current preview capabilities and limitations at the online documentation here.

Connect to your Git repo

If you are familiar with ADF’s Git integration set-up and capabilities, then you will find Fabric’s Git integration easy and with a number of nice enhancements! ADF’s approach to CI/CD included the use of ARM templates to simplify the process, but also meant that the entire factory was treated as a single entity. This made cherry-picking elements of a factory very difficult. In Fabric’s approach, each item in our workspace can optionally partake in the check-in process so this is no longer challenging.

In addition to using your own ADO Git repo, you can optionally use the built-in Deployment Pipelines feature that enables code graduation through Dev, Test, and Prod environments through a super-easy and intuitive UI, yet another nice enhancement over ADF and Synapse.

Built-in deployment pipelines

The documentation link found here has more details on current limitations, including deployment rules in deployment pipelines, which we will hook up to data pipelines soon. Other Data Factory and Fabric items like Dataflows and Data Warehouse will also become part of the CI/CD process in Fabric over time.

Have any questions or feedback? Leave a comment below!

Related blog posts

Data Factory Adds CI/CD to Fabric Data Pipelines

April 16, 2024 by Ruixin Xu

We are pleased to share a set of key updates regarding the Copilot in Microsoft Fabric experiences. The information in this blog post has also been shared with Fabric tenant administrators. Below are the highlights of the changes. This change is an important milestone to eventually allow Copilot to reach GA within this year. See … Continue reading “Copilot in MS Fabric: Soon available to more users in your organization“

April 15, 2024 by Santhosh Kumar Ravindran

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to … Continue reading “Introducing Job Queueing for Notebook in Microsoft Fabric”