Data Factory Adds CI/CD to Fabric Data Pipelines
When building successful data analytics projects, it is very important to have source control, continuous integration, continuous deployment, and collaborative development environments. Many Fabric engineers with previous Azure Synapse Analytics and Azure Data Factory experience have utilized the Git integration included in those PaaS offerings for those important capabilities. Now, we’re excited to share that we have added Git Integration and integration with built-in Deployment Pipelines to Data Factory data pipelines in Fabric as a public preview!
CI/CD features to utilize your own Git repo in Azure DevOps, or to use the built-in Deployment Pipelines in Fabric, will eventually light-up and become available to all Fabric items. Now that data pipelines can be used with these features, read more about the current preview capabilities and limitations at the online documentation here.
If you are familiar with ADF’s Git integration set-up and capabilities, then you will find Fabric’s Git integration easy and with a number of nice enhancements! ADF’s approach to CI/CD included the use of ARM templates to simplify the process, but also meant that the entire factory was treated as a single entity. This made cherry-picking elements of a factory very difficult. In Fabric’s approach, each item in our workspace can optionally partake in the check-in process so this is no longer challenging.
In addition to using your own ADO Git repo, you can optionally use the built-in Deployment Pipelines feature that enables code graduation through Dev, Test, and Prod environments through a super-easy and intuitive UI, yet another nice enhancement over ADF and Synapse.
The documentation link found here has more details on current limitations, including deployment rules in deployment pipelines, which we will hook up to data pipelines soon. Other Data Factory and Fabric items like Dataflows and Data Warehouse will also become part of the CI/CD process in Fabric over time.
Have any questions or feedback? Leave a comment below!
