Microsoft Fabric Updates Blog

Data Factory Increases Maximum Activities Per Pipeline to 80

Data Factory pipeline developer create exciting and interesting data integration and ETL workflows for their data analytics projects. Because Data Factory is a platform service that is shared across ADF, Synapse, and Fabric, we had been limiting the number of activities in a single pipeline to 40 as a way to avoid resource exhaustion.

However, just this week, we have doubled the limit on number of activities you may define in a pipeline from 40 to 80. With more freedom to develop, we want to empower you to create more powerful, versatile, and resilient data pipelines for all your business needs. We are excited to see what you come up with, harnessing the power of 40 more activities per pipeline!

What’s the limit about & why did we raise it?

To ensure the resiliency and reliability of data pipelines, Data Factory places a limit on maximum number of activities that a pipeline may define. For the longest time, the limit has always been 40 activities per pipeline. Today, we are doubling it to 80, with future plans to raise it even further for our developers. The limit applies to the number of activities defined, not actually run. For instance, in the following example with conditional branching, there are 3 activities defined, even though, realistically speaking, in any pipeline run, only 2 will actually run.

Conditional Branching in Pipeline

We understand that our customers want to build resilient and useful data pipelines for their business needs, and sometimes, the 40 activities limit may come in the way of development. Hence, we are doubling the ceiling limit and giving you 40 more activities in a pipeline. 

When to add more activities?

We strongly encourage customers to use the additional 40 activities to build error handling capabilities. For instance, send an email to my on-call alias when Copy activities failed, otherwise proceed.

Simple Error Handling If Else Scenario

Or build a try-catch block that attempts to move the data if it’s ready or move on otherwise. 

Try Catch Block – only attempts to run the first activity.

Build for Resilience and Retries!

We do not, however, encourage you to build a sequential pipeline, with 80 activities one after another. Please be aware that data pipelines, just like any other piece of software, can sometimes encounter failures. For instance, when the connection to your SQL server is throttling, and a copy activity cannot complete in time.

In those cases, you need to retry and restart the pipeline. Please bear this information in mind, as you develop your pipeline: keep the actual steps within a pipeline to a reasonable amount. Production engineers will thank you to keep their lives simple 🙂

Final Thoughts

With the power of data pipelines, we want you to be able to build and deliver business impact for your end users. We excited to see what you come up with, harnessing the power of 40 more activities.

Have any questions or feedback? Leave a comment below!

Related blog posts

Data Factory Increases Maximum Activities Per Pipeline to 80

April 16, 2024 by Ruixin Xu

We are pleased to share a set of key updates regarding the Copilot in Microsoft Fabric experiences. The information in this blog post has also been shared with Fabric tenant administrators. Below are the highlights of the changes. This change is an important milestone to eventually allow Copilot to reach GA within this year. See … Continue reading “Copilot in MS Fabric: Soon available to more users in your organization“

April 15, 2024 by Santhosh Kumar Ravindran

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to … Continue reading “Introducing Job Queueing for Notebook in Microsoft Fabric”