High Concurrency mode for notebooks in pipelines (Generally Available)
High Concurrency mode for notebooks in pipelines is now generally available (GA)! This powerful feature enhances enterprise data ingestion and transformation by optimizing session sharing within one of the most widely used orchestration mechanisms. With this release, we’re also introducing Comprehensive Monitoring for High-Concurrency Spark Applications, bringing deeper visibility and control to your workloads.
Key benefits
Faster session start
High Concurrency mode dramatically improves the session start experience, reducing the time to ~5 seconds for shared notebooks. This is approximately 30 times faster than traditional methods, leading to significant performance improvements in pipeline execution.
Session Tags for targeted management
We’ve introduced session tags, allowing users to efficiently assign notebooks to specific High Concurrency sessions for better organization and resource utilization.
Optimized pipeline execution & cost savings
By sharing a single session across multiple notebooks, High Concurrency mode reduces both compute costs and execution times. You only pay for a single session, minimizing queuing issues during peak usage hours and ensuring a smoother workflow.
For example, a pipeline with five notebook steps, each taking 5 minutes to execute, would traditionally require separate Spark session startups (3 minutes each), leading to a total runtime of ~40 minutes. With High Concurrency mode, the time is reduced to ~28 minutes, representing a 30% performance improvement.
New features in general availability
With the GA release, we are introducing Comprehensive Monitoring for High-Concurrency Spark Applications, providing enhanced visibility and logging capabilities.
1. Mapping of Spark Jobs and stages to respective notebooks
Understand the performance of individual notebook steps by mapping Spark jobs and execution stages directly to their originating notebooks. This granular insight enables better debugging and performance tuning.
2. Log Segregation for individual notebooks
Each notebook now maintains its own distinct logs, making it easier to track execution progress, troubleshoot errors, and analyze without interference from other notebooks sharing the same session.
3. Notebook snapshots
Now, snapshots of all notebooks involved in a High Concurrency session are captured, allowing users to debug active and historical executions to explore statement level results and understand past runs with greater clarity.
How to enable High Concurrency Mode for notebooks in pipelines
To enable High Concurrency mode for your Fabric Spark workspace, follow these steps:
- Navigate to Workspace Settings in your Fabric workspace.
- Select the Data Engineer/Science section.
- Go to the Spark Compute menu.
- Open the High Concurrency tab.
- Enable the option ‘For pipeline running multiple notebooks‘.
- Click Save.
Once enabled, all Spark sessions triggered by notebook steps within a pipeline will be automatically packed into a shared High Concurrency session, boosting efficiency and performance.
Why choose High Concurrency Mode?
By adopting High Concurrency mode, you can achieve:
- Faster pipeline execution with minimal session startup time.
- Lower compute costs by sharing Spark sessions across multiple notebooks.
- Improved monitoring & debugging with job mapping, log segregation, and notebook snapshots.
To learn more, please refer to our documentation, High Concurrency Mode for notebooks in pipelines.
For more information on High Concurrency mode, check out the Overview of High Concurrency Mode in Microsoft Fabric documentation and join the conversation on the Fabric Community.
