Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric
We are excited to announce a new high concurrency mode in Fabric for Data Engineering and Data Science. This allows users to share Spark compute across multiple notebooks within a workspace which means that you can run multiple Spark notebooks simultaneously on the same Spark session without compromising performance or security when paying for a single session. High concurrency mode offers an instant run experience avoiding session start delays and ~30X faster session start experience for the shared notebooks when running on custom pools.
What is High Concurrency Mode?
High concurrency mode allows sharing of Spark compute across multiple notebooks and allows their queries to execute in parallel. In this shared mode, the resources and configurations of each notebook are isolated from each other. As the session sharing is always within a single user boundary, users cannot access or modify the data or variables of another user’s high concurrency session.
High concurrency mode also leverages FAIR allocation to optimize the resource utilization and performance of the notebooks and ensures that each notebook gets a fair share of the executors available for the Spark application.
Why Use High Concurrency Mode?
High concurrency mode offers several benefits for Fabric Spark users, such as:
- Faster and easier Spark session start: You don’t have to wait for the Spark pool to spin up or configure the node sizes when you start a Spark session. As the session is already warmed up and running, attaching a new notebook to an existing Spark session gives a session start experience within ~5 seconds. You can also use custom Spark pools, which allow you to size the nodes, enable autoscaling, and dynamically allocate executors based on your Spark job requirements and with custom pools you would get a 30X faster session start experience for shared notebooks.
- Enhanced security and isolation: You can ensure that each user or query has its own isolated Spark session, which prevents data leakage or tampering.
- Do more by paying less: Achieve better compute cost savings by sharing a single session across multiple notebooks for your Data Engineering or Data Science workloads and only get billed only for the single session.
How to Enable High Concurrency Mode?
To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps:
- Go to the workspace settings in your Fabric workspace.
- Navigate to the Data Engineer/Science section.
- Select the Spark Compute menu
- Enable the High Concurrency Mode Option if its disabled (This option should be enabled by default for all Fabric Workspaces)
4. Save your changes.
Once you enable high concurrency mode, you can run your notebooks in High Concurrency mode from the notebook menu ribbon.
To learn more about using high concurrency in notebooks read: Sharing Spark Compute across Notebooks with High Concurrency Mode in Fabric.
For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft Fabric
