Microsoft Fabric Updates Blog

Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric

We are excited to announce a new high concurrency mode in Fabric for Data Engineering and Data Science. This allows users to share Spark compute across multiple notebooks within a workspace which means that you can run multiple Spark notebooks simultaneously on the same Spark session without compromising performance or security when paying for a single session. High concurrency mode offers an instant run experience avoiding session start delays and ~30X faster session start experience for the shared notebooks when running on custom pools.

What is High Concurrency Mode?

High concurrency mode allows sharing of Spark compute across multiple notebooks and allows their queries to execute in parallel. In this shared mode, the resources and configurations of each notebook are isolated from each other. As the session sharing is always within a single user boundary, users cannot access or modify the data or variables of another user’s high concurrency session.

High concurrency mode also leverages FAIR allocation to optimize the resource utilization and performance of the notebooks and ensures that each notebook gets a fair share of the executors available for the Spark application.

Why Use High Concurrency Mode?

High concurrency mode offers several benefits for Fabric Spark users, such as:

  • Faster and easier Spark session start: You don’t have to wait for the Spark pool to spin up or configure the node sizes when you start a Spark session. As the session is already warmed up and running, attaching a new notebook to an existing Spark session gives a session start experience within ~5 seconds. You can also use custom Spark pools, which allow you to size the nodes, enable autoscaling, and dynamically allocate executors based on your Spark job requirements and with custom pools you would get a 30X faster session start experience for shared notebooks.
  • Enhanced security and isolation: You can ensure that each user or query has its own isolated Spark session, which prevents data leakage or tampering.
  • Do more by paying less: Achieve better compute cost savings by sharing a single session across multiple notebooks for your Data Engineering or Data Science workloads and only get billed only for the single session.

How to Enable High Concurrency Mode?

To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps:

  1. Go to the workspace settings in your Fabric workspace.

A screenshot of a computer

Description automatically generated

  1. Navigate to the Data Engineer/Science section.
  2. Select the Spark Compute menu
  3. Enable the High Concurrency Mode Option if its disabled (This option should be enabled by default for all Fabric Workspaces)

4. Save your changes.

Once you enable high concurrency mode, you can run your notebooks in High Concurrency mode from the notebook menu ribbon.

To learn more about using high concurrency in notebooks read: Sharing Spark Compute across Notebooks with High Concurrency Mode in Fabric.

For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft Fabric

Related blog posts

Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric

November 30, 2023 by Pawel Potasinski

Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. The Microsoft Fabric team presents you with the “Microsoft Fabric MVP Corner” blog post in which we highlight selected content recently created by MVPs.

November 20, 2023 by Anshul Sharma

As part of the One logical copy effort, we’re excited to announce that you can now enable availability of KQL Database in Delta Lake format. Delta Lake  is the unified data lake table format chosen to achieve seamless data access across all compute engines in Microsoft Fabric. The data streamed into KQL Database is stored … Continue reading “Announcing Delta Lake support in Real-Time Analytics KQL Database”