Microsoft Fabric Updates Blog

Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric

We are excited to announce a new high concurrency mode in Fabric for Data Engineering and Data Science. This allows users to share Spark compute across multiple notebooks within a workspace which means that you can run multiple Spark notebooks simultaneously on the same Spark session without compromising performance or security when paying for a single session. High concurrency mode offers an instant run experience avoiding session start delays and ~30X faster session start experience for the shared notebooks when running on custom pools.

What is High Concurrency Mode?

High concurrency mode allows sharing of Spark compute across multiple notebooks and allows their queries to execute in parallel. In this shared mode, the resources and configurations of each notebook are isolated from each other. As the session sharing is always within a single user boundary, users cannot access or modify the data or variables of another user’s high concurrency session.

High concurrency mode also leverages FAIR allocation to optimize the resource utilization and performance of the notebooks and ensures that each notebook gets a fair share of the executors available for the Spark application.

Why Use High Concurrency Mode?

High concurrency mode offers several benefits for Fabric Spark users, such as:

  • Faster and easier Spark session start: You don’t have to wait for the Spark pool to spin up or configure the node sizes when you start a Spark session. As the session is already warmed up and running, attaching a new notebook to an existing Spark session gives a session start experience within ~5 seconds. You can also use custom Spark pools, which allow you to size the nodes, enable autoscaling, and dynamically allocate executors based on your Spark job requirements and with custom pools you would get a 30X faster session start experience for shared notebooks.
  • Enhanced security and isolation: You can ensure that each user or query has its own isolated Spark session, which prevents data leakage or tampering.
  • Do more by paying less: Achieve better compute cost savings by sharing a single session across multiple notebooks for your Data Engineering or Data Science workloads and only get billed only for the single session.

How to Enable High Concurrency Mode?

To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps:

  1. Go to the workspace settings in your Fabric workspace.

A screenshot of a computer

Description automatically generated

  1. Navigate to the Data Engineer/Science section.
  2. Select the Spark Compute menu
  3. Enable the High Concurrency Mode Option if its disabled (This option should be enabled by default for all Fabric Workspaces)

4. Save your changes.

Once you enable high concurrency mode, you can run your notebooks in High Concurrency mode from the notebook menu ribbon.

To learn more about using high concurrency in notebooks read: Sharing Spark Compute across Notebooks with High Concurrency Mode in Fabric.

For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft Fabric

Gerelateerde blogberichten

Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric

oktober 7, 2024 door Alex Lin

Introducing Managed VNet Support for Fabric Eventstream! By creating a Fabric’s Managed Private Endpoint, you can now securely connect Eventstream to your Azure services, such as Azure Event Hubs or IoT Hub, within a private network or behind a firewall. This integration ensures your data is securely transmitted over a private network, enabling you to … Continue reading “Secure Data Streaming with Managed Private Endpoints in Eventstream (Preview)”

oktober 4, 2024 door Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”