Microsoft Fabric Updates Blog

Introducing Optimistic Job Admission for Fabric Spark

We are excited to announce a new feature which has been a long ask from Synapse Spark customers, Optimistic Job Admission for Spark in Microsoft Fabric.
This feature brings in more flexibility to optimize for concurrency usage (in some cases ~12X increase) and prevents job starvation. This job admission approach aims to reduce the frequency of job starvation and improve the job admission experience for Fabric users running their Data Engineering or Data Science jobs, especially during peak usage hours.  

In the current job admission policy, Fabric Spark reserves the maximum number of cores that a job may need during its execution, based on the maximum number of nodes that the job can scale up to. This ensures that the job will always have enough resources to run, but it also limits the number of concurrent jobs that can be admitted in the cluster. 

With Optimistic Job Admission, Fabric Spark only reserves the minimum number of cores that a job needs to start, based on the minimum number of nodes that the job can scale down to. This allows more jobs to be admitted if there are enough resources to meet the minimum requirements. If a job needs to scale up later, the scale up requests are approved or rejected based on the available cores in capacity.  

Optimistic Job Admission can significantly increase the max number of concurrent jobs for our customers, especially for those who use large SKUs. For example, if a customer is using F32 SKU, which has 64 Spark VCores (Based on 1 CU = 2 Spark VCores) and 192 Max Burst Cores for concurrency (based on the 3X burst multiplier), they can only run 3 jobs concurrently in the compute reservation-based job policy, assuming they have the default starter pool configuration.  

Now with Optimistic Job Admission, they can run up to ~24 jobs concurrently with the same configuration when submitted concurrently. This is an 8x improvement! 

When Spark Autoscale requests for additional nodes during runtime, the job admission control approves the scale up allowing it to scale to its max limits based on the total available cores. When Autoscale requests are rejected, these active running jobs are not impacted and continue to run with their current configuration till cores become available.

Optimistic job admission is enabled by default for all workspaces. For jobs that require compute reservation, users can disable autoscale which would result in allocation of total cores for the job.

To learn more about the optimistic job admission experience and the job scale up scenarios on Spark, please refer to our documentation Job admission and management in Fabric Spark – Microsoft Fabric | Microsoft Learn

To learn more about the throttling experience on Fabric Spark based on the Fabric capacity SKU, please refer to our documentation Concurrency limits and queueing in Microsoft Fabric Spark

To learn more about the job queueing experience in Fabric Spark, please refer to our documentation Job queueing for Fabric Spark – Microsoft Fabric | Microsoft Learn

Relaterade blogginlägg

Introducing Optimistic Job Admission for Fabric Spark

juni 14, 2024 från Guy Reginiano

Announcing triggers and alerts on Real-Time Analytics Dashboards.

juni 12, 2024 från Estera Kot

The Native Execution Engine showcases our dedication to innovation and performance, transforming data processing in Microsoft Fabric. We are excited to announce that the Native Execution Engine for Fabric Runtime 1.2 is now available in public preview. The Native Execution Engine leverages technologies such as a columnar format and vectorized processing to boost query execution … Continue reading “Public Preview of Native Execution Engine for Apache Spark on Fabric Data Engineering and Data Science”