Microsoft Fabric Updates Blog

Introducing Job Queueing for Notebook in Microsoft Fabric

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to eliminate manual retries and improve the user experience for our customers who run notebook jobs on Microsoft Fabric.

Notebook jobs are a popular way to run data analysis and machine learning workflows on Fabric. They can be triggered by pipelines or a job scheduler, depending on the user’s needs. However, in the current system, notebook jobs are not queued when the Fabric capacity is at its max utilization. They are rejected with a Capacity Limit Exceeded error, which forces the user to retry the job later when the resources are available. This can be time-consuming expensive in operations, especially for enterprise users who run many notebook jobs to resubmit the jobs that have been throttled.

With Job Queueing for Notebook Jobs, this problem is solved. Notebook jobs that are triggered by pipelines or job scheduler will be added to a queue and will be retried automatically when the capacity frees up.

The user does not need to do anything to resubmit the job when its added to the queue as its automatically retried and starts execution when the capacity is freed up. The status of these notebook jobs will be Not Started when in queued state and will be changed to In Progress when they start the execution.

Fabric Spark enforces queue sizes based on the capacity SKU size attached to a workspace, providing a queueing mechanism based on the purchased Fabric capacity SKUs.

The following section lists various queue sizes for Spark workloads based on Microsoft Fabric based on the capacity SKUs:

Fabric capacity SKUEquivalent Power BI SKUQueue limit
F24
F44
F88
F1616
F3232
F64P164
F128P2128
F256P3256
F512P4512
F10241024
F20482048
Trial CapacityP1NA
Spark queue limits based on Fabric Capacity SKUs

To learn more about the job queueing experience in Fabric Spark, please refer to our documentation Job queueing for Fabric Spark – Microsoft Fabric | Microsoft Learn

To learn more about the throttling experience on Fabric Spark based on the Fabric capacity SKU, please refer to our documentation Concurrency limits and queueing in Microsoft Fabric Spark

Related blog posts

Introducing Job Queueing for Notebook in Microsoft Fabric

July 14, 2024 by Nimrod Shalit

GitHub and GitHub Enterprise are now available for source control integration. Connect your workspace and you can start syncing your fabric content into your repository.

July 12, 2024 by Ed Lima

During Microsoft Build, in May 2024, we announced the worldwide public preview of API for GraphQL in Microsoft Fabric. With the preview, data engineers and scientists can create a GraphQL data API to connect to different Fabric data sources in seconds, use the APIs in their workflows, or share the API endpoints with app development … Continue reading “Announcing Fabric API for GraphQL pricing”