Microsoft Fabric Updates Blog

Introducing Job Queueing for Notebook in Microsoft Fabric

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to eliminate manual retries and improve the user experience for our customers who run notebook jobs on Microsoft Fabric.

Notebook jobs are a popular way to run data analysis and machine learning workflows on Fabric. They can be triggered by pipelines or a job scheduler, depending on the user’s needs. However, in the current system, notebook jobs are not queued when the Fabric capacity is at its max utilization. They are rejected with a Capacity Limit Exceeded error, which forces the user to retry the job later when the resources are available. This can be time-consuming expensive in operations, especially for enterprise users who run many notebook jobs to resubmit the jobs that have been throttled.

With Job Queueing for Notebook Jobs, this problem is solved. Notebook jobs that are triggered by pipelines or job scheduler will be added to a queue and will be retried automatically when the capacity frees up.

The user does not need to do anything to resubmit the job when its added to the queue as its automatically retried and starts execution when the capacity is freed up. The status of these notebook jobs will be Not Started when in queued state and will be changed to In Progress when they start the execution.

Fabric Spark enforces queue sizes based on the capacity SKU size attached to a workspace, providing a queueing mechanism based on the purchased Fabric capacity SKUs.

The following section lists various queue sizes for Spark workloads based on Microsoft Fabric based on the capacity SKUs:

Fabric capacity SKUEquivalent Power BI SKUQueue limit
F24
F44
F88
F1616
F3232
F64P164
F128P2128
F256P3256
F512P4512
F10241024
F20482048
Trial CapacityP1NA
Spark queue limits based on Fabric Capacity SKUs

To learn more about the job queueing experience in Fabric Spark, please refer to our documentation Job queueing for Fabric Spark – Microsoft Fabric | Microsoft Learn

To learn more about the throttling experience on Fabric Spark based on the Fabric capacity SKU, please refer to our documentation Concurrency limits and queueing in Microsoft Fabric Spark

Related blog posts

Introducing Job Queueing for Notebook in Microsoft Fabric

October 31, 2024 by Jovan Popovic

Fabric Data Warehouse is a modern data warehouse optimized for analytical data models, primarily focused on the smaller numeric, datetime, and string types that are suitable for analytics. For the textual data, Fabric DW supports the VARCHAR type that can store up to 8KB of text, which is suitable for most of the textual values … Continue reading “Announcing public preview of VARCHAR(MAX) and VARBINARY(MAX) types in Fabric Data Warehouse”

October 29, 2024 by Dandan Zhang

Managed private endpoints allow Fabric experiences to securely access data sources without exposing them to the public network or requiring complex network configurations. We announced General Availability for Managed Private Endpoint in Fabric in May of this year. Learn more here: Announcing General Availability of Fabric Private Links, Trusted Workspace Access, and Managed Private Endpoints. … Continue reading “APIs for Managed Private Endpoint are now available”