Microsoft Fabric Updates Blog

Introducing Job Queueing for Notebook in Microsoft Fabric

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to eliminate manual retries and improve the user experience for our customers who run notebook jobs on Microsoft Fabric.

Notebook jobs are a popular way to run data analysis and machine learning workflows on Fabric. They can be triggered by pipelines or a job scheduler, depending on the user’s needs. However, in the current system, notebook jobs are not queued when the Fabric capacity is at its max utilization. They are rejected with a Capacity Limit Exceeded error, which forces the user to retry the job later when the resources are available. This can be time-consuming expensive in operations, especially for enterprise users who run many notebook jobs to resubmit the jobs that have been throttled.

With Job Queueing for Notebook Jobs, this problem is solved. Notebook jobs that are triggered by pipelines or job scheduler will be added to a queue and will be retried automatically when the capacity frees up.

The user does not need to do anything to resubmit the job when its added to the queue as its automatically retried and starts execution when the capacity is freed up. The status of these notebook jobs will be Not Started when in queued state and will be changed to In Progress when they start the execution.

Fabric Spark enforces queue sizes based on the capacity SKU size attached to a workspace, providing a queueing mechanism based on the purchased Fabric capacity SKUs.

The following section lists various queue sizes for Spark workloads based on Microsoft Fabric based on the capacity SKUs:

Fabric capacity SKUEquivalent Power BI SKUQueue limit
Trial CapacityP1NA
Spark queue limits based on Fabric Capacity SKUs

To learn more about the job queueing experience in Fabric Spark, please refer to our documentation Job queueing for Fabric Spark – Microsoft Fabric | Microsoft Learn

To learn more about the throttling experience on Fabric Spark based on the Fabric capacity SKU, please refer to our documentation Concurrency limits and queueing in Microsoft Fabric Spark

Related blog posts

Introducing Job Queueing for Notebook in Microsoft Fabric

May 23, 2024 by Will Thompson (HE/HIM)

Microsoft Fabric’s new Real-Time hub makes it easier than ever to discover, preview, and use streaming data from around your organization. But the data in these streams is only as useful as the actions and changes it can drive in your business. Data Activator provides a no-code experience for automatically taking actions when patterns or … Continue reading “Alerting and acting on data from the Real-Time hub”

May 23, 2024 by Salil Kanade

We are excited to announce the public preview release of Copilot for Data Warehouse in Microsoft Fabric, a groundbreaking AI assistant designed to transform your data warehousing tasks. Data warehouse development can be daunting for SQL developers, especially under tight timelines where insights are needed “yesterday”.  Developers may spend hours writing code, building schemas, documentation, … Continue reading “Announcing the Public Preview of Copilot for Data Warehouse in Microsoft Fabric”