Announcing Surge Protection (Preview)
We’re excited to announce the preview of Surge Protection, in this first release, capacity admins can set limits on background usage within a capacity. The preview is rolling out now and should be available within the next week.
What is surge protection?
Surge protection helps protect capacities from excess usage by background workloads. It acts as a resource governor, rejecting background operations when the capacity reaches a limit set by the capacity admin. Use surge protection to limit the amount of background usage on the capacity. This helps protect interactive operations like viewing reports and allows the capacity to self-heal faster if there is overuse.
How can I start using surge protection?
Surge protection is configured in the per capacity settings page. It’s available for Microsoft Fabric Capacity (F SKU), Power BI Premium (P SKU), Power BI Embedded (A SKU), and EM SKUs.
To configure surge protection:
- Open the Fabric Admin portal and select Capacities.
- Select a specific capacity.
- Expand Surge Protection.
- Select Enable surge protection.
- Set a Background rejection threshold.
- Set a Background recovery threshold.
- Select Apply.
Surge protection activates when the Background Rejection threshold is reached or exceeded. It then starts rejecting new background operations. This isn’t a hard limit, which we’ll explain later in this blog.
When the Background Recovery threshold is reached, surge protection stops rejecting operations. The capacity then resumes accepting new background operations.
In the chart, surge protection limits the 24 hours background % initially to 40% and then later to 60%. Then it decreases because surge protection rejects new background jobs. Surge protection starts to accept new background jobs when the background recovery threshold of 30% is reached, resulting in an increase in 24 hours background %.
There’s also a small spike over 70% in the chart, that happened when surge protection was briefly turned off.
Before you set the two thresholds, check your capacity’s 24 hours background %. Avoid setting the value too low too quickly to prevent impacting users.
How do I know surge protection is rejecting background jobs?
When surge protection rejects a job, users see the error message CapacityLimitExceeded or CapacityLimitExceededSurgeProtection.
The example indicates that surge protection rejected a Notebook run, as in Monitoring Hub.
Capacity admins can see surge protection in action using the capacity metrics app. On the compute page, we’ve added system events that reflect throttling on the capacity.
The example shows an Overloaded event with the reason set to SurgeProtectionActive. Surge protection emits the event when the background rejection threshold is reached. After the capacity is no longer throttled, surge protection emits an Active event with reason NotOverloaded.
In addition, we added events for all throttling that can occur like interactive delays, rejections, and background rejections that are triggered by reaching the capacity’s limits. This makes it much easier to track throttling on the capacity.
Lastly, on the timepoint page in the capacity metrics app, the background operations for timerange table shows the operation status of RejectedSurgeProtection or Rejected.
Considerations
Surge protection gives capacity admins another tool to help manage their capacities.
Key considerations as you start using surge protection.
- Surge protection’s background rejection threshold is not a hard limit on 24-hour background percentage. This is because surge protection rejects new operations but doesn’t cancel jobs in progress. You’ll see that the 24-hour background percentage goes up beyond the background rejection threshold as those in progress jobs complete. The amount it goes beyond the threshold depends on your capacity’s usage pattern.
- If you’re trying to eliminate interactive rejections or delays by using surge protection, you’ll need to tune your background rejection thresholds. This could take several iterations to set the values that work best for your capacity.
- Surge protection rejects background operations. Many UI experiences in Fabric use background operations or depend on background operations to complete. These will also be rejected when surge protection is active.
Lastly, it’s important to use surge protection alongside traditional approaches like optimizing your capacity and scaling-out capacities. Your best defense against throttling is a correctly sized capacity. If you have mission critical solutions, it’s best to put them in a dedicated capacity that’s sized correctly for the expected usage.
We’re excited to take this first step on the surge protection journey. We’d love to hear your feedback. If you have suggestions to improve surge protection, head over to our ideas page.
Next Steps
Surge Protection – Microsoft Fabric | Microsoft Learn
Evaluate and optimize your Microsoft Fabric capacity – Microsoft Fabric | Microsoft Learn
Understand the metrics app compute page – Microsoft Fabric | Microsoft Learn
