Understanding Fabric KQL DB Capacity
With the free preview usage of Fabric workloads (other than Power BI) coming to an end on October 1, 2023 it’s important for everyone to understand how a KQL Database is billed in the SaaS world of Microsoft Fabric.
In Microsoft Fabric, you can purchase a Fabric SKU which comes with a set of Capacity Units (CU). Back in June 2023 we announced the availability of Fabric SKUs. Workspaces are assigned to capacities and everything inside a workspace such as KQL Database, Warehouse, Lakehouse, Dataset, etc., consume capacity units based on consumption.
The storage costs for data in OneLake will be billed separately from the Fabric capacity usage at a pay-as-you-go rate. OneLake storage pricing is comparable to Azure ADLS (Azure Data Lake Storage) pricing.
KQL Database and Capacity Usage
A KQL Database will utilize capacity via Operations that can be monitored using the Microsoft Fabric Capacity Metrics.
KQL Database Consumption
This is the number of seconds that your KQL database is active in relation to the number of virtual cores used by your database. For example, if my database uses 4 virtual cores and is active for 10 minutes then you will utilize 2,400 seconds of Capacity Units. An auto-scale mechanism is utilized to determine the size of your KQL database. This ensures the most cost-optimized and best performance based on your usage pattern.
OneLake Transactions
Based on the number of reads and write transactions that your KQL Database performs, capacity units will be utilized from your Capacity. This will utilize the OneLake Read and Write meters which are equivalent to read and write operations on Azure ADLS Storage Accounts.
KQL Database and OneLake Storage
Data ingested into a KQL database is compressed and indexed. This compressed data is stored in two tiers of storage. This is your premium cache capacity which will be charged against OneLake Cache Storage and your standard tier of storage that will be charged against One Lake Standard Storage.
OneLake Standard Storage: This is the persistent storage, so if my retention is set to 365 days then all 365 days will be kept in OneLake Standard Storage. Mirroring a database or table allows you to have one logical copy of your data in OneLake. In cases where you mirror data to OneLake you will not get charged for OneLake Storage in both the KQL Database and OneLake.
OneLake Cache Storage: This is your high performance tier that provides the best query performance. The caching policy controls the amount of data that is stored in your OneLake Cache.
Operation Details
|
Operations Name |
Type |
Release |
|
KustoUpTime |
Capacity |
October 1 |
|
KustoOneLakeRead |
Capacity |
GA |
|
KustoOneLakeWrite |
Capacity |
GA |
|
KustoStandardStorage |
Storage |
Post October 1/Pre GA |
|
KustoCacheStorage |
Storage |
GA |
Monitoring Fabric Capacity
The Microsoft Fabric Capacity Metrics app provides visibility into capacity usage for all Fabric workspaces tied to a capacity. It is used by capacity administrators to monitor the performance of workloads and their usage, compared to purchased capacity. Consider the following use cases:
- Identify CU usage trends across warehouse items within a capacity.
- Monitor overload information, and drill through specific timepoints to identify users/operations causing spikes.
KQL Database operation
The operation categories seen in this view are:
Here you can see that the Demo22 database that is in the RTA Field Demos workspace was active for 60 seconds and utilized 255 CU (s). From this information this particular KQL Database was utilizing 4.25 virtual cores during this time (255/60).
Timepoint – Explore graph
This graph in the Microsoft Fabric Capacity Metrics app shows utilization of resources compared to capacity purchased. 100% of utilization represents the full throughput of a capacity SKU and is shared by all Fabric experiences. This is represented by the yellow dotted line. The Logarithmic scale option enables the Explore button, which opens a detailed drill through page.
In general, operations are classified either as interactive or background, and denoted by color. All operations in KQL Database category are reported as interactive.
Timepoint – Drill through graph
This table in the Microsoft Fabric Capacity Metrics app provides a detailed view of utilization at specific timepoints. The amount of capacity provided by the given SKU per 30-second period is shown, along with the breakdown of interactive and background operations.
For more information on Fabric warehouse usage reporting, please check out our documentation Billing and utilization reporting.
Monitoring OneLake Storage
In the same Microsoft Fabric Capacity Metrics app that was utilized to monitor your capacity usage you can also monitor your OneLake Storage usage.
As storage is billed for the GB stored in OneLake, this tab gives you details on the GB stored at a Microsoft Fabric Workspace level. You are able to see the overall storage along with the accumulation of storage overtime.
**NOTE: The current OneLake tab only shows OneLake Standard storage as this is the only storage currently being reported. As we introduce OneLake Cache storage this app will be updated accordingly**
