Announcing Eventhouse Query Acceleration for OneLake Shortcuts (Preview)
Turbo charge queries over data in OneLake
We are excited to announce a brand-new capability Query Acceleration for OneLake Shortcuts that turbo-charges ad-hoc queries over data in OneLake.
Eventhouse is a cutting-edge database meticulously crafted to manage and store event-based data. Engineered to handle data in motion, Eventhouse seamlessly integrates indexing and partitioning into its storing process, accommodating structured, semi-structured, and free text data formats. This sophisticated design empowers high-performance analysis with minimal latency, facilitating lightning-fast ingestion and querying within seconds.
OneLake shortcuts are references from an Eventhouse that point to internal Fabric or external sources. Previously, queries run over OneLake shortcuts were less performant than on data that is ingested directly to Eventhouses due to various factors such as network calls to fetch data from storage, the absence of indexes, and more – Query Acceleration changes that.
Introducing Query Acceleration
Query acceleration indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark.
When creating a shortcut from an Eventhouse to a OneLake delta table, users can choose if they want to accelerate the shortcut. Accelerating the shortcut means equivalent ingestion into the Eventhouse: indexing, caching, other optimizations that deliver the same level of performance for accelerated shortcuts as native Eventhouse tables.
By using this feature, you can accelerate data landing in OneLake, including existing data and any new updates. This eliminates the need to manage ingestion pipelines, maintain duplicate copies of data, while ensuring that data remains in sync without additional effort.
When using accelerated OneLake shortcuts, all data management is done by the data writer and in the Eventhouse the accelerated table shortcut, like any other shortcut is read only.
Customers can expect significant performance improvements by enabling this capability, in some cases up to 50x and beyond.
When to use Query Acceleration
Here are some scenarios where this feature proves invaluable:
1. Query data in OneLake at blazing fast speed – When you have existing workloads that are uploading data and managing it in storage (in a different cloud or region), and you would like to query some or all of it at blazing fast speed.
2. Mesh historical data with real-time streams – When you want to seamlessly combine data landing in OneLake directly with real-time streams coming into Eventhouse without compromising on query speeds.
3. Leverage dimension data managed by other items – Often high value and small volume data is hosted in SQL servers, Cosmos DB, Snowflake or other systems that can be mirrored into OneLake. Accelerated OneLake shortcuts are excellent in making this data easily consumable for joins and enrichment in the Evenhouse query. As dimension data is often significantly smaller than activity data, the additional cost associated with that usage is typically minimal, but the performance gain is huge.
How to enable Query Acceleration
To enable query acceleration on a new shortcut, follow these steps:
- Browse to an existing KQL database.
- Select New > OneLake shortcut.
3. Select a source
4. Under Internal sources, select Microsoft OneLake (or any other source where your data resides).
5. Select the item you want to connect to, and then select Next.
6.Expand Tables, and select a specific table to connect to.
7. Select Next.
8. Toggle the Accelerate button to On.
9. Select Create.
10. Query the data using external_table(‘Shortcut name‘) syntax.
It might take few minutes to notice the performance gain on existing data as Eventhouse will take some time to create initial indexes.
Costs/Billing
Enabling Query Acceleration does come with some additional costs. The accelerated data will be charged under OneLake Premium cache meter, similar to native Eventhouse tables. You can control the amount of data to accelerate by configuring number of days to cache. Indexing activity may also count towards CU consumption.
These charges will appear in the Fabric metrics app under the Eventhouse where the accelerated shortcut is created.
Conclusion
Query Acceleration is a powerful feature designed to enhance your data querying capabilities on PetaBytes of data. By understanding when and how to use this feature, you can significantly improve the efficiency and speed of your data operations – whether you are dealing with large datasets, complex queries, or real-time analytics, Query Acceleration provides the performance boost you need to stay ahead.
Get started now: Query Acceleration for OneLake Shortcuts in Eventhouse
Demo
What’s next?
There will be many more exciting developments as we continue to innovate and expand the capabilities of Real-Time Intelligence. Learn more about all the features and follow step-by-step tutorial. Join the conversation and vote for your favorite features.
Over the next month we’ll be releasing a series of blog posts that dive into all the capabilities further. Stay tuned to for more!
