Microsoft Fabric Updates Blog

Reduce egress costs with S3 shortcuts in OneLake

Fabric allows workloads to easily access data across clouds through OneLake shortcuts. Define a shortcut once and use it with Power BI reports, SQL, Spark and Kusto. This ease of consumption allows users to start analyzing their data in minutes rather than hours or even days, but this can also lead to increased egress charges.

Cloud providers like Azure, AWS and GCP have extra networking fees when the data goes out of their infrastructure. The fees are usually measured in cents per gigabyte and look minor on the pricing guides, but they can become significant in the context of big data analytics. For this reason, we are happy to announce that OneLake shortcuts to S3 now has caching functionality.

A diagram of a cloud computing system

Description automatically generated

With caching for S3 shortcuts, egress costs are greatly reduced. As files are read through an S3 shortcut, the files are stored in a cache for the Fabric workspace. Subsequent read requests are served from cache rather than S3 directly. Cached files have a retention period of 24 hours. Each time the file is accessed the retention period is reset. If the file in S3 is more recent than the file in the cache, the request is served from S3 and the updated file will be stored in cache. If a file hasn’t been accessed for more than 24hrs it is purged from the cache.

A diagram of a project

Description automatically generated

How to enable caching for S3 shortcut:

Cache for S3 shortcuts is enabled at a workspace level. Workspace admins can start by opening a Fabric workspace and selecting “Workspace settings

In the workspace settings panel, select the “OneLake” tab. Switch the toggle for “Enable Cache for S3 Shortcuts” to “On”. Then click the “Save” button.

A screenshot of a computer

Description automatically generated

Caching is now enabled for all S3 shortcuts in that workspace. Any existing or new S3 shortcut will utilize the cache.

For a reminder on creating S3 shortcuts, follow our guide here: Create an Amazon S3 shortcut – Microsoft Fabric | Microsoft Learn

Related blog posts

Reduce egress costs with S3 shortcuts in OneLake

June 4, 2024 by Anshul Sharma

As part of the One logical copy promise, we’re excited to announce that OneLake availability of Eventhouse in Delta Lake format is Generally Available.  Eventhouse is a cutting-edge database workspace meticulously crafted to manage and store event-based data. Engineered to handle data in motion, Eventhouse seamlessly integrates indexing and partitioning into its storing process, accommodating … Continue reading “Eventhouse OneLake Availability is Generally Available”

May 31, 2024 by Dandan Zhang

As more and more enterprises store and analyze data on the cloud, the need for securing sensitive data has become paramount. Microsoft Fabric offers security at different levels – for instance, access control using workspace roles/permissions and granular security at the data layer. In addition to these, Network security provides a critical level of isolation, … Continue reading “Announcing General Availability of Fabric Private Links, Trusted Workspace Access, and Managed Private Endpoints”