Shortcuts to Google Cloud Storage, now available in Public Preview
We are excited to announce that you can now create OneLake shortcuts to your Google Cloud Storage (GCS) buckets. With the addition of GCS, you can now utilize cross-cloud shortcuts to analyze your data across all three major cloud platforms. Shortcuts in OneLake allow you to connect to your existing data through a single unified name space without having to copy or move data. Just open a Lakehouse in Fabric, create a shortcut to GCS, AWS S3 or ADLS Gen2 and immediately start analyzing your data through Spark, SQL and Power BI.
How to create GCS shortcuts
Open a Lakehouse in Fabric and create a new shortcut. If your target data is already in the Delta/Parquet format, create the shortcut in the “Tables” section of the Lakehouse. This will allow metadata synchronization across Spark, SQL and Power BI. If you target data is in another format, create the shortcut in the “Files” section of the Lakehouse.
Under External Sources, select Google Cloud Storage.
Configure your connection settings.
In the connection URL, you can specify either the global endpoint for the GCS storage service or a bucket specific endpoint.
- Global endpoint: https://storage.googleapis.com
- Bucket specific endpoint: https://<BucketName>.storage.googleapis.com
GCS shortcuts utilize HMAC keys for authorization with Google. HMAC keys can be created for both user accounts and service accounts. Follow the guide here to create an HMAC key: Manage HMAC keys for service accounts.
The account associated with the HMAC key must have permission to access the data within GCS. If you provided a bucket specific URL, the account needs the following permissions:
- storage.objects.get
- stoage.objects.list
If you provided the global endpoint in the URL, the account would need the following additional permissions:
- storage.buckets.list
Once your connection is configured, you can browse your storage account in GCS.
In this view, you can select multiple target locations to create shortcuts to.
The review page allows you to see all of the selections you made.
Here you can use the pencil button to rename any of your selections. The trash button can be used to remove any unwanted selections. Click create and your shortcuts will appear in your lakehouse.
Now start analyzing your data! Open the SQL endpoint and start writing new queries. Create a new Notebook and analyze your data with Spark. Open the semantic model and create a new Power BI report.
Caching
Just like AWS S3 shortcuts, GCS shortcuts also support caching. With caching enabled, egress costs are greatly reduced. As files are read through the GCS shortcut, the files are stored in a cache for the Fabric workspace. Subsequent read requests are served from cache rather than GCS directly.
Cache enablement is configured at a workspace level. To enable shortcut caching, workspace admins can start by opening a Fabric workspace and selecting “Workspace settings”
In the workspace settings panel, select the “OneLake” tab. Switch the toggle for “Enable Cache for Shortcuts” to “On”. Then click the “Save” button.
Caching is now enabled for all GCS, S3 and S3 compatible shortcuts in that workspace. To learn more about shortcut caching see our blog here: Reduce egress costs with S3 shortcuts in OneLake | Microsoft Fabric Blog | Microsoft Fabric.
