Private ADLS Gen2 access made easy with OneLake Shortcuts: a step-by-step guide
Microsoft Fabric provides the capability to streamline data access through OneLake Shortcuts. OneLake Shortcuts can significantly reduce data sprawl, enhances data interoperability and accessibility, promotes self-service without the need for ETL/ELT processes, and can improve Power BI semantic model performance with Direct Lake mode.
A common question from our customers, particularly those in regulated industries, is how to leverage their existing ADLS Gen2 enterprise data lake within Fabric, especially with private access enabled on their storage accounts. Our solution is Trusted Workspace Access, which allows secure shortcuts to data in ADLS Gen2, even when protected by a firewall or when public access is entirely disabled.
In this article, we will:
- Create a Fabric Workspace.
- Create a Fabric Workspace Identity.
- Create an ADLS Gen2 storage account with Public Access disabled (Private IP only).
- Configure Trusted Workspace Access.
- Configure the OneLake shortcut to the Private Access storage account using the Fabric Workspace Identity as the authentication.
Note: In this article, we walk through a step-by-step process of creating all the components net new. You can absolutely leverage your existing architecture to utilize the features showcased.
Background
Before we dive into the step-by-step guide for creating the OneLake shortcut to the private access-enabled ADLS Gen2 account, let’s first cover some background information on the various components involved.
What are OneLake Shortcuts?
OneLake Shortcuts enable you to unify your data across various domains, clouds, and accounts, creating a single virtual data lake for your entire enterprise. This allows all Fabric experiences and analytical engines to directly connect to your existing data sources, such as Azure, Amazon Web Services (AWS), Google Cloud, On-premises data, Iceberg tables, Dataverse, and OneLake, through a unified namespace. OneLake manages all permissions and credentials, so you don’t need to configure each Fabric workload separately to connect to each data source. Additionally, shortcuts help eliminate redundant data copies and reduce process latency associated with data staging.
In OneLake, shortcuts are objects that reference other storage locations, which can be either internal or external to OneLake. You can Think OneLake Shortcuts like desktop shortcuts. The destination a shortcut points to is called the target path, while the location where the shortcut is accessed is known as the shortcut path. Shortcuts appear as folders within OneLake and can be utilized by any workload or service with access to OneLake. Functioning like symbolic links, shortcuts are independent of their targets. Deleting a shortcut does not affect the target, but moving, renaming, or deleting the target path can break the shortcut. You are able to create OneLake shortcuts in either a Lakehouse or KQL Database.
For our walkthrough, we will use a Lakehouse, as demonstrated in the screenshot.
For a deeper dive on OneLake shortcuts, please visit the documentation page which includes the information in the previous text: Unify data sources with OneLake shortcuts
What is Direct Lake Mode?
OneLake shortcuts also enable the use of Direct Lake Mode within Power BI Semantic Models. While this article does not cover the setup process, it’s important to understand how shortcuts facilitate this feature.
Direct Lake is a storage mode option for tables in a Power BI semantic model stored in a Microsoft Fabric workspace. It is optimized for handling large volumes of data that can be quickly loaded into memory from Delta tables, which store their data in Parquet files within OneLake—the unified store for all analytics data. Once loaded into memory, the semantic model allows for high-performance queries. Direct Lake eliminates the need to import data into the model, which can be slow and costly. Instead, it references the metadata in OneLake to access the latest data versions without reloading the model, ensuring up-to-date data without the expense of frequent refreshes.
You can use Direct Lake storage mode to connect to the tables or views of a single Fabric lakehouse or Fabric warehouse.
You might be wondering how this works with external data residing in ADLS Gen2 or other external locations. Delta tables store their data in Parquet files, typically within a lakehouse that a Direct Lake semantic model uses to load data. However, Parquet files can also be stored externally. By using a OneLake shortcut, you can reference external Parquet files stored in locations such as Azure Data Lake Storage (ADLS) Gen2, Amazon S3 storage accounts, or Dataverse. Compute engines generally access these Parquet files by querying Delta tables. This setup allows you to leverage Direct Lake mode with data stored externally, such as in ADLS Gen2. With Trusted Workspace Access, you can even utilize this feature when ADLS Gen2 is secured behind a private endpoint.
For more information on Direct Lake Mode and the source of the above information, please refer to the provided links:
Direct Lake overview
Understand storage for Direct Lake semantic models
What is Trusted Workspace Access?
Fabric enables secure access to firewall-enabled or private endpoint-enabled Azure Data Lake Storage (ADLS) Gen2 accounts. Fabric workspaces with a workspace identity can securely access ADLS Gen2 accounts, whether public network access is enabled from selected virtual networks and IP addresses or public access is disabled. You can restrict ADLS Gen2 access to specific Fabric workspaces. It’s important to note that this setup only grants network visibility from the specified Fabric workspace to ADLS Gen2. Proper authentication and authorization, controlled by RBAC, are still required to access the appropriate locations. Authorization is supported using Microsoft Entra credentials for organizational accounts or service principals.
Trusted Workspace Access leverages the ADLS Gen2 feature known as a Resource Instance Exception. This firewall exception allows traffic from a specific resource—in this case, the specified Fabric workspace only. Resource instance exceptions are an existing ADLS Gen2 security and network feature that Fabric utilizes. For more information about resource instance rules for ADLS Gen2, refer to Grant access from Azure resource instances.
Once configured you will be able to use this connection for ADLS Gen2 shortcuts, Fabric Data pipelines to directly access the privately secured ADLS Gen2 account or use T-SQL Copy statements to leverage the Lakehouse shortcut to ingest into a Fabric Data Warehouse.
For more information and the source above refer to:
Trusted workspace access in Microsoft Fabric
Configure Azure Storage firewalls and virtual networks
Note: Even when accessing ADLS Gen2 behind a private endpoint, a firewall exception is still required. Behind the scenes, the firewall exception is checked and enforced, even for privately accessible ADLS Gen2 account. Therefore, the trusted workspace process is necessary for any ADLS Gen2 account network configuration for public access with firewall enabled and public access disabled.
Step-by-Step walkthrough
Fabric Workspace Creation
First, you need to have a Fabric capacity provisioned, Trusted Workspace Access is only available in FSKUs. I have one created in the Azure Portal called ‘fabriccapacityprod’ and set to be an F2.
Next, I will create a new Fabric Workspace and assign this capacity to it. Within Fabric, navigate to Workspaces and select ‘New Workspace’.
I have named my new workspace ‘awesome_FabricWorkspace’, and I will assign my Fabric capacity above to the workspace, then select ‘Apply’.
Create a Lakehouse
With the workspace created, we will create a Lakehouse that will be used to access the ADLS Gen2 shortcut that we will create later.
Within the workspace, we will select ‘New item’ and search for Lakehouses. The Lakehouse ‘LH_TrustedWorkspaceDemo’ is created.
Create a Workspace Identity
To use Trusted Workspace Access, we will need to create a workspace identity. Navigate to the ‘Workspace settings’ and select ‘Workspace identity’. Once on the screen you will select ‘Create’.
After waiting a few seconds, the page refresh with our workspace identity.
Create ADLS Gen2 Storage Account
Now that the Fabric side of the house has been set up (minus the shortcut we have yet to create), we can create our ADLS Gen2 storage account.
In the Azure Portal, we will create a new Storage Account called ‘adlsawesomesa’. I decided to leave everything in the default except for below:
- Basics:
- Region: I made sure to align the region to the same region as my Fabric capacity. That is not a requirement to make this work, but it is best practice to ensure the regions line up to minimize latency and potential egress costs.
- Redundancy: I changed to LRS since this is a simple demo and no reason to have a different redundancy for cost savings.
- Advanced:
- Hierarchical Namespace: Check this box.
- Networking:
- I leave this as public access to start just to make it easier to upload sample data. We will edit the networking after the deployment to a private endpoint.
Note: If creating a new ADLS Gen2 Storage Account, ensure it meets your organization standards.
Create a Storage Container and Upload Sample File
With our access all squared away, the next step we have is to upload some data to our newly created storage account. This requires us to create a new storage container within the account to house our files. Within the ADLS Gen2 storage account, navigate to the ‘Data storage’ hierarchy and the ‘Containers’ blade. Select ‘+ Container’ and type the name of your container (‘amazingcontainer’), then select ‘Create’.
Once the container has been created, then I uploaded a sample CSV file to test the connectivity of the shortcut later.
Note: I uploaded files with the storage account’s network settings set to public access. This was to make it very easy to upload a file. With private endpoint enabled, we would need a lot of extra steps to upload that are irrelevant for this how-to.
Grant the Fabric Workspace Identity Access to the Storage Account
Next, we will grant access to the Fabric Workspace Identity we created previously to the storage account. You may also use a user account or service principal authentication.
The user account or service principal (inclusive of the workspace identity) used for authentication in the shortcut should have Azure RBAC roles on the storage account. The principal must have a Storage Blob Data Contributor, Storage Blob Data owner, or Storage Blob Data Reader role at the storage account scope, or a Storage Blob Delegator role at the storage account scope together with access at the folder level within the container. Access at the folder level can be provided through an RBAC role at the container level or through specific folder-level access. Find the permission requirements for Trusted Workspace Access with this link: Fabric Trusted Workspace Access Storage Account Prerequisites
To follow least privilege guidance, we will only grant ‘Storage Blob Data Reader role’ at the container level. This will scope the Fabric Workspace’s identity to one specific container.
Within the container, select ‘Access Control (IAM)’ and ‘Add role assignment’.
We will grant the workspace identity the RBAC role ‘Storage Blob Data Reader’, select ‘Next’.
In this scenario, we are just reading the data from ADLS, scoping to just one container. Shortcuts are one directional, meaning there is no write back capability to the shortcut source from Fabric.
Under ‘Assign access to’ select ‘User, group, or service principal’. Then select ‘+ Select members’. Type the name of your Fabric Workspace Identity (aka your workspace name) and select it, next, select ‘Review and assign’.
Update Network Settings on the ADLS Gen2 Storage Account
Now it is time update the network settings on the storage account we created to deny public access and use a private endpoint.
First navigate to the ‘Security + networking’ hierarchy on the resource. Then select ‘Networking’, then ‘Public network access’ select ‘Disabled’, and finally select ‘Save’.
Now when I navigate back to the container where I uploaded the files, I am not longer able to access the container from the portal because of the public access being disabled. You will get the same ‘403’ error if you were to create the shortcut from Fabric in this moment as well.
Enable Trusted Workspace Access
With everything configured on the Fabric side, a sample file in the storage account, and the storage account set behind a private endpoint, it’s time to enable trusted workspace access. To achieve this, we will deploy a custom JSON ARM template or PowerShell script that modifies the firewall to allow a resource exception specifically for our Fabric workspace. This will enable traffic from the configured Fabric workspace to securely connect to ADLS, even with the private endpoint enabled on the storage account. We will specifically cover and walkthrough deploying a JSON ARM template.
There is no option to do this in the UI. You can apply other resource instances rules on ADLS Gen2 for other services, but Fabric is code only.
JSON ARM Template
Here is a sample of the JSON ARM template that will be used, the code for where to find the values to add. This template is also available within the documentation: Trusted workspace access in Microsoft Fabric – Microsoft Fabric | Microsoft Learn
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2023-01-01",
"name": "<storage account name>",
"id": "/subscriptions/<subscription id of storage account>/resourceGroups/<resource group name>/providers/Microsoft.Storage/storageAccounts/<storage account name>",
"location": "<region>",
"kind": "StorageV2",
"properties": {
"networkAcls": {
"resourceAccessRules": [
{
"tenantId": "<tenantid>",
"resourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/Fabric/providers/Microsoft.Fabric/workspaces/<workspace-id>"
}]
}
}
}
]
}Values in JSON Explained:
- “name”: “<storage account name>”
- Storage Account Name where the resource instance rule is being applied
- “id”: “/subscriptions/<subscription id of storage account>/resourceGroups/<resource group name>/providers/Microsoft.Storage/storageAccounts/<storage account name>”
- Subscription ID, resource group name, storage account name where the resource instance rule is being applied
- All can be found on the Storage Account resource overview page.
- “location”: “<region>”
- Region of the Storage Account where the resource instance rule is being applied.
- Can be found on the Storage Account resource overview page.
- Under “properties”…”tenantId”: “<tenantid>”
- Tenant ID for your storage account where the resource instance rule is being applied
- Can be found on the front page of Microsoft Entra in the Azure Portal. You can search this in the top bar.
- “resourceId”: “/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/Fabric/providers/Microsoft.Fabric/workspaces/<workspace-id>”
- YES, ALL 0s IN THE SUBSCRIPTION IS CORRECT
- Workspace ID is the Fabric Workspace ID where the shortcut is being created. Refer to the screenshot image for where to find this.
Workspace ID location -> in Fabric, go to the workspace that is being used for connectivity/resource instance exception. The workspace ID is the ID in the URL after “…/groups/”. It is blocked out for this demo.
Powershell Template
$resourceId = "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/Fabric/providers/Microsoft.Fabric/workspaces/<YOUR_WORKSPACE_GUID>"
$tenantId = "<YOUR_TENANT_ID>"
$resourceGroupName = "<RESOURCE_GROUP_OF_STORAGE_ACCOUNT>"
$accountName = "<STORAGE_ACCOUNT_NAME>"
Add-AzStorageAccountNetworkRule -ResourceGroupName $resourceGroupName -Name $accountName -TenantId $tenantId -ResourceId $resourceId
Yes, all 0s in the subscription is correct. Unless otherwise stated, where there are all 0s, they remain. The JSON ARM template has a separate spot to place the subscription ID in addition to the all 0s entry.
Deploying the ARM Template
With the ARM template for the resource instance rule populated, it is time to deploy it. To do that, you can navigate to the Azure Portal and in the search bar, search ‘Deploy a custom template’.
Next, select ‘Build your own template in the editor’.
Paste the template with updated values into the editor and select ‘Save’.
Next select the resource group for the deployment, then ‘Review + create’, this will deploy the script, and wait for it to deploy successfully.
Verify The Firewall Exception Has Been Applied
An optional step is to verify that the firewall resource instance exception has been applied. Navigate back to the storage account that has been updated.
Navigate to the networking blade again and change the value in the ‘Public network access’ section from ‘Disabled’ to ‘Enabled from selected virtual networks and IP addresses’ (Do not save, this is just to visualize the firewall rule exception has been applied ).
Now under the ‘Resource instances’, there should be an entry for ‘Microsoft.Fabric/workspaces’ with the Fabric workspace ID populated.
Reminder that even when accessing ADLS Gen2 behind a private endpoint, a firewall exception is still required. Behind the scenes, the firewall exception is checked and enforced, even for privately accessible ADLS Gen2 accounts. Therefore, the trusted workspace process is necessary for any ADLS Gen2 network configuration for public access with firewall enabled and public access disabled.
Create a OneLake Shortcut to the ADLS Gen2 with Public Access Disabled
The final step in our process is to create the shortcut to access the sample file. With public access disabled on the ADLS Gen2 account, we will leverage the resource instance exception configured in the previous steps to establish a secure connection and utilize the shortcut.
Navigate back to Fabric and open the Lakehouse previously created.
There are multiple ways to create a new shortcut. You can do this from the first screen when opening the Lakehouse, selecting ‘Get Data’ on the top bar, or by using the method we will demonstrate: selecting the ellipsis on files or tables within the explorer pane.
Next select ‘Azure Data Lake Storage Gen2’.
Select ‘Create new connection’ and place the Data Lake storage URL in the URL section. (https://<storage account name>.dfs.core.windows.net/ )
From the ‘Authentication kind’ select ‘Workspace identity’.
Once you select ‘Next’ and see the containers, it indicates that the connection and authentication were successful. If the connection or authentication fails, you will not progress to this screen.
Now select the files to shortcut to.
Select ‘Create’.
Access Your Data!
Now you can visualize the data within Fabric via a OneLake shortcut.
In a notebook you can visualize the data as if it were a physical file in OneLake, but it is residing behind a private IP in ADLS Gen2.
Conclusion
In conclusion, this guide has shown you how to securely access data in an ADLS Gen2 storage account, whether it’s behind a private endpoint or a firewall-enabled public endpoint with Trusted Workspace Access. By using shortcuts for seamless data sharing, leveraging Direct Lake mode, utilizing trusted workspace connections within Fabric Pipelines, or employing the T-SQL COPY statement to load data into a Data Warehouse, you can maximize your existing infrastructure within Fabric. This approach enables you to Direct Lake mode, reduces data duplication, and opens up numerous possibilities within Fabric—all without compromising security with Trusted Workspace Access.
