Microsoft Fabric Updates Blog

Using Microsoft Fabric’s Lakehouse Data and prompt flow in Azure Machine Learning Service to create RAG applications

Microsoft Fabric’s Lakehouse helps us better unified management of enterprise-level data environments. In the process of transforming to AI, we cannot do without the assistance of these enterprise data. In my previous blog, I mentioned how to build RAG applications based on data in the Microsoft Fabric environment. In this post, I will introduce how to build a RAG application through prompt flow in a more professional machine learning environment – Azure Machine Learning Service combined with Microsoft Fabric’s Lakehouse data.

Azure Machine Learning Service is a machine learning platform that I enjoy using, covering the machine learning process from data, training, testing, deployment, monitoring, etc. We can very quickly introduce Microsoft Fabric Lakehouse data to Azure Machine Learning Service through a short script.

1. Get the ABFS Path of Lakehouse in Microsoft Fabric.

Choose Your Microsoft Fabric’s Lakehouse, Click Files -> Properties.

Copy ABFS Path

abfss://<One Lake workspace name><Lakehouse ID>/Files

2. Create a new Notebook in your local machine. Execute the following code to import Lakehouse data into Azure Machine Learning Service

! pip install azure-ai-ml -U 
! pip install mltable azureml-dataprep[pandas] -U 
! pip install azureml-fsspec -U

from import MLClient
from azure.identity import DefaultAzureCredential
from import OneLakeDatastore, OneLakeArtifact

subscription_id = "Your Azure Subscription ID" 
resource_group = "Your Azure Machine Learning Service Workspace Resource Group" 
workspace = "Your Azure Machine Learning Service Workspace Name"

ml_client = MLClient(
    DefaultAzureCredential(), subscription_id, resource_group, workspace

artifact = OneLakeArtifact(
    name=<Lakehouse ID>, 
store = OneLakeDatastore(
    description="Credential-less OneLake datastore.",
    one_lake_workspace_name=<One Lake workspace name>,


3. Test the data to see if it is imported successfully.

from import AssetTypes, InputOutputModes
from azureml.fsspec import AzureMachineLearningFileSystem

uri = 'azureml://subscriptions/<Your Azure Subscription ID >/resourcegroups/<Your Azure Machine Learning Service Resource Group>/workspaces/<Your Azure Machine Learning Service Workspace Name>/datastores/onelake_lh_for_azureml'

# create the filesystem

fs = AzureMachineLearningFileSystem(uri)

with'Files/csv/sales.csv') as f:
    data = f.readlines()

You can select Data from Azure Machine Learning Service to see if the relevant data is imported successfully.

from import Data
import pandas as pd
import mltable

csv_path = 'azureml://datastores/onelake_lh_for_azureml/paths/Files/csv'
my_csv_data = Data(

csv_data ="csv_data_source", version="1.0.0")

path = {
  'folder': csv_data.path

tbl = mltable.from_delimited_files(paths=[path])

df = pd.read_csv( csv_data.path + '/sales.csv')


Of course, you can also check the data in the workspace of Azure Machine Learning Service to see if it is synchronized well.

In the previous content we used Semantic Kernel. In this blog, we use prompt flow to build the application. Prompt flow is a development tool designed to streamline the entire development cycle of AI applications powered by Large Language Models (LLMs). As the momentum for LLM-based AI applications continues to grow across the globe, Prompt flow provides a comprehensive solution that simplifies the process of prototyping, experimenting, iterating, and deploying your AI applications. If you’re looking for a versatile and intuitive development tool that will streamline your LLM-based AI application development, then prompt flow is the perfect solution for you.

The biggest feature of prompt flow is to help the Prompt project to be better integrated into the project. Especially in stabilizing the output of LLM, it allows you to choose the best Prompt and combine it with LLM for effective work.

Prompt flow development applications can be developed on Azure Machine Learning Service, on the command line, or on Visual Studio Code. It is recommended that you develop on Visual Studio Code. Firstly, you need to install prompt flow for VS Code extensions.

After successful installation, click on the prompt flow extensions on the left sidebar and select Installation Dependencies. When the environment is successfully configured, you can choose to create and build the Prompt flow application.

Prompt flow can support different connections, such as Azure OpenAI Service, Azure Cognitive Search, Azure Content Safety and also support Custom Connections. You can set relevant content according to your needs.

Custom Connection is often used. You can set some link configurations, mainly in the form of key-value pairs.

Use prompt flow to quickly build a flow for enterprise data. The following are implementations for structured data and unstructured data, as well as a simple example of the Chat flow process. All of this data all comes from our Azure Machine Learning Service (imported from Microsoft Fabric Lakehouse)

This is a RAG application for unstructured data and structured data built by prompt flow

You can download samples in my GitHub Repo

Related blog posts

Using Microsoft Fabric’s Lakehouse Data and prompt flow in Azure Machine Learning Service to create RAG applications

July 3, 2024 by Ruixin Xu

We are excited to announce new and exciting updates to Semantic Link over the last a few months:  Semantic Link is preinstalled in the default runtime  At the Fabric Conference in March 2024, we announced that Semantic Link was generally available. We have now included Semantic Link in the default runtime by default. If you … Continue reading “Semantic Link Updates June 2024 “

June 26, 2024 by Amir Jafari

We are excited to share guidance for how you can use Microsoft Fabric to turn your data into knowledge for Generative AI applications. This guide will walk you through implementing a RAG (Retrieval Augmented Generation) system in Microsoft Fabric using Azure OpenAI and Azure AI Search. By the end, you’ll be more familiar with how to … Continue reading “Building Custom AI Applications with Microsoft Fabric: Implementing Retrieval Augmented Generation for Enhanced Language Models”