Get started with User Data Functions in Microsoft Fabric (Private preview)
We are excited to give you a sneak peek of User Data Functions in Microsoft Fabric! User Data Functions gives developers and data engineers the ability to easily write and run applications that integrate with resources in the Fabric Platform. You can contact us and request to join our private preview program.
Data engineering often presents challenges with data quality or complex data analytics processing in data pipelines, and using ETL tools may present limited flexibility and ability to customize to your needs. This is where User data functions can be used to run data transformation tasks and perform complex business logic by connecting to your data sources and other workloads in Fabric.
During Preview, you will be able to use the following features:
- Use the Fabric portal to create, view and test your Fabric User Data Functions.
- Use the Visual Studio Code extension to create and edit your functions in C#.
- Connect to the following Fabric-native data sources: Data Warehouse, Lakehouse and Mirrored Databases.
In this blog post, you will learn how to create your first User Data Functions and connect to a Fabric-native data source.
What is a User Data Functions item?
The User data functions item in Fabric is a collection of individual functions that share the same code file and configuration settings. They can also access the Fabric resources available in your Workspace and any other ones that you have access to. You can create many functions under the User data functions artifact to handle different tasks in your application.
Create your first User Data Functions
First, you need a Fabric workspace to get started. Follow the steps in this article to Create a workspace in Microsoft Fabric.
Once you have a workspace, navigate to the Data Engineering experience by clicking on the bottom left corner icon on the sidebar. Click on “User data functions (Preview)” from the menu to create a new User data functions artifact. This will open the Functions explorer where you can click on “New function” to automatically create a function called “HelloFabric”. This function contains boilerplate code to get started. You can always see a read-only view of the code in the “Code” tab of the Functions explorer.
You can run your function in the “Test” tab by providing all the required parameters and clicking on the “Run” button. The type and name of the parameters will be automatically sorted out by Fabric, even if there are multiple parameters. You can also view the output of your function, in addition to any logs captured during its execution.
Your User Data Functions artifact is now ready to connect to your data sources in Microsoft Fabric! Keep on reading to learn how.
Connect your User Data Functions to your data in Microsoft Fabric
User Data Functions supports connecting to the following Fabric data sources:
- Warehouse
- Lakehouse
- Mirrored Databases
We will use a Warehouse in this example, but the process to create a connection is similar for any of the other data sources.
To get started, first create a Warehouse in Microsoft Fabric. If you have an existing Warehouse, make sure that it is in the same workspace of your User Data Functions item or that you have access to the workspace that it is located in.
Once the data warehouse is created, you can now create a connection to it by going to the Functions explorer and clicking on Manage connections.
This will open a side panel with a list of configured connections. Any connections added to this list will be visible to all your functions in this User Data Functions item. Select the “+ Add” button to add a new data connection. This will open the “Get data” view where you will be able to select any compatible data sources that your function already has access to.
Selecting any of these data sources will automatically create a connection alias for your data source. You can use this alias in the programming model under the FabricItemInput(“alias-name”) parameter of your function code. In our case, we used the alias “myfabricwarehouse”.
Edit, debug, and publish your data functions using VS Code
As part of the private preview, you will also have access to the following VS code extensions:
- Fabric Workspaces: This extension allows you to manage your Fabric workspace inside VS Code.
- Fabric Data development: This extension gives you the capabilities to author, manage, and publish your data functions from VS Code.
Once these extensions are enabled, you can select “Open in VS Code” in the Functions explorer. Your functions will open in the Explorer in VS Code and you can do the following:
- You can edit the existing functions in the FabricFunctions.cs file.
- You can create new functions by adding them inside of the FabricFunctions class in the FabricFunctions.cs file.
- You can start debugging by pressing F5. You can also enable breakpoints in your code.
When debugging, you will see the Run & Debug Functions tool, which you can use to locally run your functions, even with parameters. Your functions will also call any Fabric data sources that you have configured.
You can add as many functions as you want inside user data functions artifact. These functions may connect to data sources in Fabric or can be used to schema validation, filtering data, checking for null values, removing duplicate data and more. After making the changes and testing it locally, you can publish the function to Fabric by navigating to the Fabric extension on the sidebar and doing a right click on your User Data Functions item.
All changes and any newly added functions will now be visible in Fabric and ready to be invoked by any application or to be used inside Fabric.
Conclusion
If you are interested in trying out User data functions, reach out to the Product team to join our Preview program. There are lots more exciting features in our roadmap for User data functions along with improvements in our programming model focused to making it easier to author your data functions and ease of development in VS Code.
Please share your feedback or questions and keep an eye on Fabric blog for more exciting updates on User data functions in Fabric.
To learn more about Microsoft Fabric, check out the following resources:
