Data Factory June 2023 Monthly Update
Welcome to the first edition of the Microsoft Fabric Data Factory monthly update!
In the first week of each month, visit our blog site to see what our team has been up to. This blog will cover the Data Factory experience in Fabric: Dataflows Gen2 and Data Pipelines.
Read on to find all the announcements we have for you for this month!
Contents
Dataflows Gen2
Data pipelines
Dataflows Gen2
Power Query Editor
Manage Connections (Preview)
A highly requested feature that brings you a list of your connections used in your Dataflow and the ability to disconnect any of them from your Dataflow. You will find Manage Connections inside of the Home tab of the Power Query Editor.
This new feature currently only offers a way to disconnect any connection that you have in your dataflow.
During the rest of the calendar year 2023, we’ll be releasing new capabilities to this experience, such as the ability to add and edit connections in your Dataflow.
To learn more, read Manage connections (Preview) – Power Query | Microsoft Learn.
Column pair suggestion for Merge operations (Preview)
One of the most common transformations done in Power Query is the Merge operation which provides you with a way to combine data from two tables using a logic that pairs column(s) from one table to the other.
In some scenarios, it might be a bit challenging with new projects that have tables with a high number of columns to understand how columns should be paired between tables. With the new suggestions inside of the Merge dialog, you can click the lightbulb on the top right corner of the dialog and get a list of possible ways to pair the columns between both tables.
To learn more, read Merge queries overview – Power Query | Microsoft Learn.
Power Query templates (Preview)
Power Query is a data preparation tool that is available in multiple products such as Power BI Dataflows, Power Platform Dataflows, Excel for Windows, Excel for Mac and many others. Traditionally, to share your queries you could copy and paste your queries as shown in the article on share queries. Today we are releasing into public preview the ability to export and import templates inside of Data Factory Dataflows Gen2.
This consists of the export template functionality available across all Power Query Online hosts with a button in the Home tab with the name Export template that effectively exports your Power Query script and its associated metadata.
Note: Excel for Windows also has an export template functionality that can be leveraged to create a Power Query template file. Read more on Create Power Platform dataflows from queries in Microsoft Excel (Preview) – Power Query | Microsoft Learn.
The ability to import a template is currently only available for Dataflows Gen2 in Microsoft Fabric. This functionality is presented as a clickable link in the main canvas of the Power Query editor.
To learn more, read Power Query template (preview) – Power Query | Microsoft Learn.
Data pipelines
Activities
Fail Activity
We’re excited to announce that the Fail activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Fail activity to throw an error in your pipeline intentionally after specific activity outputs.
To learn more, read Execute a Fail activity | Microsoft Learn.
Web Activity
We’re excited to announce that the Web activity is now available to use in your Data Factory data pipelines! With the Web activity, you can call custom REST endpoints and pass information to be used in your pipeline.
To learn more, read Web Activity | Microsoft Learn.
Webhook Activity
We’re excited to announce that the Webhook activity is now available to use in your Data Factory data pipelines! Using the Webhook activity, you will be able to call a custom endpoint and pass additional information in your pipelines. Your data pipeline will wait until the activity completes before running downstream activities.
To learn more, read Webhook activity | Microsoft Learn.
Pipeline Management
Capacity integration
To help with you better understand and manage your Fabric data integration core consumption, you can now view your data pipelines usage. Capacity administrators can monitor the usage reported from the Admin Monitoring workspace under Feature Usage and Adoption.
Note: The Fabric Trial capacity is still free, only the consumption of pipeline (units) is being reported.
To learn more, read Feature usage and adoption report – Microsoft Fabric | Microsoft Learn.
Soft deletion now supported
We’ve added soft deletion support for data pipelines when deleting a workspace. This will allow you to recover a pipeline within a deleted workspace within 30 days. Administrators can restore data pipelines by restoring a workspace from the Admin portal.
To learn more, read Manage workspaces – Microsoft Fabric | Microsoft Learn.
Productivity
Design improvements – error messaging
We’ve heard your feedback and have made design improvements to better help you identify and troubleshoot errors, making it easier to take corrective actions.
You can find improvements in:
- Activity error messages
- Pipeline authoring failures
Detect file format in dataset
Now, you can auto-detect the file format in your datasets, making it quicker to set up your data connections.
Navigate to Data Warehouse directly from your pipeline
To make developing between workloads easier, you can now navigate to your Data Warehouse directly from your data pipeline.
Navigate to KQL Database directly from your pipeline
To make multi-tasking between artifacts easier, you can now navigate to your KQL Database directly from your data pipeline.
Preview destination data
When developing your data pipelines, you can now view preview the data in your selected destination. You will be able to see if your selected destination has any data and what that data schema looks like.
Support for type conversion
We’ve added support for type version for Data Warehouse and Snowflake when you enabled staging. This allows for better reliability when copying your data.
To enable staging, navigate to the Settings tab in your Copy activity.
Known issues
We are actively listening to the feedback that we receive from you, and we are taking action to improve our product.
You can head over to the new Known Issues page to learn more about the current issues in Data Factory, check their statuses and all issues that have been recently fixed.
We encourage you to keep engaging with us through our Community Forum and share your feedback with us about Data Factory.
Sign up for the free trial and get started with Data Factory today!
Other resources
- Join the Fabric community to post your questions, share your feedback, and learn from others.
- Visit Microsoft Fabric Ideas to submit feedback and suggestions for improvements and vote on your peers’ ideas!
- Check our Known Issues page for up to date on product fixes!
Have any questions or feedback? Leave a comment below!
