Microsoft Fabric Updates Blog

Data Factory June 2023 Monthly Update

Welcome to the first edition of the Microsoft Fabric Data Factory monthly update!

In the first week of each month, visit our blog site to see what our team has been up to. This blog will cover the Data Factory experience in Fabric: Dataflows Gen2 and Data Pipelines.

Read on to find all the announcements we have for you for this month! 

Contents

Dataflows Gen2

Data pipelines

Dataflows Gen2

Power Query Editor

Manage Connections (Preview)

A highly requested feature that brings you a list of your connections used in your Dataflow and the ability to disconnect any of them from your Dataflow. You will find Manage Connections inside of the Home tab of the Power Query Editor.

This new feature currently only offers a way to disconnect any connection that you have in your dataflow.

Screenshot of the Manage connections dialog showcasing multiple connections on its page.

During the rest of the calendar year 2023, we’ll be releasing new capabilities to this experience, such as the ability to add and edit connections in your Dataflow.

To learn more, read Manage connections (Preview) – Power Query | Microsoft Learn.

Column pair suggestion for Merge operations (Preview)

Screenshot of Merge dialog showcasing the suggested pairing of columns for the Sales and Countries tables where they are joined by the CountryID columns in both tables.

One of the most common transformations done in Power Query is the Merge operation which provides you with a way to combine data from two tables using a logic that pairs column(s) from one table to the other.

In some scenarios, it might be a bit challenging with new projects that have tables with a high number of columns to understand how columns should be paired between tables. With the new suggestions inside of the Merge dialog, you can click the lightbulb on the top right corner of the dialog and get a list of possible ways to pair the columns between both tables.

To learn more, read Merge queries overview – Power Query | Microsoft Learn.

Power Query templates (Preview)

Power Query is a data preparation tool that is available in multiple products such as Power BI Dataflows, Power Platform Dataflows, Excel for Windows, Excel for Mac and many others. Traditionally, to share your queries you could copy and paste your queries as shown in the article on share queries. Today we are releasing into public preview the ability to export and import templates inside of Data Factory Dataflows Gen2.

This consists of the export template functionality available across all Power Query Online hosts with a button in the Home tab with the name Export template that effectively exports your Power Query script and its associated metadata.

Screenshot that shows the Export Template button located inside the Share group of the Home tab in the ribbon.

Note: Excel for Windows also has an export template functionality that can be leveraged to create a Power Query template file. Read more on Create Power Platform dataflows from queries in Microsoft Excel (Preview) – Power Query | Microsoft Learn.

The ability to import a template is currently only available for Dataflows Gen2 in Microsoft Fabric. This functionality is presented as a clickable link in the main canvas of the Power Query editor.

To learn more, read Power Query template (preview) – Power Query | Microsoft Learn.

Data pipelines

Activities

Fail Activity

We’re excited to announce that the Fail activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Fail activity to throw an error in your pipeline intentionally after specific activity outputs.

To learn more, read Execute a Fail activity | Microsoft Learn.

Web Activity

We’re excited to announce that the Web activity is now available to use in your Data Factory data pipelines! With the Web activity, you can call custom REST endpoints and pass information to be used in your pipeline.

To learn more, read Web Activity | Microsoft Learn.

Webhook Activity

We’re excited to announce that the Webhook activity is now available to use in your Data Factory data pipelines! Using the Webhook activity, you will be able to call a custom endpoint and pass additional information in your pipelines. Your data pipeline will wait until the activity completes before running downstream activities.

To learn more, read Webhook activity | Microsoft Learn.

Pipeline Management

Capacity integration

To help with you better understand and manage your Fabric data integration core consumption, you can now view your data pipelines usage. Capacity administrators can monitor the usage reported from the Admin Monitoring workspace under Feature Usage and Adoption.  

Note: The Fabric Trial capacity is still free, only the consumption of pipeline (units) is being reported.

To learn more, read Feature usage and adoption report – Microsoft Fabric | Microsoft Learn.

Soft deletion now supported

We’ve added soft deletion support for data pipelines when deleting a workspace. This will allow you to recover a pipeline within a deleted workspace within 30 days. Administrators can restore data pipelines by restoring a workspace from the Admin portal

To learn more, read Manage workspaces – Microsoft Fabric | Microsoft Learn.

Productivity

Design improvements – error messaging

We’ve heard your feedback and have made design improvements to better help you identify and troubleshoot errors, making it easier to take corrective actions.

You can find improvements in:

  • Activity error messages
  • Pipeline authoring failures

Detect file format in dataset

Now, you can auto-detect the file format in your datasets, making it quicker to set up your data connections.

Navigate to Data Warehouse directly from your pipeline

To make developing between workloads easier, you can now navigate to your Data Warehouse directly from your data pipeline.

Navigate to KQL Database directly from your pipeline

To make multi-tasking between artifacts easier, you can now navigate to your KQL Database directly from your data pipeline.

Preview destination data

When developing your data pipelines, you can now view preview the data in your selected destination. You will be able to see if your selected destination has any data and what that data schema looks like.

Support for type conversion

We’ve added support for type version for Data Warehouse and Snowflake when you enabled staging. This allows for better reliability when copying your data.

To enable staging, navigate to the Settings tab in your Copy activity.

Known issues

We are actively listening to the feedback that we receive from you, and we are taking action to improve our product.

You can head over to the new Known Issues page to learn more about the current issues in Data Factory, check their statuses and all issues that have been recently fixed.

We encourage you to keep engaging with us through our Community Forum and share your feedback with us about Data Factory.

Sign up for the free trial and get started with Data Factory today!

Other resources

  • Join the Fabric community to post your questions, share your feedback, and learn from others.
  • Visit Microsoft Fabric Ideas to submit feedback and suggestions for improvements and vote on your peers’ ideas!
  • Check our Known Issues page for up to date on product fixes!

Have any questions or feedback? Leave a comment below!

Related blog posts

Data Factory June 2023 Monthly Update

April 16, 2024 by Ruixin Xu

We are pleased to share a set of key updates regarding the Copilot in Microsoft Fabric experiences. The information in this blog post has also been shared with Fabric tenant administrators. Below are the highlights of the changes. This change is an important milestone to eventually allow Copilot to reach GA within this year. See … Continue reading “Copilot in MS Fabric: Soon available to more users in your organization“

April 15, 2024 by Santhosh Kumar Ravindran

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to … Continue reading “Introducing Job Queueing for Notebook in Microsoft Fabric”