Unify your data estate for the era of AI with Fabric Data Factory
Microsoft Fabric provides industry leading data integration capabilities that are unique in the market. Fabric is built on OneLake, which provides data integration at the root, with shortcuts and mirroring that enable Zero-ETL approaches to data unification. Fabric Data Factory provides the industry’s largest, most widely adopted data integration capability offered as a single, cohesive SaaS led, multi-cloud ready experience. Together, these capabilities enable Fabric customers to break down the silos within data estates to unlock the true value of data no matter where it resides (within Azure and Fabric, across clouds, behind firewalls).
Fabric Data Factory is built on tried and tested capability that is unique in the industry and the most widely adopted data integration stack in the industry. Fabric Data Factory is a fusion of standalone experiences that are now brought into a cohesive offering: Azure Data Factory (Pro-grade data integration in Azure) and Power Query (Citizen Data Integration in Power BI, Excel, Dynamics). The metrics below provide some insight into the momentum we are seeing with our data integration capability.
We are always humbled by the scale of Data Factory – with 22 billion orchestration runs per month and 500+ Petabytes of data moved every month. We are also grateful for our large community of Power Query users that have stayed with us and grown to what is the largest self-serve data preparation user community in the world. Data Factory’s unique hybrid architecture enables line of sight to on-prem data sources through our cloud-native experiences via an extremely large footprint of 790,000+ on-premises gateways deployed across our customer base.
As part of the Microsoft Fabric Community Conference in Vienna, we are eager to bring you the next wave of product innovation that will further customer adoption through new features, substantial updates to pricing and better enterprise-readiness to meet the needs of our largest and most complex deployments.
Pricing & Performance Updates to Dataflow Gen2
Dataflow Gen2 is the most modern, most scalable implementation of Power Query based self-service data preparation. Dataflow Gen2 is built on Fabric OneLake and scaled via Fabric’s compute engines and is Copilot enabled. Together, these capabilities provide for the most broadly approachable high scale self-serve data preparation in the market. However, we also hear you loud and clear in that Dataflow Gen2 is too expensive. We are introducing some meaningful developments with fresh changes on this front.
First, we are making a change to the overall pricing model of Dataflow Gen2:
- We are lowering the base rate of Dataflow Gen2 overall. The base rate of Dataflow Gen2 will drop down from 16CU to 12 CU.
- We are introducing a tiered approach that will make Dataflow Gen2 much more palatable to long running jobs. Jobs that take longer than 10 minutes to complete will see a base rate of 1.5CU for any part of the job that exceeds 10 minutes.
Second, we are introducing dramatic performance improvements to Dataflow Gen2 through a Modern PQ evaluator. And adding the ability to parallelize query runs through partitioning. Together, both these improvements yield dramatic runtime performance that will have an overall positive impact not only on the user experience but will yield further reduction in cost.
Third, we are further improving the user experience within the design-time experience by adding a ‘Preview only Steps’ feature that will improve feature your design time experience by delivering faster iterations with query editing.
These pricing and performance enhancements are effective immediately for Dataflow Gen2 (CI/CD) operations. To benefit, upgrade non-CI/CD items using ‘Save as Dataflow Gen2 (CI/CD)’.
For more information on Dataflow Gen2 pricing for Data Factory in Microsoft Fabric and What is Dataflow Gen2? refer to the documentation.
Petabyte scale, cross-cloud data movement with Copy job
Data Factory provides connectors to over 170+ data sources and destinations. Pipelines, Copy job and Dataflow Gen2 include data sources and data destinations that are able to move data efficiently and reliably across a multitude of data sources, destinations and clouds. On-Premises Gateway and VNet Gateway enable data access across firewalls and network boundaries.
Copy job in Data Factory enables simple yet powerful data movement at petabyte scale. Copy job is intended to include every business-critical data source and destination across Microsoft and non-Microsoft sources/targets.
The full set of connectors supported today is illustrated in the following visual example; we’re expanding this area on a weekly basis.
Additionally, we are announcing the following new capabilities in Copy job:
Preview
- Copy job activity – Enables orchestration of Copy job via pipelines.
- Connection Parameterization via variable library for Copy job – Streamlines CI/CD process for multiple environments without modifying Copy job.
- Fabric Lakehouse Change Data Feeds (CDF) – Enables Fabric Lakehouse change data to be read by Copy job for incremental copy.
- Change Data Capture (CDC) can be merged into SQL Database and Snowflake destinations – Easily merge your change data into SQL Database in Fabric and Snowflake with Copy job.
- Copy Assistant in pipeline is now powered by Copy job – Setting up Copy in pipeline is easier than ever with the updated Copy Assistant powered by Copy job.
Generally Available
- VNet Gateway support in Copy job and Copy Activity – Enable data access across firewalls and network boundaries with Copy job and Copy Activity.
- Incremental Copy Reset – Enables re-set/re-seed of an incremental copy.
- Iceberg and JSON file formats support with Copy job – Copying Iceberg and JSON formats are now supported with Copy job.
- Multiple-schedules support with Copy job – Set multiple-schedules on a single Copy job to optimize your data movement needs.
- Database views support in Copy job – Database views can now be used as the basis for both full and incremental copy.
- New enterprise connectors for Copy job and Copy Activity – Enterprise data sources such as AWS RDS for Oracle, PostgreSQL, Cassendra, Greenplum, HDFS, Informix, Microsoft Access, Presto, Teradata are available for Copy job and pipeline.
To learn more, refer to the What is Copy job in Data Factory for Microsoft Fabric? documentation.
In addition to Copy job, Dataflow Gen2 enables new connectors for seamless data movement as part of self-serve data transformation. We are announcing the following new data destinations in Dataflow Gen2.
Preview
- Azure Data Lake Storage Gen2: Write the results of Dataflow into CSV files in ADLS Gen2.
- Lakehouse Files: Write the results of Dataflow into CSV files in Fabric Lakehouse.
Generally Available
- SharePoint Files CSV: Write the results of a Dataflow as a CSV file in a SharePoint Library.
Coming Soon
- Snowflake Database: Write the results of a Dataflow into Snowflake database
- SharePoint Files Excel: Write the results of Dataflow as an Excel file in SharePoint Library.
Pro-Code & Low-Code Data Orchestration
Data Factory provides pipelines for Low-code data orchestration. With its rich library of Activities as well as options for scheduled and triggered execution, pipelines offer highly reliable orchestration to enable a broad range of data engineering scenarios.
Preview
- Copy job activity – Orchestrate Copy jobs via pipeline.
- Expression Evaluator – When writing pipeline expressions, inspect your results in-line during design time.
- ADF pipeline Upgrade utility – Open source Powershell utility to migrate pipelines from ADF to Fabric Data Factory.
- Workspace Monitoring for pipelines – Pipeline execution logs are now available in workspace monitoring providing a real-time observability data store for you to create queries and reports.
- Multiple schedules per pipeline – Automate your pipeline workflows on differing cadences on a given pipeline.
Generally Available
- Invoke pipeline activity – Invoke pipelines across Fabric, Azure Data Factory, Synapse.
- Email & Teams Activities – Send notifications using email and Teams.
- Functions Activity – Execute Fabric functions.
- Dataflow activity- Orchestrate your Dataflows, includes built-in support for Dataflow Gen2 parameters.
- Pipeline Variable libraries – Variable libraries provide global pipeline support for metadata drive pipelines and makes it easy to change values across environments to support your CICD processes.
- SPN/Workspace Identity support in activities – Automate pipelines using SPN or Workspace identity.
Data Factory also provides Pro-Code data integration via Managed Airflow. By providing a serverless model for airflow runtimes, Data Factory removes the burden of having to manage and oversee airflow clusters – enabling customers to focus on the core orchestration task at hand.
Generally Available
- Airflow SPN UI – Effortlessly add Airflow SPN connections to your Apache Airflow projects without writing code.
- Add Notebook and pipeline execution to your DAG – Simple, one-click code template that will automatically add the code to your DAGs to call native Fabric components like notebooks and pipelines.
Learn more about all capabilities we are bringing to Orchestration.
Database Mirroring
Mirroring enables simple, zero-copy, zero-ETL based replication of any data into OneLake – no matter the cloud, database, vendor, or the engine serving the data. We are thrilled to announce several new data sources and improvements to existing data sources in our mirroring lineup.
Preview
- Google BigQuery – Mirror your Google BigQuery data alongside other cloud data sources, enabling cross-cloud querying, unified semantic models, and integrated analytics and AI workloads within the Fabric ecosystem.
- Oracle (including Exadata) – Mirror your Oracle data alongside other cloud data sources, enabling cross-cloud querying, unified semantic models, and integrated analytics and AI workloads within the Fabric ecosystem. Oracle Mirroring supports on-prem data as well as data from Oracle Cloud Infrastructure and Oracle Exadata.
- Workspace level Private Link – Use Workspace Private Link to securely access Fabric workspaces (including mirrored database items) with granular network security control
- Mirrored Databases available for Data Agent – Use Data Agent to chat with data in any Mirrored Database for quick insights
Generally Available
- Azure SQL Managed Instance – Mirror your Azure SQL Managed Instance data alongside other cloud data sources, enabling cross-cloud querying, unified semantic models, and integrated analytics and AI workloads within the Fabric ecosystem.
- VNet Gateway & On-Prem Gateway for accessing databases behind firewalls – Mirror data behind a firewall in Snowflake, Azure SQL Database, Azure SQL Managed Instance.
- Workspace Identity Authentication for Azure SQL Database – Use credential-free Workspace Identity authentication to connect to and mirror your Azure SQL Database.
- Create Semantic Models in Power BI from Mirrored Database – Create a semantic model directly from a mirrored database
Learn more about the announcements for Mirroring
AI-powered Data Integration
Data Factory is AI-powered via Copilots. Copilot enables you to author, debug and monitor pipelines and dataflows with minimal effort. Additionally, copilots are available in-line within the experiences to enable targeted scenarios – such as custom columns via NL prompts.
Today, we are announcing some amazing new capabilities that take Data Factory further with AI-led data integration:
Preview
- AI-enhanced Modern Get Data experience – Easily ingest and transform data with natural language as part of the data discovery and connection experience.
Generally Available
- Support Natural language to generate custom columns in Dataflow Gen2 -Express custom column definitions using natural language.
- Explain Dataflow Gen2 queries and steps: Derive natural language explanation of any Dataflow query to better understand what a query / step is doing.
- Chat with Copilot to create new connections, and connect to existing connection resources.
- Use Copilot-generated summaries to document data pipelines, activities, and Dataflow queries and steps.
Coming Soon
- AI-powered Data Transformation prompts in Dataflow Gen2 – Soon, you will be able to use natural language prompts to apply transformations like Sentiment Analysis, Summarization, Categorization, and more. It has never been easier to transform & enrich your data with intelligence.
To learn more about AI-powered Data Integration, refer to Get started with Copilot in Fabric in the Data Factory.
Mission-Critical Data Integration
Data Factory is mission critical by design and includes key capability to support the needs of the most demanding enterprise environments. We are thrilled to announce the addition of several new capabilities to provide security, isolation and outbound access protection. The following features are now available to support mission-critical enterprise needs:
Preview
- Workspace Identity authentication support for connectors – Enables secure, seamless connector access using workspace identity, reducing credential sprawl.
- Workspace Private Link support for Fabric Data Factory – Allows using Dataflows Gen2, Pipelines, Copy Job in workspace that are protected by Private Link for inbound access, making data integration more secure.
- PowerShell for On-Premises Gateway takeover – Automates gateway ownership transfer for streamlined administration and disaster recovery.
- PowerShell for configuring installation path of On-Premises Data Gateway – Offers flexibility in gateway deployment by allowing custom installation paths.
Generally Available
- Azure Key Vault integration in Connections – Centralizes secret management for connections by allowing secure credential storage and retrieval via Azure Key Vault.
- VNet Gateway support for Data pipeline, Copy job – Allows secure data movement through virtual networks, ensuring compliance and network isolation.
- Connections and Gateways API – Enables programmatic management of connections and gateways for scalable automation and integration with SPN support.
- VNet Gateway shut down and restart controls – Provides operational control over gateway lifecycle to optimize resource usage and security.
Coming Soon
- Snowflake Key-Pair authentication – Strengthens security and automation for Snowflake connections with key-pair-based authentication.
To learn more, refer to Security in Data Factory
Summary
This release represents a substantial step for Fabric Data Factory, with a breadth of product capability as well as pricing adjustments that are all based on the feedback from you and our customer community at large. We can’t wait to see what you build on top of these additions.
Stay tuned for a series of in-depth blog posts over the next month, getting into the details of many of these features announced here. We hope to see you on our blogs, forums and Ideas channels as you work with Fabric Data Factory!
Resources
Get started with Fabric Data Factory today!
To learn more, refer to Data Factory documentation.
