Simplifying data movement across multiple clouds with Copy job – Enhancements on incremental copy and change data capture
Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement, whether you’re moving data across clouds, from on-premises systems, or between services. With native support for multiple delivery styles, including bulk copy, incremental copy, and change data capture (CDC) replication, Copy job offers the flexibility to handle a wide range of data movement scenarios—all through an intuitive, easy-to-use experience.
This update introduces several enhancements to native incremental copy and change data capture (CDC) capabilities in Copy job, including support for additional data source stores for incremental copy, expanded CDC support for SAP Datasphere Outbound for Amazon S3 and Google Cloud Storage (in addition to Microsoft Cloud Storage), and Column Mapping support for CDC in Copy job.
Expanded source store support for Incremental Copy
We are expanding Copy job with broader multi‑cloud connectivity and stronger incremental copy support. Incremental copy in Copy job now supports additional connectors, including:
- Google Big Query
- Google Cloud Storage
- DB2
- ODBC
- Fabric Lakehouse table
- Folder
- Azure files
- SharePoint List
- Amazon RDS for SQL Server
- Amazon RDS for Oracle
- Azure Data Explorer
Incremental copy from Fabric Lakehouse now supports both CDF and watermark-based methods
When performing incremental copy from a Fabric Lakehouse table, we strongly recommend using CDF (Delta Change Data Feed) to capture row inserts, updates, and deletions, and replicate them to supported destinations.
However, you can now also optionally use watermark-based incremental copy without enabling CDF. In this mode, you can select a watermark column for each table to identify changes.
To enable this, go to the Advanced Settings button after creating the Copy job, where you will have the option to switch from CDF to using a watermark column.
SAP Datasphere Outbound for Amazon S3 and Google Cloud Storage
Previously, you could use SAP Datasphere Outbound for ADLS Gen2 in Copy job to perform CDC replication from SAP to any supported destination. For more details, see Tutorial: Copy job with SAP Datasphere Outbound (Preview) to get more details.
You can now also use SAP Datasphere Outbound for Amazon S3 and SAP Datasphere Outbound for Google Cloud Storage, expanding staging storage support across multiple clouds and allowing you to choose the option that best fits your scenario.
Column Mapping in CDC in Copy job
You can now map columns from source to destination when performing CDC replication in Copy job. This is useful when you want to rename columns, change data types, or otherwise customize the schema in the destination store.
Column mapping now is supported across all data movement patterns in Copy job, including full copy, watermark-based incremental copy and CDC replication.
Additional Resources
To learn more, explore Microsoft Fabric Copy Job documentation.
Submit your feedback on Fabric Ideas and join the conversation in the Fabric Community.
If you have a question or want to share your feedback, please leave us a comment below!
