Truncate Destination, Queries and Multiple Folders – Simplifying Data Ingestion with Copy Job
Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement, whether you’re moving data across clouds, from on-premises systems, or between services. With native support for multiple delivery styles, including bulk copy, incremental copy, and change data capture (CDC) replication, Copy job offers the flexibility to handle a wide range of data movement scenarios—all through an intuitive, easy-to-use experience.
This update introduces several enhancements related to truncate destination before full copy, full & incremental copy data with database queries and copy multiple folders in one Copy job.
Truncate Destination before Full Copy
You can now optionally truncate destination data before the full load, ensuring your source and destination are fully synchronized without duplicates.
By default, Copy job does not delete any data in your destination. When you enable this option:
- The first run of incremental copy will truncate all data in the destination before loading the full dataset.
- Subsequent incremental copies will continue to append or merge data without affecting existing records.
- If you later reset incremental copy to full copy, enabling this option will again clear the destination before loading.
This approach not only ensures that the your destination remains clean, fully synchronized, and free of duplicates, but also delivers significant performance improvements during full loads, providing a reliable foundation for your data ingestion solution.
Full & Incremental Copy Subsets of data with Database Queries
You can now copy subsets of data from your tables using database queries, unlocking a wide range of data ingestion scenarios.
- Copy only data for a specific region from a table with a region column to ensure compliance in data ingestion.
- Copy only the top N rows for testing or sampling.
More importantly, this feature supports both full and incremental copies on table subsets based on your custom queries, allowing flexible data selection and filtering before loading. This makes your data ingestion more efficient, precise, and tailored to your needs.
Start by trying it with Azure SQL DB — support for more connectors will be added soon.
Copy Multiple Folders in one Copy job
You can now copy multiple file objects, including multiple folders or a combination of folders and individual files, in a single Copy job. This makes your development work more efficient, eliminating the need to create multiple Copy jobs to achieve the same result.
Additional Resources
To learn more, explore Microsoft Fabric Copy Job documentation.
Submit your feedback on Fabric Ideas and join the conversation in the Fabric Community.
If you have a question or want to share your feedback, please leave us a comment.
