Best-in-class connectivity and data movement with Data Factory in Fabric
In the fast-evolving data integration landscape, Data Factory continues to enhance the existing connectors to provide high-throughput data ingestion experience with no-code, low-code experience. With a focus on improving connector efficiency and expanding capabilities, recent updates bring significant advancements to a number of connectors.
These improvements focus on:
- Innovation: Continue to ship new innovations, features to Fabric connectors to ensure the data integration between source and desitination for a seamless data movement experience.
- Efficiency: Focus on the data movement efficiency on both read and write to ensure the enterprise readiness and cost optimization.
- Quality, security and reliability: We commit to ship the best-in-class connectivity which means we will not stop iterating on the connector quality, reliability and proactively ship the security enhancement and integration with the Fabric platform to help customers be focusing on the business and data insight.
Latest innovations
1. Lakehouse connector now supports deletion vector for delta tables in data pipelines
The Lakehouse connector in Data Factory has been upgraded to provide deeper integration with delta table. Two major new capabilities enhance data processing workflows:
1.1. Support for deletion vector
Delta table uses deletion vector to track deleted records efficiently without physically removing them from storage. With this new feature in the Lakehouse connector, users can:
- Read Delta tables while respecting deletion vector, ensuring that deleted records are automatically excluded from queries.
- Improve performance by leveraging soft deletions instead of physical file modifications, making data updates and maintenance more efficient.
- Enable compliance with data retention policies by retaining historical data for auditability while ensuring deleted records are filtered out from active queries.
These enhancements ensure that data engineers can work with delta tables more efficiently, improving data governance, performance, and maintainability.
- Learn more in the Configure Lakehouse in a copy activity documentation.
2. Performance improvement in Salesforce connector in data pipelines
Salesforce is a critical data source for many organizations, housing valuable customers and business data. To enhance data movement efficiency, Data Factory has introduced performance optimization in the Salesforce connector for data pipelines. The optimization now allows you to fetch the data concurrently from Salesforce by leveraging the parallelism read and processing capability, thus significantly reducing extraction times for large datasets.
- Learn more in the Salesforce objects connector overview documentation.
3. New Snowflake connector on ADBC (Preview)
We are thrilled to announce the new Snowflake connector in dataflow gen2 to enhance the native integration with Snowflake (available in Preview). It uses Arrow Database Connectivity (ADBC) to connect to and retrieve data from Snowflake which improves performance especially for large result sets. As we continue to enhance and add new capabilities to this connector, we encourage you to upgrade to the latest version to try it out and provide feedback.
- Learn more New Snowflake connector implementation (Preview) documentation.
4. New and updated Certified Connectors for Power BI and Dataflows
As a developer and data source owner, you are able to create connectors using the Power Query SDK and have them certify through the Data Factory Connector Certification Program. Certifying a Data Factory connector makes the connector available publicly, out-of-box, Microsoft Fabric Data Factory and Microsoft Power BI in the following experiences
This month we are happy to list the newly and updated certified connectors that are part of the Microsoft Data Factory Connector Certification Program. Be sure to check the documentation for each of these connectors so you can see what’s new with each of them.
New connectors
- ADP Analytics
- Dynatrace Grail DQL
Updated connectors
- Anaplan
- Asana
- BQE Core
- BuildingConnected
- Delta Sharing
- SolarWinds Service Desk
- Supermetrics
- Windsor
- Worksplace Analytics
- Zendesk Data
We’re committed to continuously offering and improving the connectivity in Data Factory to make data ingestion simpler, smarter, and faster. Stay tuned for learning more enhancements in the Connector overview documentation. Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community.
Fabric Data Factory team
