AI-powered development with Data Factory Microsoft Fabric
In today’s data-driven landscape, organizations are constantly seeking ways to streamline their data integration processes, enhance productivity, and democratize access to powerful data engineering capabilities with Copilot for Data Factory. At Microsoft, we’re committed to empowering data engineers and analysts with intelligent tools that reduce complexity and accelerate development. We’re excited to share the latest advancements in AI-powered development for data integration in Microsoft Fabric.
Copilot in Dataflows Gen2
In September 2024, we made Copilot experiences in Dataflows Gen2 generally available.
The key Copilot capabilities in Dataflows Gen2 include:
- Generate new transformation steps: describe in natural language the transformation you need, and Copilot generates the appropriate steps.
- Generate new queries: create entirely new queries that may include sample data or references to existing queries using natural language.
- Query summarization: get comprehensive summaries of queries and their applied steps to better understand data transformations.
Copilot in Data pipelines
Copilot in Data Pipelines with three powerful capabilities (Generally Available):
- Pipeline generation: generate complete pipelines from natural language descriptions, ideal for rapid prototyping, creating standard ETL/ELT patterns, and developing complex orchestration flows.
- Error message assistant: troubleshoot Data pipeline issues with clear error explanation capability and actionable troubleshooting guidance.
- Pipeline explanation: explain your complex pipeline with the summary of content and relations of activities within the pipeline. You can use this to generate documentation for existing pipelines, receive plain-language explanations of pipeline logic and document best practices embedded within the pipeline.
Enhancements to Copilot in Data pipelines
Pipeline generation
- Iterative pipeline refinement: move beyond single-shot generation to incrementally update pipelines based on evolving business needs. Add, modify specific activities while preserving the overall pipeline structure.
- Branch logic enhancement: improve conditional flows and advanced dependency chains through natural language requests.
- Parameter and variable optimization: refine pipeline parameters and variables through conversation.
Error message assistant
- Activity-level error analysis: use Copilot chat to troubleshooting a specific failed activity run within complex pipeline.
- Root cause identification: get deeper insights into underlying issues beyond surface-level error messages.
- Resolution suggestions: receive actionable recommendations to fix identified problems.
Pipeline explanation
- Selective component explanation: request explanations for specific activities or sections rather than entire pipelines.
- Audience-tailored documentation: generate documentation suited for different stakeholders (technical teams, business users, etc.).
- Dependency visualization: better understand upstream and downstream impacts of pipeline components.
General usability improvements
- Enhanced natural language understanding: more flexible phrasing recognition when describing pipeline needs.
- Conversation history awareness: build on previous interactions for more coherent assistance.
- Faster response times: optimized performance for quicker feedback loops.
Democratizing AI for data integration
These Copilot for Data Factory advancements represent our commitment to democratizing AI and adding tangible value to data engineers’ daily work. It is a truly collaborative assistant that grows with your data integration needs. By embedding AI capabilities directly into the data integration experience, we’re:
- Lowering barriers to entry for data integration development.
- Accelerating time-to-value for data projects.
- Enhancing collaboration between technical and business teams.
- Freeing data engineers to focus on higher-value tasks.
As we continue to innovate, our focus remains on ensuring AI serves as a trusted assistant that amplifies human expertise rather than replacing it.
We invite you to explore the Copilot for Data Factory in Microsoft Fabric and share your experiences with us.
