Microsoft Fabric Copilot for Data Science and Data Engineering
Microsoft Fabric Copilot for Data Science and Data Engineering began its public preview journey in November 2023, during the Microsoft Ignite Conference. This tool, built to act as a collaborative assistant for data professionals, has already started to transform data exploration, preparation, and machine learning tasks, boosting productivity of data scientists and data engineers in Fabric. This week we are also happy to announce the worldwide availability of Copilot for Data Science and Data Engineering.
What can Copilot for Data Science and Data Engineering do for you?
The value of Microsoft Fabric Copilot lies in its ability to adapt to the user’s workflow, providing assistance tailored to the context of your data and project. Here are examples of the value it provides:
- For data exploration: Copilot can swiftly generate code for plots and statistics using common python libraries such as plotly and seaborn, giving immediate insight into the underlying data patterns.
- For predictive analytics: It can suggest the most appropriate models and also tailor them to your dataset in the Lakehouse, leading to quicker iterations and results.
- As a learning tool: Copilot’s generated code can serve as a reference for code implementation and advanced techniques, aiding in the continuous professional development of the user.
- Enhancing collaboration: Saved interactions with Copilot can be shared among team members, promoting a collaborative and productive work environment.
- Seamless assistance: No matter where you are, whether inside a notebook cell or chatting with Copilot in the Chat panel, Copilot understands and shares context for input, output, and data awareness. This way, you can stay focused and ask for help without context within your workflow.
If you are a data professional who uses Fabric notebooks, you can now leverage the power of Microsoft Fabric Copilot. Through natural language processing, Copilot allows you to generate code, receive guidance, and gain insights from data more quickly than ever before! This is achieved via two main features: the Copilot Chat panel and Copilot inside notebook cells (Chat Magics).
Chat with Copilot in the chat panel
The Copilot Chat Panel serves as an interactive AI assistant within Microsoft Fabric’s notebook environment. Here’s how it adds detailed value to various stages of a data professional’s workflow:
Interactive Q&A for Instant Insights: You can query your datasets interactively. For example, by asking, “What is the distribution of sales across different regions?” Copilot can generate the code for you to complete the relevant analysis, providing a visual distribution or summary statistics. This natural language interface simplifies the data exploration process and enables you to have a conversational experience with your data.
Context-Aware Code Suggestions: The chat panel understands the state of the user’s notebook. If a data frame is loaded with sales data, you might ask, “Can you clean this data for analysis?” Copilot would generate a code snippet that applies filters, handle missing values, or remove duplicates based on the specific characteristics of the data in context.
Seamless Progression of Analysis: While working through a notebook, you might reach a point where the next steps are unclear. The user can leverage the chat panel to ask, “What should I do next after visualizing this data?” Copilot could suggest various statistical tests to validate hypotheses or recommend machine learning models that suit the data’s features.
Use Chat Magics in your notebook
We have introduced a developer friendly and non-intrusive way of interacting with Copilot directly in code-cells. Here’s a closer look at how it works:
Natural Language answers as output: With magics, you can embed Copilot’s AI capabilities directly into your code cells. For instance, %%chat magic could be used to initiate a conversation about the user’s data within a cell. You may write, %%chat Analyze the pandas dataframe and recommend a few machine learning models, and Copilot would provide a natural language explanation and set of insights. Copilot may also provide sample code snippets that you may be able build upon in a new cell.
Code Generation in-cell: The %%code magic command allows you to convert a natural language instruction into executable code right inside a cell. For instance, if you type %%code Can you provide me with linear regression model code on my dataframe , Copilot would understand the context, generate the code to create a linear regression model, and present the code snippet ready for execution.
Learn more and Get started
Microsoft Fabric Copilot for Data Science and Data Engineering has the potential to revolutionize the way data professionals complete their tasks, providing a layer of intelligence that augments skills and streamlines workflows. Its integration into Fabric notebooks is a testament to the future of collaborative, AI-enhanced data analysis and model building. This tool does not just automate tasks; it acts as a partner that brings data professionals closer to goals of efficient data management, insightful analysis, and innovative machine learning solutions. To learn more, please visit our documentation here.
