Run Spark Job Definitions in Pipelines with Service Principal or Workspace Identity
The Spark job definition activity in Microsoft Fabric Data Factory pipelines now supports connection property, unlocking a more secure and production-ready way to run your SJDs.
What’s New?
With this update, you can configure Notebook activities to run as Service Principal (SPN) or Workspace Identity (WI). These authentication methods are our recommended approach for production environments, ensuring:
- Operational reliability: Avoid issues tied to user credentials, such as password changes or account deactivation.
- Enterprise-grade security: Service-based authentication reduces risk and simplifies compliance.
- Consistent automation: Keep your pipelines running smoothly without manual intervention.
Why it Matters
Previously, many customers relied on user authentication for pipeline runs, which can lead to broken workflows when users leave the organization or tokens expire. By enabling SPN and WI through the connection property, you gain:
- Scalable orchestration for notebooks in complex workflows.
- Better governance with centralized identity management.
- Future-proof automation for production workloads.
How to Get Started
- In your pipeline, add a Spark job definition activity.
- Under Connection, you can configure a new connection or use an existing one.
- Provide the required credentials or identity configuration (you can use SPN and WI now).
- Run your pipeline and enjoy a secure, automated experience.
To learn more, check out our documentation on the Spark Job Definition activity.
