Announcing the preview of the REST API for Livy for Data Engineering.
The Fabric Livy endpoint lets users submit and execute their Spark code on the Spark compute within a designated Fabric workspace, eliminating the need to create a Notebook or Spark Job Definition artifacts. This integration with a specific Lakehouse artifact ensures straightforward access to data stored on OneLake. Additionally, Livy API offers the ability to customize the execution environment through its integration with the Environment artifact.
When a request is sent to the Fabric Livy endpoint, the user-submitted code can be executed in two different modes:
Session Job:
- A Livy session job entails establishing a Spark session that remains active throughout the interaction with the Livy API. This is particularly useful for interactive and iterative workloads.
- A Spark session starts when a job is submitted and lasts until the user ends it or the system terminates it after 20 minutes of inactivity. Throughout the session, multiple jobs can run, sharing state and cached data between runs.
Batch Job:
- A Livy batch job entails submitting a Spark application for a single execution. In contrast to a Livy session job, a batch job does not sustain an ongoing Spark session.
- With Livy batch jobs each job initiates a new Spark session, which ends when the job finishes. This approach works well for tasks that don’t rely on previous computations or require maintaining state between jobs.
The endpoint of the session job API would look like: https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/ batches
The endpoint of the session job API would look like: https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/sessions
ws_id: this is the workspace in which the hosting Lakehouse artifact belongs to. lakehouse_id: this is the artifact id of the hosting Lakehouse. All the capacity consumption history from this batch API call will be associated with this artifact.
To access the Livy API endpoint, you need to create a Lakehouse artifact. Once it’s set up, you’ll find the Livy API endpoint in the settings panel.
Our Livy API documentation provides more details and shows you how to create your first Livy batch or session job.
