From Files to Delta Tables—Parquet & JSON data ingestion simplified with Shortcut Transformations
Picture this: A data engineer at a global enterprise starts the day with a familiar challenge – ‘Why does ingesting Parquet and JSON files always feel like a battle?’.
Data engineers are dealing with millions of records stored in compressed Parquet files and deeply nested JSON logs. Every schema change means hours of debugging. Every new compression format like Snappy, Gzip, Brotli adds another layer of complexity. And when upstream teams delete old files? Their ETL pipeline breaks. This is the reality for thousands of data teams.
Traditional approaches often require building and maintaining complex ETL pipelines, which can slow down development and increase operational overhead. Complex schemas, nested structures, and evolving data models often require custom ETL pipeline configurations, schema mapping logic, and constant monitoring and maintenance.
Shortcut Transformations – A Game-Changer for File Data Ingestion
With Shortcut Transformations, we’re changing the game. After introducing CSV to Delta transformations and AI Transformations for unstructured files, we’re excited to announce support for Parquet and JSON, bringing the same simplicity and efficiency to these widely used formats, making data ingestion from files seamless and reliable.
Instead of writing custom Spark jobs and maintaining fragile ETL pipelines, the data engineer opens Microsoft Fabric and clicks New Table Shortcut. In minutes, raw files are transformed into Delta tables without touching a single line of code.
How this happens
- Automatic Schema handling intelligently infers and aligns schemas from Parquet and JSON files to Delta tables, handling nested structures without manual flattening.
- Deep Flattening Capability decodes nested structures up to 5 levels deep, including:
- Structs (up to 5 levels).
- Arrays of structs (up to 5 levels).
- Arrays of arrays of structs (flattened one level deep).
- Broad Format support & Compression support means no more worrying about .json, .jsonl, and .ndjson—even combining all three in a single transformation with smart file handling in place. No more worrying about compressed parquet files. Whether Snappy, LZ4, Gzip, Brotli, or Zstandard, Shortcut transformation ingests them all seamlessly. Output tables are snappy compressed.
- Schema validation with flexibility ensures validation of flattened columns against the target schema; mismatches are safely ignored & logged as error which is visible to users for analysis.
- Case-Insensitive schema handling treats uppercase and lowercase column names equivalently for hassle-free mapping.
- Column Order Independence automatically handles column order differences, ensuring correct mapping and writing to Delta tables.
- Continuous Sync (managed change detection & file ingestion) keep Delta tables fresh. New files appear almost instantly, and deleted files are removed automatically—without custom change detection logic.
- Reliability Built-In delivers high concurrency, ACID transactions, time travel, and optimized queries thereby analytics stay consistent and fast.
- Managed Error handling captures success/failure/warning messages with error codes and explanatory error messages in a JSON file in monitoring view to troubleshoot and fix file data issues during ingestion.
- Effortless file source selection in OneLake provides a low-code experience to configure source folders across various cloud data sources.
Real-World Impact Across Personas
- Retail Data Analyst Persona: Global online retailers ingest daily product catalog updates and campaign performance data from multiple vendors. Shortcut Transformations handle compression quirks across Parquet and JSON effortlessly, ensuring dashboards remain fresh without manual intervention.
- Finance Compliance Persona: Banks flatten deeply nested JSON transaction logs from multiple payment gateways without writing custom Spark code. This guarantees ACID compliance and enables real-time fraud detection while meeting strict regulatory requirements.
- Healthcare Data Ops Persona: Hospitals ingest mixed JSON formats from patient monitoring devices and IoT sensors in one go. Shortcut Transformations provide clear error logs for troubleshooting and ensure schema consistency for downstream analytics and compliance audits.
- Manufacturing IoT Persona: Factories stream sensor data from thousands of machines in JSON and Parquet formats. Shortcut Transformations convert these into Delta tables for predictive maintenance models, reducing downtime and improving operational efficiency
- Energy & Utilities Persona: Power companies ingest weather-driven demand forecasts and smart meter readings in varying formats. Shortcut Transformations simplify ingestion and enable Materialized Lake Views to aggregate data for load balancing and outage prediction.
Getting Started is Simple
Step 1: Select or create a new Fabric Lakehouse.
Step 2: Select New Table Shortcut and choose your source (e.g., Azure Data Lake, Azure Blob Storage, Dataverse, Amazon S3, GCP, SharePoint, OneDrive etc.).
Step 3: Pick your file folder, verify transforms in the shortcut transformation wizard and save your Shortcut.
Step 4: Track refreshes in Manage Shortcuts hub.
Step 5: View logs in the monitoring view for complete transparency.
Coming soon
Stay tuned for updates and feature enhancements in Shortcut Transformations, including the following:
- Excel to Delta Transformations: Excel files as source format with multi-tab data processing & schema inference across tabs in an excel file.
- Dynamic Schema Evolution: When new columns appear or existing ones change or data type changes, Shortcut Transformations adapt automatically—no redeployment required.
- Schema Definition & Preview: User can define schema, filter columns, change data types and preview schema in advance.
- Customize ingestion & error handling behavior: Ability to customize handling duplicate records, define error behavior (drop rows or enforce value or NULL), data & timestamp format etc.
- Enhancement to monitoring & troubleshooting: Rows written, Rows read, # of files in sync, precision in refresh duration, Consolidated log files for troubleshooting planned to be available to users shortly.
Resources
To learn more, refer to the Shortcut transformations documentation.
Provide your feedback and suggest ideas via Fabric Ideas.
