Set It and Forget It Target File Size Optimization What if you could enable a single setting and never worry about file size tuning again? Or if your tables automatically adjusted their optimal file sizes as they grew from megabytes to terabytes, without any manual intervention? Today’s data teams face a familiar challenge. Too small, … Continue reading “Adaptive Target File Size Management in Fabric Spark”
We’re introducing a new feature that gives you more granular control over your Spark compute resources in Microsoft Fabric: The Job-Level Bursting Switch. This highly anticipated addition empowers capacity administrators to fine-tune how Spark jobs utilize burst capacity, optimizing for either peak performance or higher concurrency based on your specific workload needs. Microsoft Fabric’s Compute … Continue reading “Introducing the Job-Level Bursting Switch in Microsoft Fabric”
The OPENROWSET function that can read JSONL format empowers you to easily read and ingest JSONL files – for example log files, social media streams, machine learning datasets, configuration files, and other semi-structured sources. With the versatile OPENROWSET T-SQL function, you can reference and query JSONL files as if they were tables, eliminating the need … Continue reading “Query and ingest JSONL files in Data Warehouse and SQL Analytics Endpoint for Lakehouse (Generally Available)”
The introduction of data sources and relative paths in the OPENROWSET function unlocks a new level of simplicity and flexibility for querying files in your Lakehouse and beyond. By leveraging external data sources, you can write cleaner, more maintainable SQL queries and easily access data wherever it resides — making data engineering and analytics more … Continue reading “Simplifying file access in OPENROWSET using data sources and relative paths (Preview)”
End Write Amplification and Automate Your Table Maintenance Compaction is one the most necessary but also challenging aspects of managing a Lakehouse architecture. Similar to file systems and even relational databases, unless closely managed, data will get fragmented over time, and can lead to excessive compute costs. The OPTIMIZE command exists to solve for this … Continue reading “Introducing Optimized Compaction in Fabric Spark”
