Set It and Forget It Target File Size Optimization What if you could enable a single setting and never worry about file size tuning again? Or if your tables automatically adjusted their optimal file sizes as they grew from megabytes to terabytes, without any manual intervention? Today’s data teams face a familiar challenge. Too small, … Continue reading “Adaptive Target File Size Management in Fabric Spark”
We’re introducing a new feature that gives you more granular control over your Spark compute resources in Microsoft Fabric: The Job-Level Bursting Switch. This highly anticipated addition empowers capacity administrators to fine-tune how Spark jobs utilize burst capacity, optimizing for either peak performance or higher concurrency based on your specific workload needs. Microsoft Fabric’s Compute … Continue reading “Introducing the Job-Level Bursting Switch in Microsoft Fabric”
Microsoft OneLake is the unified data lake for your entire organization, built into Microsoft Fabric. It provides a single, open, and secure foundation for all your analytics workloads – eliminating data silos and simplifying data management across domains. The preview of Microsoft OneLake Table APIs, a new way to programmatically manage and interact with your … Continue reading “OneLake Table APIs (Preview)”
End Write Amplification and Automate Your Table Maintenance Compaction is one the most necessary but also challenging aspects of managing a Lakehouse architecture. Similar to file systems and even relational databases, unless closely managed, data will get fragmented over time, and can lead to excessive compute costs. The OPTIMIZE command exists to solve for this … Continue reading “Introducing Optimized Compaction in Fabric Spark”
Fabric Spark connector for SQL databases (Azure SQL databases, Azure SQL Managed Instances, Fabric SQL databases and SQL Server in Azure VM) in the Fabric Spark runtime is now available. This connector enables Spark developers and data scientists to access and work with data from SQL database engines using a simplified Spark API. The connector … Continue reading “Spark Connector for SQL databases (Preview)”
