We have made Azure Synapse Runtime for Apache Spark 3.5 for our Azure Synapse Spark customers generally available for customers to start using it for their production workloads, while they get ready and prepare for migrating to Microsoft Fabric Spark. What This Mean for You You can now create Azure Synapse Runtime for Apache Spark … Continue reading “Azure Synapse Runtime for Apache Spark 3.5 (Generally Available)”
Set It and Forget It Target File Size Optimization What if you could enable a single setting and never worry about file size tuning again? Or if your tables automatically adjusted their optimal file sizes as they grew from megabytes to terabytes, without any manual intervention? Today’s data teams face a familiar challenge. Too small, … Continue reading “Adaptive Target File Size Management in Fabric Spark”
We’re introducing a new feature that gives you more granular control over your Spark compute resources in Microsoft Fabric: The Job-Level Bursting Switch. This highly anticipated addition empowers capacity administrators to fine-tune how Spark jobs utilize burst capacity, optimizing for either peak performance or higher concurrency based on your specific workload needs. Microsoft Fabric’s Compute … Continue reading “Introducing the Job-Level Bursting Switch in Microsoft Fabric”
End Write Amplification and Automate Your Table Maintenance Compaction is one the most necessary but also challenging aspects of managing a Lakehouse architecture. Similar to file systems and even relational databases, unless closely managed, data will get fragmented over time, and can lead to excessive compute costs. The OPTIMIZE command exists to solve for this … Continue reading “Introducing Optimized Compaction in Fabric Spark”
Fabric Spark connector for SQL databases (Azure SQL databases, Azure SQL Managed Instances, Fabric SQL databases and SQL Server in Azure VM) in the Fabric Spark runtime is now available. This connector enables Spark developers and data scientists to access and work with data from SQL database engines using a simplified Spark API. The connector … Continue reading “Spark Connector for SQL databases (Preview)”
