image generation

Databricks has enhanced its managed Postgres service using lakebase architecture, achieving write throughput improvements of up to five times. Traditional Postgres durability mechanisms, such as full page writes (FPW), impose overhead that can inflate Write-Ahead Log (WAL) volume by up to 15 times in write-heavy scenarios. The lakebase architecture decouples compute from storage, allowing compute nodes to stream WAL records to a distributed quorum of safekeepers, mitigating the risk of torn pages. Databricks has addressed read performance challenges by transferring image generation to the storage layer, which reconstructs data pages by identifying the latest materialized image and replaying corresponding WAL deltas. This results in a 94% reduction in WAL traffic and significant performance enhancements, with write throughput increasing by over 4.5 times on a 32-vCPU instance and WAL generation decreasing from 58KB per transaction to under 4KB. In production settings, steady-state WAL generation dropped from 30 MB/s to 1 MB/s, and read latencies improved by 30% to 50%. The optimization has been seamlessly integrated across Databricks' Serverless and Neon databases without requiring restarts or interruptions for customers.