Autoscaling Lakebase Postgres

Choosing a database instance size without prior knowledge of the workload has long been a conventional approach, often leading to inefficiencies and excessive compute usage. This issue is particularly pressing in today’s landscape, where compute resources are increasingly viewed as a luxury. Lakebase Postgres addresses this challenge by eliminating the need for manual sizing through its innovative autoscaling feature. This functionality is powered by in-place VM resizing and a sophisticated algorithm that continuously monitors CPU, memory, and the database’s working set.

The architectural requirement

In traditional Postgres setups, the database operates as a stateful process bound to a specific machine and its disks. Consequently, any resizing or replacement of that machine constitutes a database operation, as it directly affects both execution and durable state. In contrast, the Lakebase Postgres architecture distinctly separates these responsibilities:

• The compute layer is responsible for running Postgres and executing queries, utilizing RAM and local NVMe for low-latency access, while not owning any durable state.
• The storage layer manages durability and historical data. WAL is replicated by safekeepers on SSDs, pageservers (also SSDs) reconstruct page versions, and object storage maintains the long-term immutable record. (For a deeper dive into the storage aspect, refer to our detailed blog post.)

This architectural separation allows a compute node to start, stop, move, or resize independently of the underlying database, establishing a crucial foundation for autoscaling implementation. The process involves two key components: determining when to adjust capacity and how to execute these changes without interrupting Postgres operations.

Part I: The algorithm

The three autoscaling signals

The Lakebase Postgres autoscaling algorithm relies on three primary signals, each contributing to the target compute size:

1. CPU load: cpuGoalCU
2. Memory use: memGoalCU
3. Compute-cache working set size: lfcGoalCU

The final scaling target is determined by the largest of these three signals, constrained by the minimum and maximum compute sizes configured by the user for that database.

CPU (cpuGoalCU)

Monitoring CPU load is the most straightforward aspect of the autoscaling algorithm. The system observes the processor’s workload closely:

• Every five seconds, the autoscaler-agent reads the one-minute CPU load average from the VM.
• The CPU goal aims to maintain this load at or below 90% of available capacity.
• When the load exceeds this threshold, cpuGoalCU increases; conversely, it decreases when sustained load falls.

This method of using a one-minute average helps filter out transient fluctuations while still responding to significant demand changes. However, relying solely on CPU metrics is insufficient for effective autoscaling, as low CPU usage may not accurately reflect performance issues stemming from memory or cache constraints.

Memory (memGoalCU)

Memory management presents a different challenge. While brief CPU demand spikes can slow queries, excessive memory allocation can lead to process termination by the kernel. Thus, the autoscaler monitors memory at two distinct frequencies:

• Every five seconds, the autoscaler-agent reviews overall memory metrics from the VM.
• Every 100 milliseconds, the vm-monitor checks memory usage specifically by Postgres.

The memory goal is to keep usage below 75% of allocated RAM, ensuring sufficient headroom for new allocations and other processes. The vm-monitor also assesses every proposed downscale to confirm that sufficient memory remains for active processes.

The compute cache (lfcGoalCU)

The third signal evaluates whether the workload’s active data can be efficiently accessed by Postgres. Given the separation of storage and compute in Lakebase Postgres, when a page is unavailable locally, the compute requests it from the pageserver, caching the returned page for future access. This disk-backed cache, originally termed the Local File Cache (LFC), acts as a resizable extension of Postgres shared buffers. When compute resources expand, the vm-monitor adjusts the cache size accordingly.

Performance can significantly improve once the working set fits within local memory, highlighting a limitation of CPU-only autoscaling: cache misses can lead to network delays, reducing CPU utilization. Thus, Lakebase Postgres incorporates a third autoscaling signal that directly estimates the working set size.

Zooming in: how we estimate the Postgres working set

The working set comprises the database and index pages accessed repeatedly over a specified period. To accurately count these pages without excessive memory use, the algorithm employs HyperLogLog, a probabilistic cardinality estimator that efficiently estimates the number of distinct items in a set.

For each page access, the standard HyperLogLog implementation:

1. Hashes the page identifier.
2. Uses the initial bits of the hash to select a register.
3. Counts the leading zeroes in the remaining bits.
4. Updates the selected register if this observation exceeds its previous value.

However, a conventional HyperLogLog only grows, making it unsuitable for autoscaling, which requires a time-sensitive estimate of the current workload’s distinct pages. To address this, Lakebase Postgres modifies the HyperLogLog registers to store timestamps instead of simple bits.

Adding time to HyperLogLog

In this modified approach, the estimator records the current timestamp at each position. To estimate cardinality since a specific time T, it considers positions updated after T as set and older positions as unset. This allows for estimates of distinct pages accessed within various time frames, such as:

• Distinct pages accessed in the last minute
• Distinct pages accessed in the last five minutes
• Distinct pages accessed in the last hour

Every 20 seconds, the autoscaler-agent gathers working-set estimates across these time windows, but determining the optimal window presents its own challenges.

Choosing the working set time window

There is no one-size-fits-all window for describing a database’s current working set. Short windows may respond quickly to workload changes but risk excessive cache eviction, while longer windows may retain memory for outdated workloads. The algorithm addresses this by analyzing how the working set evolves over time. For steady workloads, the estimated number of pages initially rises and then stabilizes. Conversely, following a heavy workload, a longer window will capture the spike in estimates as it includes data from the previous workload.

The implementation begins its analysis after five minutes, preventing immediate downsizing during brief pauses while still allowing for adjustments based on workload shifts.

Projecting cache growth

To further enhance performance, the algorithm also projects future working-set growth. It examines the increase in estimates across time intervals and allocates sufficient cache to accommodate expected growth. This projection is limited to a short time frame to avoid amplifying brief spikes that could lead to oscillation in compute resources.

The projected size ultimately becomes lfcGoalCU, with the algorithm aiming to fit the working set within the available memory allocated for the compute cache, capped at 75% of the compute’s RAM.

Part II: Resizing the running compute

With the scaling target established, the next step involves adjusting the CPU and memory of a running VM without disrupting Postgres operations. Each Postgres instance within Lakebase Postgres operates in its own virtual machine within a Kubernetes cluster, allowing for strong isolation and dynamic resource allocation.

Four key components collaborate to facilitate compute resizing:

1. The autoscaler-agent operates on every Kubernetes node, collecting metrics, calculating target sizes, and initiating scaling actions.
2. The vm-monitor runs within each VM, closely monitoring Postgres memory, validating downscale requests, and resizing the compute cache.
3. A modified Kubernetes scheduler maintains a comprehensive view of available resources, ensuring that upscale requests do not lead to memory overcommitment.
4. NeonVM implements the changes, serving as a custom Kubernetes resource and controller capable of dynamically adjusting CPU and memory allocations for running VMs.

Scaling up

When one of the three goals indicates a need for additional compute resources, the scaling-up process unfolds as follows:

1. The autoscaler-agent computes the new target based on CPU, memory, and working-set goals.
2. The Kubernetes scheduler verifies that the node can accommodate the request without exceeding memory limits.
3. Upon approval, the autoscaler-agent updates the NeonVM resource.
4. The NeonVM controller adds the required CPU and memory to the running VM.
5. The vm-monitor expands the compute cache to utilize the newly allocated capacity.

The scheduler serves as the definitive authority for resource allocation, ensuring that autoscaling requests do not conflict with other Kubernetes scheduling activities.

Scaling down

The downscaling process mirrors the components of scaling up, with an additional verification step within the VM. The vm-monitor ensures that sufficient memory remains for Postgres and the guest operating system before proceeding with any downscale.

It is crucial to recognize that scaling down is as important as scaling up. Many autoscaling systems are quick to add capacity but slow to retract it, resulting in oversized databases long after demand has subsided. Lakebase Postgres prioritizes timely adjustments in both directions to minimize unnecessary costs.

Run it

To experience the benefits of Lakebase Postgres and put its autoscaling capabilities to the test, consider deploying it in your environment. Get started here.

Lakebase Postgres can operate as a standalone database and seamlessly integrates with the broader Databricks Data + AI Platform, encompassing Unity Catalog governance, lakehouse analytics, notebooks, and AI workflows.