The long-standing divide between operational and analytical databases is finally dissolving. Databricks is demonstrating this shift by integrating Backstage, Spotify’s internal developer portal, with its Lakebase. This move effectively transitions Backstage from its traditional PostgreSQL database to Databricks’ unified data platform. Historically, operational (OLTP) and analytical (OLAP) databases necessitated separate infrastructures, budgets, and on-call rotations due to their fundamental differences in storage, compute, and failure modes. However, with advancements in shared storage, serverless computing, workload-isolated compute, and catalog-level governance, these distinctions are evolving into mere access patterns on a singular foundation.
The Setup: Pointing Backstage at Lakebase
Lakebase now provides a serverless PostgreSQL interface, powered by Neon’s architecture, directly within the Databricks Platform. By utilizing the wire-protocol PostgreSQL, Backstage operates seamlessly, remaining oblivious to the fact that it is not connected to a conventional RDS instance. The integration process involved updating Backstage’s configuration to direct it to Lakebase and substituting its default in-memory search with PgSearchEngine. A notable challenge was the authentication process; Lakebase requires OAuth JWTs rather than the traditional Databricks Personal Access Tokens. The `databricks postgres generate-database-credential` CLI command generates short-lived, scoped JWTs, which are suitable for applications and CI/CD. For this proof-of-concept, a cron script was implemented to refresh these credentials every 50 minutes to manage expiration. Once authentication was addressed, Backstage’s Knex migrations executed smoothly, bringing the portal online.
Branching Changes the Database Development Cycle
The true innovation lies in how this integration transforms the database development lifecycle. Traditional PostgreSQL often imposes a slow and costly tempo on teams, particularly for tasks like schema migration testing. Creating a copy of a conventional database for testing can be a tedious and expensive process, often leading teams to bypass thorough validation and rely on maintenance windows. The ability to create near-instant database branches fundamentally alters this dynamic. With Lakebase’s copy-on-write architecture, branching creates a pointer to existing data, diverging only upon write, thus making the operation instantaneous.
This capability empowers developers to shift their focus from asking “is this change safe enough?” to “which copy of production should I test this on first?” This aligns with the concept that Databricks Postgres branches like Git, facilitating rapid iteration. A minor caveat exists: the API mandates that all parameters be nested within a ‘spec’ object, and a time-to-live (ttl), expire_time, or no_expiry must be specified. The control plane acknowledged the request almost instantaneously, resulting in a ~63 MB clone of the Backstage catalog being created in just over a second.
Point-in-Time Recovery: The Undo Button
Branching and Point-in-Time Recovery (PITR) share a fundamental principle. To test recovery, the ‘final_entities’ table was wiped clean. A recovery branch was then established from a timestamp just seconds prior to the deletion. The entire recovery process took less than four seconds. Verification confirmed that all 32 entities were restored, while the production environment remained untouched, demonstrating the isolation of branches. Notably, Lakebase snapped to the nearest WAL record, 12 seconds before the requested timestamp, providing WAL-level granularity for time-sensitive recovery—an essential feature for understanding Point-in-Time Recovery database operations. When the state of a database becomes a cheap, forkable artifact, risky operations can undergo a dry run, and incidents can be reversed with an instant undo button.
From Infrastructure Capability to Developer Workflow
The technical feasibility of database branching has been established: instant clones and rapid recovery with a real application that remains unaware of the underlying changes. The next challenge is to seamlessly integrate this capability into the daily developer workflow, making it as natural as branching code. This is where significant enhancements in developer productivity can be realized. The objective is to render database branching invisible—automatic, rather than requiring explicit commands. Efforts are underway to develop VS Code extensions that will synchronize Git and database branches automatically.
The traditional sprint cycle typically involves creating Git branches, writing extensive mock objects for database interfaces, and testing against mocked or in-memory databases. This often leads to the discovery of schema migration issues only after deployment to staging environments. With database branching, the developer workflow undergoes a transformation: Git branches trigger automatic, sub-second Lakebase database branches. Developers can write and test code against live database data from the outset, with integration tests running against the actual database, eliminating the need for mocks.
Multiple solutions can be explored, and rollbacks become trivial. Pull requests can initiate CI to create their own database branch, validating both code and schema. QA teams benefit from disposable branches for destructive testing. Merging a pull request allows CD pipelines to migrate changes upstream while cleaning up branches, code, and data. Mock objects, which can account for 20-30% of test code, may soon become obsolete. Staging environment collisions will fade away, and the common “works on my machine” problem will be resolved. Database changes identified during development will be inexpensive to rectify. Instant branches facilitate performance tests, while disposable branches streamline functional tests, and dedicated running branches enhance User Acceptance Testing (UAT). The costly infrastructure built around a now-obsolete constraint—slow, expensive database copies—will no longer be necessary.
The pressing question for development teams is how much of their sprint time is consumed by workarounds for a constraint that has ceased to exist.
Part 2 of this series will delve into the implications for security and compliance as operational databases are integrated into Databricks’ Unity Catalog.