At the recent Data and AI Summit, a groundbreaking innovation was unveiled: the introduction of lakebases, a new category of operational databases designed specifically for the development of intelligent applications. Today, we are thrilled to announce the Public Preview of Databricks Lakebase, the first fully-managed Postgres database tailored for data applications and artificial intelligence.
Organizations are increasingly merging their operational and analytical data to create intelligent applications. These applications serve features and models, facilitate standalone operations, or analyze operational data within a lakehouse. However, many customers face challenges related to provisioning, scaling, and a modern developer experience, primarily because traditional databases have seen little innovation over the past few decades.
Introducing Lakebase
Operational databases, particularly OLTP systems, have remained largely unchanged since the 1990s. Even in cloud environments, these legacy systems are often slow and costly to manage. Typically, operational databases exist in separate stacks from analytics platforms, leading to silos that hinder the integration of transactional and analytical data. Furthermore, they do not align with the modern development workflows essential for AI-driven projects, which often require distinct databases for development, testing, staging, and production, each needing separate provisioning and maintenance.
Databricks Lakebase emerges as a pioneering solution, built on open-source standards and featuring a highly scalable architecture that separates compute from storage. This design is specifically optimized for contemporary application development, allowing seamless integration of operational, analytical, and AI stacks within the lakehouse environment.
Built on open source Postgres
Over the past seven years, Postgres has ascended to become the preferred database within the developer community, recognized for its open-source nature and vibrant ecosystem of extensions. Supported by a robust community of libraries, tools, and frameworks, Postgres is familiar to engineers, and foundational models are often trained on extensive datasets from this ecosystem, making it an ideal choice for intelligent applications.
Lakebase supports popular extensions such as PostGIS and pgvector, providing a rich array of capabilities that development teams will find intuitive and accessible.
Separation of Compute and Storage
Lakebase employs an architecture that distinctly separates compute and storage, enabling independent scaling while ensuring low latency (less than 10ms) and high concurrency transactions (over 10k queries per second). Managed entirely by Databricks, Lakebase eliminates the need for infrastructure provisioning and maintenance, thus streamlining both infrastructure and development processes. This allows teams to accelerate their workflows without sacrificing control or reliability.
- High availability with readable secondaries: Multi-zone high availability safeguards against zonal failures by provisioning secondary compute resources across zones, which can also be made readable to enhance isolation and horizontal scaling of read workloads.
- Data storage and recovery: All transactions are securely stored in encrypted storage that is regionally durable, protecting against single zone failures. Point-in-time recovery is supported through a data protection window offering up to 35 days of recovery time.
- Branching for isolated test environments or point-in-time recovery: Utilizing copy-on-write branching, Lakebase allows for the instant creation of zero-copy clones of the database, with dedicated compute resources for each branch. This enables independent management of child branches, which can be created based on current or historical data, facilitating isolated testing or recovery operations.
Modern DevEx, Built for AI
Lakebase is constructed on Neon technology, which introduces copy-on-write branching and autoscaling serverless compute. This innovative branching capability allows developers to create new databases with identical data and schema as existing databases without impacting the original, all while being cost-effective by avoiding data duplication. Serverless compute autoscaling ensures sub-second start times and scales dynamically according to demand, even allowing for zero scaling to optimize costs.
This combination of serverless autoscaling and branching fundamentally transforms the development landscape for applications. Developers can swiftly create database branches that correspond to each git branch, eliminating the need for setting up new database instances or managing multiple databases for development and testing.
For both developers and agents, this means ephemeral database environments can be rapidly established, utilized, and decommissioned with minimal cost and effort. The full Neon developer experience in Lakebase, along with additional exciting features, will be available soon.
Integrated with the lakehouse
Lakebase seamlessly integrates a transactional database layer with the lakehouse, inheriting the operational maturity of the Databricks Platform, which includes observability, security, and access controls. It synchronizes with Unity Catalog managed tables, simplifying the combination of operational, analytical, and AI workloads without the need for custom ETL pipelines. This integration enables the development of intelligent applications that leverage features or predictions generated in the lakehouse while simultaneously updating the analytical layer with fresh operational data, all within a unified platform.
- Fully managed data synchronization: Configurable data synchronization pipelines offer a straightforward, scalable method for managing data between Unity Catalog managed tables and Lakebase, with options for one-off snapshots, triggered, or continuous synchronization.
- Feature and model serving: Lakebase serves as the online feature store for machine learning features and models, while the lakehouse functions as the offline store for training and analysis.
- Unified governance: Native integration with Unity Catalog and Databricks identity simplifies access control across the platform, allowing for consistent identity management across operational and analytical users.
- Databricks Apps integration: Lakebase supports full-stack applications on Databricks, powering transactional interactions.
- Unified development environment: The Databricks SQL Editor allows for direct querying of Lakebase and data browsing.
- Built-in monitoring: Key database metrics, including transactions per second, open connections, and resource utilization, are readily available.
- Network security: Lakebase is integrated with Databricks’ enterprise network security features, such as PrivateLink and IP ACLs, ensuring consistent network security.
- Multi-cloud: Lakebase is accessible across various cloud providers without the need for replatforming. Currently, it is available on Azure and AWS, with plans to support Google Cloud Platform in the future.
Customers are using Lakebase
With hundreds of customers participating in the Private Preview program, the range of use cases has been impressive:
- Serving data and/or features from the lakehouse: Applications such as personalized recommendations and customer segmentation.
- Building applications and agents: Solutions for order processing, interactive workflow sign-offs, and chatbots.
- Analyzing operational data in the lakehouse: Syncing data for historical order analysis or chatbot training data.
At Heineken, our goal is to become the best-connected brewer. To do that, we needed a way to unify all of our datasets to accelerate the path from data to value. Databricks has long been our foundation for analytics, creating insights such as product recommendations and supply chain enhancements. Our analytical data platform is now evolving to be an operational AI data platform and needs to deliver those insights to applications at low latency.
—Jelle Van Etten, Head of Global Data Platform, Heineken
At Tibber, empowering customers to take control of their energy consumption requires a flexible data infrastructure. Lakebase’s integration with Databricks makes it easy to serve analytical and transactional data helping us deliver real-time insights to our customers.
— Niklas Nordansjö, Data Platform Lead, Tibber AS
A strong partner network enhances Lakebase customers’ ability to collaborate with existing technology partners and system integrators for data integration, business intelligence, and governance. We are proud to have a remarkable group of industry launch partners for Lakebase.
At dbt Labs, we’re changing how data engineering is done. With Databricks’ new Lakebase, our joint customers will now be able to combine low-latency, transactional data and analytical data into one platform on Databricks. This will help us both deliver enterprise-scale AI for our customers. We can’t wait to usher in the new era of analytics with Databricks.
— Ryan Segar, Chief Product Officer, dbt Labs
Lakebase merges the familiarity and extensibility of Postgres with the scalability of a modern serverless architecture, all while providing a modern developer experience and the unified data capabilities of the lakehouse, alongside the operational maturity of the Databricks Data Intelligence Platform. By integrating these elements into a single, fully managed offering, Lakebase empowers teams to construct intelligent, data-driven applications without the operational complexities typically associated with transactional systems.
Lakebase is currently available in Public Preview, with pricing details accessible here. For those looking to develop applications that integrate analytics and AI, Lakebase is the essential component of your tech stack, poised to enhance development and streamline operations. Workspace or Account administrators can enable it directly from their Databricks Workspace. Experience it today!
