NewApp Digest
search bot

tables

From monolith to Lakebase to LTAP: rethinking the database from storage up
Tech Optimizer
July 2, 2026

From monolith to Lakebase to LTAP: rethinking the database from storage up

Sixteen years ago, the author began their PhD at UC Berkeley and was advised to focus on analytics rather than OLTP databases, which were considered solved. This led to the creation of Apache Spark and Databricks. However, during the development of Databricks, it was found that OLTP databases were not solved problems; they were difficult to scale and fragile. This realization led to the development of Lakebase, a serverless Postgres database designed with modern technology. Lakebase architecture separates the write-ahead log (WAL) and data files into independent, scalable services. The WAL is externalized to a service called SafeKeeper, which uses Paxos-based replication for durability, while data files are managed by PageServer, which stores them in cloud object storage. This design addresses challenges such as data loss, scaling, and performance interference between transactional and analytical workloads. Lakebase maintains compatibility with Postgres, offers unlimited storage, serverless compute, durable writes, and simpler high availability. It also introduces LTAP (Lake Transactional/Analytical Processing), which allows both transactional and analytical processing to operate on a single data copy in real time, eliminating the need for separate data copies and reducing costs. LTAP utilizes a unified storage layer that allows data to be materialized in both row and columnar formats, optimizing it for both transactional and analytical workloads. The system ensures that analytics can access the most current data without affecting transactional performance by using a log sequence number (LSN) to retrieve the latest changes. Unlike traditional CDC approaches, LTAP requires no explicit table replication, as all data is stored in a single governed copy. This architecture circumvents common issues faced by hybrid transactional/analytical processing (HTAP) systems, such as incomplete feature sets, lack of ecosystem support, and performance contention. The Lakebase architecture has unlocked capabilities like unlimited storage, elastic compute, durable writes, and instant branching, with ongoing developments anticipated in the future.
silver Android smartphone
Tech Optimizer
July 2, 2026

From monolith to Lakebase to LTAP: rethinking the database from storage up

Sixteen years ago, the author began their PhD at UC Berkeley, where they were advised to focus on analytics rather than OLTP databases, which were considered a solved problem. However, upon developing Databricks, they found that OLTP databases were cumbersome, challenging to scale, and fragile. This led to the creation of Lakebase, a serverless Postgres database designed to address these issues by externalizing the write-ahead log (WAL) and data files into independent services. Lakebase architecture separates the WAL into a distributed service called SafeKeeper and data files into another service called PageServer, allowing for improved durability, scalability, and performance. This architecture eliminates data loss risks associated with disk failures and misconfigurations, simplifies high availability, and allows for elastic compute resources. Lakebase also introduces LTAP (Lake Transactional/Analytical Processing), which enables transactions and analytics to operate on a single data copy in real-time, avoiding the need for separate data copies and reducing costs. The system retains Postgres's ACID semantics while storing data in open columnar formats, making it accessible to both transactional and analytical engines without the delays and complexities of traditional replication methods. The architecture allows for unlimited storage, serverless compute, and instant branching, while ensuring that analytics can read the most current transactional data without affecting transactional workloads. LTAP aims to unify transactional and analytical processing by leveraging distinct compute engines for each workload while integrating at the storage layer.
Microsoft hits back at viral MacBook Neo mocking Windows hardware, but the Windows 11 reputation is the real fight
Winsage
July 1, 2026

Microsoft hits back at viral MacBook Neo mocking Windows hardware, but the Windows 11 reputation is the real fight

A viral video comparing Apple's MacBook Neo to a Windows gaming laptop, the HP Victus, received 5.4 million views and was criticized for being an unfair comparison. Microsoft responded by showcasing the Dell XPS 13, emphasizing its features such as a touchscreen, robust build quality, and a starting price of 9 for students. The Dell XPS 13 specifications include a 13.4-inch 2.5K touch display, Intel Core 5 320 processor, 8GB of RAM, and up to 512GB SSD storage. The XPS 13 is positioned as a superior alternative to the MacBook Neo, offering more flexibility in RAM configurations and features that appeal to ultraportable laptop consumers. Despite the hardware advantages of the XPS 13, Microsoft faces challenges with the perception of Windows 11, which may affect consumer sentiment.
Meta pauses controversial employee-tracking program after security review
Tech Optimizer
June 23, 2026

Meta pauses controversial employee-tracking program after security review

Meta has suspended its employee-tracking program after an internal security review revealed excessive accessibility to sensitive data collected from staff laptops. The program, part of the Model Capability Initiative (MCI), aimed to gather detailed information on employee interactions with work devices, including mouse movements, click locations, keystrokes, and screen content. Concerns arose regarding the privacy and security of the collected data, which included AI prompts, transcriptions, private conversations, and performance-related information. The initiative faced backlash, particularly after an engineer criticized "laptop surveillance," leading to a petition for its termination. The monitoring software was deployed on US workers’ laptops without an opt-out option, capturing comprehensive behavioral datasets. The situation highlighted significant legal and regulatory challenges, as well as the risks associated with managing sensitive data. Access controls, data minimization, and retention policies are critical to mitigate potential breaches.
The database that refused to die: How Postgres survived its own creators
Tech Optimizer
June 22, 2026

The database that refused to die: How Postgres survived its own creators

Postgres, originally developed by Michael Stonebraker in the early 1980s, is an open-source database system that evolved from Ingres. It was designed to handle complex data types and introduced user-defined data types, operators, and functions, leading to the support for abstract data types (ADTs). The initial commercialization of Postgres occurred through a startup named Illustra, later acquired by Informix. In 1995, graduate students Andrew Yu and Jolly Chen revived Postgres, transitioning it from QUEL to SQL, resulting in Postgre95, which evolved into PostgreSQL. Today, Postgres is one of the most popular database systems globally, known for its extensibility and high code quality. However, it currently lacks features like file-level encryption (TDE), which are standard in commercial systems, relying instead on the operating system for encryption. Efforts to implement TDE have faced challenges due to the complexity of required code changes.
PostgreSQL 18 on Amazon Aurora and Amazon RDS: Performance enhancements
Tech Optimizer
June 20, 2026

PostgreSQL 18 on Amazon Aurora and Amazon RDS: Performance enhancements

PostgreSQL 18 addresses common performance challenges for users, including managing query performance across composite indexes, diagnosing memory spills in materialized Common Table Expressions (CTEs), and upgrading major versions without plan regressions. Key enhancements include skip scan optimization for multicolumn indexes, improved EXPLAIN functionality, and optimizer statistics that persist through major version upgrades. Skip scan optimization allows PostgreSQL to efficiently utilize multicolumn B-tree indexes even when leading columns are not specified in the WHERE clause, significantly improving query performance. The EXPLAIN command has been enhanced to include buffer statistics by default, providing deeper insights into query execution and resource usage. PostgreSQL 18 also introduces visibility into the storage of materialized nodes in query plans, indicating whether intermediate results were stored in memory or spilled to disk. A new metric, Index Searches, has been added to EXPLAIN ANALYZE output, indicating how many times the database traversed the index tree during query execution. Additionally, Self-Join Elimination (SJE) automatically detects and removes unnecessary inner joins of a table to itself, optimizing query performance. The autovacuum mechanism has been improved with the introduction of autovacuum_vacuum_max_threshold, which caps the number of dead tuples that can accumulate before autovacuum triggers a VACUUM, addressing issues with large tables. The vacuum_truncate parameter provides a server-wide control point to disable VACUUM’s file truncation behavior, reducing locking issues on busy systems. PostgreSQL 18 also separates the allocation of autovacuum worker slots from their usage, allowing for dynamic adjustments to autovacuum_max_workers without requiring a server restart. Finally, new columns in pg_stat_all_tables track cumulative time spent on maintenance operations, providing better insights into maintenance overhead for each table.
How Five PostgreSQL Optimizations Sped Up Our Dashboard
Tech Optimizer
June 20, 2026

How Five PostgreSQL Optimizations Sped Up Our Dashboard

The dashboard operates on a Django monolith with PostgreSQL and is transitioning to ClickHouse for denormalization. The initial p50 metric was 0.7 seconds, but the p95 was 8 seconds, which was reduced to 1 second. Observability tools were established to monitor performance, and slow HTTP requests were identified using OpenTelemetry traces. Optimization techniques included late joining, asynchronous counting, creating a PostgreSQL replica for read operations, and improving full-text search. Denormalization was explored to enhance filtering performance by creating composite indexes. The production stack was upgraded to PostgreSQL 18, which provided incremental performance improvements. The final p95 value achieved was 1 second, below the target of 3 seconds.
Snowflake Postgres Unifies Your Apps, Analytics and AI
Tech Optimizer
June 18, 2026

Snowflake Postgres Unifies Your Apps, Analytics and AI

Every enterprise operates in two realms: one for real-time applications that process orders and engage customers, and another for analytics platforms that extract insights and drive AI. Snowflake is introducing Snowflake Postgres to bridge these realms with two key features: 1. Data mirroring, which is an always-on replication feature between Postgres and Snowflake, set to enter public preview soon. 2. Postgres for data lakes, allowing synchronization with analytics using open formats like Iceberg, which will be generally available shortly. These features aim to simplify the connection between transactional and analytical data, reducing the need for complex ETL pipelines. Customer feedback indicates that transferring data between OLTP and OLAP databases is the most challenging infrastructure task, leading to costs and issues such as data inconsistencies and delayed decision-making. Snowflake Postgres offers a simplified integration method with low-latency data mirroring that automatically maintains target tables in Snowflake to reflect the current state of source tables in Postgres. This setup can be configured easily through various interfaces or a single SQL command.
Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres
Tech Optimizer
June 18, 2026

Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres

Lakebase Search is a hybrid vector and full-text retrieval system integrated into Lakebase, now in beta on AWS and Azure. It utilizes two Postgres extensions: lakebase_vector and lakebase_text, allowing agents to operate on a single data backend. Agents manage four times more databases than human users and require real-time access to indexed data. The system features a tiered architecture that stores cold data in cost-effective object storage while keeping active data in local NVMe, significantly reducing costs. The lakebase_vector extension offers 32x compression for vectors, allowing a billion vectors to fit into under 10GB of RAM. The lakebase_text extension provides BM25 relevance ranking without high RAM usage. Benchmarking shows that Lakebase Search can efficiently handle large-scale workloads, achieving high recall and low latency with reduced resource requirements compared to traditional architectures. The system allows for continuous search experimentation and dedicated retrieval engines for each agent, enhancing operational efficiency and scalability.
Search