Cloud database provider Turso is embarking on an ambitious journey to create a Postgres-compatible implementation, building upon its existing SQLite-compatible database crafted from scratch in Rust. Turso’s CEO, Glauber Costa, expressed confidence in the foundational integrity of Postgres, stating, “I don’t think there is anything fundamentally wrong with Postgres. If there was something wrong with Postgres, we wouldn’t rewrite it.” However, he believes that, much like SQLite, Postgres can benefit from modernization to better suit the cloud-native landscape.
Turso’s aspirations extend beyond merely adding another entry to the Rust ecosystem. The company’s SQLite reimplementation, initially codenamed Limbo and now known as Turso, employs a virtual machine architecture that the team envisions could accommodate various database frontends. Recently, Turso launched its Postgres initiative, hinting that this architecture might eventually support frontends for other systems, including MySQL and Redis. Costa likened Turso’s vision to that of LLVM, aiming for “one modern, reliable core; many database frontends compiled down onto it.”
From fork to rewrite
The inception of Turso stemmed from Costa and co-founder Pekka Enberg examining SQLite’s architecture while transitioning from their roles at ScyllaDB to establish a startup named ChiselStrike in San Francisco. Faced with the need for an online database, they recognized SQLite as a suitable option, albeit one that required enhancements for cloud service compatibility. Originally developed in 2000 by software engineer D. Richard Hipp, SQLite has maintained a minimalist scope, resisting external contributions and feature bloat, akin to how Andrew Tanenbaum managed Minix, which inspired Linus Torvalds to create Linux.
ChiselStrike initially forked SQLite into libSQL, enabling its developers to modify the database system and allowing outside contributors to introduce their own features. Early contributions focused on enhancements without altering SQLite’s core, a task that proved challenging. More substantial modifications, such as implementing a richer type system, necessitated a complete rewrite, which was initially dubbed Limbo. Eventually, ChiselStrike pivoted to a SQLite-based cloud service and rebranded as Turso. Limbo evolved into an official project, adopting the Turso name, inspired by the mythical Finnish sea creature Iku-Turso, and laid the groundwork for future database engines.
Behind the SQL
In their rewrite of SQLite, Costa and Enberg concentrated on its virtual machine architecture, concluding that this approach could be adapted into a more versatile database execution engine. The software translates SQL queries into a custom bytecode language known as the Virtual Database Engine (VDBE). They constructed Turso using a similar virtual machine design, based on VDBE, which could potentially emulate not just SQLite but any database system. “Look close enough, and every SQL database is just a fancy collection of B-Trees with a bunch of Indexes,” Costa noted. Features unique to Postgres could be implemented using VDBE.
To showcase the capabilities of their virtual machine, the duo created a demo that runs the classic video game DOOM from the 1990s in WebAssembly, with each rendered frame returned as a result row. Other database virtualization approaches exist, such as Google’s Spanner, which supports various modalities, and Microsoft’s Cosmos database service, built on a similar foundation.
However, Costa clarified that Spanner and Cosmos operate at a higher level than Turso. In contrast, Turso’s architecture is more akin to Apache DataFusion, which is also developed in Rust. “The idea of a database that can speak multiple models is nothing new,” Costa remarked. “What is different about Turso is that it sits at the lowest possible level, exactly one layer above the storage engine.”
Postgres on top, Turso under
Enberg and Costa have developed a Postgres-compatible prototype named pgmicro that sits atop Turso. The next phase involves preparing this prototype for production use. “Essentially we would offer a database that speaks Postgres, that plays the same role as Postgres, that is familiar to the developers in the Postgres ecosystem, but it’s built in a different way underneath,” Costa explained. Once sufficiently refined, this Postgres variant of Turso aims to run most existing applications without requiring modifications, although complete compatibility with the original Postgres is not guaranteed. The company also plans to introduce enhancements, such as automatically updated materialized views, a feature frequently requested by the Postgres community.
One legacy aspect of Postgres that may be reconsidered is its one-process-per-connection model, which can be computationally expensive. “It’s worth a fresh look,” Costa stated. While the Postgres project will demand significant effort, it is not insurmountable, and the new features can be integrated back into Turso for use in other database variants in the future. On the commercial front, Turso is developing a database-as-a-cloud service that will provide customers with any database flavor they desire, all operating on the Turso platform.
Rust and AI on Turso
The trend of rewriting open-source software in Rust has gained momentum, with both official and unofficial rewrites of projects like Kubernetes, the Bun JavaScript toolchain, and various components of Linux underway. Much of this work has been facilitated by AI, which has demonstrated a solid understanding of Rust fundamentals. However, Costa does not attribute excessive significance to the choice of Rust for Turso’s rewrite. While Rust provides memory safety, many Postgres bugs are unrelated to memory safety. Costa emphasized that Rust’s compatibility with AI, which Turso utilizes extensively, was a factor, but the decision to use Rust was made prior to the widespread adoption of AI coding agents. “I never understood the fascination programmers have with programming languages,” he remarked, adding, “For us, Rust is just a tool.”
