elegance

Making data from transactional databases accessible to analytical databases is essential in modern data architecture, but it faces challenges such as fragile tooling, high costs, and complex operations. Snowflake has developed a Postgres service that addresses these issues by reimagining Postgres replication. Postgres is a strong operational database, but its change data capture (CDC) capabilities need improvement. Many data pipelines are fragile due to complexities in managing continuous data flow, schema changes, snapshots, and failures. To enhance user experience with Snowflake Postgres, a complete reinvention of the replication process was necessary. A new feature called data mirroring is in public preview, allowing resilient data replication into Snowflake with low cost, minimal lag, and transactional consistency. This feature directly pushes changes from Postgres into Apache Iceberg™ tables in transactional batches, which are then automatically applied to Snowflake tables without extra infrastructure. Change data capture records changes from a transactional database for replay on another system, primarily using logical decoding in Postgres. This method decodes WAL records into logical operations, but the client bears the responsibility for subsequent steps like backfilling, managing schema changes, and handling failures. Built-in logical replication in Postgres only addresses some complexities. A limitation of logical decoding is that the external system does not recognize the state of Postgres, making it difficult to detect schema changes or correlate snapshots with changes. The solution is to push changes from Postgres into a data lake, specifically into Iceberg tables using compressed Parquet format, with object stores like Amazon S3 as the destination. The mirroring process uses a new Postgres extension called snowflake_cdc, which continuously pushes batches of changes into per-table change logs and a meta log. This extension understands the Postgres environment, coordinating schema changes and complex transactions while ensuring alignment between snapshots and changes.