data management

Postgres extensions, specifically pg_lake, pg_partman, and pg_incremental, provide a vendor-agnostic, open-source solution for managing high-performance time series data. PostgreSQL serves as the foundation, while pg_partman automates time partition management, pg_lake connects Postgres with data lakes for offloading cold data, and pg_incremental processes append-only data in batches. These extensions are maintained by the Postgres team at Snowflake. An example system for monitoring temperature readings uses local storage for recent data and transitions cold data to an Apache Iceberg table on S3. The process includes creating a partitioned table in Postgres, establishing an Iceberg table, using pg_incremental for data appending, eliminating old partitions with pg_partman, and querying from both local and cold tables to optimize storage and costs. A time-partitioned table enhances performance by allowing efficient deletion of outdated data, speeding up queries, and reducing fragmentation.