partition management Archives

Tech Optimizer

July 31, 2025

Cloudflare Chooses PostgreSQL Extension Over Specialized OLAP for 100K Row/Second Analytics

Cloudflare's team chose TimescaleDB over ClickHouse for its Zero Trust product suite to achieve a balance between managing analytical and configuration data and the performance of an OLAP system. They developed Digital Experience Monitoring (DEX), an observability platform that collects structured logs from WARP clients and presents them through a dashboard. Initially using ClickHouse since 2017, they faced limitations with its MergeTree engine, which struggled with numerous small writes from millions of devices. To expedite DEX's release, they used PostgreSQL, managing 200 inserts per second at launch, which scaled to 1,000 inserts/sec as usage grew, leading to performance degradation. They improved query performance by precomputing aggregates, achieving a 1000x improvement. Transitioning to TimescaleDB allowed for better management of time-stamped data with automated aggregation and data retention. Benchmarking showed performance enhancements of 5x to 35x compared to traditional PostgreSQL tables. Following DEX's success, TimescaleDB has been adopted for other Cloudflare initiatives, including Zero Trust Analytics & Reporting.

Tech Optimizer

July 8, 2025

How TimescaleDB helped us scale analytics and reporting

At Cloudflare, PostgreSQL and ClickHouse are used as primary databases for transactional and analytical workloads. PostgreSQL is preferred for product development due to its speed, versatility, and reliability, with hundreds of instances running in various configurations. ClickHouse, introduced in 2017, allows ingestion of tens of millions of rows per second with millisecond-level query performance but comes with trade-offs. The Digital Experience Monitoring (DEX) product was developed with a focus on simplicity and rapid shipping, utilizing a team of three engineers and collaboration with other teams. DEX's initial launch included fleet status logs uploaded every two minutes and synthetic tests with usage caps, leading to a successful MVP launch ahead of schedule. The architecture included an HTTP API, PostgreSQL for storing configurations, and a React UI in the Cloudflare Dashboard. PostgreSQL was chosen for analytics due to its ability to handle structured logs, with a table created for device state logs lacking a primary key but optimized with unique and standard indexes for common queries. The UPSERT query was used for data insertion, and the order of columns in multicolumn indexes was optimized for performance. After launching DEX, the system scaled to 1,000 inserts per second, leading to performance degradation. Precomputed aggregates were implemented to improve query performance, resulting in up to a 1000x improvement. Table partitioning was considered but ultimately not pursued due to its manual management requirements. TimescaleDB was explored as an alternative to enhance query performance, offering features like automatic partition management, continuous aggregates, and compression. A side-by-side comparison with PostgreSQL showed significant performance improvements, particularly for longer time windows. TimescaleDB's integration allowed for efficient data retention and aggregation, simplifying the infrastructure. Following the success of DEX, other teams at Cloudflare began using TimescaleDB for analytics and reporting, consolidating data from various sources and managing high ingestion rates effectively by optimizing data handling techniques such as switching to COPY for bulk inserts and adjusting replication settings.

Tech Optimizer

May 21, 2025

Achieve up to 1.7 times higher write throughput and 1.38 times better price performance with Amazon Aurora PostgreSQL on AWS Graviton4-based R8g instances

Upgrading to Graviton4-based R8g instances with Aurora PostgreSQL-Compatible 17.4 in an Aurora I/O-Optimized cluster configuration results in significant performance improvements. The new instances provide up to 1.7 times higher write throughput, 1.38 times better price-performance, and reduce commit latency by up to 46% on r8g.16xlarge instances and 38% on r8g.2xlarge instances compared to Graviton2-based R6g instances. The Amazon Aurora PostgreSQL-Compatible Edition now supports AWS Graviton4-based R8g instances and PostgreSQL 17.4, which introduces performance enhancements for I/O-Optimized configurations, optimizing write operations and batch processing. R8g instances offer up to 192 vCPUs and 1.5 TB of memory, supporting larger configurations and providing up to 50 Gbps of network bandwidth. PostgreSQL 17 includes vacuum improvements, eliminates the need to drop logical replication slots during upgrades, and expands SQL/JSON standards. Aurora PostgreSQL-Compatible separates compute from storage, enabling independent scaling and maintaining six-way replication for durability, while processing changes as log records to reduce I/O operations. Performance benchmarks using HammerDB show improvements in throughput and commit latency across various workloads. For small workloads on 2xlarge instance size, throughput increased by 50.25% and commit latency improved by 33.87%. For medium workloads on 16xlarge instance size, throughput increased by 30% and commit latency improved by 17.44%. The most significant performance benefits arise from combining hardware upgrades from Graviton2 to Graviton4 with database engine upgrades from PostgreSQL 15.10 to 17.4. For small workloads, throughput increased by 70% and commit latency improved by 38.71%. For medium workloads, throughput increased by 70% and commit latency improved by 46.67%. Cost efficiency is also enhanced, with a 38% improvement in price performance and a 61.26% improvement in price-performance ratio when comparing Graviton2 and Graviton4 instances. Reserved Instances for Graviton4-based R8g instances offer additional cost-optimization opportunities.

Winsage

March 24, 2025

5 reasons why GParted is better than any Windows alternative for partition management

GParted is a free and open-source partition management tool available on Linux, allowing users to boot directly from a flash drive without installation. It supports over a dozen file systems, including HFS, HFS+, BTRFS, and F2FS, providing versatility beyond Windows' Disk Management, which is limited to NTFS, exFAT, and FAT32. GParted enables users to manage partition flags easily, a feature not available in Windows. While it does not offer direct partition recovery options, users can use the Terminal in GParted Live to execute disk tests for locating lost partitions. GParted can run from a live USB across multiple operating systems, including Windows, Linux, and macOS, making it a flexible choice for partition management. It allows users to shrink partitions, create new ones, and recover lost partitions and files, although it is recommended for users comfortable with partition management.