partitioning Archives

Tech Optimizer

February 19, 2026

How OpenAI Scaled to 800 Million Users on PostgreSQL

OpenAI has scaled PostgreSQL to support over 800 million active users of ChatGPT, making it one of the largest PostgreSQL deployments globally. The database can handle millions of concurrent connections and a very high volume of requests per second. OpenAI employs several strategies to optimize performance: 1. **Connection Pooling with PgBouncer**: Reduced database connections from 10,000 to 200, enhancing efficiency by a factor of 50. 2. **Read Replicas**: Distributes read requests across multiple replicas while the primary database handles writes. 3. **Horizontal Sharding**: Partitions data across multiple instances based on a shard key, such as user_id or tenant_id. 4. **Query Optimization**: Analyzes slow queries and creates appropriate indexes to maintain performance. 5. **Connection Management**: Implements timeouts and connection limits to prevent overload. 6. **Caching**: Uses application-level caching with Redis to reduce database load. 7. **Monitoring and Observability**: Tracks key metrics like connection counts and query latency to identify issues early. These strategies enable OpenAI to maintain performance and reliability for a large user base.

Tech Optimizer

November 20, 2025

How to implement Dynamic SQL in PostgreSQL 10

Dynamic SQL in PostgreSQL allows developers to streamline querying processes and reduce repetitive tasks. It is implemented using the EXECUTE statement, which evaluates a string input. Dynamic SQL can be used for executing DDL statements, such as analyzing tables within a schema, and for DML statements, enabling interactive data manipulation, like filtering results in a billing application. PL/pgSQL caching can lead to inefficiencies in dynamic SQL scenarios where execution plans vary. Security risks, particularly SQL injection vulnerabilities, are associated with dynamic SQL. Recommended practices to mitigate these risks include using the USING clause for parameterized statements, employing the format function for query construction, and utilizing functions like quote_ident() and quote_literal() for proper formatting.

Tech Optimizer

November 7, 2025

PostgreSQL 13’s Sunset: Why Database Pros Must Upgrade Before the Lights Go Out

Organizations using PostgreSQL 13 must upgrade before its end-of-life on November 13, 2025, as this will result in the cessation of security patches, bug fixes, and official support. Continuing to use an unsupported version exposes systems to vulnerabilities, which can lead to data breaches and compliance challenges. The last minor release for PostgreSQL 13 was 13.21 in May 2025. Upgrading to newer versions, such as PostgreSQL 16 or 17, offers performance improvements and enhanced features. Strategies for upgrading include using tools like pg_upgrade, pg_dump/pg_restore, and logical replication to minimize downtime. Compatibility issues may arise due to deprecated functions in PostgreSQL 13, necessitating code reviews. Managed services like those from Percona can provide support beyond EOL. The costs of not upgrading can be significant, with potential downtime from security breaches often exceeding migration expenses.

Tech Optimizer

November 2, 2025

Meet the Team Behind Snowflake Postgres

In June 2025, Snowflake acquired Crunchy Data to integrate Postgres into its offerings. Crunchy Data, founded in 2012, focused on enhancing Postgres for enterprise environments, creating a security-first distribution called Crunchy Certified PostgreSQL. They introduced the Postgres Operator for Kubernetes in 2017 and launched Crunchy Bridge, a fully managed cloud Postgres service, in 2020. The Crunchy Data team has made significant contributions to the Postgres ecosystem, including enhancements to core functionality and extensions like PostGIS and pg_partman. Key team members have held leadership roles in the PostgreSQL community. Snowflake Postgres aims to unify transactional and analytical workloads, allowing organizations to run applications at scale while leveraging data for analytics, machine learning, and AI.

Winsage

October 5, 2025

How to switch your Windows 10 PC to Linux Mint – for free

The countdown for Windows 10 support is underway, prompting users to consider alternatives like Linux Mint, especially if they are hesitant about upgrading to Windows 11 or if their hardware does not meet the requirements. Linux has become more user-friendly, and Linux Mint is recommended for both newcomers and experienced users. To switch to Linux Mint, users need a Windows PC, an internet connection, and a USB stick with at least 4GB of storage, with an estimated installation time of about 2 hours. Users should compile a list of essential programs and check for Linux-compatible alternatives. Many cloud-based applications work on Linux, and Windows 10 can be run within Linux using Oracle VirtualBox for incompatible programs. The installation process involves downloading the Linux Mint ISO from the website, preparing a USB drive, and adjusting the boot order in the BIOS/UEFI settings. Users should disable Secure Boot to ensure a smooth installation. After rebooting, the Mint interface will appear, and the installation wizard will guide users through the setup process, including language preferences and partitioning the hard drive. Enabling Timeshift for system backups and using the Backup Tool for personal files is recommended. After installation, users should check for additional drivers and install multimedia codecs. Linux Mint requires a minimum of 2GB of RAM and 20GB of disk space and is compatible with most modern hardware. Running both Linux Mint and Windows on the same PC is possible, either by dual-booting or using a virtual machine. Most modern hardware is compatible with Linux, and testing Mint through a live session on a USB stick can confirm compatibility. Software-as-a-service applications generally work well on Linux, while older programs may require additional tools. Many popular applications are available on Linux, and Steam-based games perform well.

Winsage

October 3, 2025

How to switch your Windows 10 PC to Linux Mint

Windows 10 support is ending, prompting users to consider alternatives like Linux Mint, which is compatible with PCs running Windows 10 and allows exploration without altering the existing setup. To switch to Linux Mint, users need a Windows PC, internet connection, and a USB stick or DVD-RW with at least 4GB of storage, with an estimated installation time of about 2 hours. Users should back up important files before installation and may need to disable Windows Fast Startup, BitLocker, and Secure Boot for a smoother transition. Linux Mint requires 2GB of RAM (4GB recommended) and 20GB of disk space (100GB recommended), and it can run on older machines. Most modern hardware is compatible with Linux, and many cloud-based applications like Microsoft 365 work on Linux. Users can test Mint in a live session before installation to check hardware compatibility.

AppWizard

October 1, 2025

How to enable TPM, Secure Boot, HVCI, and VBS for Battlefield 6 and Black Ops 7

Battlefield 6 requires gamers to meet advanced security requirements to access the game, including TPM 2.0, Secure Boot, HVCI, and VBS. TPM 2.0 is a hardware-based security feature found on most modern motherboards, and if absent, users may need to research their motherboard for upgrade options. Secure Boot and TPM 2.0 can be checked using the Steam beta client or through the Device Manager and System Information panel. If disabled, they can typically be enabled via the BIOS/EFI system. Windows must operate in UEFI mode with the GUID partition table for these features to function correctly, and if installed in legacy mode, reinstallation may be necessary. HVCI protects against malicious code and can be checked via the memory integrity settings. VBS can be verified through the System Information panel, indicating if it is running.

Winsage

September 29, 2025

I used DISM to create a Windows installer with all my apps, and here’s how it works

To back up a Windows 11 image, start with a fresh installation and use the Sysprep tool to generalize the system, avoiding Microsoft Store apps to prevent Sysprep failure. Prepare two external drives: one for storing the Windows image (minimum 8GB) and another bootable drive configured with Windows Preinstallation Environment (Windows PE). Download the Windows Assessment and Deployment Kit (ADK) and the Windows PE add-on to create the bootable USB drive. Enter Audit Mode by pressing Ctrl+Shift+F3 at the out-of-box experience screen, disable app updates, and install desired applications for the entire system. Use Sysprep to enter Out of Box Experience (OOBE), check Generalize, and select Shutdown. Boot from the Windows PE drive, use diskpart to identify drive letters, and capture the image with the DISM command. To apply the image on a new PC, create a diskpart script for partitioning and a batch script for applying the image. Boot the new PC from the Windows PE drive, run the diskpart script to create partitions, and execute the batch script to apply the image. This method is efficient for deploying consistent applications across multiple PCs.

Tech Optimizer

September 23, 2025

Global indexes for partitions in Postgres Pro: uniqueness without hacks

Partitioning in PostgreSQL divides large tables into smaller segments called partitions, improving query performance by enabling partition pruning, which allows the DBMS to scan only relevant partitions. However, local indexes on partitioned tables have limitations, such as inability to enforce global uniqueness and inefficiencies when querying across multiple partitions. Global indexes, introduced in Postgres Pro Enterprise, operate at the partitioned table level, ensuring global uniqueness and improved query performance. They are created for the entire table and include data from all partitions, allowing for faster queries but slower writes due to uniqueness checks. Global indexes do not automatically remove entries for deleted rows and require manual VACUUM to clean up obsolete entries. Current limitations include lack of support for concurrent operations, expressions in index creation, and certain foreign key constraints. Despite these limitations, global indexes can enforce global uniqueness and enhance query performance for non-partitioning keys.

Tech Optimizer

August 26, 2025

How we loaded a petabyte into PostgreSQL before New Year — and what happened next

A team decided to test PostgreSQL's capabilities with a one-petabyte database, setting a challenge for December 10, with a report due by January 20. They initially sought cloud storage but found no provider could meet their requirement for a single chunk of storage, leading them to rent physical servers. They used Shardman, a distributed database engine, to manage the database across seven servers. They employed the YCSB benchmark for performance testing and faced various challenges, including data loading issues and hardware problems. By January 15, they reached 863 terabytes of data generation, falling short of their petabyte goal, and documented their findings for future reference.