PostgreSQL Archives

Tech Optimizer

July 22, 2026

Migrating mission-critical payments at Nubank to Amazon Aurora PostgreSQL

Nubank, a digital banking platform with approximately 135 million customers in Brazil, Mexico, and Colombia, faced challenges with its payment infrastructure due to inefficiencies in managing 7.5 TB of self-hosted PostgreSQL databases. Queries took over 13 minutes to execute, and replication lags exceeded four minutes during peak times. To address these issues, Nubank evaluated database solutions based on developer productivity, operational efficiency, performance reliability, and scalability. They selected Amazon Aurora PostgreSQL-Compatible Edition for its compatibility, performance improvements, and automated features. The migration to Aurora was facilitated by AWS Database Migration Service (AWS DMS), which minimized downtime and allowed for efficient transition. Post-migration, query performance improved significantly, with some queries executing up to 1,900 times faster, and overall end-to-end service latency decreased, enhancing customer experience. The migration resulted in a 25 percent cost reduction and ensured compliance with regulatory requirements.

Tech Optimizer

July 22, 2026

Google boosts AlloyDB vector search with HNSW preview

Google has introduced a preview of its columnar engine-accelerated HNSW feature for AlloyDB, enhancing vector search throughput by up to four times for users of the pgvector extension in PostgreSQL. This improvement is aimed at approximate nearest neighbor searches using HNSW across large datasets. AlloyDB utilizes a columnar engine that allows HNSW indexes to reside in memory, improving throughput and recall metrics. In benchmark tests on the GloVe 100 Angular dataset, queries per second increased by approximately 4.2x to 4.9x, with recall improving from about 0.78 to over 0.94 at a throughput of around 350 queries per second. The columnar engine's design bypasses traditional PostgreSQL buffer management bottlenecks, enhancing efficiency during graph traversal. The memory architecture is distinct from basic caching techniques, and the feature requires no changes to application code. Users must enable the columnar engine and index caching flags to utilize this feature. Benchmarks were conducted on an AlloyDB C4A machine with 16 virtual CPUs.

Tech Optimizer

July 21, 2026

I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer

The author transitioned from a data analyst role to a data engineer, creating a 12-month self-study roadmap focused on learning by building projects. The first project was a GitHub ETL pipeline, which evolved from a simple Python script to a more complex system using SQLite and GitHub Actions for automation. The author realized that the challenges of data engineering lay in system design and orchestration rather than just writing ETL logic. For the second project, the author built an automated RSS ingestion pipeline to explore engineering decisions in creating a reliable data pipeline. The project emphasized the importance of separating application responsibilities from orchestration tasks, leading to the use of Docker for consistent execution environments and Kestra for orchestration. The author validated each component of the project incrementally: first the Python ETL, then PostgreSQL, followed by Docker, and finally Kestra. This approach ensured that each layer functioned correctly before adding complexity. The final architecture included distinct responsibilities for each component: Python for data processing, PostgreSQL for data storage, Docker for execution, and Kestra for orchestration. The author learned that effective engineering involves considering system reliability, error handling, and observability, shifting the focus from merely writing code to designing entire systems. The experience transformed the author's mindset, emphasizing the importance of incremental building and the separation of execution and orchestration responsibilities.

Tech Optimizer

July 18, 2026

North Korea Buried Four-Stage Malware in Flag Images: Zero Antivirus Detections

North Korea's Contagious Interview hackers have been using a deceptive strategy to target developers by posing as recruiters and embedding malware in SVG files. Elastic Security Labs discovered that the attackers hid malicious payloads within HTML comment blocks of these files, allowing the malware to evade antivirus detection. At the time of the findings, no antivirus engines flagged the compromised repositories, which included trojanized GitHub repositories disguised as coding challenges. The malware executed automatically at server startup and deployed four modules: a browser credential and cryptocurrency wallet stealer, a file stealer, a remote access Trojan, and a clipboard monitor. The campaign, tracked as REF9403, is part of the ongoing Contagious Interview operation attributed to North Korea's Lazarus Group, which aims to generate revenue through cryptocurrency theft. Developers are advised to audit any projects run from unsolicited sources and to monitor specific domains associated with the attack.

Tech Optimizer

July 16, 2026

Google Cloud boosts AlloyDB search for CJK text with Gemini

Google Cloud has enhanced AlloyDB by integrating AlloyDB AI Functions with Gemini models to improve search indexing for logographic languages like Chinese, Japanese, and Korean. This update addresses challenges in database search functionality, particularly for languages written in continuous scripts, where traditional PostgreSQL text parsing struggles. Developers can now invoke Gemini models directly from SQL to facilitate word segmentation and stop-word removal before creating full-text indexes, offering a more efficient alternative to traditional methods. The architecture maintains raw text, segmented text, search vectors, and embeddings in a single table, with generated columns ensuring automatic updates to search vectors and embeddings. A batching process using a PL/pgSQL stored procedure allows for efficient processing of extensive document collections by aggregating rows into arrays for model requests, thus reducing row-locking issues. Google Cloud provides tailored text search configurations for different datasets, suggesting a simple configuration for solely Chinese text and an English configuration for bilingual datasets. This preprocessing logic can also be applied during query execution, allowing user searches to be directed to Gemini for text segmentation and stop-word elimination. AlloyDB supports RUM indexes on search vectors, which store lexeme positions for computing relevance and word distance, enabling rapid query responses. Additionally, a hybrid search methodology combines full-text and vector search using a ScaNN index on embeddings, merging results through Reciprocal Rank Fusion. This development allows for robust multilingual search and vector retrieval solutions within the database environment, eliminating the need for external microservices.

Tech Optimizer

July 12, 2026

What Is Serverless PostgreSQL?

Serverless PostgreSQL is a fully managed cloud database model that separates compute and storage, allowing them to scale independently and automatically based on demand. It eliminates the need for manual infrastructure provisioning and capacity planning, charging only for active usage. Unlike traditional PostgreSQL setups, which require continuous resource allocation and manual scaling, serverless PostgreSQL provisions resources on demand and can scale down to zero during idle periods. Serverless PostgreSQL integrates with serverless compute platforms, enabling analytical queries to access the same data within a unified architecture. Key differences between traditional and serverless PostgreSQL include manual versus automatic provisioning and scaling, fixed versus usage-based billing, and high versus reduced operational overhead. Lakebase architecture is an emerging model that combines transactional databases with lakehouse foundations, allowing operational and analytical workloads to coexist on a single platform. This architecture minimizes data duplication and simplifies access, enhancing data management and analysis. Serverless PostgreSQL operates on a cloud-native architecture that enhances efficiency by allowing compute and storage to scale autonomously. It features scale-to-zero behavior, where compute resources are suspended when inactive and reactivated upon new queries. Major providers include Databricks Lakebase, Amazon Aurora Serverless v2, and Neon, each offering varying capabilities and integrations. Pricing for serverless PostgreSQL typically includes charges for compute resources, storage, and data transfer, with costs fluctuating based on workload activity. Cold start latency is a performance consideration, as reactivating compute resources can introduce delays. Strategies to mitigate this include keeping resources partially active or selecting providers with minimal cold start impacts. Serverless PostgreSQL is well-suited for OLTP workloads, while lakebase architecture is better for AI development, variable workloads, and environments requiring rapid iteration. Setting up serverless PostgreSQL involves choosing a provider, creating a database instance, and configuring access settings. It can also be used alongside serverless compute platforms for analytics, further extending its capabilities.

Tech Optimizer

July 12, 2026

Running pgvector in production on Amazon Aurora PostgreSQL

Running pgvector on Amazon Aurora PostgreSQL-Compatible Edition offers a vector store with operational capabilities, high availability, and scalability. It is favored for Retrieval Augmented Generation (RAG) workloads transitioning to production, but increased traffic introduces challenges like query latency and memory management. Key operational practices for pgvector workloads include selecting the appropriate index type (HNSW or IVFFlat), establishing a baseline schema, choosing a suitable distance operator, scaling the index through quantization and partitioning, and preparing for churn and observability. The prerequisites for using pgvector include an Aurora PostgreSQL-Compatible cluster with specific PostgreSQL versions and the vector extension enabled. The embedding model used in examples is Amazon Titan Text Embeddings V2, which produces 1024-dimensional embeddings. pgvector supports two Approximate Nearest Neighbor (ANN) index types: HNSW, which is efficient for querying and allows for incremental insertions, and IVFFlat, which is less resource-intensive but requires rebuilding if data changes. There are scenarios where forgoing an index is beneficial, such as small datasets or partitioned datasets requiring 100% recall. A baseline schema for a multi-tenant document store includes creating a table for documents with an embedding vector and establishing indexes for tenant IDs and embeddings using HNSW. The recommended parameters for HNSW include m = 16 and ef_construction = 128. Scaling to millions of vectors involves quantization, tuning HNSW parameters, and partitioning. Aurora Optimized Reads can extend effective cache capacity, and managing index churn is crucial for maintaining performance. Observability metrics include query-level statistics, instance-level metrics, and custom application-defined metrics. To clean up after testing, it is advisable to drop the created indexes and tables, and delete the Aurora PostgreSQL-Compatible cluster and any manual snapshots taken during testing.

Tech Optimizer

July 10, 2026

Google Cloud: PostgreSQL community contribution updates

Google Cloud actively participates in the PostgreSQL ecosystem by supporting community-driven events and contributing to open-source initiatives. Recent key events include: - **PGConf.dev 2026**: Featured strategic discussions on logical replication and global index architecture, with a consensus to adopt a deparsing-based approach for DDL replication. Dilip Kumar presented on global indexes. - **PGConf India 2026**: Attracted over 580 participants, featuring various sessions including keynotes and technical talks by Google Cloud contributors. - **PGDay Paris & PGDay France 2026**: Matt Cornillon was involved in organizing PGDay France, with sessions led by him and Yves Colin. - **PGDay FOSDEM 2026**: Focused on AI-assisted workflows in PostgreSQL development, with a technical talk by Matt Cornillon. - **PGConf Belgium 2026**: The session was selected as supplementary material for a database exam, indicating student engagement. - **Nordic PG Day 2026**: Google participated as a Partner-level sponsor and hosted a dedicated booth. - **Swiss PGDay 2026**: Featured a demonstration on processing vectors in PostgreSQL. - **Postgres Conference 2026 San Jose**: Google sponsored the event, with Vikas Arora discussing PostgreSQL adaptations for AI workloads. Community leadership roles included: - Dilip Kumar on the Program Committee for PGConf.dev 2026 and the Paper Selection Committee for PGConf India 2026. - Matt Cornillon on the organization committee for PGDay France, and Yves Colin on the Program Committee. Acknowledgment was given to various contributors for their dedication to PostgreSQL conferences.

Tech Optimizer

July 9, 2026

DBA2: Configuration and Monitoring of PostgreSQL 16 is now available in English

Postgres Professional has translated the DBA2: Configuration and Monitoring of PostgreSQL 16 course into English. This four-day program is designed for PostgreSQL administrators with basic Unix skills and knowledge equivalent to the DBA1 course. The curriculum covers topics such as Multi-Version Concurrency Control (MVCC), isolation levels, vacuuming, buffer cache management, write-ahead logging (WAL), and server upgrades. Participants will learn to configure PostgreSQL parameters, monitor the server, manage localization settings, and handle extensions. The course materials include a student guide for setting up a virtual machine and performing practical tasks. The translation was done by Elena Sharafutdinova, with assistance from Ilya Bashtanov and Alexander Meleshko.

Tech Optimizer

July 8, 2026

PostgreSQL Extensions Are an Architecture Decision, Not an Afterthought

A common issue in database migrations is the unplanned accumulation of extensions in PostgreSQL, leading to extension sprawl. Teams often install extensions without documenting the rationale, resulting in a complex web of dependencies that complicates future upgrades and removals. The installation process involves PostgreSQL accessing a control file that details the extension's version and dependencies, which can lead to multiple extensions being installed unintentionally. Upgrading and removing extensions are often neglected, causing risks such as the loss of dependent objects. Extensions typically default to the public schema, which can become cluttered; relocating them to dedicated schemas can improve organization. Trusted Language Extensions (TLE) allow non-privileged users to utilize procedural languages in managed databases without needing superuser access. Key extensions recommended for use include pg_stat_statements, pg_trgm, hstore, citext, and PostGIS, each serving specific use cases. Proper governance is essential for managing extensions, including documenting their purpose, ownership, and dependencies, to prevent operational surprises.