PostgreSQL Archives

Tech Optimizer

July 24, 2026

How we pushed CDC into Postgres, and turned replication into clockwork

Making data from transactional databases accessible to analytical databases is essential in modern data architecture, but it faces challenges such as fragile tooling, high costs, and complex operations. Snowflake has developed a Postgres service that addresses these issues by reimagining Postgres replication. Postgres is a strong operational database, but its change data capture (CDC) capabilities need improvement. Many data pipelines are fragile due to complexities in managing continuous data flow, schema changes, snapshots, and failures. To enhance user experience with Snowflake Postgres, a complete reinvention of the replication process was necessary. A new feature called data mirroring is in public preview, allowing resilient data replication into Snowflake with low cost, minimal lag, and transactional consistency. This feature directly pushes changes from Postgres into Apache Iceberg™ tables in transactional batches, which are then automatically applied to Snowflake tables without extra infrastructure. Change data capture records changes from a transactional database for replay on another system, primarily using logical decoding in Postgres. This method decodes WAL records into logical operations, but the client bears the responsibility for subsequent steps like backfilling, managing schema changes, and handling failures. Built-in logical replication in Postgres only addresses some complexities. A limitation of logical decoding is that the external system does not recognize the state of Postgres, making it difficult to detect schema changes or correlate snapshots with changes. The solution is to push changes from Postgres into a data lake, specifically into Iceberg tables using compressed Parquet format, with object stores like Amazon S3 as the destination. The mirroring process uses a new Postgres extension called snowflake_cdc, which continuously pushes batches of changes into per-table change logs and a meta log. This extension understands the Postgres environment, coordinating schema changes and complex transactions while ensuring alignment between snapshots and changes.

Tech Optimizer

July 23, 2026

Optimizing PostgreSQL on Azure directly in Visual Studio Code

Poor database performance can lead to missed SLAs, delayed releases, customer dissatisfaction, and lost revenue. Microsoft has enhanced PostgreSQL on Azure, transforming it into a fully managed platform that meets enterprise demands. Azure Database for PostgreSQL and the new Azure HorizonDB offer significant performance improvements, with HorizonDB being three times faster than self-managed PostgreSQL. The PostgreSQL extension for Visual Studio Code integrates performance management into development workflows, providing tools for managing PostgreSQL throughout its lifecycle. Key features include a server metrics dashboard that displays performance indicators, Azure Advisor recommendations for actionable insights, improved query plan visualization, and AI-assisted query analysis. The extension also enhances schema design and query formulation experiences, ensuring secure and governed interactions with PostgreSQL. Azure HorizonDB is in public preview, designed for AI-native workloads. These advancements aim to reduce friction, enhance clarity, and enable faster actions for enterprise teams managing PostgreSQL at scale.

Tech Optimizer

July 23, 2026

Simplify AI agent orchestration with Lakebase Postgres

The auditing process has traditionally been labor-intensive, requiring extensive document reviews. CLA (CliftonLarsonAllen LLP) partnered with the Databricks Forward Deployed Engineering team to develop a document processing application that reduces extraction time from hours to minutes while maintaining quality. The application is built on the Databricks platform, utilizing technologies like Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. The application addresses challenges in document parsing, including unpredictable per-task latency, rate-limit-aware throttling, workload prioritization, cost attribution per task, and real-time progress visibility. It eliminates the need for multiple specialized systems by integrating components such as a web application for user interaction, Lakebase for state management, an orchestrator for task management, and AI agents for document processing. The task queue is supported by two Postgres tables in Lakebase, tracking task status and execution attempts. Concurrency safety is ensured through a locking mechanism, and crash recovery is facilitated by lease-based locking. The orchestrator implements rate-limit-aware throttling and idempotent webhook callbacks to manage task processing effectively. An operator dashboard provides real-time metrics on task status, agent performance, and workload costs, with live updates through Postgres LISTEN/NOTIFY events. Cost attribution is managed at the application level, allowing for detailed monitoring of expenditures. Lakebase enhances the orchestration process with features like autoscaling compute, OAuth-rotated authentication, Unity Catalog integration, and support for branching and snapshots.

Tech Optimizer

July 22, 2026

Migrating mission-critical payments at Nubank to Amazon Aurora PostgreSQL

Nubank, a digital banking platform with approximately 135 million customers in Brazil, Mexico, and Colombia, faced challenges with its payment infrastructure due to inefficiencies in managing 7.5 TB of self-hosted PostgreSQL databases. Queries took over 13 minutes to execute, and replication lags exceeded four minutes during peak times. To address these issues, Nubank evaluated database solutions based on developer productivity, operational efficiency, performance reliability, and scalability. They selected Amazon Aurora PostgreSQL-Compatible Edition for its compatibility, performance improvements, and automated features. The migration to Aurora was facilitated by AWS Database Migration Service (AWS DMS), which minimized downtime and allowed for efficient transition. Post-migration, query performance improved significantly, with some queries executing up to 1,900 times faster, and overall end-to-end service latency decreased, enhancing customer experience. The migration resulted in a 25 percent cost reduction and ensured compliance with regulatory requirements.

Tech Optimizer

July 22, 2026

Google boosts AlloyDB vector search with HNSW preview

Google has introduced a preview of its columnar engine-accelerated HNSW feature for AlloyDB, enhancing vector search throughput by up to four times for users of the pgvector extension in PostgreSQL. This improvement is aimed at approximate nearest neighbor searches using HNSW across large datasets. AlloyDB utilizes a columnar engine that allows HNSW indexes to reside in memory, improving throughput and recall metrics. In benchmark tests on the GloVe 100 Angular dataset, queries per second increased by approximately 4.2x to 4.9x, with recall improving from about 0.78 to over 0.94 at a throughput of around 350 queries per second. The columnar engine's design bypasses traditional PostgreSQL buffer management bottlenecks, enhancing efficiency during graph traversal. The memory architecture is distinct from basic caching techniques, and the feature requires no changes to application code. Users must enable the columnar engine and index caching flags to utilize this feature. Benchmarks were conducted on an AlloyDB C4A machine with 16 virtual CPUs.

Tech Optimizer

July 21, 2026

I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer

The author transitioned from a data analyst role to a data engineer, creating a 12-month self-study roadmap focused on learning by building projects. The first project was a GitHub ETL pipeline, which evolved from a simple Python script to a more complex system using SQLite and GitHub Actions for automation. The author realized that the challenges of data engineering lay in system design and orchestration rather than just writing ETL logic. For the second project, the author built an automated RSS ingestion pipeline to explore engineering decisions in creating a reliable data pipeline. The project emphasized the importance of separating application responsibilities from orchestration tasks, leading to the use of Docker for consistent execution environments and Kestra for orchestration. The author validated each component of the project incrementally: first the Python ETL, then PostgreSQL, followed by Docker, and finally Kestra. This approach ensured that each layer functioned correctly before adding complexity. The final architecture included distinct responsibilities for each component: Python for data processing, PostgreSQL for data storage, Docker for execution, and Kestra for orchestration. The author learned that effective engineering involves considering system reliability, error handling, and observability, shifting the focus from merely writing code to designing entire systems. The experience transformed the author's mindset, emphasizing the importance of incremental building and the separation of execution and orchestration responsibilities.

Tech Optimizer

July 18, 2026

North Korea Buried Four-Stage Malware in Flag Images: Zero Antivirus Detections

North Korea's Contagious Interview hackers have been using a deceptive strategy to target developers by posing as recruiters and embedding malware in SVG files. Elastic Security Labs discovered that the attackers hid malicious payloads within HTML comment blocks of these files, allowing the malware to evade antivirus detection. At the time of the findings, no antivirus engines flagged the compromised repositories, which included trojanized GitHub repositories disguised as coding challenges. The malware executed automatically at server startup and deployed four modules: a browser credential and cryptocurrency wallet stealer, a file stealer, a remote access Trojan, and a clipboard monitor. The campaign, tracked as REF9403, is part of the ongoing Contagious Interview operation attributed to North Korea's Lazarus Group, which aims to generate revenue through cryptocurrency theft. Developers are advised to audit any projects run from unsolicited sources and to monitor specific domains associated with the attack.

Tech Optimizer

July 16, 2026

Google Cloud boosts AlloyDB search for CJK text with Gemini

Google Cloud has enhanced AlloyDB by integrating AlloyDB AI Functions with Gemini models to improve search indexing for logographic languages like Chinese, Japanese, and Korean. This update addresses challenges in database search functionality, particularly for languages written in continuous scripts, where traditional PostgreSQL text parsing struggles. Developers can now invoke Gemini models directly from SQL to facilitate word segmentation and stop-word removal before creating full-text indexes, offering a more efficient alternative to traditional methods. The architecture maintains raw text, segmented text, search vectors, and embeddings in a single table, with generated columns ensuring automatic updates to search vectors and embeddings. A batching process using a PL/pgSQL stored procedure allows for efficient processing of extensive document collections by aggregating rows into arrays for model requests, thus reducing row-locking issues. Google Cloud provides tailored text search configurations for different datasets, suggesting a simple configuration for solely Chinese text and an English configuration for bilingual datasets. This preprocessing logic can also be applied during query execution, allowing user searches to be directed to Gemini for text segmentation and stop-word elimination. AlloyDB supports RUM indexes on search vectors, which store lexeme positions for computing relevance and word distance, enabling rapid query responses. Additionally, a hybrid search methodology combines full-text and vector search using a ScaNN index on embeddings, merging results through Reciprocal Rank Fusion. This development allows for robust multilingual search and vector retrieval solutions within the database environment, eliminating the need for external microservices.

Tech Optimizer

July 12, 2026

What Is Serverless PostgreSQL?

Serverless PostgreSQL is a fully managed cloud database model that separates compute and storage, allowing them to scale independently and automatically based on demand. It eliminates the need for manual infrastructure provisioning and capacity planning, charging only for active usage. Unlike traditional PostgreSQL setups, which require continuous resource allocation and manual scaling, serverless PostgreSQL provisions resources on demand and can scale down to zero during idle periods. Serverless PostgreSQL integrates with serverless compute platforms, enabling analytical queries to access the same data within a unified architecture. Key differences between traditional and serverless PostgreSQL include manual versus automatic provisioning and scaling, fixed versus usage-based billing, and high versus reduced operational overhead. Lakebase architecture is an emerging model that combines transactional databases with lakehouse foundations, allowing operational and analytical workloads to coexist on a single platform. This architecture minimizes data duplication and simplifies access, enhancing data management and analysis. Serverless PostgreSQL operates on a cloud-native architecture that enhances efficiency by allowing compute and storage to scale autonomously. It features scale-to-zero behavior, where compute resources are suspended when inactive and reactivated upon new queries. Major providers include Databricks Lakebase, Amazon Aurora Serverless v2, and Neon, each offering varying capabilities and integrations. Pricing for serverless PostgreSQL typically includes charges for compute resources, storage, and data transfer, with costs fluctuating based on workload activity. Cold start latency is a performance consideration, as reactivating compute resources can introduce delays. Strategies to mitigate this include keeping resources partially active or selecting providers with minimal cold start impacts. Serverless PostgreSQL is well-suited for OLTP workloads, while lakebase architecture is better for AI development, variable workloads, and environments requiring rapid iteration. Setting up serverless PostgreSQL involves choosing a provider, creating a database instance, and configuring access settings. It can also be used alongside serverless compute platforms for analytics, further extending its capabilities.

Tech Optimizer

July 12, 2026

Running pgvector in production on Amazon Aurora PostgreSQL

Running pgvector on Amazon Aurora PostgreSQL-Compatible Edition offers a vector store with operational capabilities, high availability, and scalability. It is favored for Retrieval Augmented Generation (RAG) workloads transitioning to production, but increased traffic introduces challenges like query latency and memory management. Key operational practices for pgvector workloads include selecting the appropriate index type (HNSW or IVFFlat), establishing a baseline schema, choosing a suitable distance operator, scaling the index through quantization and partitioning, and preparing for churn and observability. The prerequisites for using pgvector include an Aurora PostgreSQL-Compatible cluster with specific PostgreSQL versions and the vector extension enabled. The embedding model used in examples is Amazon Titan Text Embeddings V2, which produces 1024-dimensional embeddings. pgvector supports two Approximate Nearest Neighbor (ANN) index types: HNSW, which is efficient for querying and allows for incremental insertions, and IVFFlat, which is less resource-intensive but requires rebuilding if data changes. There are scenarios where forgoing an index is beneficial, such as small datasets or partitioned datasets requiring 100% recall. A baseline schema for a multi-tenant document store includes creating a table for documents with an embedding vector and establishing indexes for tenant IDs and embeddings using HNSW. The recommended parameters for HNSW include m = 16 and ef_construction = 128. Scaling to millions of vectors involves quantization, tuning HNSW parameters, and partitioning. Aurora Optimized Reads can extend effective cache capacity, and managing index churn is crucial for maintaining performance. Observability metrics include query-level statistics, instance-level metrics, and custom application-defined metrics. To clean up after testing, it is advisable to drop the created indexes and tables, and delete the Aurora PostgreSQL-Compatible cluster and any manual snapshots taken during testing.