Amazon Web Services Inc. has introduced a significant enhancement to its Aurora PostgreSQL database management system, enabling users to directly query the widely adopted Apache Iceberg data lake. This innovative feature allows applications to seamlessly integrate live transactions with historical records, all without the need for data duplication or the complexities of extract/transform/load (ETL) pipelines.
Integration of DuckDB Analytical Engine
At the heart of this capability is the embedded DuckDB analytical engine, which supports data stored in both Iceberg and Apache Parquet formats. Customers can leverage their existing PostgreSQL applications, tools, and endpoints to query operational records alongside data residing in Amazon S3, including S3 Tables. This integration follows AWS’s recent acquisition of DuckLabs B.V., the creators of the open-source database, further solidifying AWS’s commitment to enhancing data analytics capabilities.
AWS positions this feature as a means to streamline application development, significantly reducing the engineering workload typically associated with maintaining data pipelines. Potential applications include real-time dashboards, enriched transaction records, and AI agents that require access to both current and archived data.
Previously, merging recent transactions in Aurora with historical records stored in S3 necessitated the use of reverse ETL pipelines, which often led to data duplication, increased infrastructure costs, and ongoing synchronization challenges. As Esra Kayabali, a principal solutions architect at AWS, noted, “This challenge only grows as you increasingly embed AI agents into your applications, where it is impractical to predict and pre-replicate every dataset an agent might need.”
Efficient Query Processing
DuckDB’s ability to process analytical scans directly within Aurora eliminates the need for additional network hops during query processing. This allows a single query to access both data lake records and live operational data, including uncommitted writes. AWS emphasizes that this integration exemplifies its broader strategy to incorporate the DuckDB engine across its services.
The feature also supports external catalogs that comply with the Iceberg Representational State Transfer Catalog specification. Through federation with the AWS Glue serverless data integration service, customers can register an external catalog with Glue and create foreign tables that reference its data. This functionality enables applications to join Aurora records with Iceberg tables registered across multiple catalogs.
Optimized Data Handling
To enhance performance, Aurora intelligently filters records and selects relevant columns during query execution, while also caching frequently accessed data. Developers can monitor metrics such as rows scanned, bytes read from S3, and cache hits, providing insights into query efficiency.
In a practical demonstration, Kayabali illustrated how a query could combine seven days of customer transactions in Aurora with five years of historical transactions stored in a Parquet file in S3. Aurora automatically inferred the schema of the historical table from file metadata, thereby eliminating the need for manual column definitions.
For workloads demanding single-digit-millisecond latency, customers have the option to copy selected data lake records into native Aurora tables using standard SQL commands. Read queries can be executed on either the cluster’s writer or a read replica, effectively offloading analytical scans from operational workloads. Commands that materialize data are processed on the writer.
To enable this capability, customers must utilize the aurora_analytics extension and an AWS Identity and Access Management role that grants access to S3 and Glue. The feature is compatible with Aurora PostgreSQL versions 17 and 18, starting with versions 17.11 and 18.6, respectively.
AWS has made this feature available across all commercial AWS regions without imposing an additional charge. Customers will incur costs only for the incremental Aurora computing resources consumed by their queries and for S3 requests used to access files.
Photo: AWS
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence, and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity, and more.
- 11.4k+ theCUBE alumni — Connect with over 11,400 tech and business leaders shaping the future through a unique trusted-based network.
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights, and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI, and theCUBE SuperStudios, SiliconANGLE Media operates at the intersection of media, technology, and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach over 15 million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
