team

The article discusses a serverless pipeline for creating an irreversibly redacted and queryable archive of Amazon RDS for PostgreSQL audit logs. It sanitizes sensitive information, such as Social Security numbers, credit card details, and email addresses, before storing the clean logs in Amazon S3, which can be queried using Amazon Athena. The deployment process is facilitated by a single AWS CloudFormation template. The challenge is that pgAudit logs sensitive data in cleartext to Amazon CloudWatch Logs, which poses compliance risks under regulations like GDPR and HIPAA. While CloudWatch Logs Data Protection offers reversible masking of PII, this is insufficient for compliance needs requiring permanent separation of PII from audit evidence. The proposed pipeline creates an irreversibly redacted copy stored in Amazon S3, ensuring original values cannot be restored. The deployment requires an active AWS account with specific permissions, AWS CLI v2, a compatible AWS region, and confirmation of service quotas. The CloudFormation template has four main parameters for deployment, including ProjectName, DBInstanceClass, DBAllocatedStorage, and EnvironmentType, with sensible defaults provided. Three deployment options are available: via the AWS console, AWS CLI, or a deploy script. After deployment, users can generate test data, trigger the redaction pipeline, and verify results in Athena. The process includes steps to create a PostgreSQL table, insert records with PII, run queries, and monitor the execution of the redaction pipeline. The article emphasizes that unredacted PII remains in the source CloudWatch Logs, recommending enabling CloudWatch Logs Data Protection for added security. To clean up resources and avoid charges, users must empty the S3 buckets and delete the stack. Cost considerations for the pipeline include infrastructure costs and the expense of achieving irreversibility.