Connecting .NET Lambda to Amazon Aurora PostgreSQL via RDS Proxy

Many organizations are currently navigating the complexities of modernizing legacy .NET applications while simultaneously reducing database costs and enhancing scalability. This dual challenge often arises during migrations to AWS Lambda and transitions from SQL Server to Amazon Aurora PostgreSQL-Compatible Edition. This article delves into the process of connecting Lambda functions to Aurora PostgreSQL using Amazon RDS Proxy. We will explore the configuration of AWS Secrets Manager, the setup of RDS Proxy, and the creation of a C# Lambda function that incorporates secure credential caching. A GitHub repository is provided, containing a YAML-format AWS CloudFormation template for provisioning the essential components, alongside a C# sample function. A step-by-step guide for deploying the Lambda function is also included.

Solution overview

The architecture diagram illustrates a .NET Lambda function interfacing with Aurora PostgreSQL via RDS Proxy, with AWS Secrets Manager managing secure credential storage.

AWS Lambda with RDS Proxy and Aurora PostgreSQL Architecture

When establishing connections between Lambda functions and Aurora PostgreSQL, RDS Proxy plays a crucial role by providing connection pooling, which is essential for handling serverless scaling patterns and minimizing database connection overhead. Credentials for the database are securely stored in AWS Secrets Manager. For production scenarios, it is advisable to enable automatic credential rotation to enhance security without necessitating manual updates.

In this example, the architecture is simplified by situating Lambda, RDS Proxy, and Aurora PostgreSQL within the same VPC private subnet. In a production environment, separating these components across multiple subnets may offer better isolation and security.

The solution leverages Npgsql, an open-source .NET data provider, to facilitate interaction between the Lambda-based C# application and PostgreSQL databases. The connection flow initiates with the Lambda function linking through RDS Proxy, which manages connections to Aurora PostgreSQL and retrieves credentials from Secrets Manager via a configured VPC endpoint.

Prerequisites

This walkthrough utilizes a CloudFormation template to create all necessary resources. The template provisions:

  • VPC with private subnets
  • Aurora PostgreSQL cluster
  • RDS Proxy: blog-rds-proxy
  • Secrets Manager VPC endpoint
  • Two secrets in AWS Secrets Manager:
    • blog-apg-secret
    • blog-rds-proxy-secret
  • Windows EC2 instance (t3.medium with Microsoft Windows Server 2022 English Full Base AMI)
  • IAM roles and supporting resources

The template organizes these resources within a single stack to prevent conflicts with existing resources and to streamline maintenance.

The RDS Proxy is configured with the following parameters:

  • Engine family: PostgreSQL
  • Idle client connection timeout: 1800 seconds
  • IAM authentication: Disabled
  • Secrets Manager secret: blog-apg-secret
  • Require Transport Layer Security: Enabled
  • Enhanced logging: Disabled

Note: In this walkthrough, IAM authentication is disabled on the RDS Proxy for simplicity. The proxy authenticates to Aurora PostgreSQL using credentials stored in AWS Secrets Manager. In production environments, enabling IAM authentication is recommended for centralized management of database access through IAM policies, rather than relying solely on database credentials.

For detailed configuration options, refer to the AWS RDS Proxy documentation.

For the Windows EC2 instance, you need to specify one parameter:

  • Password for the Windows Administrator account (8-32 characters with letters, numbers, and symbols)

    Note: When prompted for the Windows Administrator password during stack creation, choose a strong password and store it securely. This password is defined as a NoEcho parameter, ensuring it is masked in the AWS CloudFormation console and API responses, visible only to administrators with direct access to the EC2 instance.
  • Latest AMI ID. (No need to input it; you can leave it as default)

The template provisions a Windows EC2 instance with the following pre-installed components:

To connect to the underlying Windows EC2 instance, I will utilize Fleet Manager Remote Desktop. The CloudFormation template configures the AWS Systems Manager (SSM) Agent, allowing direct access to the instance via AWS Systems Manager Fleet Manager, even though it resides in a private subnet.

  1. Select the EC2 instance and click Connect.
  2. Select the RDP client tab, then choose Connecting using Fleet Manager, and select Fleet Manager Remote Desktop.
    Note: To optimize costs, the CloudFormation template employs a t3.medium instance type. Since Fleet Manager Remote Desktop requires the SSM Agent to be fully operational and registered with AWS Systems Manager, you may need to wait 10–20 minutes after instance launch before the “Connect using Fleet Manager” option becomes available.
  3. Enter the following credentials:
    • Username: Administrator
    • Password: ******* (This password can be found in the Outputs section of the CloudFormation template)
  4. Click Connect to log in to the Windows EC2 instance.

To test the solution, you will need to create a sample table named employee and populate it with test data. Follow these steps:

  1. Connect to Windows EC2 using Fleet Manager
  2. Open a Command Prompt window and execute the following command to retrieve the database connection information from the blog-rds-proxy-secret in AWS Secrets Manager:
    aws secretsmanager get-secret-value --secret-id blog-rds-proxy-secret
  3. Utilize pgAdmin on the Windows EC2 instance to connect to the Aurora PostgreSQL database.
  4. Execute the following SQL statements in the public schema:
    CREATE TABLE employee (
        emp_id SERIAL PRIMARY KEY, 
        first_name VARCHAR(50) NOT NULL, 
        last_name VARCHAR(50) NOT NULL,  
        job_title VARCHAR(50),           
        salary NUMERIC(10, 2),           
        hire_date DATE DEFAULT CURRENT_DATE 
    );
    INSERT INTO employee (first_name, last_name, job_title, salary) 
    VALUES 
        ('Alice', 'Smith', 'Software Engineer', 75000.00),
        ('Bob', 'Johnson', 'Data Analyst', 65000.00);
    SELECT * FROM employee;

Create a new Lambda project

A new .NET project can be created using the .NET Lambda Global CLI on the provisioned Windows EC2 instance. The .NET Lambda Global Tools extension (Amazon.Lambda.Tools), included with the .NET CLI, offers a cross-system solution for creating, packaging, and deploying .NET-based Lambda functions.

  1. Open Command Prompt on the Windows EC2 instance, and execute the following commands to create a new Lambda project:
    mkdir C:MyApp
    cd C:MyApp
    dotnet new lambda.EmptyFunction --name demo-CS-LambdaDbFunc
    cd demo-CS-LambdaDbFunc/src/demo-CS-LambdaDbFunc/
    dotnet add package Npgsql
    dotnet add package Newtonsoft.Json
    dotnet add package AWSSDK.SecretsManager.Caching --version 1.0.6
  2. Review the source code.

Upon creating a new Lambda function project, add the following packages:

  • Npgsql: Facilitates connections to PostgreSQL databases.
  • Newtonsoft.Json: A widely-used JSON framework for .NET, supporting serialization and deserialization to convert .NET objects to JSON format and vice versa.

The following code snippet demonstrates the use of the AWS Secrets Manager client-side caching library to enhance application availability, reduce latency, and lower the cost of retrieving secrets:

private SecretsManagerCache cache = new SecretsManagerCache();
public SecretsManagerService()
{
    var client = new AmazonSecretsManagerClient();
    cache = new SecretsManagerCache(client);
}   // Retrieves the secret containing the username and password
public async Task GetCredentialsAsync(string secretName)
{
    try
    {
        string secretString = await cache.GetSecretString(secretName);
        var secret = JsonConvert.DeserializeObject(secretString);
        return (secret.host, secret.port, secret.dbname, secret.username, secret.password);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error retrieving secret: {ex.Message}");
        throw;
    }
}

The complete C# source code can be found in Function.cs on GitHub.

  1. Deploy the project to Lambda.

Since the IAM user blog-user, provisioned by the CloudFormation template, has been granted the necessary permissions, executing the command dotnet lambda deploy-function will present a list of available IAM roles. Select an IAM role for deploying the Lambda C# code to AWS, such as xxxx-BlogLambdaRole-YYYY.

dotnet lambda deploy-function demo-CS-LambdaDbFunc
...
Select IAM Role that to provide AWS credentials to your code:
    1) XXXX-BlogLambdaRole-YYYY
    2) *
1
New Lambda function created

Configure the new Lambda function

The Lambda function is created and deployed via the AWS Lambda project with the .NET Core CLI, with final configuration performed through the AWS CLI:

aws lambda update-function-configuration ^
    --function-name demo-CS-LambdaDbFunc ^
    --vpc-config "SubnetIds=,,,SecurityGroupIds="

The Physical IDs can be found mapped to the following Logical IDs on the Resources tab of the CloudFormation stack:

  • BlogPrivateSubnet1
  • BlogPrivateSubnet2
  • BlogPrivateSubnet3
  • BlogLambdaSG

Test the Lambda function

  1. Invoke the Lambda function:
aws lambda invoke ^
  --function-name demo-CS-LambdaDbFunc ^
  --payload "{"key1": "blog-rds-proxy-secret"}" ^
  --cli-binary-format raw-in-base64-out ^
  response.json
  1. Retrieve the log stream name:
aws logs describe-log-streams ^
  --log-group-name "/aws/lambda/demo-CS-LambdaDbFunc" ^
  --order-by LastEventTime ^
  --descending ^
  --max-items 1 ^
  --query "logStreams[0].logStreamName" ^
  --output text
  1. Utilize the log stream name to display the Amazon CloudWatch log for the test result:
aws logs get-log-events ^
  --log-group-name "/aws/lambda/demo-CS-LambdaDbFunc" ^
  --log-stream-name "YOUR_STREAM_NAME" ^
  --query "events[*].message" ^
  --output text

CloudWatch Log Result

Clean up

To prevent ongoing charges, it is essential to delete the resources created during this process:

  1. Delete the Lambda function:
    CREATE TABLE employee (
        emp_id SERIAL PRIMARY KEY, 
        first_name VARCHAR(50) NOT NULL, 
        last_name VARCHAR(50) NOT NULL,  
        job_title VARCHAR(50),           
        salary NUMERIC(10, 2),           
        hire_date DATE DEFAULT CURRENT_DATE 
    );
    INSERT INTO employee (first_name, last_name, job_title, salary) 
    VALUES 
        ('Alice', 'Smith', 'Software Engineer', 75000.00),
        ('Bob', 'Johnson', 'Data Analyst', 65000.00);
    SELECT * FROM employee;

    Note: Wait a few minutes before proceeding, as Lambda ENIs can take some time to fully detach after function deletion.

  2. Delete the CloudFormation stack:
    • Navigate to the CloudFormation console.
    • Select and delete the associated stack.
    • Note: Wait for complete stack deletion before proceeding with other tasks.
  3. (Optional) Remove Elastic Network Interfaces (ENIs):

    If the CloudFormation stack was not fully deleted, check for and manually remove any remaining Lambda-associated ENIs:

    • Navigate to the EC2 console and select Network Interfaces.
    • Filter by Interface Type: lambda
    • Delete any remaining ENIs associated with your Lambda function.

Important: Always verify that all resources are properly deleted to avoid unexpected charges.

Tech Optimizer
Connecting .NET Lambda to Amazon Aurora PostgreSQL via RDS Proxy