Creating Cluster Instance triggers SecretsManager update

[j-fulbright]

Describe what happened

When using aws.rds.Cluster with the option for manageMasterUserPassword enabled, a secret gets created in Secrets Manager. When creating instances that will be attached to that cluster utilizing the aws.rds.ClusterInstance component, this appears to trigger an update to the secret created by the cluster.

There is no option to control that within ClusterInstance.

This is resulting in us retrieving credentials incorrectly, as they're being changed in mid-deploy by the instances. We pull out from the Secrets Manager, but it doesn't seem to matter when we pull them, it is always the value the cluster set.

This was found while setting up a command to disable auto-rotation for the time being and realizing it kept re-enabling itself.

Without some way to control this on ClusterInstance, I'm not sure how to stop it from happening or to make sure we retrieve the correct secrets.

Sample program

const clusterArgs = {
            clusterIdentifier: `db-${rdsStackName}-${stack}`,
            engineMode: aws.rds.EngineMode.Provisioned,
            engine: aws.rds.EngineType.AuroraPostgresql,
            engineVersion: '16.6', // Changing this will result in outages
            engineLifecycleSupport: 'open-source-rds-extended-support-disabled',

            masterUsername: 'masteruser',
            manageMasterUserPassword: true,

            // VCPU
            serverlessv2ScalingConfiguration: {
                minCapacity: args.minCpuCapacity || 1,
                maxCapacity: args.maxCpuCapacity || 2,
            },
            skipFinalSnapshot: true,
            vpcSecurityGroupIds: [args.securityGroup.id],
            dbSubnetGroupName: dbSubnetGroup.name,

            //
            deletionProtection: isProd ? true : false,
}        

const cluster = new aws.rds.Cluster(`db-our-stack`, clusterArgs, {
          parent: this,
 });

            const instance = new aws.rds.ClusterInstance(
                instanceName,
                {
                    identifier: instanceName,
                    clusterIdentifier: cluster.id,
                    instanceClass: 'db.serverless',
                    engine: cluster.engine.apply((engine) => engine as aws.rds.EngineType),
                    dbSubnetGroupName: cluster.dbSubnetGroupName,
                    publiclyAccessible: false,
                    autoMinorVersionUpgrade: false,
                },
                {
                    parent: cluster,
                    provider: opts?.provider,
                },
            );

Log output

No response

Affected Resource(s)

aws.rds.Cluster
aws.rds.ClusterInstance

Output of pulumi about

CLI
Version 3.145.0
Go Version go1.23.4
Go Compiler gc

Plugins
KIND NAME VERSION
resource aws 6.66.2
resource aws-native 1.22.0
resource awsx 2.19.0
resource command 1.0.1
resource docker 4.5.8
resource docker 3.6.1
language nodejs 3.145.0

Host
OS darwin
Version 15.2
Arch arm64

Dependencies:
NAME VERSION
@pulumi/aws 6.66.2
@pulumi/awsx 2.19.0
@pulumi/command 1.0.1
@pulumi/pulumi 3.145.0
@eslint/js 9.17.0
@types/node 22.9.1
eslint 9.17.0
globals 15.14.0
typescript 5.5.4
typescript-eslint 8.19.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[j-fulbright]

Reviewing of the upstream Terraform seems to indicate that this is expected for instances that are not considered replicas.. And I am not seeing a variable for this exposed

[flostadler]

Hey @j-fulbright, AWS RDS runs an initial rotation when the first instance is created. There's no option to configure this on ClusterInstance because the secret is managed on the cluster level for Aurora.

There's nothing we can do about this on the Pulumi side, but it should be possible to retrieve the correct secret by retrieving the secret after the ClusterInstance has been created.
How are you retrieving the secret right now? You would need to make the retrieval depend on the ClusterInstance to retrieve the correct secret version.
This could look this, in my case instance and instanceReader are my two ClusterInstances:

aws.secretsmanager.getSecretVersionOutput({
        secretId: "SECRET_ID",
    }, { dependsOn: [instance, instanceReader] });

Note: Beware that the screenshots you've posted are not from the upstream terraform provider, but instead from an Terraform module. While this can give good indications for how to use the provider (e.g. by translating the terraform code to pulumi, see), it doesn't speak for the implementation of the provider itself.

[flostadler]

Here's a full example I used for testing this:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
    numberOfAvailabilityZones: 2,
});

const dbSubnetGroup = new aws.rds.SubnetGroup(`flo-aurora-subnet-group`, {
    subnetIds: vpc.privateSubnetIds,
});

const cluster = new aws.rds.Cluster(`flo-aurora-cluster`, {
    clusterIdentifier: `flo-aurora-cluster`,
    engineMode: aws.rds.EngineMode.Provisioned,
    engine: aws.rds.EngineType.AuroraPostgresql,
    engineVersion: '16.6',
    engineLifecycleSupport: 'open-source-rds-extended-support-disabled',

    masterUsername: 'masteruser',
    manageMasterUserPassword: true,

    // VCPU
    serverlessv2ScalingConfiguration: {
        minCapacity: 1,
        maxCapacity: 2,
    },
    skipFinalSnapshot: true,
    dbSubnetGroupName: dbSubnetGroup.name,
});

const instance = new aws.rds.ClusterInstance(
    `flo-aurora-instance`,
    {
        identifier: "flo-aurora-instance",
        clusterIdentifier: cluster.id,
        instanceClass: 'db.serverless',
        engine: cluster.engine.apply((engine) => engine as aws.rds.EngineType),
        dbSubnetGroupName: cluster.dbSubnetGroupName,
        publiclyAccessible: false,
        autoMinorVersionUpgrade: false,
    }
);

const instanceReader = new aws.rds.ClusterInstance(
    `flo-aurora-instance-reader`,
    {
        identifier: "flo-aurora-instance-reader",
        clusterIdentifier: cluster.id,
        instanceClass: 'db.serverless',
        engine: cluster.engine.apply((engine) => engine as aws.rds.EngineType),
        dbSubnetGroupName: cluster.dbSubnetGroupName,
        publiclyAccessible: false,
        autoMinorVersionUpgrade: false,
    }
);

export const masterUserSecretArn = cluster.masterUserSecrets.apply((secret) => secret[0].secretArn);
const secretId = masterUserSecretArn.apply((secret) => {
    // Extract the secret ID by:
    // 1. Split by ":" and take the last part
    // 2. Split that by "-", remove the last segment, and rejoin with "-"
    const parts = secret.split(':').pop()?.split('-') || [];
    return parts.slice(0, -1).join('-');
});

export const dbPassword = pulumi.secret(aws.secretsmanager.getSecretVersionOutput({
    secretId: secretId,
}, { dependsOn: [instance, instanceReader] }).secretString);

[j-fulbright]

@flostadler I will look at your example and see how it flows accordingly for us. I will update afterwards

[j-fulbright]

@flostadler I think i am going to go ahead with closure here. I've reworked our code to better flow since we were trying to do some asynchronous stuff that didn't seem to work well, and likely was to blame.