RFC: Setting SSM Parameters and Secret Manager Secrets #3040

stephenbawks · 2023-09-04T16:35:53Z

Is this related to an existing feature request or issue?

Which Powertools for AWS Lambda (Python) utility does this relate to?

Parameters

Summary

There are many use cases where the Parameter store is used with PowerTools everywhere. It's an invaluable tool for getting SSM Parameters and the ability to cache them has been fantastic to save time going back to SSM repeatedly. There is one thing that I find to be missing and that is the ability to put SSM parameters as well as the ability to write secrets back to Secrets Manager as well.

Use case

Parameter Store

Adding the function to put_parameter back to SSM with the ability to make sure it's also being cached when it is written back. This should also include the ability to have the overwrite flag as optional. When parameters are being over-written to Parameter store you need to set the overwrite flag because it's set to false by default.

Secrets Manager

Would also like to add the ability to write values to the Secrets Store as well. There are many uses, especially in Lambda where I need to cache/write something secret, (JWT Tokens for example) so that multiple Lambda's are not all going back to get a new token every time a new one is invoked. "Caching" that value in Secret Manager (or as a SecureString in Parameter Store) is a good use case for this.

Proposal

Tasks

Define the API for creating/updating secrets/parameters
Define whether we will cache
Create a PoC to prove the experience.

Parameter Store

For getting values in SSM, Powertools is using get_parameter. I am proposing that we add a new option for put_paramter for the ability to write a value back to a parameter in SSM.

By default, you have to specify the overwrite=true to write a value to a parameter if there is already an existing value. I am wondering if this should be set to true by default or if we should be setting that on every time you need to overwrite a value. When it is written to Parameter store the put_parameter will return a response that includes the new version of the parameter. So in theory there is already the ability to go back versions and it will hold up to 100 of the previous versions before it will start rolling over the oldest.

Secret Manager

Adding the ability to create, set, and update secrets in Secret Manager.

I think create() makes sense for creating a new one.

set() might be translated into boto3 as put_secret_value. The newly created secret would become the latest version of the secret.

update() sounds very similar to set() but could be used for changing the version of the secret that would be retrieved.

update(secret_name="mySuperSecret", secret_value="toomanysecrets")

update(secret_name="mySuperSecret", secret_value="toomanysecrets", stage="development")

get() would be nice to have with the optional way to get different versions of a secret.

generate_random_secret() is also a nice feature for Secrets Manager. This might be a really nice way for people to generate random secrets that can be added to Powertools and saving the need for our fellow Powertools-ers to write custom code to generate random strings or secrets.

Out of scope

Secrets Manager

I think being able to cache that value would be great but I may have to think through if they are updating a secret with a different version.

I have been thinking about how this might eventually be used for versioning around being able to do "canary" or even cutovers to new secrets. But that is probably future work....

Potential challenges

Secrets Manager

When it comes to having the SecretsManager class also cache the new value, I think it would be great but I may have to think through if they are updating a secret with a different version. Do we cache the new version AND the old version if it exists? What would be the behavior around this?

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

This feature request meets Powertools for AWS Lambda (Python) Tenets
Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

The text was updated successfully, but these errors were encountered:

leandrodamascena · 2023-09-04T22:32:37Z

hey @stephenbawks! Congrats on writing your first RFC, this is an incredible work of ideas and solutions, and I really appreciate this work. I will comment on each block you wrote and leave some new ideas.

From here on, I will use `boto3.methodname()` to identify methods in the boto3 SDK and `method()` to identify new methods that will be created according to this RFC.

Parameter Store

For getting values in SSM, Powertools is using get_parameter. I am proposing that we add a new option for put_paramter for the ability to write a value back to a parameter in SSM.

By default, you have to specify the overwrite=true to write a value to a parameter if there is already an existing value. I am wondering if this should be set to true by default or if we should be setting that on every time you need to overwrite a value.

I think we need to define the API interface for the create/put/set parameter. Looking at the boto3.put_parameter() method I see many options that a customer can configure, I suggest we do something like:

def set(name: str, value: str, description: str = None, overwrite: bool = False, kms_key_id: str = None, type: str = "String", tier: str = "Standard", **kwargs)

We can define the type annotations in the PR.

I don't think we should set overwrite to True by default. Doing this we might lead the customer to make an inadvertent mistake and override an existing parameter.

When it is written to Parameter store the put_parameter will return a response that includes the new version of the parameter. So in theory there is already the ability to go back versions and it will hold up to 100 of the previous versions before it will start rolling over the oldest.

I'm curious about this point. Did you add this explanation here to contextualize that we should return the version as it is a standard return from the boto3.put_parameter() API or are you thinking of doing something with it? The boto3.get_parameter() API does not support informing the version you want to retrieve, it will always retrieve the last.one.

Secret Manager

Adding the ability to create, set, and update secrets in Secret Manager.

I think create() makes sense for creating a new one.

set() might be translated into boto3 as put_secret_value. The newly created secret would become the latest version of the secret.
update() sounds very similar to set() but could be used for changing the version of the secret that would be retrieved.

I think create() is ok, but I didn't understand the difference between set() and update(), can you elaborate more, please?

get() would be nice to have with the optional way to get different versions of a secret.

We already have a method to retrieve secrets specifying the version: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/#passing-additional-sdk-arguments

generate_random_secret() is also a nice feature for Secrets Manager. This might be a really nice way for people to generate random secrets that can be added to Powertools and saving the need for our fellow Powertools-ers to write custom code to generate random strings or secrets.

I like this API and used it a lot, but in my tests, it adds about 25ms to the Lambda execution just to create a random password. I don't know if we bring much-added value just by wrapping this method in Powertools for AWS Lambda. I would like your opinion here @sthulb and @heitorlessa.

Cache

After creating/updating a secret and a parameter I think we should cache this for 5 seconds and if the customer uses parameters.get_parameter("param")/parameters.get_secret("secret") this comes from the cache and we avoid a new API call by the client, what do you think?

Next steps

I think we need to create a PoC to validate the experience. In this PoC you don't need to do everything like documentation, testing, and others, just how it will look when a customer uses it. You can take this as an example:

from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools import Logger

logger = Logger()

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    
    endpoint_parameter = parameters.set_parameter(name="/lambda-powertools/endpoint_comments", value="xyz", ....)
    
    logger.info(endpoint_parameter)

Tasks

Define the API for creating/updating secrets/parameters
Define whether we will cache
Create a PoC to prove the experience.

Thank you once more for your dedicated effort in creating this RFC.

sthulb · 2023-09-05T09:53:11Z

I think the set() and update() functions should probably be merged into one unless there's a significant use case for why both exist. Versioning alone could be another flag in set()/update() depending which one stays.

As for generate_random_secret() func, I'd probably avoid implementing this – there's a tonne of use cases that need to be thought about before implementation – perhaps scope for another RFC in the future. Things to think about for this feature:

does it need to be FIPS compliant?
do we care about secret length?
can we just read from /dev/(u)random?
do we need to need to generate to a pattern (letters, numbers, symbols and quantity of each)?

stephenbawks · 2023-09-05T15:25:42Z

@sthulb @leandrodamascena

Yeah, they are pretty close to what they are doing. I think one question I still have is if we are using set() for writing a secret to Secrets Manager, should we also use set() for a similar syntax for SSM Parameter store? In SSM Parameter store, the boto3 command is put_parameter(). I have mixed feelings about Secrets Manager being set() and the SSM Parameter store it would be put().

Wondering if we should make those both set() since people are typically using them interchangeably.

stephenbawks · 2023-09-05T15:30:56Z

I think the set() and update() functions should probably be merged into one unless there's a significant use case for why both exist. Versioning alone could be another flag in set()/update() depending which one stays.

As for generate_random_secret() func, I'd probably avoid implementing this – there's a tonne of use cases that need to be thought about before implementation – perhaps scope for another RFC in the future. Things to think about for this feature:

does it need to be FIPS compliant?

do we care about secret length?

can we just read from /dev/(u)random?

do we need to need to generate to a pattern (letters, numbers, symbols and quantity of each)?

In regards to the random password, those are all optional parameters in the command.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/get_random_password.html

In terms of making a password compliant with certain standards, you could just abstract that all way into a simple ENUM variable.

For example, "Give me a FIPS complaint password" and it is coded with those standards but that may be more than we want to maintain since those do change over time.

aradyaron · 2023-09-07T06:31:12Z

hey @stephenbawks! Congrats on writing your first RFC, this is an incredible work of ideas and solutions, and I really appreciate this work. I will comment on each block you wrote and leave some new ideas.

From here on, I will use boto3.methodname() to identify methods in the boto3 SDK and method() to identify new methods that will be created according to this RFC.

Parameter Store

For getting values in SSM, Powertools is using get_parameter. I am proposing that we add a new option for put_paramter for the ability to write a value back to a parameter in SSM.
By default, you have to specify the overwrite=true to write a value to a parameter if there is already an existing value. I am wondering if this should be set to true by default or if we should be setting that on every time you need to overwrite a value.

I think we need to define the API interface for the create/put/set parameter. Looking at the boto3.put_parameter() method I see many options that a customer can configure, I suggest we do something like:
def set(name: str, value: str, description: str = None, overwrite: bool = False, kms_key_id: str = None, type: str = "String", tier: str = "Standard", **kwargs) 
We can define the type annotations in the PR.

I don't think we should set overwrite to True by default. Doing this we might lead the customer to make an inadvertent mistake and override an existing parameter.

When it is written to Parameter store the put_parameter will return a response that includes the new version of the parameter. So in theory there is already the ability to go back versions and it will hold up to 100 of the previous versions before it will start rolling over the oldest.

I'm curious about this point. Did you add this explanation here to contextualize that we should return the version as it is a standard return from the boto3.put_parameter() API or are you thinking of doing something with it? The boto3.get_parameter() API does not support informing the version you want to retrieve, it will always retrieve the last.one.

Secret Manager

Adding the ability to create, set, and update secrets in Secret Manager.
I think create() makes sense for creating a new one.
set() might be translated into boto3 as put_secret_value. The newly created secret would become the latest version of the secret.
update() sounds very similar to set() but could be used for changing the version of the secret that would be retrieved.

I think create() is ok, but I didn't understand the difference between set() and update(), can you elaborate more, please?

get() would be nice to have with the optional way to get different versions of a secret.

We already have a method to retrieve secrets specifying the version: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/#passing-additional-sdk-arguments

generate_random_secret() is also a nice feature for Secrets Manager. This might be a really nice way for people to generate random secrets that can be added to Powertools and saving the need for our fellow Powertools-ers to write custom code to generate random strings or secrets.

I like this API and used it a lot, but in my tests, it adds about 25ms to the Lambda execution just to create a random password. I don't know if we bring much-added value just by wrapping this method in Powertools for AWS Lambda. I would like your opinion here @sthulb and @heitorlessa.

Cache

After creating/updating a secret and a parameter I think we should cache this for 5 seconds and if the customer uses parameters.get_parameter("param")/parameters.get_secret("secret") this comes from the cache and we avoid a new API call by the client, what do you think?

Next steps

I think we need to create a PoC to validate the experience. In this PoC you don't need to do everything like documentation, testing, and others, just how it will look when a customer uses it. You can take this as an example:
from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools import Logger

logger = Logger()

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    
    endpoint_parameter = parameters.set_parameter(name="/lambda-powertools/endpoint_comments", value="xyz", ....)
    
    logger.info(endpoint_parameter)
Tasks

Define the API for creating/updating secrets/parameters

Define whether we will cache

Create a PoC to prove the experience.

Thank you once more for your dedicated effort in creating this RFC.

@leandrodamascena What do you think of adding '*' in order to enforce key value args, this will help users to avoid mistakes when using this capability

def set(name: str, value: str, *, description: str = None, overwrite: bool = False, kms_key_id: str = None, type: str = "String", tier: str = "Standard", **kwargs)

heitorlessa · 2023-09-11T15:08:58Z

Reviewing tomorrow 👊

heitorlessa · 2023-09-12T07:51:49Z

Firstly, thank you @stephenbawks on your first RFC 🎉

Quick thoughts to complement what others shared earlier:

Ignore higher level functions for now and focus on Provider first. SecretsManager.set() rather than set_secret()
Separate PR to make it super quick to review and add respective docs. The devil is in the details and docs are key to avoid confusion since we still need to agree on overwrite default. Docs-driven-development can enlighten us quickly in the absence of customer feedback.
Customers can use standard library secrets module to generate secrets. get_random_secret is yet another API, full network round trip, and customers setting a secret will highly likely already know the value it should be at runtime (otherwise IaC/Console for safety). It's a can of worms.
Regroup in this RFC after the implementation to agree on whether we need a higher level function and what name that would be. We can shout out to get customer feedback
Agree with @aradyaron on enforcing keyword-arguments for new APIs. Only reason we didn't do it was backwards compatibility and it also makes upgrading harder for people (need to do in waves and carefully).

One of the things I wish we've done in the early design of high-level functions in Parameters was to accept a provider= option, and be super careful about higher-level parameters (decrypt=True) that might be too coupled with existing providers.

For example, If we were doing Parameters today, I wouldn't add a higher-level function for AppConfig like get_app_config. It's easy to get confused with get_parameter.

stephenbawks · 2023-09-14T11:40:53Z

One more thing I have noticed is that the get() on SSM Parameters has the ability to transform objects. This is a transform on objects on the way "out" (get) of SSM, not intended for objects on the way "into" (put/set) the SSM Parameter store.

So I thinking of removing the transform option on set() or at least just letting it default to None at the moment. For SSM there are only three options for putting into a SSM Parameter.

String
StringList
SecureString

String is obviously just a string. StringList is just a comma separated set of string values. Secure String is just how SSM encodes the value on the Parameter Store side.

Another option I can think of is adding another option to the transform that accepts a List that will turn that into a comma-separated string on the way into SSM and possibly convert it back into a List on the way out of SSM?

Wondering what feedback others might have on this.

stephenbawks · 2023-09-15T13:40:56Z

@aradyaron

Here is what I am leaning towards.

def set_parameter(
    name: str,
    value: str,
    description: Optional[str] = None,
    type: Optional[Literal["String", "StringList", "SecureString"]] = "String",
    overwrite: bool = False,
    tier: Optional[Literal["Standard", "Advanced", "Intelligent-Tiering"]] = "Standard",
    max_age: Optional[int] = None,
    kms_key_id: Optional[str] = None,
    **sdk_options,
) -> str:

aradyaron · 2023-09-16T06:53:30Z

@aradyaron

Here is what I am leaning towards.

def set_parameter(
    name: str,
    value: str,
    description: Optional[str] = None,
    type: Optional[Literal["String", "StringList", "SecureString"]] = "String",
    overwrite: bool = False,
    tier: Optional[Literal["Standard", "Advanced", "Intelligent-Tiering"]] = "Standard",
    max_age: Optional[int] = None,
    kms_key_id: Optional[str] = None,
    **sdk_options,
) -> str:

Overall This looks pretty got IMO.

Is there a need for max_age parameter in set parameter?
the get_parameter utilities has a transform ability. Do you think you API should enable the transformation to string from dict/bytes?

I suggest the following API

def set_parameter(
    name: str,
    value: Union[str, dict, bytes],
    *,
    description: Optional[str] = None,
    type: Optional[Literal["String", "StringList", "SecureString"]] = "String",
    overwrite: bool = False,
    tier: Optional[Literal["Standard", "Advanced", "Intelligent-Tiering"]] = "Standard",
    kms_key_id: Optional[str] = None,
    **sdk_options,
) -> str:

heitorlessa · 2023-09-16T17:53:09Z

UX tips - Write a complete example for each parameter type that a customer would read - Be mindful of “Optional[]”, you don’t need it if the parameter can have a default value - Code will be read by others, make that easier and intuitive to read and to write The minute you try writing an example for each parameter type it’ll become easier to realize improvements you can make here, including set vs put_parameter, cache, transform. Readme driven development is king for an ergonomic and accessible API

…

On Sat, 16 Sep 2023 at 08:53, aradyaron ***@***.***> wrote: @aradyaron <https://github.com/aradyaron> Here is what I am leaning towards. def set_parameter( name: str, value: str, description: Optional[str] = None, type: Optional[Literal["String", "StringList", "SecureString"]] = "String", overwrite: bool = False, tier: Optional[Literal["Standard", "Advanced", "Intelligent-Tiering"]] = "Standard", max_age: Optional[int] = None, kms_key_id: Optional[str] = None, **sdk_options, ) -> str: Overall This looks pretty got IMO. - Is there a need for max_age parameter in set parameter? - the get_parameter utilities has a transform ability. Do you think you API should enable the transformation to string from dict/bytes? I suggest the following API def set_parameter( name: str, value: Union[str, dict, bytes], *, description: Optional[str] = None, type: Optional[Literal["String", "StringList", "SecureString"]] = "String", overwrite: bool = False, tier: Optional[Literal["Standard", "Advanced", "Intelligent-Tiering"]] = "Standard", kms_key_id: Optional[str] = None, **sdk_options, ) -> str: — Reply to this email directly, view it on GitHub <#3040 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBFFL377C25NU3VIJ6DX2VEHLANCNFSM6AAAAAA4KU5T2E> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

aradyaron · 2023-09-21T17:14:53Z

@stephenbawks What do you think of adding an option to control the default upon constructing the SSMProvider?
For example:

parameter_type
tier
kms_key_id

This for example will allow to use a secure by default of parameter_type="SecureString"

leandrodamascena · 2023-09-21T17:28:44Z

Hey everyone! I was attending an external event and I'm reading the new comments on this RFC. I hope to provide feedback tomorrow and move forward to finish this RFC and work on the opened PR.

heitorlessa · 2024-02-27T14:26:48Z

Thank you everyone for the wealth of context. I've just enumerated the activities we need to get done to release it in the next two weeks or so.

#2858 (comment)

github-actions · 2024-03-22T14:02:00Z

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

stephenbawks added RFC triage Pending triage from maintainers labels Sep 4, 2023

github-project-automation bot added this to Powertools for AWS Lambda (Python) Sep 4, 2023

github-project-automation bot moved this to Triage in Powertools for AWS Lambda (Python) Sep 4, 2023

stephenbawks mentioned this issue Sep 4, 2023

Feature request: Setting Parameters and Secrets #2826

Closed

2 tasks

leandrodamascena added parameters Parameters utility and removed triage Pending triage from maintainers labels Sep 4, 2023

leandrodamascena self-assigned this Sep 4, 2023

leandrodamascena moved this from Triage to Pending review in Powertools for AWS Lambda (Python) Sep 4, 2023

leandrodamascena pinned this issue Sep 4, 2023

leandrodamascena mentioned this issue Sep 4, 2023

feat(auth): SigV4 Signing for calls that are taking IAM Auth #2435

Open

7 tasks

leandrodamascena linked a pull request Sep 4, 2023 that will close this issue

feat(parameters): add feature for creating and updating Parameters and Secrets #2858

Merged

7 tasks

aradyaron mentioned this issue Sep 26, 2023

feat(parameters): add feature for creating and updating Parameters and Secrets #2858

Merged

7 tasks

heitorlessa moved this from Pending review to Working on it in Powertools for AWS Lambda (Python) Feb 27, 2024

heitorlessa assigned heitorlessa and unassigned leandrodamascena Feb 27, 2024

leandrodamascena closed this as completed in #2858 Mar 22, 2024

github-project-automation bot moved this from Working on it to Coming soon in Powertools for AWS Lambda (Python) Mar 22, 2024

leandrodamascena unpinned this issue Mar 22, 2024

leandrodamascena moved this from Coming soon to Shipped in Powertools for AWS Lambda (Python) Mar 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Setting SSM Parameters and Secret Manager Secrets #3040

RFC: Setting SSM Parameters and Secret Manager Secrets #3040

stephenbawks commented Sep 4, 2023 •

edited by leandrodamascena

Loading

leandrodamascena commented Sep 4, 2023

Parameter Store

Secret Manager

sthulb commented Sep 5, 2023

stephenbawks commented Sep 5, 2023 •

edited

Loading

stephenbawks commented Sep 5, 2023 •

edited

Loading

aradyaron commented Sep 7, 2023

From here on, I will use `boto3.methodname()` to identify methods in the boto3 SDK and `method()` to identify new methods that will be created according to this RFC.

Parameter Store

Secret Manager

Cache

Next steps

Tasks

heitorlessa commented Sep 11, 2023

heitorlessa commented Sep 12, 2023 •

edited

Loading

stephenbawks commented Sep 14, 2023 •

edited

Loading

stephenbawks commented Sep 15, 2023

aradyaron commented Sep 16, 2023

heitorlessa commented Sep 16, 2023 via email

aradyaron commented Sep 21, 2023

leandrodamascena commented Sep 21, 2023

heitorlessa commented Feb 27, 2024

github-actions bot commented Mar 22, 2024

RFC: Setting SSM Parameters and Secret Manager Secrets #3040

RFC: Setting SSM Parameters and Secret Manager Secrets #3040

Comments

stephenbawks commented Sep 4, 2023 • edited by leandrodamascena Loading

Is this related to an existing feature request or issue?

Which Powertools for AWS Lambda (Python) utility does this relate to?

Summary

Use case

Parameter Store

Secrets Manager

Proposal

Tasks

Parameter Store

Secret Manager

Out of scope

Secrets Manager

Potential challenges

Secrets Manager

Dependencies and Integrations

Alternative solutions

Acknowledgment

leandrodamascena commented Sep 4, 2023

From here on, I will use boto3.methodname() to identify methods in the boto3 SDK and method() to identify new methods that will be created according to this RFC.

Parameter Store

Secret Manager

Cache

Next steps

Tasks

sthulb commented Sep 5, 2023

stephenbawks commented Sep 5, 2023 • edited Loading

stephenbawks commented Sep 5, 2023 • edited Loading

aradyaron commented Sep 7, 2023

From here on, I will use boto3.methodname() to identify methods in the boto3 SDK and method() to identify new methods that will be created according to this RFC.

Parameter Store

Secret Manager

Cache

Next steps

Tasks

heitorlessa commented Sep 11, 2023

heitorlessa commented Sep 12, 2023 • edited Loading

stephenbawks commented Sep 14, 2023 • edited Loading

stephenbawks commented Sep 15, 2023

aradyaron commented Sep 16, 2023

heitorlessa commented Sep 16, 2023 via email

aradyaron commented Sep 21, 2023

leandrodamascena commented Sep 21, 2023

heitorlessa commented Feb 27, 2024

github-actions bot commented Mar 22, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

stephenbawks commented Sep 4, 2023 •

edited by leandrodamascena

Loading

From here on, I will use `boto3.methodname()` to identify methods in the boto3 SDK and `method()` to identify new methods that will be created according to this RFC.

stephenbawks commented Sep 5, 2023 •

edited

Loading

stephenbawks commented Sep 5, 2023 •

edited

Loading

From here on, I will use `boto3.methodname()` to identify methods in the boto3 SDK and `method()` to identify new methods that will be created according to this RFC.

heitorlessa commented Sep 12, 2023 •

edited

Loading

stephenbawks commented Sep 14, 2023 •

edited

Loading