Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parameters): add feature for creating and updating Parameters and Secrets #2858

Merged
merged 85 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 61 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
0369d80
gotta start somewhere
stephenbawks Jul 27, 2023
5551118
small tweaks
stephenbawks Aug 1, 2023
6e0c7de
adding set param
stephenbawks Sep 2, 2023
330de35
adding exception classes
stephenbawks Sep 2, 2023
929ae7f
adding set parameter
stephenbawks Sep 2, 2023
2ae911e
fixing set
stephenbawks Sep 2, 2023
aedab55
few more updates
stephenbawks Sep 11, 2023
1548690
Merge branch 'develop' into setparams
stephenbawks Sep 11, 2023
e154a5d
missing value
stephenbawks Sep 11, 2023
805f3d8
adding documentation
stephenbawks Sep 11, 2023
12b8850
one more update
stephenbawks Sep 11, 2023
f9323ea
question about transform yet
stephenbawks Sep 14, 2023
a0b32b8
remove optional transform
stephenbawks Sep 15, 2023
ab6065e
updates
stephenbawks Sep 16, 2023
6d2012c
updating put secret value
stephenbawks Sep 16, 2023
c6a1088
fixing value on return secret
stephenbawks Sep 16, 2023
609dbc9
cleaning up naming and examples
stephenbawks Sep 16, 2023
0abb31b
fix example and return type
stephenbawks Sep 16, 2023
05b8eed
Merge branch 'develop' into setparams
stephenbawks Sep 18, 2023
2992989
update
stephenbawks Sep 18, 2023
887f3c4
fix a few new warnings
stephenbawks Sep 18, 2023
98c7d5f
simplying secret value
stephenbawks Sep 19, 2023
d837154
small tweaks
stephenbawks Sep 21, 2023
13c9cf7
cleaning up
stephenbawks Sep 21, 2023
5b91705
missed one
stephenbawks Sep 21, 2023
7bbcaa7
Merge branch 'develop' into setparams
heitorlessa Sep 21, 2023
529c059
cleaning up ruff
stephenbawks Sep 22, 2023
20dac0c
Merge branch 'develop' into setparams
leandrodamascena Sep 25, 2023
0c03c82
couple of modifications
stephenbawks Sep 25, 2023
2d98d04
forgot to remove this here
stephenbawks Sep 25, 2023
7d406de
fix
stephenbawks Sep 25, 2023
be6f7af
adding create when not existing
stephenbawks Sep 28, 2023
8fffa58
fix name
stephenbawks Sep 28, 2023
b576bb7
create flag
stephenbawks Sep 28, 2023
4396b81
Update aws_lambda_powertools/utilities/parameters/secrets.py
stephenbawks Sep 28, 2023
78e9f45
couple refinements
stephenbawks Sep 28, 2023
668ba3d
updating docstrings
stephenbawks Sep 28, 2023
7413067
💄 fix example again
stephenbawks Sep 28, 2023
2539382
📝 documentation is seriously tough
stephenbawks Sep 28, 2023
682dd39
adding examples and documentation for set_param
stephenbawks Sep 28, 2023
d264b65
trying to add some docs
stephenbawks Sep 29, 2023
7a488bc
creating "realistic" example
stephenbawks Sep 30, 2023
3b2d67c
Merge branch 'develop' into setparams
leandrodamascena Oct 13, 2023
6915419
updating example
stephenbawks Oct 14, 2023
4920274
making example more appropiate
stephenbawks Oct 14, 2023
abce28e
missed one
stephenbawks Oct 14, 2023
20925ec
Merge branch 'develop' into setparams
leandrodamascena Oct 15, 2023
d49f89f
couple tests so far
stephenbawks Oct 19, 2023
bee00a4
Merge branch 'develop' into setparams
sthulb Dec 6, 2023
973b3a9
Merge branch 'develop' into setparams
leandrodamascena Jan 10, 2024
3c05392
Merge branch 'develop' into setparams
heitorlessa Feb 15, 2024
90fa67a
Merge branch 'develop' into setparams
stephenbawks Feb 17, 2024
f9a2944
changing variable name
stephenbawks Feb 17, 2024
d55a9b4
removing the create logic for the time being
stephenbawks Feb 17, 2024
9ead47a
remove create option
stephenbawks Feb 19, 2024
8a52d21
Merge branch 'develop' into setparams
stephenbawks Feb 19, 2024
647e3b8
Merge branch 'develop' into setparams
heitorlessa Feb 26, 2024
7783aad
chore: remove set as mandatory method (temporarily)
heitorlessa Feb 26, 2024
7325bc7
fix(parameters): make cache aware of single vs multiple calls
heitorlessa Jul 25, 2023
8704337
chore: cleanup, add test for single and nested
heitorlessa Jul 25, 2023
5098ed9
Merge branch 'develop' into setparams
heitorlessa Feb 27, 2024
cd116bb
refactor: implement set() minimum contract, and set() in ssm
heitorlessa Feb 27, 2024
5ccb953
refactor: use name over path in set_parameter
heitorlessa Feb 27, 2024
e28b548
Merge remote-tracking branch 'upstream/develop' into setparams
leandrodamascena Mar 20, 2024
d9cf5fd
Adding docstring
leandrodamascena Mar 20, 2024
ae8975f
Fixing description type + adding TypeDict for set_parameter
leandrodamascena Mar 20, 2024
6502229
Refactoring tests
leandrodamascena Mar 20, 2024
3e79eac
Adding correct exception for Secrets + fixing examples
leandrodamascena Mar 20, 2024
401cdf8
Making SonarCloud happy
leandrodamascena Mar 20, 2024
44710ba
Changing return from setSecret + fix mypy issues
leandrodamascena Mar 20, 2024
31777b1
Increasing coverage
leandrodamascena Mar 20, 2024
b89c9b8
Merge branch 'develop' into setparams
leandrodamascena Mar 20, 2024
c016d24
Increasing coverage
leandrodamascena Mar 20, 2024
3bf45df
Refactoring secrets
leandrodamascena Mar 20, 2024
57eb44b
Improving docstring
leandrodamascena Mar 21, 2024
df0ffc9
Adding more tests
leandrodamascena Mar 21, 2024
0efe6d7
Making SonarCloud happy
leandrodamascena Mar 21, 2024
6fae9dd
Adding more tests and docs
leandrodamascena Mar 21, 2024
4b0af25
Merge branch 'develop' into setparams
leandrodamascena Mar 21, 2024
f65570f
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
bf5f05b
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
4cebf75
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
273c544
Addressing Ruben's feedback
leandrodamascena Mar 22, 2024
a9a3c41
Merge branch 'develop' into setparams
leandrodamascena Mar 22, 2024
ac78c0d
Merge branch 'develop' into setparams
leandrodamascena Mar 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions aws_lambda_powertools/utilities/parameters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
from .base import BaseProvider, clear_caches
from .dynamodb import DynamoDBProvider
from .exceptions import GetParameterError, TransformParameterError
from .secrets import SecretsProvider, get_secret
from .ssm import SSMProvider, get_parameter, get_parameters, get_parameters_by_name
from .secrets import SecretsProvider, get_secret, set_secret
from .ssm import SSMProvider, get_parameter, set_parameter, get_parameters, get_parameters_by_name

__all__ = [
"AppConfigProvider",
Expand All @@ -21,8 +21,10 @@
"TransformParameterError",
"get_app_config",
"get_parameter",
"set_parameter",
"get_parameters",
"get_parameters_by_name",
"get_secret",
"set_secret",
"clear_caches",
]
6 changes: 6 additions & 0 deletions aws_lambda_powertools/utilities/parameters/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,12 @@ def _get(self, name: str, **sdk_options) -> Union[str, bytes, Dict[str, Any]]:
"""
raise NotImplementedError()

def _set(self, name: str, **sdk_options) -> Union[str, bytes]:
"""
Sets a parameter value from the underlying parameter store
"""
raise NotImplementedError()

def get_multiple(
self,
path: str,
Expand Down
4 changes: 4 additions & 0 deletions aws_lambda_powertools/utilities/parameters/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,7 @@ class GetParameterError(Exception):

class TransformParameterError(Exception):
"""When a provider fails to transform a parameter value"""


class SetParameterError(Exception):
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved
"""When a provider raises an exception on setting a parameter"""
137 changes: 137 additions & 0 deletions aws_lambda_powertools/utilities/parameters/secrets.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,15 @@
AWS Secrets Manager parameter retrieval and caching utility
"""

from __future__ import annotations

import json
import os
from typing import TYPE_CHECKING, Any, Dict, Optional, Union

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

if TYPE_CHECKING:
from mypy_boto3_secretsmanager import SecretsManagerClient
Expand All @@ -15,6 +19,7 @@
from aws_lambda_powertools.shared.functions import resolve_max_age

from .base import DEFAULT_MAX_AGE_SECS, DEFAULT_PROVIDERS, BaseProvider
from .exceptions import SetParameterError


class SecretsProvider(BaseProvider):
Expand Down Expand Up @@ -115,6 +120,65 @@ def _get_multiple(self, path: str, **sdk_options) -> Dict[str, str]:
"""
raise NotImplementedError()

def _set(
self,
name: str,
value: Union[str, dict, bytes],
*, # force keyword arguments
client_request_token: Optional[str] = None,
version_stages: Optional[list[str]] = None,
**sdk_options,
) -> str:
"""
Modifies the details of a secret, including metadata and the secret value.

Parameters
----------
name: str
The ARN or name of the secret to add a new version to.
value: str or bytes
Specifies text data that you want to encrypt and store in this new version of the secret.
client_request_token: str, optional
This value helps ensure idempotency. Recommended that you generate
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved
a UUID-type value to ensure uniqueness within the specified secret.
This value becomes the VersionId of the new version. This field is
autopopulated if not provided.
version_stages: list[str], optional
Specifies a list of staging labels that are attached to this version of the secret.
sdk_options: dict, optional
Dictionary of options that will be passed to the Secrets Manager update_secret API call

Returns:
-------
Version ID of the newly created version of the secret.

URLs:
-------
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/put_secret_value.html
"""

sdk_options["SecretId"] = name

if isinstance(value, dict):
value = json.dumps(value)
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved

if isinstance(value, bytes):
sdk_options["SecretBinary"] = value
else:
sdk_options["SecretString"] = value

if version_stages:
sdk_options["VersionStages"] = version_stages
if client_request_token:
sdk_options["ClientRequestToken"] = client_request_token
rubenfonseca marked this conversation as resolved.
Show resolved Hide resolved

try:
value = self.client.put_secret_value(**sdk_options)
return value["VersionId"]
except ClientError as exc:
if exc.response["Error"]["Code"] != "ResourceNotFoundException":
raise SetParameterError(str(exc)) from exc
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved


def get_secret(
name: str,
Expand Down Expand Up @@ -182,3 +246,76 @@ def get_secret(
force_fetch=force_fetch,
**sdk_options,
)


def set_secret(
name: str,
value: Union[str, bytes],
*, # force keyword arguments
client_request_token: Optional[str] = None,
version_stages: Optional[list[str]] = None,
**sdk_options,
) -> str:
"""
Retrieve a parameter value from AWS Secrets Manager

Parameters
----------
name: str
Name of the parameter
value: str or bytes
Secret value to set
client_request_token: str, optional
This value helps ensure idempotency. Recommended that you generate
a UUID-type value to ensure uniqueness within the specified secret.
This value becomes the VersionId of the new version. This field is
autopopulated if not provided.
version_stages: list[str], optional
A list of staging labels that are attached to this version of the secret.
sdk_options: dict, optional
Dictionary of options that will be passed to the get_secret_value call

Raises
------
SetParameterError
When the secrets provider fails to set a secret value or secret binary for
a given name.

Returns:
-------
Version ID of the newly created version of the secret.

URLs:
-------
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/put_secret_value.html

Example
-------
**Sets a secret***

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> parameters.set_secret(name="llamas-are-awesome", value="supers3cr3tllam@passw0rd")

**Sets a secret and includes an client_request_token**

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> parameters.set_secret(
name="my-secret",
value='{"password": "supers3cr3tllam@passw0rd"}',
client_request_token="61f2af5f-5f75-44b1-a29f-0cc37af55b11"
)
"""

# Only create the provider if this function is called at least once
if "secrets" not in DEFAULT_PROVIDERS:
DEFAULT_PROVIDERS["secrets"] = SecretsProvider()

return DEFAULT_PROVIDERS["secrets"]._set(
name=name,
value=value,
client_request_token=client_request_token,
version_stages=version_stages,
**sdk_options,
)
78 changes: 78 additions & 0 deletions aws_lambda_powertools/utilities/parameters/ssm.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
from mypy_boto3_ssm import SSMClient
from mypy_boto3_ssm.type_defs import GetParametersResultTypeDef

SSM_PARAMETER_TYPES = Literal["String", "StringList", "SecureString"]
SSM_PARAMETER_TIER = Literal["Standard", "Advanced", "Intelligent-Tiering"]


class SSMProvider(BaseProvider):
"""
Expand Down Expand Up @@ -811,6 +814,81 @@ def get_parameters(
)


def set_parameter(
path: str,
value: str,
*, # force keyword arguments
parameter_type: SSM_PARAMETER_TYPES = "String",
overwrite: bool = False,
tier: SSM_PARAMETER_TIER = "Standard",
description: Optional[str] = None,
kms_key_id: Optional[str] = None,
**sdk_options,
) -> int:
"""
Retrieve a parameter value from AWS Systems Manager (SSM) Parameter Store

Parameters
----------
path: str
The fully qualified name includes the complete hierarchy of the parameter path and name.
value: str
The parameter value
parameter_type: str, optional
Type of the parameter. Allowed values are String, StringList, and SecureString
overwrite: bool, optional
If the parameter value should be overwritten, False by default
tier: str, optional
The parameter tier to use. Allowed values are Standard, Advanced, and Intelligent-Tiering
description: str, optional
The description of the parameter
kms_key_id: str, optional
The KMS key id to use to encrypt the parameter
sdk_options: dict, optional
Dictionary of options that will be passed to the Parameter Store get_parameter API call

Returns:
--------
The version (integer) of the parameter that was set

Raises
------
SetParameterError
When the parameter provider fails to retrieve a parameter value for
a given name.

URLs:
-------
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm/client/put_parameter.html

Example
-------
**Sets a parameter value from Systems Manager Parameter Store**

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> response = parameters.set_parameter(path="/my/example/parameter", value="More Powertools")
>>>
>>> print(response)
123
"""

# Only create the provider if this function is called at least once
if "ssm" not in DEFAULT_PROVIDERS:
DEFAULT_PROVIDERS["ssm"] = SSMProvider()

return DEFAULT_PROVIDERS["ssm"]._set(
path,
value,
parameter_type=parameter_type,
overwrite=overwrite,
tier=tier,
description=description,
kms_key_id=kms_key_id,
**sdk_options,
)


@overload
def get_parameters_by_name(
parameters: Dict[str, Dict],
Expand Down
29 changes: 29 additions & 0 deletions docs/utilities/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@ This utility requires additional permissions to work as expected.
| SSM | **`get_parameter`**, **`SSMProvider.get`** | **`ssm:GetParameter`** |
| SSM | **`get_parameters`**, **`SSMProvider.get_multiple`** | **`ssm:GetParametersByPath`** |
| SSM | **`get_parameters_by_name`**, **`SSMProvider.get_parameters_by_name`** | **`ssm:GetParameter`** and **`ssm:GetParameters`** |
| SSM | **`set_parameter`** | **`ssm:PutParameter`** |
| SSM | If using **`decrypt=True`** | You must add an additional permission **`kms:Decrypt`** |
| Secrets | **`get_secret`**, **`SecretsProvider.get`** | **`secretsmanager:GetSecretValue`** |
| Secrets | **`set_secret`**, **`SecretsProvider.get`** | **`secretsmanager:PutSecretValue`** and or **`secretsmanager:CreateSecret`** |
| DynamoDB | **`DynamoDBProvider.get`** | **`dynamodb:GetItem`** |
| DynamoDB | **`DynamoDBProvider.get_multiple`** | **`dynamodb:Query`** |
| AppConfig | **`get_app_config`**, **`AppConfigProvider.get_app_config`** | **`appconfig:GetLatestConfiguration`** and **`appconfig:StartConfigurationSession`** |
Expand Down Expand Up @@ -84,6 +86,23 @@ For multiple parameters, you can use either:
--8<-- "examples/parameters/src/get_parameter_by_name_error_handling.py"
```

### Setting parameters

You can set a parameter using the `set_parameter` high-level function. This will create a new parameter if it doesn't exist.

=== "getting_started_set_single_ssm_parameter.py"
```python hl_lines="8"
--8<-- "examples/parameters/src/getting_started_set_single_ssm_parameter.py"
```

=== "getting_started_set_ssm_parameter_overwrite.py"
There are occasions where sometimes you are setting a parameter and then you may need to update that parameter later on. In this case, you can use the `overwrite` parameter to overwrite the parameter value if it already exists. If you do not set this parameter, then the parameter will not be overwritten and an exception will be raised.
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved

```python hl_lines="8"
--8<-- "examples/parameters/src/getting_started_set_ssm_parameter_overwrite.py"
```


### Fetching secrets

You can fetch secrets stored in Secrets Manager using `get_secret`.
Expand All @@ -93,6 +112,16 @@ You can fetch secrets stored in Secrets Manager using `get_secret`.
--8<-- "examples/parameters/src/getting_started_secret.py"
```

### Setting secrets

You can set secrets stored in Secrets Manager using `set_secret`.

=== "getting_started_secret.py"
```python hl_lines="5 15"
--8<-- "examples/parameters/src/getting_started_setting_secret.py"
```


### Fetching app configurations

You can fetch application configurations in AWS AppConfig using `get_app_config`.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.typing import LambdaContext


def lambda_handler(event: dict, context: LambdaContext) -> dict:
try:
# Set a single parameter, returns the version ID of the parameter
parameter_version = parameters.set_parameter(path="/mySuper/Parameter", value="PowerToolsIsAwesome") # type: ignore[assignment] # noqa: E501

return {"mySuperParameterVersion": parameter_version, "statusCode": 200}
except parameters.exceptions.SetParameterError as error:
return {"comments": None, "message": str(error), "statusCode": 400}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.typing import LambdaContext


def lambda_handler(event: dict, context: LambdaContext) -> dict:
try:
# Set a single parameter, but overwrite if it already exists
# by default, overwrite is False, explicitly set it to True
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved
updating_parameter = parameters.set_parameter(path="/mySuper/Parameter", value="PowerToolsIsAwesome", overwrite=True) # type: ignore[assignment] # noqa: E501

return {"mySuperParameterVersion": updating_parameter, "statusCode": 200}
except parameters.exceptions.SetParameterError as error:
return {"comments": None, "message": str(error), "statusCode": 400}
Loading
Loading