Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ThrottlingException #40

Merged
merged 4 commits into from
Jan 27, 2023
Merged

Fix ThrottlingException #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
25ec77c
Temporary comment task that lists all the buckets
alinabuzachis Jan 18, 2023
a491f20
Add aws_retry=True to avoid ThrottlingException
alinabuzachis Jan 19, 2023
36a2d70
Consistent use of wait=true
alinabuzachis Jan 26, 2023
056451d
Attempt to disbale s3 integration tests
alinabuzachis Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions plugins/module_utils/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ def wait_until_resource_request_success(self, request_token: str):
)

@to_sync
@AWSRetry.jittered_backoff(
catch_extra_error_codes=["ThrottlingException"], retries=10
)
async def list_resources(
self, type_name: str, identifiers: Optional[List] = None
) -> List:
Expand Down Expand Up @@ -144,7 +147,7 @@ async def list_resources(
if "NextToken" in response:
params["NextToken"] = response["NextToken"]
try:
response = self.client.list_resources(**params)
response = self.client.list_resources(**params, aws_retry=True)
except (
botocore.exceptions.BotoCoreError,
botocore.exceptions.ClientError,
Expand Down Expand Up @@ -173,6 +176,9 @@ async def list_resources(

return results

@AWSRetry.jittered_backoff(
catch_extra_error_codes=["ThrottlingException"], retries=10
)
def list_resource_requests(self, params: Iterable) -> List:
"""
Returns existing resource operation requests using specific filters.
Expand All @@ -186,7 +192,9 @@ def list_resource_requests(self, params: Iterable) -> List:
if "NextToken" in response:
params["NextToken"] = response["NextToken"]
try:
response = self.client.list_resource_requests(**params)
response = self.client.list_resource_requests(
**params, aws_retry=True
)
except (
botocore.exceptions.BotoCoreError,
botocore.exceptions.ClientError,
Expand Down Expand Up @@ -387,7 +395,7 @@ def delete_resource(self, type_name: str, identifier: str) -> bool:
self.wait_for_in_progress_requests(in_progress_requests, identifier)
try:
response = self.client.delete_resource(
TypeName=type_name, Identifier=identifier
TypeName=type_name, Identifier=identifier, aws_retry=True
)
except self.client.exceptions.ResourceNotFoundException:
# If the resource has been deleted by an IN PROGRESS delete operation
Expand All @@ -414,7 +422,8 @@ def ensure_request_status(self, response: Dict) -> bool:
if response and response["ProgressEvent"]["OperationStatus"] == "PENDING":
try:
response = self.client.get_resource_request_status(
RequestToken=response["ProgressEvent"]["RequestToken"]
RequestToken=response["ProgressEvent"]["RequestToken"],
aws_retry=True,
)
except (
botocore.exceptions.BotoCoreError,
Expand Down Expand Up @@ -461,7 +470,7 @@ def update_resource(
try:
for e in in_progress_requests:
self.client.cancel_resource_request(
RequestToken=e["RequestToken"]
RequestToken=e["RequestToken"], aws_retry=True
)
except (
botocore.exceptions.BotoCoreError,
Expand All @@ -488,6 +497,7 @@ def update_resource(
TypeName=type_name,
Identifier=identifier,
PatchDocument=str(patch),
aws_retry=True,
)
except (
botocore.exceptions.BotoCoreError,
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/targets/s3/aliases
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
cloud/aws
zuul/aws/cloud_control

disabled # Temporary disabled because they throw several ThrottlingExceptions and it needs further investigation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please make sure we have a ticket to remind us to come back to this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an issue here #42. We do not have the jira label in this repo. If necessary, I can manually create a Jira ticket as well.

2 changes: 1 addition & 1 deletion tests/integration/targets/s3/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
- assert:
that:
- _result is success

- name: List S3 buckets
amazon.cloud.s3_bucket:
state: list
Expand Down
11 changes: 4 additions & 7 deletions tests/integration/targets/s3/tasks/tagging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
amazon.cloud.s3_bucket:
tags: '{{ first_tags }}'
purge_tags: true
wait: true
register: _result
- name: assert that update succeeded
assert:
Expand Down Expand Up @@ -90,8 +91,6 @@
- _result is not changed
- _result.result.properties.tags == first_tags

###

- name: test updating tags with purge on amazon.cloud.s3_bucket (check mode)
amazon.cloud.s3_bucket:
tags: '{{ second_tags }}'
Expand All @@ -107,6 +106,7 @@
amazon.cloud.s3_bucket:
tags: '{{ second_tags }}'
purge_tags: true
wait: true
register: _result
- name: assert that update succeeded
assert:
Expand Down Expand Up @@ -136,12 +136,11 @@
- _result is not changed
- _result.result.properties.tags == second_tags

#

- name: test updating tags without purge on amazon.cloud.s3_bucket (check mode)
amazon.cloud.s3_bucket:
tags: '{{ third_tags }}'
purge_tags: false
wait: true
register: _result
check_mode: yes
- name: assert that update succeeded
Expand Down Expand Up @@ -195,7 +194,6 @@
- _result is not changed
- _result.result.properties.tags == final_tags


- name: test no tags param amazon.cloud.s3_bucket
amazon.cloud.s3_bucket: {}
register: _result
Expand All @@ -205,8 +203,6 @@
- _result is not changed
- _result.result.properties.tags == final_tags

###

- name: test removing tags from amazon.cloud.s3_bucket (check mode)
amazon.cloud.s3_bucket:
tags: {}
Expand All @@ -222,6 +218,7 @@
amazon.cloud.s3_bucket:
tags: {}
purge_tags: true
wait: true
register: _result
- name: assert that update succeeded
assert:
Expand Down