Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(s3): support default credentials provider in AWS S3 storage #37

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

rohilsurana
Copy link

@rohilsurana rohilsurana commented Jan 13, 2025

Problem

Currently the only way to set AWS credentials is via setting the static credentials as environment variables or by passing as a flag when executing.
Some systems already have credentials setup using different ways provided by aws like credentials file, config file or EC2 profiles.

Solution

This PR adds support for using AWS S3 credentials from default credentials provider chain in the SDK.
The order for the AWS SDK golang is documented at [1].
The order for duckdb is documented at [2].

Changes

  • Adds a new configuration AWS_CREDENTIALS_TYPE with options STATIC and DEFAULT.
    • STATIC maintains the old behaviour and uses the supplied static credentials.
    • DEFAULT uses the AWS SDK and DuckDB default credentials chain.
  • AWS client config changes for syncing tables
  • DuckDB create secret query changes for creating S3 secret as per new config.
  • Update README with details of the new config.

[1] AWS SDK Golang - https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials
[2] DuckDB S3 API Support - https://duckdb.org/docs/extensions/httpfs/s3api.html#credential_chain-provider

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant