Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added GCS support guide to template files #233

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+58 −38
Open

Added GCS support guide to template files #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8832d39
Added documentation for GCS support
Nov 15, 2024
9a7d62e
fixed grammer errors
ammarhusain-EMR Nov 15, 2024
e0f2ca5
updated the docs
ammarhusain-EMR Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions getting-started/templates/systemlink-secrets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,17 +276,21 @@ dataframeservice:
##
feedservice:
secrets:
## Access key information for S3/MinIO access.
## If using MinIO, use the user and password from the MinIO configuration above here. Otherwise, S3 credentials will need to be provided.
## Access key information for S3/MinIO/GCS access.
## If using MinIO, use the user and password from the MinIO configuration above here. Otherwise, S3/GCS credentials will need to be provided.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, in recent PRs we've removed references to MinIO since its not longer supported and included in our top level Helm chart. Review the latest versions of these templates to ensure your additions align with this change.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, the comment applies to both files changed in this PR and not just this single line.

## If using GCS, use HMAC keys for authentication.
## Refer the Google documentation to create HMAC keys: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create
##
s3:
## Access key ID for the S3 instance. Override if not using MinIO.
## If using GCS, use the Access ID of the HMAC key as accessKeyID.
##
accessKeyId: *minioUser
## Access key for the S3 instance. Override if not using MinIO.
## If using GCS, use the Secret of the HMAC key as accessKey.
##
accessKey: *minioPassword
## Default region for the S3 instance. Override if not using MinIO.
## Default region for the S3/GCS instance. Override if not using MinIO.
##
defaultRegion: &minioDefaultRegion "us-east-1"
## Credentials for the MongoDB cluster.
Expand All @@ -313,17 +317,21 @@ fileingestion:
## Cryptographic key to be used for encryption of download tokens. This key should have a length of at least 32 bytes.
##
downloadTokenEncryptionKey: "" # <ATTENTION>
## Access key information for S3/MinIO access.
## If using MinIO, use the user and password from the MinIO configuration above here. Otherwise, S3 credentials will need to be provided.
## Access key information for S3/MinIO/GCS access.
## If using MinIO, use the user and password from the MinIO configuration above here. Otherwise, S3/GCS credentials will need to be provided.
## If using GCS, use HMAC keys for authentication.
## Refer the Google documentation to create HMAC keys: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create
##
s3:
## Access key ID for the S3 instance. Override if not using MinIO.
## If using GCS, use the Access ID of the HMAC key as accessKeyID.
##
accessKeyId: *minioUser
## Access key for the S3 instance. Override if not using MinIO.
## If using GCS, use the Secret of the HMAC key as accessKey.
##
accessKey: *minioPassword
## Default region for the S3 instance. Override if not using MinIO.
## Default region for the S3/GCS instance. Override if not using MinIO.
##
defaultRegion: *minioDefaultRegion
## Credentials for the MongoDB cluster.
Expand All @@ -344,14 +352,18 @@ fileingestion:
##
nbexecservice:
secrets:
## Access key information for S3/MinIO access.
## If you are using MinIO, the user and password are the same as the MinIO configuration. If you are not using MinIO, specify S3 credentials.
## Access key information for S3/MinIO/GCS access.
## If you are using MinIO, the user and password are the same as the MinIO configuration. If you are not using MinIO, specify S3/GCS credentials.
## If you are using GCS, use HMAC keys for authentication.
## Refer the Google documentation to create HMAC keys: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create
##
s3:
## Access key ID for the S3 instance. Override if you are not using MinIO.
## If using GCS, use the Access ID of the HMAC key as accessKeyID.
##
accessKeyId: *minioUser
## Access key for the S3 instance. Override if you are not using MinIO.
## If using GCS, use the Secret of the HMAC key as accessKey.
##
accessKey: *minioPassword
mongodb:
Expand Down
68 changes: 38 additions & 30 deletions getting-started/templates/systemlink-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -843,39 +843,41 @@ saltmaster:
## Feed configuration.
##
feedservice:
## Configure S3/MinIO access.
## Configure S3/MinIO/GCS access.
##
s3:
## Secret name for S3 credentials.
## Secret name for S3/GCS credentials.
##
secretName: "feeds-s3-credentials"
## The name of the S3 or MinIO bucket that the service should connect to.
## The name of the S3/MinIO/GCS bucket that the service should connect to.
##
bucket: "systemlink-feeds"
## S3 connection scheme.
## S3/GCS connection scheme.
##
scheme: "http://"
## Set this value to connect to an external S3 instance.
# <ATTENTION> To connect to an external S3 bucket, set the host here as well as the scheme and port.
# <ATTENTION> To connect to an external S3/GCS bucket, set the host here as well as the scheme and port.
# <ATTENTION> To connect to an external S3 bucket, set the host as "s3.amazonaws.com"
# <ATTENTION> To connect to an external GCS bucket, set the host as "storage.googleapis.com".
##
host: ""
## Set this value to connect to an S3 instance which is internal to the cluster. Ignored if host is set.
##
service: *minioServiceName
## S3 Port
## S3/GCS Port
# <ATTENTION> This must be overridden if not using the SLE MinIO instance.
##
port: *minioPort
## S3 Region
# <ATTENTION> This must be set to the region of the S3 instance.
## S3/GCS Region
# <ATTENTION> This must be set to the region of the S3/GCS instance.
##
region: "us-east-1"
## Proxy configuration to be used when the service needs to go through a proxy to have access to external services like ni.com.
httpProxy:
## @param httpProxy.address Address of the HTTP proxy in the $host:$port format. Example: "1.1.1.1:2222"
##
address: ""
## @param httpProxy.includeS3HostInNoProxy Set to true to include the S3 host in the noProxy list. Use this if the S3 host is internal to the cluster so we don't need to go through the proxy to access it.
## @param httpProxy.includeS3HostInNoProxy Set to true to include the S3/GCS host in the noProxy list. Use this if the S3/GCS host is internal to the cluster so we don't need to go through the proxy to access it.
##
includeS3HostInNoProxy: false
## @param httpProxy.includeAPIHostsInNoProxy Set to true to include the API hosts in the noProxy list. Use this if the API host is internal to the cluster so we don't need to go through the proxy to access it.
Expand All @@ -899,34 +901,36 @@ fileingestion:
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx.ingress.kubernetes.io/proxy-buffering: "off"

## Configure S3/MinIO access.
## Configure S3/MinIO/GCS access.
##
s3:
## Secret name for S3 credentials.
## Secret name for S3/GCS credentials.
##
secretName: "fileingestion-s3-credentials"
## The name of the S3 or MinIO bucket that the service should connect to.
## The name of the S3/MinIO/GCS bucket that the service should connect to.
##
bucket: "systemlink-file-ingestion"
## Set this to true to limit each user to a maximum of 1Gb of file storage.
##
storageLimitsEnabled: false
## S3 connection scheme.
## S3/GCS connection scheme.
##
scheme: "http://"
## Set this value to connect to an external S3 instance.
# <ATTENTION> To connect to an external S3 bucket, set the host here as well as the scheme and port.
## Set this value to connect to an external S3/GCS instance.
# <ATTENTION> To connect to an external S3/GCS bucket, set the host here as well as the scheme and port.
# <ATTENTION> To connect to an external S3 bucket, set the host as "s3.amazonaws.com".
# <ATTENTION> To connect to an external GCS bucket, set the host as "storage.googleapis.com"
##
host: ""
## Set this value to connect to an S3 instance which is internal to the cluster. Ignored if host is set.
##
service: *minioServiceName
## S3 Port
## S3/GCS Port
# <ATTENTION> This must be overridden if not using the SLE MinIO instance.
##
port: *minioPort
## S3 Region
# <ATTENTION> This must be set to the region of the S3 instance.
## S3/GCS Region
# <ATTENTION> This must be set to the region of the S3/GCS instance.
##
region: "us-east-1"
## Configure rate limiting. Limits are enforced per-user. Each replica of the file ingestion service
Expand Down Expand Up @@ -1027,48 +1031,52 @@ argoworkflows:
##
nbexecservice:
maxNumberOfWorkflowsToSchedule: *workflowParallelism
## Configure S3/MinIO access.
## Configure S3/MinIO/GCS access.
##
s3:
## Secret name for S3 credentials.
## Secret name for S3/MinIO/GCS credentials.
##
secretName: "nbexecservice-s3-credentials"
## The name of the S3 or MinIO bucket that the service should connect to.
## The name of the S3/MinIO/GCS bucket that the service should connect to.
##
bucket: "systemlink-executions"
## S3 connection scheme.
## S3/GCS connection scheme.
##
scheme: "http://"
## Set this value to connect to an external S3 instance.
# <ATTENTION> To connect to an external S3 bucket, set the host here as well as the scheme and port.
## Set this value to connect to an external S3/GCS instance.
# <ATTENTION> To connect to an external S3/GCS bucket, set the host here as well as the scheme and port.
# <ATTENTION> To connect to an external S3 bucket, set the host as "s3.amazonaws.com"
# <ATTENTION> To connect to an external GCS bucket, set the host as "storage.googleapis.com"
##
host: ""
## Set this value to connect to an S3 instance which is internal to the cluster. Ignored if host is set.
##
service: *minioServiceName
## S3 Port
## S3/GCS Port
# <ATTENTION> This must be overridden if not using the SLE MinIO instance.
port: *minioPort
## S3 Region
## S3/GCS Region
# <ATTENTION> This must be set to the region of the S3 instance.
##
region: "us-east-1"
argo:
## Configure S3/MinIO access.
## Configure S3/MinIO/GCS access.
##
artifactRepository:
s3:
## The name of the S3 or MinIO bucket that the service should connect to.
## The name of the S3/MinIO/GCS bucket that the service should connect to.
##
bucket: "systemlink-notebook-execution"
## Set this value to connect to an external S3 instance.
# <ATTENTION> To connect to an external S3 bucket, set the host here as well as the insecure and port.
# <ATTENTION> To connect to an external S3/GCS bucket, set the host here as well as the insecure and port.
# <ATTENTION> To connect to an external S3 bucket, set the host as "s3.amazonaws.com"
# <ATTENTION> To connect to an external GCS bucket, set the host as "storage.googleapis.com"
##
host: ""
## Set this value to connect to an S3 instance which is internal to the cluster. Ignored if host is set.
##
service: *minioServiceName
## S3 Port
## S3/GCS Port
# <ATTENTION> This must be overridden if not using the SLE MinIO instance.
##
port: *minioPort
Expand Down