forked from 2i2c-org/infrastructure
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Setup workload identity via terraform
The GKE config connector was helpful in letting us deploy Google Cloud Service Accounts with permissions for cloud storage directly just from helm. However, it has been difficult to debug, and in 2i2c-org#669 we decided to move away from it and towards creating these cloud resources via Terraform. This commit adds: - Terraform code that will create a Google Service Account, bind it to a given Kubernetes Service Account, for a list of hub namespaces passed in. This means that some hub initial deployments now *can not be done just with CD*, but need manual work with terraform. I think this would be any hub that wants to use requestor pays or scratch buckets. This would need to be documented. - Move meom-ige to use this new scheme. metadata concealment (https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata#concealment) which is what we were using earlier as alternative to config-connector + workload identity, is no longer supported by the terraform google provider. In b7b42ce, we changed the default from 'SECURE' to 'UNSPECIFIED', but it looks like 'UNSPECIFIED' really means 'use workload identity' haha. When 2i2c-org#1124 was deployed to meom-ige yesterday, it seems to have enabled workload identity, causing cloud access to stop working, leading to https://2i2c.freshdesk.com/a/tickets/107. Further investigation on what happened here is needed, but I've currently fixed it by just deploying this change for meom-ige. - All hubs are given access to all buckets we create. This is inadequete, and needs to be more fine grained. Ref 2i2c-org#669 Ref 2i2c-org#1046
- Loading branch information
Showing
6 changed files
with
129 additions
and
70 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # User pods need to authenticate to cloud APIs - particularly around storage. | ||
| # On GKE, Workload Identity (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) | ||
| # is the canonical way to do this securely. A Google Cloud Service Account (GSA) | ||
| # is created and given appropriate rights, and then bound to a Kubernetes Service Account (KSA) | ||
| # via workload identity. All pods that then mount this kubernetes service account (named user-sa) | ||
| # get the cloud permissions assigned to the Google Cloud Service Account. | ||
| # | ||
| # Since each cluster can contain multiple hubs, we need to tell terraform which hubs we want | ||
| # to equip with the KSA that has cloud credentials. Terraform will create this Kubernetes | ||
| # Service Account (and the namespace, if it does not exist). | ||
|
|
||
| resource "google_service_account" "workload_sa" { | ||
| for_each = var.workload_identity_enabled_hubs | ||
| account_id = "${var.prefix}-${each.value}-workload-sa" | ||
| display_name = "Service account for user pods in hub ${each.value} in ${var.prefix}" | ||
| project = var.project_id | ||
| } | ||
|
|
||
| # To access GCS buckets with requestor pays, the calling code needs | ||
| # to have serviceusage.services.use permission. We create a role | ||
| # granting just this to provide the workload SA, so user pods can | ||
| # use it. See https://cloud.google.com/storage/docs/requester-pays | ||
| # for more info | ||
| resource "google_project_iam_custom_role" "workload_role" { | ||
| // Role names can't contain -, so we swap them out. BOO | ||
| role_id = replace("${var.prefix}_workload_sa_role", "-", "_") | ||
| project = var.project_id | ||
| title = "Identify as project role for users in ${var.prefix}" | ||
| description = "Minimal role for hub users on ${var.prefix} to identify as current project" | ||
| permissions = ["serviceusage.services.use"] | ||
| } | ||
|
|
||
| resource "google_project_iam_member" "workload_binding" { | ||
| for_each = var.workload_identity_enabled_hubs | ||
| project = var.project_id | ||
| role = google_project_iam_custom_role.workload_role.name | ||
| member = "serviceAccount:${google_service_account.workload_sa[each.value].email}" | ||
| } | ||
|
|
||
| # Bind the Kubernetes Service Accounts to their appropriate Google Cloud Service Accounts | ||
| resource "google_service_account_iam_binding" "workload_identity_binding" { | ||
| for_each = var.workload_identity_enabled_hubs | ||
| service_account_id = google_service_account.workload_sa[each.value].id | ||
| role = "roles/iam.workloadIdentityUser" | ||
| members = [ | ||
| "serviceAccount:${var.project_id}.svc.id.goog[${each.value}/user-sa]" | ||
| ] | ||
| } | ||
|
|
||
| # Create the Service Account in the Kubernetes Namespace | ||
| # FIXME: We might need to create the k8s namespace here some of the time, but then who is | ||
| # responsible for that - terraform or helm (via our deployer?) | ||
| resource "kubernetes_service_account" "workload_kubernetes_sa" { | ||
| for_each = var.workload_identity_enabled_hubs | ||
|
|
||
| metadata { | ||
| name = "user-sa" | ||
| namespace = each.value | ||
| annotations = { | ||
| "iam.gke.io/gcp-service-account" = google_service_account.workload_sa[each.value].email | ||
| } | ||
| } | ||
| } |