release v0.4.56 from PR #744

Worklytics · Jul 4, 2024 · b7102fb · b7102fb
2 parents 1d824ca + 95f389b
commit b7102fb
Show file tree

Hide file tree

Showing 35 changed files with 351 additions and 172 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,30 +7,21 @@ Changes to be including in future/planned release notes will be added here.
 
 ## Next
 
-## 0.5 *future, subject to change!!*
-  - RULES only via config management, never env variable
-  - Eliminate "fall-through" configs.
-     - `PATH_TO_SHARED_CONFIG` - env var that locates shared parameters within the config store.
-     - `PATH_TO_CONNECTOR_CONFIG` - env var that locates connector-specific parameters within the
-       config store.
-  - Expect distinct paths for the shared and connector scopes, to support more  straight-forward IAM
-    policies.
-     - eg, `PSOXY_SHARED` and `PSOXY_GCAL`, to allow IAM policies such as "read `PSOXY_SHARED*`" and
-        "read+write `PSOXY_GCAL*`" (if shared secrets have common prefix with connector secrets,
-        then wildcard policy to read shared also grants read of secrets across all connectors)
-  - keys/salts per value kind (PII, item id, etc)
-
-## [0.4.55]https://github.com/Worklytics/psoxy/release/tag/v0.4.55
+## [0.4.56](https://github.com/Worklytics/psoxy/release/tag/v0.4.56)
+ - due to refactoring, users of Microsoft connectors may see some moves of resources in Terraform
+   plan; these will be no-ops.
+
+## [0.4.55](https://github.com/Worklytics/psoxy/release/tag/v0.4.55)
 - For Microsoft Connectors:
   - reference to `msgraph` service principal has been replaced with `data` instead of `terraform` resource.
     For that reason you will see changes in plan related to these resource, but those changes are only related on the resources for Terraform
     state as the kind (*data* from *resource*) has been changed. No change will be done in real Entra ID for that.
   - dropping support for `/beta` endpoints in Microsoft Graph API. All endpoints are now using `/v1.0` version.
 
-## [0.4.53]https://github.com/Worklytics/psoxy/release/tag/v0.4.53)
-  - As Microsoft Azure Active Directory has been [renamed](https://learn.microsoft.com/en-us/entra/fundamentals/new-name) 
-    to Microsoft Entra ID, there is a new connector `msft-entra-id`. Old connector `azure-ad` is deprecated and in case of 
-    new connection, the new one `msft-entra-id` should be used. 
+## [0.4.53](https://github.com/Worklytics/psoxy/release/tag/v0.4.53)
+  - As Microsoft Azure Active Directory has been [renamed](https://learn.microsoft.com/en-us/entra/fundamentals/new-name)
+    to Microsoft Entra ID, there is a new connector `msft-entra-id`. Old connector `azure-ad` is deprecated and in case of
+    new connection, the new one `msft-entra-id` should be used.
 
 ## [0.4.52](https://github.com/Worklytics/psoxy/release/tag/v0.4.52)
   - BREAKING: default behavior for sub-addressing aka "plus addressing" of emails has changed; the
@@ -50,8 +41,8 @@ Changes to be including in future/planned release notes will be added here.
     if you fork an example > 0.4.52 and are attempting to migrate a proxy deployment initially built
     with modules or examples from < 0.4.52, you should explicitly add `email_canonicalization = "STRICT"`
     in your `terraform.tfvars`
-  - GCP: Existing GCP functions are using *Container Registry* for building their internal docker image where the psoxy code is deployed. However, 
-    this is [deprecated since May 2023 and starting Feb 2024](https://cloud.google.com/container-registry/docs/deprecations/container-registry-deprecation) it 
+  - GCP: Existing GCP functions are using *Container Registry* for building their internal docker image where the psoxy code is deployed. However,
+    this is [deprecated since May 2023 and starting Feb 2024](https://cloud.google.com/container-registry/docs/deprecations/container-registry-deprecation) it
     is required that functions use *Artifact Registry* instead. All deployments made since this version will use *Artifact Registry*
     default repository for storing all psoxy images. Any previous version before this version will work without any issue.
 
@@ -323,4 +314,19 @@ Features:
 Upgrade Notes:
   - secret management has been refactored; you may see indications of some secrets being moved, or
     even destroyed and recreated. If you plan shows SALT or ENCRYPTION_KEY as being destroyed,
-    **DO NOT** apply the plan and contact Worklytics support for assistance.
+    **DO NOT** apply the plan and contact Worklytics support for assistance.
+
+# Planned
+
+## 0.5 *future, subject to change!!*
+- RULES only via config management, never env variable
+- Eliminate "fall-through" configs.
+    - `PATH_TO_SHARED_CONFIG` - env var that locates shared parameters within the config store.
+    - `PATH_TO_CONNECTOR_CONFIG` - env var that locates connector-specific parameters within the
+      config store.
+- Expect distinct paths for the shared and connector scopes, to support more  straight-forward IAM
+  policies.
+    - eg, `PSOXY_SHARED` and `PSOXY_GCAL`, to allow IAM policies such as "read `PSOXY_SHARED*`" and
+      "read+write `PSOXY_GCAL*`" (if shared secrets have common prefix with connector secrets,
+      then wildcard policy to read shared also grants read of secrets across all connectors)
+- keys/salts per value kind (PII, item id, etc)
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -2,12 +2,12 @@
 
 * [PSOXY](README.md)
 * [Overview](overview.md)
+  * [Authentication](authentication-authorization)
 * [Install Prerequisites](prereqs-ubuntu.md)
 * [AWS](aws/README.md)
   * [Getting Started](aws/getting-started.md)
   * [Authentication and Authorization](aws/authentication-authorization.md)
   * [Getting Started with Cloud Shell](aws/cloud-shell.md)
-  * [AWS Development](aws/development.md)
   * [Encryption Keys in AWS](aws/encryption-keys.md)
   * [Protips](aws/protips.md)
   * [Guides](aws/guides/README.md)
@@ -17,6 +17,7 @@
     * [Lambdas on a VPC](aws/guides/lambdas-on-vpc.md)
     * [Least-Privileged Provisioning Role](aws/guides/least-privileged-provisioning-role.md)
   * [AWS Troubleshooting](aws/troubleshooting.md)
+  * [AWS Development](aws/development.md)
 * [GCP](gcp/README.md)
   * [Getting Started](gcp/getting-started.md)
   * [Authentication and Authorization](gcp/authentication-authorization.md)

diff --git a/docs/authentication-authorization.md b/docs/authentication-authorization.md
@@ -0,0 +1,53 @@
+# API Mode Authentication and Authorization
+
+There are two connection legs to consider with regard to authentication and authorization in API
+mode:
+1. between Worklytics and the proxy (your host cloud)
+2. between the proxy and the data source API
+
+Eg, Worklytics initiates an API request to the proxy (1); which, after validating the request,
+forwards it to the data source API on behalf of Worklytics (2), adding its additional authentication
+information.
+
+## Worklytics to Proxy (1)
+
+Worklytics is **authorized** to access your proxy instance via an Identity and Access Management (IAM)
+policy which you must configure in your host platform. The exact details vary by cloud provider:
+  - [AWS](aws/authentication-authorization.md)
+  - [GCP](gcp/authentication-authorization.md)
+
+Worklytics **authenticates** in all cases via Workload Identity Federation; as your Worklytics tenant is
+running natively in the cloud, it can leverage the cloud provider's native IAM service to establish
+identity which can be asserted to other services in the cloud.
+
+## Proxy to Data Source API (2)
+
+Although exact details vary by data source, most utilize some form of [OAuth 2.0](https://oauth.net/2/)
+for authorization and authentication.
+
+A data source admin (eg, a Google Workspace admin) must **authorize** the proxy to access the data
+source via the data source's admin console. This typically involves creating a new OAuth 2.0 client
+and granting that client a set of [oauth scopes](https://oauth.net/2/scope/) required to support the
+API calls that will be made on behalf of Worklytics.  A detailed list of scopes required for each
+data source is specified in the documentation of each connector.
+
+See https://docs.worklytics.co/psoxy#supported-data-sources
+
+The proxy **authenticates** itself for calls to the data source using one of the supported OAuth 2.0
+mechanisms, see [https://oauth.net/2/client-authentication/]. Most commonly, these are [Client Credentials](https://oauth.net/2/grant-types/client-credentials/)
+or [Workload Identity Federation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation).
+
+In particular, a quick overview for common sources:
+  - Microsoft 365 sources authenticate via Workload Identity Federation
+  - Google Workspace sources authenticate via Client Credentials (a GCP Service Account key)
+  - GitHub authenticates via Client Credentials (a GitHub App client id + key)
+  - Jira authenticates via Client Credentials (a Jira App client id + secret)
+  - Slack authenticates via Client Credentials (a Slack App token)
+  - Salesforce authenticates via Client Credentials (a Salesforce App client id + secret)
+  - Zoom authenticates via Client Credentials (a Zoom App client id + secret)
+
+In all cases relying on secrets (a key, client secret, token, etc) to authenticate, these values
+are stored in the secret store implementation of your Host cloud provider (eg, GCP Secret Manager)
+and **never** passed to or accessed by Worklytics.  Worklytics has no means to directly connect to
+any of your data sources.
+
diff --git a/docs/aws/authentication-authorization.md b/docs/aws/authentication-authorization.md
@@ -1,8 +1,11 @@
 # Authentication and Authorization in AWS Deployments of Psoxy
 
-This page provides an overview of how proxy authenticates and confirms authorization of clients
+This page provides an overview of how Psoxy authenticates and confirms authorization of clients
 (Worklytics tenants).
 
+For general overview of how Psoxy is authorized to access data sources, and authenticates when
+making API calls to those sources, see [API Mode Authentication and Authorization](../authentication-authorization.md).
+
 ## Authentication
 
 Each Worklytics tenant operates as a unique GCP service account within Google Cloud. GCP issues an

diff --git a/docs/gcp/authentication-authorization.md b/docs/gcp/authentication-authorization.md
@@ -3,6 +3,9 @@
 This page provides an overview of how psoxy authenticates and confirms authorization of clients
 (Worklytics tenants) to access data for GCP-hosted deployments.
 
+For general overview of how Psoxy is authorized to access data sources, and authenticates when
+making API calls to those sources, see [API Mode Authentication and Authorization](../authentication-authorization.md).
+
 ## Authentication
 
 As Worklytics tenants run inside GCP, they are implicitly authenticated by GCP. No secrets or keys

diff --git a/infra/examples-dev/aws-all/google-workspace.tf b/infra/examples-dev/aws-all/google-workspace.tf
@@ -8,7 +8,7 @@ provider "google" {
 
 module "worklytics_connectors_google_workspace" {
   source = "../../modules/worklytics-connectors-google-workspace"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.56"
 
   providers = {
     google = google.google_workspace

diff --git a/infra/examples-dev/aws-all/main.tf b/infra/examples-dev/aws-all/main.tf
@@ -21,7 +21,7 @@ terraform {
 # general cases
 module "worklytics_connectors" {
   source = "../../modules/worklytics-connectors"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors?ref=v0.4.56"
 
   enabled_connectors               = var.enabled_connectors
   jira_cloud_id                    = var.jira_cloud_id
@@ -35,6 +35,8 @@ module "worklytics_connectors" {
   github_organization              = var.github_organization
   github_example_repository        = var.github_example_repository
   salesforce_example_account_id    = var.salesforce_example_account_id
+  todos_as_local_files             = var.todos_as_local_files
+  todo_step                        = 1
 }
 
 # sources which require additional dependencies are split into distinct Terraform files, following
@@ -99,7 +101,7 @@ locals {
 
 module "psoxy" {
   source = "../../modules/aws-host"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-host?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-host?ref=v0.4.56"
 
   environment_name                     = var.environment_name
   aws_account_id                       = var.aws_account_id
@@ -133,6 +135,7 @@ module "psoxy" {
   custom_bulk_connector_rules          = var.custom_bulk_connector_rules
   custom_bulk_connector_arguments      = var.custom_bulk_connector_arguments
   todo_step                            = local.max_auth_todo_step
+  todos_as_local_files                 = var.todos_as_local_files
 
 
   #  vpc_config = {
@@ -155,17 +158,18 @@ module "connection_in_worklytics" {
   for_each = local.all_instances
 
   source = "../../modules/worklytics-psoxy-connection-aws"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-psoxy-connection-aws?ref=v0.4.55"
-
-  psoxy_instance_id  = each.key
-  worklytics_host    = var.worklytics_host
-  aws_region         = var.aws_region
-  aws_role_arn       = module.psoxy.caller_role_arn
-  psoxy_endpoint_url = try(each.value.endpoint_url, null)
-  bucket_name        = try(each.value.sanitized_bucket, null)
-  connector_id       = try(local.all_connectors[each.key].worklytics_connector_id, "")
-  display_name       = try(local.all_connectors[each.key].worklytics_connector_name, "${local.all_connectors[each.key].display_name} via Psoxy")
-  todo_step          = module.psoxy.next_todo_step
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-psoxy-connection-aws?ref=v0.4.56"
+
+  psoxy_instance_id    = each.key
+  worklytics_host      = var.worklytics_host
+  aws_region           = var.aws_region
+  aws_role_arn         = module.psoxy.caller_role_arn
+  psoxy_endpoint_url   = try(each.value.endpoint_url, null)
+  bucket_name          = try(each.value.sanitized_bucket, null)
+  connector_id         = try(local.all_connectors[each.key].worklytics_connector_id, "")
+  display_name         = try(local.all_connectors[each.key].worklytics_connector_name, "${local.all_connectors[each.key].display_name} via Psoxy")
+  todo_step            = module.psoxy.next_todo_step
+  todos_as_local_files = var.todos_as_local_files
 
   connector_settings_to_provide = try(each.value.settings_to_provide, {})
 

diff --git a/infra/examples-dev/aws-all/msft-365-variables.tf b/infra/examples-dev/aws-all/msft-365-variables.tf
@@ -7,10 +7,16 @@ variable "msft_tenant_id" {
 
 variable "msft_owners_email" {
   type        = set(string)
-  description = "(Only if config includes MSFT connectors). Optionally, set of emails to apply as owners on AAD apps apart from current logged user"
+  description = "(Only if config includes MSFT connectors). Optionally, set of emails to apply as owners on AAD apps apart from current logged user. Ignored if `existing_app_object_id` provided."
   default     = []
 }
 
+variable "msft_connector_app_object_id" {
+  type        = string
+  description = "BETA; if provided, the app corresponding to this object id will be used instead of creating new ones per source. User must ensure that roles/scopes are appropriate for the connector"
+  default     = null
+}
+
 variable "example_msft_user_guid" {
   type        = string
   description = "example MSFT user guid (uuid) for test API calls (OPTIONAL)"
@@ -45,4 +51,4 @@ variable "msft_teams_example_call_record_guid" {
   type        = string
   description = "example of MSFT Id (GUID) of a Teams Call Record for test API calls (OPTIONAL)"
   default     = "{EXAMPLE_MSFT_TEAMS_CALL_RECORD_GUID}"
-}
+}
diff --git a/infra/examples-dev/aws-all/msft-365.tf b/infra/examples-dev/aws-all/msft-365.tf
@@ -2,8 +2,7 @@
 
 module "worklytics_connectors_msft_365" {
   source = "../../modules/worklytics-connectors-msft-365"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-msft-365?ref=v0.4.55"
-
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-msft-365?ref=v0.4.56"
 
   enabled_connectors                  = var.enabled_connectors
   environment_id                      = var.environment_name
@@ -15,6 +14,7 @@ module "worklytics_connectors_msft_365" {
   msft_teams_example_chat_guid        = var.msft_teams_example_chat_guid
   msft_teams_example_call_guid        = var.msft_teams_example_call_guid
   msft_teams_example_call_record_guid = var.msft_teams_example_call_record_guid
+  msft_connector_app_object_id        = var.msft_connector_app_object_id
   todos_as_local_files                = var.todos_as_local_files
   todo_step                           = 1
 }
@@ -47,24 +47,37 @@ module "cognito_identity_pool" {
   count = local.msft_365_enabled ? 1 : 0 # only provision identity pool if MSFT-365 connectors are enabled
 
   source = "../../modules/aws-cognito-pool"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-pool?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-pool?ref=v0.4.56"
 
   developer_provider_name = local.developer_provider_name
   name                    = "${local.env_qualifier}-azure-ad-federation"
 }
 
+locals {
+
+  provision_entraid_apps = var.msft_connector_app_object_id == null
+  # either ONE shared, or ONE per connector
+  shared_connector = local.provision_entraid_apps ? null : module.worklytics_connectors_msft_365.enabled_api_connectors[keys(module.worklytics_connectors_msft_365.enabled_api_connectors)[0]]
+  cognito_identity_login_ids = local.provision_entraid_apps ? {
+      for k, msft_connector in module.worklytics_connectors_msft_365.enabled_api_connectors :
+      k => msft_connector.connector.client_id
+    } : {
+      "shared" : local.shared_connector.connector.client_id
+    }
+}
+
 module "cognito_identity" {
   count = local.msft_365_enabled ? 1 : 0 # only provision identity pool if MSFT-365 connectors are enabled
 
   source = "../../modules/aws-cognito-identity-cli"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-identity-cli?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-identity-cli?ref=v0.4.56"
 
   aws_region       = data.aws_region.current.id
   aws_role         = var.aws_assume_role_arn
   identity_pool_id = module.cognito_identity_pool[0].pool_id
   login_ids = {
-    for k, v in module.worklytics_connectors_msft_365.enabled_api_connectors :
-    k => "${local.developer_provider_name}=${v.connector.application_id}"
+    for k, client_id in local.cognito_identity_login_ids :
+     k => "${local.developer_provider_name}=${client_id}"
   }
 }
 
@@ -75,13 +88,28 @@ resource "aws_iam_role_policy_attachment" "cognito_lambda_policy" {
   policy_arn = module.cognito_identity_pool[0].policy_arn
 }
 
+locals {
+
+  enabled_to_entraid_object = { for k, msft_connector in module.worklytics_connectors_msft_365.enabled_api_connectors : k => {
+      connector_id: msft_connector.connector.id
+      display_name: msft_connector.display_name
+    }
+  }
+  shared_to_entraid_object = {
+    "shared" : {
+      connector_id: try(local.shared_connector.connector.id, null),
+      display_name: "Shared"
+    }
+  }
+}
+
 module "msft_connection_auth_federation" {
-  for_each = module.worklytics_connectors_msft_365.enabled_api_connectors
+  for_each = local.provision_entraid_apps ? local.enabled_to_entraid_object : local.shared_to_entraid_object
 
   source = "../../modules/azuread-federated-credentials"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/azuread-federated-credentials?ref=v0.4.55"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/azuread-federated-credentials?ref=v0.4.56"
 
-  application_object_id = each.value.connector.id
+  application_object_id = each.value.connector_id
   display_name          = "${local.env_qualifier}AccessFromAWS"
   description           = "AWS federation to be used for ${local.env_qualifier} Connectors - ${each.value.display_name}${var.connector_display_name_suffix}"
   issuer                = "https://cognito-identity.amazonaws.com"
@@ -96,7 +124,7 @@ locals {
       environment_variables = merge(try(msft_connector.environment_variables, {}),
         {
           IDENTITY_POOL_ID  = module.cognito_identity_pool[0].pool_id,
-          IDENTITY_ID       = module.cognito_identity[0].identity_id[k],
+          IDENTITY_ID       = try(module.cognito_identity[0].identity_id[k], module.cognito_identity[0].identity_id["shared"]),
           DEVELOPER_NAME_ID = local.developer_provider_name
         }
       )