Merge pull request #210 from bento-platform/releases/v15.1

Release v15.1

README.md

@@ -23,7 +23,7 @@ that make up the Bento platform.
 
 ## Requirements
 - Docker >= 19.03.8
-- Docker Compose >= 2.14.0 (plugin form: you should have the `docker compose` command available, without a dash)
+- Docker Compose >= 2.14.0, <2.24 (plugin form: you should have the `docker compose` command available, without a dash)
 - Python >= 3.9 (for `bentoctl`); the services require Python 3.10 but this is included in their Docker images. 
 
 
@@ -35,6 +35,7 @@ that make up the Bento platform.
 * [Installation](./docs/installation.md)
 * [Development](./docs/development.md)
 * [Troubleshooting guide](./docs/troubleshooting.md)
+* [Deployment](./docs/deployment.md)
 
 ### Data ingestion and usage
 
@@ -43,6 +44,7 @@ that make up the Bento platform.
 
 ### Migration documents
 
+* [v15 to v15.1](./docs/migrating_to_15_1.md)
 * [v14 to v15](./docs/migrating_to_15.md)
   * [Converting Phenopackets from V1 to V2 using `bentoctl`](./docs/phenopackets_v1_to_v2.md) 
 * [v13 to v14](./docs/migrating_to_14.md)

bentoctl.bash

@@ -1,20 +1,29 @@
 #!/usr/bin/env bash
 
-set -o allexport
+# This file is a shim between the user and the `py_bentoctl` management CLI module. Its job is two-fold:
+#  - Load any environment variables from the various .env and .bash files, which configure a given Bento deployment.
+#  - Pass the configured environment, and CLI arguments, to the Python module.
 
-# Sourcing order
-#  1. default_config.env (declare some defaults for configurable variables / make them exist)
-#  2. local.env          (overrides for default config)
-#  3. bento.env          (uses some local.env values)
-#  4. local.env *again*  (overrides for bento.env values, e.g. versions)
+# Automatically export any variables set in this shell to processes launched from this shell (i.e., `py_bentoctl`):
+set -o allexport
 
+# ENVIRONMENT CONFIGURATION --------------------------------------------------------------------------------------------
+# Sourcing order:
+#  1. default_config.env     (declare some defaults for configurable variables / make them exist)
 source etc/default_config.env
-if [[ -f "local.env" ]]; then
-  source local.env
-fi
+#  2. local.env              (overrides for default config)
+if [[ -f "local.env" ]]; then source local.env; fi
+#  3. bento.env              (uses some local.env values)
 source etc/bento.env
-if [[ -f "local.env" ]]; then
-  source local.env
-fi
+#  4. local.env *again*      (overrides for bento.env values, e.g. versions)
+if [[ -f "local.env" ]]; then source local.env; fi
+#  5. bento_post_config.bash (script, rather than .env, for branching decisions based on environment)
+source etc/bento_post_config.bash
+# ----------------------------------------------------------------------------------------------------------------------
 
+# After configuring the environment, take any CLI arguments that were passed to this script and forward them to the
+# Python bentoctl module, which will perform the actual execution:
 python3 -m py_bentoctl.entry "$@"
+
+# Reverse change to variable-setting behaviour from the beginning of the script:
+set +o allexport

docker-compose.yaml

@@ -12,10 +12,13 @@ services:
       - BENTOV2_DOMAIN
       - BENTOV2_PORTAL_DOMAIN
       - BENTOV2_CBIOPORTAL_DOMAIN
+      - BENTO_DOMAIN_REDIRECT
 
       # Feature flags:
       - BENTOV2_USE_EXTERNAL_IDP
       - BENTOV2_USE_BENTO_PUBLIC
+      - BENTO_USE_DOMAIN_REDIRECT
+      - BENTO_GATEWAY_USE_TLS
       - BENTO_BEACON_ENABLED
       - BENTO_CBIOPORTAL_ENABLED
       - BENTO_GOHAN_ENABLED
@@ -29,6 +32,8 @@ services:
       - BENTOV2_GATEWAY_INTERNAL_PRIVKEY_RELATIVE_PATH
       - BENTOV2_GATEWAY_INTERNAL_CBIOPORTAL_FULLCHAIN_RELATIVE_PATH
       - BENTOV2_GATEWAY_INTERNAL_CBIOPORTAL_PRIVKEY_RELATIVE_PATH
+      - BENTO_GATEWAY_INTERNAL_REDIRECT_FULLCHAIN_RELATIVE_PATH
+      - BENTO_GATEWAY_INTERNAL_REDIRECT_PRIVKEY_RELATIVE_PATH
 
       - BENTOV2_GATEWAY_PUBLIC_ALLOW_FRAME_DOMAINS
 

docs/deployment.md

@@ -0,0 +1,101 @@
+# Deployment
+
+This section details the required steps to deploy Bento on a server and expose it to the internet.
+The instructions assume you have a server with a public IP and DNS records pointing to it for the domains you will use.
+
+A bento deployment usually has up to 4 domains or subdomains, this section uses the following examples:
+```bash
+BENTOV2_DOMAIN=bento.example.com
+BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
+BENTOV2_AUTH_DOMAIN=auth.${BENTOV2_DOMAIN}
+BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+```
+
+For a real deployment, make sure that your `local.env` file uses valid domain names for which SSL certificates 
+can be obtained.
+
+## Certificates paths
+
+In production mode, Bento must use valid SSL certificates instead of self-signed ones.
+Before requesting certificates, make sure that you override the following environment 
+variables in your `local.env` file.
+
+```bash
+BENTOV2_AUTH_FULLCHAIN_RELATIVE_PATH=/live/$BENTOV2_AUTH_DOMAIN/fullchain.pem
+BENTOV2_AUTH_PRIVKEY_RELATIVE_PATH=/live/$BENTOV2_AUTH_DOMAIN/privkey.pem
+BENTOV2_GATEWAY_INTERNAL_PORTAL_FULLCHAIN_RELATIVE_PATH=/live/$BENTOV2_PORTAL_DOMAIN/fullchain.pem
+BENTOV2_GATEWAY_INTERNAL_PORTAL_PRIVKEY_RELATIVE_PATH=/live/$BENTOV2_PORTAL_DOMAIN/privkey.pem
+BENTOV2_GATEWAY_INTERNAL_FULLCHAIN_RELATIVE_PATH=/live/$BENTOV2_DOMAIN/fullchain.pem
+BENTOV2_GATEWAY_INTERNAL_PRIVKEY_RELATIVE_PATH=/live/$BENTOV2_DOMAIN/privkey.pem
+```
+
+## Obtain certificates
+
+On a fresh Bento install, the initial SSL certificates can be obtained with a convenience script.
+
+```bash
+bash ./etc/scripts/init_certs_only.sh
+
+# Answer CertBot prompts
+# Does a --dry-run first
+
+# Check the certificates
+ls ./certs/gateway/letsencrypt
+ls ./certs/auth/letsencrypt
+```
+
+## Renew certificates
+
+To renew the certificates after they are issued, use the convenience script made to this effect:
+
+```bash
+bash ./etc/scripts/renew_certs_and_redeploy.sh
+# Answer CertBot prompts
+# Does a --dry-run first
+
+# Check the certificates
+ls ./certs/gateway/letsencrypt/live
+ls ./certs/auth/letsencrypt/live
+```
+
+Note that this script will shutdown auth and gateway during the certificate renewal.
+
+## Permanently redirect a domain
+
+If a Bento instance must use a new domain, the gateway can redirect requests from the old domain to the new one. 
+This is advised to prevent dead links, see [301](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301) 
+response status reference.
+
+For this to work, you **must** update the old domain's DNS record to point to the new instance's IP, 
+otherwise you won't be able to obtain the certificates.
+
+First, declare these variables in the `local.env` file:
+
+```bash
+BENTO_DOMAIN_REDIRECT=old-bento.example.com     # The old domain name
+BENTO_USE_DOMAIN_REDIRECT='true'                # Feature flag for domain redirect
+BENTO_GATEWAY_INTERNAL_REDIRECT_FULLCHAIN_RELATIVE_PATH=/live/$BENTO_DOMAIN_REDIRECT/fullchain.pem
+BENTO_GATEWAY_INTERNAL_REDIRECT_PRIVKEY_RELATIVE_PATH=/live/$BENTO_DOMAIN_REDIRECT/privkey.pem
+```
+
+Obtain the certificate for the old domain with certbot:
+```bash
+# Stop gateway
+./bentoctl.bash stop gateway
+
+# Obtain certificates
+docker run -it --rm --name certbot \
+    -v "./certs/gateway/letsencrypt:/etc/letsencrypt" \
+    -v "./certs/gateway/letsencrypt/lib:/var/lib/letsencrypt" \
+    -p 80:80 -p 443:443 \
+    certbot/certbot certonly -d old-bento.example.com -d portal.old-bento.example.com
+
+# Subdomains are optional but recommended, add the ones you want redirected with the '-d' flag
+# This creates a single certificate, even if using multiple subdomains.
+ls certs/gateway/letsencrypt/live/
+
+# Start the gateway
+./bentoctl.bash run gateway
+```
+
+If all went well, the `old-bento.example.com` domain should be redirected to `bento.example.com` in a browser.

docs/development.md

@@ -193,7 +193,7 @@ Katsu database during development.
 Go to `localhost:8080` to access the login page. Fill the fields with the values shown below, using the value of
 `BENTOV2_KATSU_DB_PASSWORD` for the password field.
 
-![Adminer login](docs/img/adminer_login.png)
+![Adminer login](img/adminer_login.png)
 
 
 ## Using Elasticvue
@@ -208,7 +208,7 @@ port 9200 (e.g. http://192.168.48.2:9200), it can be found with this command:
 
 Note: the CORS instructions have already been taken care of in the `docker-compose.dev.yaml` file.
 
-![Elasticvue login](docs/img/elasticview_login.png)
+![Elasticvue login](img/elasticview_login.png)
 
 
 ## Converting Phenopackets

docs/installation.md

@@ -23,7 +23,7 @@ cp ./etc/bento_dev.env local.env
 cp ./etc/bento_deploy.env local.env
 ```
 
-Then, modify the values as seen; depending on if you're using the instance for development or deployment.
+Then, modify the values as needed; depending on if you're using the instance for development or deployment.
 
 
 #### Development example
@@ -46,9 +46,14 @@ BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
 # Feature switches ----------------------------------------------------
 BENTOV2_USE_EXTERNAL_IDP=0
 BENTOV2_USE_BENTO_PUBLIC=1
-BENTOV2_PRIVATE_MODE=false
 
-BENTO_CBIOPORTAL_ENABLED=false
+#  - Switch to enable TLS - defaults to true (i.e., use TLS):
+BENTO_GATEWAY_USE_TLS='true'
+
+BENTO_BEACON_ENABLED='true'
+BENTO_BEACON_UI_ENABLED='true'
+BENTO_CBIOPORTAL_ENABLED='false'
+BENTO_GOHAN_ENABLED='true'
 # ---------------------------------------------------------------------
 
 # set this to a data storage location, optionally within the repo itself, like: /path-to-my-bentov2-repo/data
@@ -63,6 +68,8 @@ BENTOV2_SESSION_SECRET=my-very-secret-session-secret  # !!! ADD SOMETHING MORE S
 
 #  - Set auth DB password if using a local IDP
 BENTO_AUTH_DB_PASSWORD=some-secure-password
+#  - Always set authz DB password
+BENTO_AUTHZ_DB_PASSWORD=some-other-secure-password
 
 BENTOV2_AUTH_ADMIN_USER=admin
 BENTOV2_AUTH_ADMIN_PASSWORD=admin  # !!! obviously for dev only !!!
@@ -89,6 +96,13 @@ BENTO_GIT_NAME=David  # Change this to your name
 [email protected]  # Change this to your GitHub account email
 ```
 
+You should at least fill to the following settings in dev mode (it may differ for a production setup), which are not set 
+in the example file:
+* `BENTOV2_SESSION_SECRET`
+* `BENTO_AUTH_DB_PASSWORD`
+* `BENTO_AUTHZ_DB_PASSWORD`
+* `BENTO_WES_CLIENT_SECRET`
+
 If the internal OIDC identity provider (IdP) is being used (by default, Keycloak), variables specifying default 
 credentials should also be provided. The *admin* credentials are used to connect to the Keycloak UI for authentication 
 management (adding users, getting client credentials, ...). The *test* credentials will be used to authenticate on the 
@@ -243,7 +257,8 @@ which in Keycloak should be a UUID.
 
 ### b. Create grants for the Workflow Execution Service (WES) OAuth2 client
 
-Run the following commands to set up authorization for the WES client:
+Run the following commands to set up authorization for the WES client. Don't forget to replace `ISSUER_HERE` by the 
+issuer URL!
 
 ```bash
 # This grant is a temporary hack to get permissions working for v12/v13. In the future, it should be removed.

docs/migrating_to_15.md

@@ -30,14 +30,19 @@ Before updating, perform the following steps:
 ```
 
 
-## 4. Restart Bento
+## 4. Set new environment variables
+
+In `local.env`, set a secure value for `BENTO_REFERENCE_DB_PASSWORD`.
+
+
+## 5. Restart Bento
 
 ```bash
 ./bentoctl.bash start
 ```
 
 
-## 5. Add permissions
+## 6. Add permissions
 
 Add permissions as needed for reference service content management:
 

docs/migrating_to_15_1.md

@@ -0,0 +1,28 @@
+# Migrating to Bento v15.1
+
+Migrating to version 15.1 from version 15 should be straightforward.
+
+
+## 1. Update `bentoctl` Python requirements
+
+Make sure you've entered into the `bentoctl` Python virtual environment, if you've set one up:
+
+```bash
+source env/bin/activate
+```
+
+Then, install the latest requirements / module versions:
+
+```bash
+pip install -r requirements
+```
+
+
+## 2. Update services and restart
+
+Run the following commands to pull the latest service images and restart services as needed:
+
+```bash
+./bentoctl.bash pull
+./bentoctl.bash start
+```