E2E Workflow scripts test against Enterprise follower in K8s #349

john-odonnell · 2021-07-02T01:42:08Z

What does this PR do?

End-to-end workflow scripts can now test Conjur Kubernetes authentication against Conjur Enterprise
with follower in K8s. Uses cyberark/kubernetes-conjur-deploy
to setup a Conjur leader, standbys and follower in a GKE cluster.

Open Source / Enterprise workflow can be toggled from start script with --open-source or --enterprise flags

What ticket does this PR close?

Resolves #243

Checklists

Change log

The CHANGELOG has been updated, or
This PR does not include user-facing changes and doesn't require a CHANGELOG update

Test coverage

This PR includes new unit and integration tests to go with the code changes, or
The changes in this PR do not require tests

Documentation

Docs (e.g. READMEs) were updated in this PR, and/or there is a follow-on issue to update docs, or
This PR does not require updating any documentation

Manual tests

If you are preparing for a release, have you run the following manual tests to verify existing functionality continues to function as expected?

Manually run Kubernetes-Conjur demo with a local authn-k8s client image build

diverdane

This is GOOD stuff, thanks for taking on this challenge!
I'm about 1/2-way through, will finish later today.
Looks very good so far!

General comment, not specific to this PR, but at some point, we need to add CodeClimate to this repo.

diverdane · 2021-07-19T12:56:38Z

Jenkinsfile

@@ -47,6 +47,12 @@ pipeline {
      }
    }

+    stage('E2E Workflow Tests') {


I think this should be moved to run after the "Scan images" test stage. The general idea is to scan them before using them.
Also, can we add a "GKE" to the name of the stage (or add a sub-stage with "GKE" init, since we'll eventually have an OpenShift test) to distinguish these from the GitHub actions / KinD E2E tests?

Moved, and included a sub-stage.

bin/test-workflow/0_prep_env.sh

bin/test-workflow/1_deploy_conjur.sh

diverdane · 2021-07-19T13:55:27Z

bin/test-workflow/0_prep_env.sh

+export TEST_PLATFORM
+
+if [[ "${TEST_PLATFORM}" == "oc" ]]; then
+  PLATFORM="openshift"


Trying to figure out why we need PLATFORM in addtion to TEST_PLATFORM. Do you happen to know why both are needed?

After a little investigation, the only practical use of the long-form PLATFORM was when referencing directories of manifests (i.e. bin/test-workflow/kubernetes and bin/test-workflow/openshift which I assume is soon to come). Both were being used to run commands conditionally for Openshift.

I changed the long-form PLATFORM to MANIFEST_DIR, and TEST_PLATFORM to PLATFORM.
PLATFORM is used in conditionals.
MANIFEST_DIR is only used where the long-form "kubernetes" or "openshift" is needed.

EDIT
Going to have to roll this back. Kubernetes Conjur Deploy requires the PLATFORM envvar to differentiate between kubernetes and openshift. In the context of these tests, PLATFORM will be used to differentiate general platforms (k8s v. oc) and TEST_PLATFORM (rename to CLUSTER_TYPE, for specificity?) will be used to differentiate between sub-platforms (kind v. gke).

AH, thanks for the explanation. I was thinking that the distinction between these 2 would be something like this.
Either way, maybe we can add a comment just above this line, since I suspect other readers may wonder what the difference is here.

bin/test-workflow/2_admin_load_conjur_policies.sh

bin/test-workflow/platform_login.sh

diverdane

Looks good, nice work!!! I have mostly nit editorial comments.

bin/test-workflow/0_prep_env.sh

bin/test-workflow/6_app_build_and_push_containers.sh

diverdane · 2021-07-19T22:01:38Z

bin/test-workflow/7_app_deploy_backend.sh

@@ -37,6 +37,7 @@ $cli --namespace $TEST_APP_NAMESPACE_NAME \
  --from-file=server.key=./etc/ca-key.pem

 helm repo add bitnami https://charts.bitnami.com/bitnami
+helm repo update


Nice catch!

bin/test-workflow/8_app_deploy.sh

bin/test-workflow/stop

bin/test-workflow/utils.sh

john-odonnell · 2021-07-20T15:53:06Z

@diverdane Thanks for the review(s)! Just wondering - a lot of comments for wrapping envvar references in quotes. What does this achieve that the bare envvar reference doesn't?

diverdane · 2021-07-20T18:31:32Z

...a lot of comments for wrapping envvar references in quotes. What does this achieve that the bare envvar reference doesn't?

@john-odonnell , In Bash, it's generally good practice to wrap any variables and command substitutions (e.g. result="$(some_command and its args)"). From the "Quoting" section of the Google Bash Style Guide:

* Always quote strings containing variables, command substitutions, spaces or shell meta characters, unless careful unquoted expansion is required or it’s a shell-internal integer (see next point).

The idea is that you want to make sure the shell interprets the variable value as a single entity regardless of whether it contains spaces, etc.

If the variable is part of a larger string that's already quoted, then the quotes directly around the variable aren't needed.

I think CodeClimate would flag these, if we had it running for this repository. I think you can run CodeClimate locally from a Docker container.

diverdane

Looks FANTASTIC! Thanks for taking care of the missing quotes in existing code, that will make it easier when we turn on CodeClimate for this repo!

I'm ready to approve as soon as the kubectl install is working (and we at least upgrade the default version of kubectl).

diverdane · 2021-07-22T12:26:54Z

bin/test-workflow/0_prep_env.sh

+export CONJUR_OSS_HELM_INSTALLED="${CONJUR_OSS_HELM_INSTALLED:-true}"
+
+if [[ "$CONJUR_OSS_HELM_INSTALLED" == "true" ]]; then
+  CLUSTER_TYPE="${CLUSTER_TYPE:-kind}"


Thanks for making this name change, it's MUCH clearer!

diverdane · 2021-07-22T12:44:23Z

bin/test-workflow/Dockerfile

+
+# Install kubectl CLI
+ARG KUBECTL_CLI_URL
+RUN wget -O /usr/local/bin/kubectl ${KUBECTL_CLI_URL:-https://storage.googleapis.com/kubernetes-release/release/v1.7.6/bin/linux/amd64/kubectl} && \


We should at least update the default kubectl version to 1.20.9 or 1.21.3.
And maybe use the command from the latest kubectl install documentation (even if the new command gets redirected to storage.googleapis.com. That would look like this:

curl -LO "${KUBECTL_CLI_URL:-https://dl.k8s.io/release/v1.21.3/bin/linux/amd64/kubectl}" && \ chmod +x kubectl && \ mv kubectl /usr/local/bin/kubectl

Or, maybe even better, if we simplify things and just look for a Kubernetes version:

curl -LO https://dl.k8s.io/release/v"${KUBECTL_VERSION:-1.21.3}"/bin/linux/amd64/kubectl && \ chmod +x kubectl && \ mv kubectl /usr/local/bin/kubectl

Updated to the second version, and updated secrets.yml to account for this.

imheresamir

Thanks for taking on this challenge!

bin/test-workflow/4_admin_cluster_prep.sh

Conjur Leader and Follower in K8s cluster.

Maintains style with TEST_APP_NAMESPACE_NAME, and agrees with kubernetes-conjur-deploy required env var

diverdane

Still LGTM!!!

@john-odonnell

@john-odonnell will address concerns in next PR. @samir is okay with this.

imheresamir

LGTM

* Maintain shell on failure * Test E2E workflow against Conjur Enterprise in KinD Conjur Leader and Follower in K8s cluster. * Enterprise E2E workflow updated for GKE * E2E workflow: CONJUR_NAMESPACE to CONJUR_NAMESPACE_NAME Maintains style with TEST_APP_NAMESPACE_NAME, and agrees with kubernetes-conjur-deploy required env var * Add Jenkinsfile stage for Enterprise/GKE E2E workflow * Update CI GHA workflow * E2E Workflow README updates * PR Review updates

john-odonnell force-pushed the e2e-follower-in-k8s branch 2 times, most recently from 485cbbd to d56db09 Compare July 13, 2021 17:07

john-odonnell force-pushed the e2e-follower-in-k8s branch 5 times, most recently from a6a7127 to b4da873 Compare July 14, 2021 16:52

john-odonnell marked this pull request as ready for review July 14, 2021 17:20

john-odonnell requested review from a team as code owners July 14, 2021 17:20

diverdane reviewed Jul 19, 2021

View reviewed changes

john-odonnell force-pushed the e2e-follower-in-k8s branch 2 times, most recently from 3283e88 to 4641b55 Compare July 21, 2021 22:36

diverdane reviewed Jul 22, 2021

View reviewed changes

john-odonnell force-pushed the e2e-follower-in-k8s branch 4 times, most recently from 859dbc6 to 73a057c Compare July 22, 2021 16:18

imheresamir previously requested changes Jul 22, 2021

View reviewed changes

bin/test-workflow/4_admin_cluster_prep.sh Show resolved Hide resolved

diverdane previously approved these changes Jul 22, 2021

View reviewed changes

john-odonnell added 7 commits July 22, 2021 15:07

Maintain shell on failure

18586bf

Test E2E workflow against Conjur Enterprise in KinD

fcf70a3

Conjur Leader and Follower in K8s cluster.

Enterprise E2E workflow updated for GKE

3677551

E2E workflow: CONJUR_NAMESPACE to CONJUR_NAMESPACE_NAME

a767703

Maintains style with TEST_APP_NAMESPACE_NAME, and agrees with kubernetes-conjur-deploy required env var

Add Jenkinsfile stage for Enterprise/GKE E2E workflow

a4d36c1

Update CI GHA workflow

e002b66

E2E Workflow README updates

a6bb5bd

PR Review updates

d2d53e8

john-odonnell dismissed diverdane’s stale review via d2d53e8 July 22, 2021 19:08

john-odonnell force-pushed the e2e-follower-in-k8s branch from 73a057c to d2d53e8 Compare July 22, 2021 19:08

diverdane approved these changes Jul 22, 2021

View reviewed changes

diverdane requested a review from imheresamir July 22, 2021 19:20

imheresamir approved these changes Jul 22, 2021

View reviewed changes

john-odonnell merged commit 7a26e5d into master Jul 22, 2021

john-odonnell deleted the e2e-follower-in-k8s branch July 22, 2021 19:40

john-odonnell restored the e2e-follower-in-k8s branch July 22, 2021 19:40

imheresamir deleted the e2e-follower-in-k8s branch July 22, 2021 19:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E Workflow scripts test against Enterprise follower in K8s #349

E2E Workflow scripts test against Enterprise follower in K8s #349

john-odonnell commented Jul 2, 2021 •

edited

Loading

diverdane left a comment

diverdane Jul 19, 2021

john-odonnell Jul 21, 2021

diverdane Jul 19, 2021 •

edited

Loading

john-odonnell Jul 21, 2021 •

edited

Loading

diverdane Jul 21, 2021

diverdane left a comment

diverdane Jul 19, 2021

john-odonnell commented Jul 20, 2021

diverdane commented Jul 20, 2021

diverdane left a comment

diverdane Jul 22, 2021

diverdane Jul 22, 2021 •

edited

Loading

john-odonnell Jul 22, 2021

imheresamir left a comment

diverdane left a comment

imheresamir left a comment

E2E Workflow scripts test against Enterprise follower in K8s #349

E2E Workflow scripts test against Enterprise follower in K8s #349

Conversation

john-odonnell commented Jul 2, 2021 • edited Loading

What does this PR do?

What ticket does this PR close?

Checklists

Change log

Test coverage

Documentation

Manual tests

diverdane left a comment

Choose a reason for hiding this comment

diverdane Jul 19, 2021

Choose a reason for hiding this comment

john-odonnell Jul 21, 2021

Choose a reason for hiding this comment

diverdane Jul 19, 2021 • edited Loading

Choose a reason for hiding this comment

john-odonnell Jul 21, 2021 • edited Loading

Choose a reason for hiding this comment

diverdane Jul 21, 2021

Choose a reason for hiding this comment

diverdane left a comment

Choose a reason for hiding this comment

diverdane Jul 19, 2021

Choose a reason for hiding this comment

john-odonnell commented Jul 20, 2021

diverdane commented Jul 20, 2021

diverdane left a comment

Choose a reason for hiding this comment

diverdane Jul 22, 2021

Choose a reason for hiding this comment

diverdane Jul 22, 2021 • edited Loading

Choose a reason for hiding this comment

john-odonnell Jul 22, 2021

Choose a reason for hiding this comment

imheresamir left a comment

Choose a reason for hiding this comment

diverdane left a comment

Choose a reason for hiding this comment

imheresamir left a comment

Choose a reason for hiding this comment

john-odonnell commented Jul 2, 2021 •

edited

Loading

diverdane Jul 19, 2021 •

edited

Loading

john-odonnell Jul 21, 2021 •

edited

Loading

diverdane Jul 22, 2021 •

edited

Loading