[CDAP-21096] Add support for startupProbe in CDAP master

[vsethi09]

CDAP-21096 Add support for startupProbe in CDAP master

Context

In addition to the rolling upgrade strategy, for high availability of APIs, startup probe need to be implemented to ensure that the CDAP service pods (eg. Appfabric) accepts requests only when the server has started.

CDAP services (eg. Appfabric) run on dynamic port and announces the port after the server has started. For this command probe can be configured to check the status of system service using the URL ‘/v3/system/services//status’.

CDAP services will write the port to a temp file in the container after startup. Here is a code snippet for startup probe using the port written to the temp file:

startupProbe:
          exec:
            command:
            - sh
            - -c
            - curl -s -f -k https://localhost:$(cat /tmp/appfabric.port)/v3/system/services/appfabric/status
          failureThreshold: 2
          periodSeconds: 2
          successThreshold: 2
          timeoutSeconds: 1

Change Description

• Introduced StartupProbe as an optional field in the CDAPServiceSpec.
• The StartupProbe config of type corev1.Probe will be set to the container spec.
• Auto-generated:
  • api/v1alpha1/zz_generated.deepcopy.go
  • config/crd/bases/cdap.cdap.io_cdapmasters.yaml
• Makefile: To fix unit tests, the max request size of etcd database used by controller-runtime/tools/setup-envtest had to be increased from default 1.5 MB to 2.5 MB. This is due to the increase in size of generated CRD.

Verification

• Unit tests: make test
• Docker image: deployed locally built docker image and edited the cdapmaster manifest to add startupProbe to appfabric spec.