Unable to use our own certificates via the operator using TLS env variables in temporal server

[vinimona]

We are having a little hard time using our own set of certificates with the operator. It seems like temporal operator is always using the certificates generated by the cert-manager. We don't want to use any cert-manager and want to use the certs generated in our eco-system. Can you throw some lights on the steps (to override the config) so that we can use our certificates? I see a note here which mentions that we can use our own certificates, but it doesn't include any details on how to do that.

[alexandrevilain]

Hi!

Indeed, this feature was planned but is not developed for now.

I was waiting for user feedback before adding this feature.

Could you please give me more insights about your use-case ?

[vinimona]

Thanks for responding. Firstly, we can't use self signed certificates. I don't have a lot of details for you on the use-case, but we know where the certificates are present and we want to use those by setting the cert paths via the temporal TLS environment variables (mostly using the overrides spec for each temporal service). We are able to override the environment variables for Admintools and ui services, but not for temporal frontend/matching/history/worker services. As we understand the operator code, it appears that it is not using those environment variables for temporal server and directly using cert-manager mounted cert paths in temporal configuration. If you could enable the default mTLS configuration via the env variables, that would be great.

Also, Is there a way we can completely get rid of cert-manager? We gave it a try to install temporal operator without cert-manager, but it didn't work.

[vinimona]

Hi, Can you please share the expected timeline that this will be rolled out? We are planning to use the operator but we don't want to get stuck with using cert-manager (which has its own limitations on the issuers etc. at least for our use-case). So we are looking forward to use default setup by overriding TLS env variables. Hope to hear from you soon. Thanks again.

[alexandrevilain]

Hi!
Sorry for the delay @vinimona !
I can't give any ETA for this feature as I'm maintaining this in my free time.

[vinimona]

@alexandrevilain Thanks for responding. It is about allowing/enabling the overrides on TEMPORAL_TLS* environment variables for each of frontend/history/matching/worker/internalFrontend services, like how it is for admin-tools and temporal-ui. This will make it flexible enough to mount and read certificates from a given location. It would be great if you could still try for it.

[alexandrevilain]

It is about allowing/enabling the overrides on TEMPORAL_TLS* environment variables for each of frontend/history/matching/worker/internalFrontend services, like how it is for admin-tools and temporal-ui.

This should already be doable by following: https://alexandrevilain.github.io/temporal-operator/features/overrides/#overrides-for-all-services Did you tried it ?

[alexandrevilain]

To speak about a persistent API design to allow this, I think we have to rework current API.
Most of the cert-manager customization fields are in the spec.mTLS which is bad.
I think we can rework current API for cert-manager to get something like:

kind: TemporalCluster
spec:
  mTLS:
    provider: cert-manager
    internode:
      enabled: true
      certmanager:
        certificateDuration: 1h
    frontend:
      enabled: true
      certmanager:
        certificateDuration: 1h
        clientCertificatesDuration: 1h
    certmanager:
      certificatesDuration:
        rootCACertificate: 2h
        intermediateCAsCertificates: 1h30m
      refreshInterval: 5m
      renewBefore: 2m

This will allow addition to a new "selfsigned" mTLS provider which could provide the following API:

kind: TemporalCluster
spec:
  mTLS:
    provider: selfsigned
    internode:
      enabled: true
      selfsigned: {}
    frontend:
      enabled: true
      selfsigned: {}

Fields still need to be defined inside each selfsigned keys.

[vinimona]

It is about allowing/enabling the overrides on TEMPORAL_TLS* environment variables for each of frontend/history/matching/worker/internalFrontend services, like how it is for admin-tools and temporal-ui.

This should already be doable by following: https://alexandrevilain.github.io/temporal-operator/features/overrides/#overrides-for-all-services Did you tried it ?

This doesn't work. In fact, I raised this issue only after I tried the overrides for each service.

[vinimona]

To speak about a persistent API design to allow this, I think we have to rework current API. Most of the cert-manager customization fields are in the spec.mTLS which is bad. I think we can rework current API for cert-manager to get something like:

kind: TemporalCluster
spec:
  mTLS:
    provider: cert-manager
    internode:
      enabled: true
      certmanager:
        certificateDuration: 1h
    frontend:
      enabled: true
      certmanager:
        certificateDuration: 1h
        clientCertificatesDuration: 1h
    certmanager:
      certificatesDuration:
        rootCACertificate: 2h
        intermediateCAsCertificates: 1h30m
      refreshInterval: 5m
      renewBefore: 2m

This will allow addition to a new "selfsigned" mTLS provider which could provide the following API:

kind: TemporalCluster
spec:
  mTLS:
    provider: selfsigned
    internode:
      enabled: true
      selfsigned: {}
    frontend:
      enabled: true
      selfsigned: {}

Fields still need to be defined inside each selfsigned keys.

I would suggest to use a better naming than calling it selfsigned, it could easily be confused with self-signed certs provided by cert-manager. Again, just a suggestion. Thanks.

[alexandrevilain]

This doesn't work. In fact, I raised this issue only after I tried the overrides for each service.

Oh did not understood that, what's not working ? Could you please share me the manifest your tried ?

[vinimona]

This doesn't work. In fact, I raised this issue only after I tried the overrides for each service.

Oh did not understood that, what's not working ? Could you please share me the manifest your tried ?

So this is what I tried and it wasn't working. This configuration is always overridden by the certs generated by cert-manager.

  services:
    overrides:
      deployment:
        spec:
          template:
            spec:
              containers:
              - name: service
                env:
                  - name: TEMPORAL_TLS_SERVER_CA_CERT
                    value: /etc/temporal/certs/ca.cert
                  - name: TEMPORAL_TLS_SERVER_CERT
                    value: /etc/temporal/certs/cluster.pem
                  - name: TEMPORAL_TLS_SERVER_KEY
                    value: /etc/temporal/certs/cluster.key
                  - name: TEMPORAL_TLS_REQUIRE_CLIENT_AUTH
                    value: "true"
                  - name: TEMPORAL_TLS_FRONTEND_CERT
                    value: /etc/temporal/certs/cluster.pem
                  - name: TEMPORAL_TLS_FRONTEND_KEY
                    value: /etc/temporal/certs/cluster.key
                  - name: TEMPORAL_TLS_CLIENT1_CA_CERT
                    value: /etc/temporal/certs/ca.cert
                  - name: TEMPORAL_TLS_CLIENT2_CA_CERT
                    value: /etc/temporal/certs/ca.cert
                  - name: TEMPORAL_TLS_INTERNODE_SERVER_NAME
                    value: tls-sample
                  - name: TEMPORAL_TLS_FRONTEND_SERVER_NAME
                    value: tls-sample
                  - name: TEMPORAL_TLS_FRONTEND_DISABLE_HOST_VERIFICATION
                    value: "false"
                  - name: TEMPORAL_TLS_INTERNODE_DISABLE_HOST_VERIFICATION
                    value: "false"
                  - name: TEMPORAL_CLI_TLS_CA
                    value: /etc/temporal/certs/ca.cert
                  - name: TEMPORAL_CLI_TLS_CERT
                    value: /etc/temporal/certs/cluster.pem
                  - name: TEMPORAL_CLI_TLS_KEY
                    value: /etc/temporal/certs/cluster.key
                  - name: TEMPORAL_CLI_TLS_ENABLE_HOST_VERIFICATION
                    value: "true"
                  - name: TEMPORAL_CLI_TLS_SERVER_NAME
                    value: tls-sample
                  - name: TEMPORAL_TLS_CA
                    value: /etc/temporal/certs/ca.cert
                  - name: TEMPORAL_TLS_CERT
                    value: /etc/temporal/certs/cluster.pem
                  - name: TEMPORAL_TLS_KEY
                    value: /etc/temporal/certs/cluster.key
                  - name: TEMPORAL_TLS_ENABLE_HOST_VERIFICATION
                    value: "true"
                  - name: TEMPORAL_TLS_SERVER_NAME
                    value: tls-sample
                volumeMounts:
                  - name: certs_dir
                    mountPath: /etc/temporal/certs
              volumes:
                - name: certs_dir
                  hostPath:
                    path: "/temporal/certs"

[alexandrevilain]

This configuration is always overridden by the certs generated by cert-manager.

And what if you disable mTLS with cert-manager ?