Failing to Logging into the cluster in k8s

[odidev]

Hi Team,

I am working on dexidp/dex and I’m facing an issue in logging into the cluster. I was following this documentation.
I have used the below commands:

./bin/example-app --issuer https://dex.example.com:32000 --listen http://192.168.2.176:5555 --redirect-uri https://192.168.2.176:5555/callback   --issuer-root-ca examples/k8s/ssl/ca.pem 

Error logs are mentioned below.

Error: failed to query provider "https://dex.example.com:32000": Get "https://dex.example.com:32000/.well-known/openid-configuration": dial tcp: lookup dex.example.com: no such host 
Usage: 
  example-app [flags] 
Flags: 
      --client-id string        OAuth2 client ID of this application. (default "example-app") 
      --client-secret string    OAuth2 client secret of this application. (default "ZXhhbXBsZS1hcHAtc2VjcmV0") 
      --debug                   Print all request and responses from the OpenID Connect issuer. 
  -h, --help                    help for example-app 
      --issuer string           URL of the OpenID Connect issuer. (default "http://127.0.0.1:5556/dex") 
      --issuer-root-ca string   Root certificate authorities for the issuer. Defaults to host certs. 
      --listen string           HTTP(S) address to listen at. (default "http://127.0.0.1:5555") 
      --redirect-uri string     Callback URL for OAuth2 responses. (default "http://127.0.0.1:5555/callback") 
      --tls-cert string         X509 cert file to present when serving HTTPS. 
      --tls-key string          Private key for the HTTPS cert. 
error: failed to query provider "https://dex.example.com:32000": Get "https://dex.example.com:32000/.well-known/openid-configuration": dial tcp: lookup dex.example.com: no such host 

I have configured the kube-apiserver.yaml and the examples/k8s/dex.yml files which are listed below:-

• Kube-apiserver.yaml

apiVersion: v1 
kind: Pod 
metadata: 
  annotations: 
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.2.176:6443 
  creationTimestamp: null 
  labels: 
    component: kube-apiserver 
    tier: control-plane 
  name: kube-apiserver 
  namespace: kube-system 
spec: 
  containers: 
  - command: 
    - kube-apiserver 
    - --advertise-address=192.168.2.176 
    - --allow-privileged=true 
    - --authorization-mode=Node,RBAC 
    - --client-ca-file=/etc/kubernetes/pki/ca.crt 
    - --enable-admission-plugins=NodeRestriction 
    - --enable-bootstrap-token-auth=true 
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt 
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt 
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key 
    - --etcd-servers=https://127.0.0.1:2379 
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt 
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key 
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt 
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key 
    - --requestheader-allowed-names=front-proxy-client 
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt 
    - --requestheader-extra-headers-prefix=X-Remote-Extra- 
    - --requestheader-group-headers=X-Remote-Group 
    - --requestheader-username-headers=X-Remote-User 
    - --secure-port=6443 
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local 
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub 
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key 
    - --service-cluster-ip-range=10.96.0.0/12 
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt 
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key 
    - --oidc-issuer-url=https://dex.example.com:32000 
    - --oidc-client-id=example-app 
    - --oidc-ca-file=/etc/ssl/certs/openid-ca.pem 
    - --oidc-username-claim=email 
    - --oidc-groups-claim=groups 
    image: k8s.gcr.io/kube-apiserver:v1.22.2 
    imagePullPolicy: IfNotPresent 
    livenessProbe: 
      failureThreshold: 8 
      httpGet: 
        host: 192.168.2.176 
        path: /livez 
        port: 6443 
        scheme: HTTPS 
      initialDelaySeconds: 10 
      periodSeconds: 10 
      timeoutSeconds: 15 
    name: kube-apiserver 
    readinessProbe: 
      failureThreshold: 3 
      httpGet: 
        host: 192.168.2.176 
        path: /readyz 
        port: 6443 
        scheme: HTTPS 
      periodSeconds: 1 
      timeoutSeconds: 15 
    resources: 
      requests: 
        cpu: 250m 
    startupProbe: 
      failureThreshold: 24 
      httpGet: 
        host: 192.168.2.176 
        path: /livez 
        port: 6443 
        scheme: HTTPS 
      initialDelaySeconds: 10 
      periodSeconds: 10 
      timeoutSeconds: 15 
    volumeMounts: 
    - mountPath: /etc/ssl/certs 
      name: ca-certs 
      readOnly: true 
    - mountPath: /etc/ca-certificates 
      name: etc-ca-certificates 
      readOnly: true 
    - mountPath: /etc/pki 
      name: etc-pki 
      readOnly: true 
	    - mountPath: /etc/kubernetes/pki 
      name: k8s-certs 
      readOnly: true 
    - mountPath: /usr/local/share/ca-certificates 
      name: usr-local-share-ca-certificates 
      readOnly: true 
    - mountPath: /usr/share/ca-certificates 
      name: usr-share-ca-certificates 
      readOnly: true 
    - mountPath: /etc/ssl/certs 
      name: etc-ssl-certs 
      readOnly: true 
  hostNetwork: true 
  priorityClassName: system-node-critical 
  securityContext: 
    seccompProfile: 
      type: RuntimeDefault 
  volumes: 
  - hostPath: 
      path: /etc/ssl/certs 
      type: DirectoryOrCreate 
    name: ca-certs 
  - hostPath: 
      path: /etc/ca-certificates 
      type: DirectoryOrCreate 
    name: etc-ca-certificates 
  - hostPath: 
      path: /etc/pki 
      type: DirectoryOrCreate 
    name: etc-pki 
  - hostPath: 
      path: /etc/kubernetes/pki 
      type: DirectoryOrCreate 
    name: k8s-certs 
  - hostPath: 
      path: /usr/local/share/ca-certificates 
      type: DirectoryOrCreate 
    name: usr-local-share-ca-certificates 
  - hostPath: 
      path: /usr/share/ca-certificates 
      type: DirectoryOrCreate 
    name: usr-share-ca-certificates 
  - hostPath: 
      path: /etc/ssl/certs 
      type: DirectoryOrCreate 
    name: ca-certs 
status: {} 

• Dex.yml

--- 
apiVersion: v1 
kind: Namespace 
metadata: 
  name: dex 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  labels: 
    app: dex 
  name: dex 
  namespace: dex 
spec: 
  replicas: 3 
  selector: 
    matchLabels: 
      app: dex 
  template: 
    metadata: 
      labels: 
        app: dex 
    spec: 
      serviceAccountName: dex # This is created below 
      containers: 
      - image: ghcr.io/dexidp/dex:v2.30.0 
        name: dex 
        command: ["/usr/local/bin/dex", "serve", "/etc/dex/cfg/config.yaml"] 

        ports: 
        - name: https 
          containerPort: 5556 

        volumeMounts: 
        - name: config 
          mountPath: /etc/dex/cfg 
        - name: tls 
          mountPath: /etc/dex/tls 

        env: 
        - name: GITHUB_CLIENT_ID 
          valueFrom: 
            secretKeyRef: 
              name: github-client 
              key: client-id 
        - name: GITHUB_CLIENT_SECRET 
          valueFrom: 
            secretKeyRef: 
              name: github-client 
              key: client-secret 

        readinessProbe: 
          httpGet: 
            path: /healthz 
            port: 5556 
            scheme: HTTPS 
      volumes: 
      - name: config 
        configMap: 
          name: dex 
          items: 
          - key: config.yaml 
            path: config.yaml 
      - name: tls 
        secret: 
          secretName: dex.example.com.tls 
--- 
kind: ConfigMap 
apiVersion: v1 
metadata: 
  name: dex 
  namespace: dex 
data: 
  config.yaml: | 
    issuer: https://dex.example.com:32000 
    storage: 
      type: kubernetes 
      config: 
        inCluster: true 
    web: 
      https: 192.168.2.176:5556 
      tlsCert: /etc/dex/tls/tls.crt 
      tlsKey: /etc/dex/tls/tls.key 
    connectors: 
    - type: github 
      id: github 
      name: GitHub 
      config: 
        clientID: $GITHUB_CLIENT_ID 
        clientSecret: $GITHUB_CLIENT_SECRET 
        redirectURI: https://dex.example.com:32000/callback 
        org: kubernetes 
    oauth2: 
      skipApprovalScreen: true 

    staticClients:
    - id: example-app 
      redirectURIs: 
      - 'http://192.168.2.176:5555/callback' 
      name: 'Example App' 
      secret: ZXhhbXBsZS1hcHAtc2VjcmV0 

    enablePasswordDB: true 
    staticPasswords: 
    - email: "[email protected]" 
      # bcrypt hash of the string "password": $(echo password | htpasswd -BinC 10 admin | cut -d: -f2) 
      hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W" 
      username: "admin" 
      userID: "08a8684b-db88-4b73-90a9-3cd1661f5466" 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: dex 
  namespace: dex 
spec: 
  type: NodePort 
  ports: 
  - name: dex 
    port: 5556 
    protocol: TCP 
    targetPort: 5556 
    nodePort: 32000 
  selector: 
    app: dex 
--- 
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  labels: 
    app: dex 
  name: dex 
  namespace: dex 
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRole 
metadata: 
  name: dex 
rules: 
- apiGroups: ["dex.coreos.com"] # API group created by dex 
  resources: ["*"] 
  verbs: ["*"] 
- apiGroups: ["apiextensions.k8s.io"] 
  resources: ["customresourcedefinitions"] 
  verbs: ["create"] # To manage its own resources, dex must be able to create customresourcedefinitions 
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding 
metadata: 
  name: dex 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: ClusterRole 
  name: dex 
subjects: 
- kind: ServiceAccount 
  name: dex           # Service account assigned to the dex pod, created above 
  namespace: dex  # The namespace dex is running in 

[sagikazarmark]

The error message pretty clearly says that dex.example.com cannot be resolved.

The documentation says:

dex.example.com should be replaced by whatever DNS name or IP address dex is running under.

[odidev]

I have tried by replacing it (dex.example.com) with my local IP but still, I’m getting the connection refused error.

subham@x64server:~/dex$ ./bin/example-app --issuer http://192.168.2.176:32000 --listen http://192.168.2.176:5555 --redirect-uri https://192.168.2.176:5555/callback   --issuer-root-ca examples/k8s/ssl/ca.pem 
Error: failed to query provider "http://192.168.2.176:32000": Get "http://192.168.2.176:32000/.well-known/openid-configuration": dial tcp 192.168.2.176:32000: connect: connection refused 
Usage: 
  example-app [flags] 
Flags: 
      --client-id string        OAuth2 client ID of this application. (default "example-app") 
      --client-secret string    OAuth2 client secret of this application. (default "ZXhhbXBsZS1hcHAtc2VjcmV0") 
      --debug                   Print all request and responses from the OpenID Connect issuer. 
  -h, --help                    help for example-app 
      --issuer string           URL of the OpenID Connect issuer. (default "http://127.0.0.1:5556/dex") 
      --issuer-root-ca string   Root certificate authorities for the issuer. Defaults to host certs. 
      --listen string           HTTP(S) address to listen at. (default "http://127.0.0.1:5555") 
      --redirect-uri string     Callback URL for OAuth2 responses. (default "http://127.0.0.1:5555/callback") 
      --tls-cert string         X509 cert file to present when serving HTTPS. 
      --tls-key string          Private key for the HTTPS cert. 
error: failed to query provider "http://192.168.2.176:32000": Get "http://192.168.2.176:32000/.well-known/openid-configuration": dial tcp 192.168.2.176:32000: connect: connection refused 

[sagikazarmark]

Did you change the Dex config to listen on that address?

[odidev]

Yes, I have changed the dex configuration to listen on that IP.

[odidev]

On running this command. All the dex pods are not starting and getting stuck in creating container state. It is failing because pods are not able to mount the tls secret. Details can be found below.

xyz@xyz:~/dex/examples/k8s$ kubectl get pods -n dex
NAME                   READY   STATUS              RESTARTS   AGE
dex-7f85c695dc-cqz6f   0/1     ContainerCreating   0          10s
dex-7f85c695dc-krnml   0/1     ContainerCreating   0          10s
dex-7f85c695dc-pk6x7   0/1     ContainerCreating   0          10s

Below are the logs of dex pod which is in creating container state.

xyz@xyz:~/dex/examples/k8s$ kubectl describe pod dex-7f85c695dc-cqz6f -n dex
Name:           dex-7f85c695dc-cqz6f
Namespace:      dex
Priority:       0
Node:           minikube/192.168.49.2
Start Time:     Wed, 22 Dec 2021 05:50:40 -0500
Labels:         app=dex
                pod-template-hash=7f85c695dc
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/dex-7f85c695dc
Containers:
  dex:
    Container ID:
    Image:         ghcr.io/dexidp/dex:v2.30.0
    Image ID:
    Port:          5556/TCP
    Host Port:     0/TCP
    Command:
      /usr/local/bin/dex
      serve
      /etc/dex/cfg/config.yaml
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      http-get https://:5556/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GITHUB_CLIENT_ID:      <set to the key 'client-id' in secret 'github-client'>      Optional: false
      GITHUB_CLIENT_SECRET:  <set to the key 'client-secret' in secret 'github-client'>  Optional: false
    Mounts:
      /etc/dex/cfg from config (rw)
      /etc/dex/tls from tls (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hmt4x (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dex
    Optional:  false
  tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dex.example.com.tls
    Optional:    false
  kube-api-access-hmt4x:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    2m56s                default-scheduler  Successfully assigned dex/dex-7f85c695dc-cqz6f to minikube
  Warning  FailedMount  53s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[tls kube-api-access-hmt4x config]: timed out waiting for the condition
  Warning  FailedMount  48s (x9 over 2m55s)  kubelet            MountVolume.SetUp failed for volume "tls" : secret "dex.example.com.tls" not found

It is failing to mount the tls secret.