Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

With v1.2.5 failed to start - lib64/libc.so.6: version `GLIBC_2.34' not found #163

Closed
1337andre opened this issue Nov 14, 2023 · 3 comments
Closed

With v1.2.5 failed to start - lib64/libc.so.6: version `GLIBC_2.34' not found #163

1337andre opened this issue Nov 14, 2023 · 3 comments

Comments

@1337andre
Copy link

Hi folks,

with image quay.io/redhat-cop/namespace-configuration-operator:v1.2.5

namespace-configuration-operator failed to start

lib64/libc.so.6: version `GLIBC_2.34' not found 

#kubectl describe pod namespace-configuration-operator-67dc8fdbbb-z7xwd
Name:                 namespace-configuration-operator-67dc8fdbbb-z7xwd
Namespace:            default
Priority:             1
Priority Class Name:  low
Service Account:      controller-manager
Node:                 rxcmpk8s19.hcp-infra.blub.de/172.31.102.101
Start Time:           Tue, 14 Nov 2023 08:26:13 +0100
Labels:               app.kubernetes.io/instance=namespace-configuration-operator
                      app.kubernetes.io/name=namespace-configuration-operator
                      control-plane=namespace-configuration-operator
                      pod-template-hash=67dc8fdbbb
Annotations:          cni.projectcalico.org/containerID: 24d87fc3831cd061e868876406b8b5cb1c393f09a49b7a21ec17f6012596ab9c
                      cni.projectcalico.org/podIP: 10.4.192.202/32
                      cni.projectcalico.org/podIPs: 10.4.192.202/32
                      kubectl.kubernetes.io/restartedAt: 2023-09-01T11:43:34+02:00
Status:               Running
IP:                   10.4.192.202
IPs:
  IP:           10.4.192.202
Controlled By:  ReplicaSet/namespace-configuration-operator-67dc8fdbbb
Containers:
  kube-rbac-proxy:
    Container ID:  containerd://7c5362a8759c282136bc91d150d392d3c07dad903a7645a3d2feafcf80c76cb0
    Image:         quay.io/redhat-cop/kube-rbac-proxy:v0.11.0
    Image ID:      quay.io/redhat-cop/kube-rbac-proxy@sha256:c68135620167c41e3d9f6c1d2ca1eb8fa24312b86186d09b8010656b9d25fb47
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --tls-cert-file=/etc/certs/tls/tls.crt
      --tls-private-key-file=/etc/certs/tls/tls.key
      --v=10
    State:          Running
      Started:      Tue, 14 Nov 2023 08:26:15 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /etc/certs/tls from namespace-configuration-operator-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ln4t4 (ro)
  namespace-configuration-operator:
    Container ID:  containerd://7773f9892338e07a8f25b4fd2b2c1907598555e5fa442085716ab02f3e12a18c
    Image:         quay.io/redhat-cop/namespace-configuration-operator:v1.2.5
    Image ID:      quay.io/redhat-cop/namespace-configuration-operator@sha256:20debaa7b91aebf034a0bd2baa80d37bac2b23be5c76efc2a6edbc14f942b2b1
    Port:          <none>
    Host Port:     <none>
    Command:
      /manager
    Args:
      --leader-elect
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 14 Nov 2023 08:37:17 +0100
      Finished:     Tue, 14 Nov 2023 08:37:17 +0100
    Ready:          False
    Restart Count:  7
    Requests:
      cpu:        100m
      memory:     20Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from webhook-server-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ln4t4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  namespace-configuration-operator-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  namespace-configuration-operator-certs
    Optional:    false
  webhook-server-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-cert
    Optional:    false
  kube-api-access-ln4t4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned default/namespace-configuration-operator-67dc8fdbbb-z7xwd to rxcmpk8s19.hcp-infra.blub.de
  Normal   Pulled     14m                   kubelet            Container image "quay.io/redhat-cop/kube-rbac-proxy:v0.11.0" already present on machine
  Normal   Created    14m                   kubelet            Created container kube-rbac-proxy
  Normal   Started    14m                   kubelet            Started container kube-rbac-proxy
  Normal   Pulling    14m                   kubelet            Pulling image "quay.io/redhat-cop/namespace-configuration-operator:v1.2.5"
  Normal   Pulled     14m                   kubelet            Successfully pulled image "quay.io/redhat-cop/namespace-configuration-operator:v1.2.5" in 6.834695195s (6.834708313s including waiting)
  Normal   Created    13m (x4 over 14m)     kubelet            Created container namespace-configuration-operator
  Normal   Started    13m (x4 over 14m)     kubelet            Started container namespace-configuration-operator
  Normal   Pulled     13m (x3 over 14m)     kubelet            Container image "quay.io/redhat-cop/namespace-configuration-operator:v1.2.5" already present on machine
  Warning  BackOff    4m27s (x52 over 14m)  kubelet            Back-off restarting failed container namespace-configuration-operator in pod namespace-configuration-operator-67dc8fdbbb-z7xwd_default(4d535843-f87d-4e36-86d8-669881710c75)
@slopezz
Copy link

slopezz commented Nov 22, 2023

Failed version 1.2.5 has just been published into operatorHub community-operators
redhat-openshift-ecosystem/community-operators-prod#3593

And so the error is starting to arrive to anyone using the operator through operatorHub:

2023-11-22T09:54:58Z INFO Starting workers {"controller": "namespaceconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "NamespaceConfig", "worker count": 1}
2023-11-22T09:54:58Z INFO controllers.NamespaceConfig reconciling started {"namespaceconfig": {"name":"basic-user-namespace-monitoring"}}
2023-11-22T09:54:59Z INFO Observed a panic in reconciler: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema {"controller": "namespaceconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "NamespaceConfig", "NamespaceConfig": {"name":"basic-user-namespace-monitoring"}, "namespace": "", "name": "basic-user-namespace-monitoring", "reconcileID": "d1c4208b-33eb-444a-941c-d2af11af0852"}
panic: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema [recovered]
panic: interface conversion: validation.Schema is *validation.schemaValidation, not *validation.NullSchema
goroutine 209 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x22be620?, 0xc0041aa660?})
/opt/hostedtoolcache/go/1.21.4/x64/src/runtime/panic.go:914 +0x21f
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).validateLockedResources(0xc000854380, {0xc000990800, 0x34, 0xc000637a60?})
/home/runner/go/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/lockedresourcecontroller/locked-resource-manager.go:341 +0x6f3
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).SetResources(0xc000854380, {0xc000990800?, 0x34, 0x40})
/home/runner/go/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/lockedresourcecontroller/locked-resource-manager.go:82 +0x77
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).Restart(0xc000854380, {0x27c5e68, 0xc00075e660}, {0xc000990800, 0x34, 0x40}, {0x3677360, 0x0, 0x0}, 0x0, ...)
/home/runner/go/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/lockedresourcecontroller/locked-resource-manager.go:222 +0x16c
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResourcesWithRestConfig(0xc0000f0f20, {0x27c5e68, 0xc00075e660}, {0x27db750?, 0xc00061a1a0?}, {0xc000990800, 0x34, 0x40}, {0x3677360, 0x0, ...}, ...)
/home/runner/go/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:117 +0x3bc
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResources(...)
/home/runner/go/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:91
github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile(0xc0000f0f20, {0x27c5e68, 0xc00075e660}, {{{0x0, 0x0}, {0xc00025c880, 0x1f}}})
/home/runner/work/namespace-configuration-operator/namespace-configuration-operator/controllers/namespaceconfig_controller.go:121 +0x83c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x27c5e68?, {0x27c5e68?, 0xc00075e660?}, {{{0x0?, 0x21c61c0?}, {0xc00025c880?, 0x27b66d0?}}})
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00027d900, {0x27c5ea0, 0xc0003f2690}, {0x233bc20?, 0xc0000626c0?})
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314 +0x365
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00027d900, {0x27c5ea0, 0xc0003f2690})
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 70
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x565

@slopezz
Copy link

slopezz commented Nov 22, 2023

With this issue, we were receiving constant prometheus TargetDown alerts, because new operator version 1.2.5 cannot start, there is a panic, and Openshift has a generic alert called TargetDown that check that operator metrics endpoint is alive, which is not the case, impacting all clusters.

Until there is a fix on a newer version, I have mitigated the issue by now:

  • I disabled ArgoCD autosync
  • Uninstall failed operator version 1.2.5
  • Forcing old operator version 1.2.4, with Manual approval. That way, once manually accepted and installed correct 1.2.4, it won't jump automatically to failed 1.2.5 because required manual approval
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: namespace-configuration-operator
spec:
  channel: alpha
  installPlanApproval: Manual
  startingCSV: namespace-configuration-operator.v1.2.4
  name: namespace-configuration-operator
  source: community-operators
  sourceNamespace: openshift-marketplace

@1337andre
Copy link
Author

seems to be fixed with v1.2.6
#169

/cc @slopezz @raffaelespazzoli

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@1337andre @slopezz