TiDB Pods stay unhealthy after configuring the port parameter in TiDB configuration

[kos-team]

Bug Report

What version of Kubernetes are you using?
Client Version: v1.31.1
Kustomize Version: v5.4.2

What version of TiDB Operator are you using?
v1.6.0

What did you do?
We deployed a tidb cluster with 3 replicas of pd, tikv and tidb. After the cluster is fully up and healthy, we changed the spec.tidb.config and set port to 80.
The last replica of TiDB restarted. However, it stayed in the unhealthy state due to the error reported by Readiness probe: Readiness probe failed: dial tcp 10.244.3.8:4000: connect: connection refused. And the operator keeps waiting it and hangs.

The root cause of this bug is similar to #6014. The operator hardcodes the port number used to query the readiness of the TiDB here:

tidb-operator/pkg/manager/member/tidb_member_manager.go

Line 1229 in 1867f39

Port: intstr.FromInt(4000),

How to reproduce

1. Deploy a TiDB cluster with TiProxy enabled, for example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: test-cluster
spec:
  configUpdateStrategy: RollingUpdate
  enableDynamicConfiguration: true
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    config: "[dashboard]\n  internal-proxy = true\n"
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 10Gi
  pvReclaimPolicy: Retain
  tidb:
    baseImage: pingcap/tidb
    config: '
      [performance]

      tcp-keep-alive = true
      '
    maxFailoverCount: 0
    replicas: 3
    service:
      externalTrafficPolicy: Local
      type: NodePort
  tikv:
    baseImage: pingcap/tikv
    config: 'log-level = "info"

      '
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 100Gi
  timezone: UTC
  version: v8.1.0

2. Add port to the spec.tidb.config

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: test-cluster
spec:
  configUpdateStrategy: RollingUpdate
  enableDynamicConfiguration: true
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    config: "[dashboard]\n  internal-proxy = true\n"
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 10Gi
  pvReclaimPolicy: Retain
  tidb:
    baseImage: pingcap/tidb
    config: 'port = 80

      [performance]

      tcp-keep-alive = true
      '
    maxFailoverCount: 0
    replicas: 3
    service:
      externalTrafficPolicy: Local
      type: NodePort
  tikv:
    baseImage: pingcap/tikv
    config: 'log-level = "info"

      '
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 100Gi
  timezone: UTC
  version: v8.1.0

What did you expect to see?
We expected to see the tidb rolling update and using the new port.

What did you see instead?
The last replica of tidb restarted. However, it stayed in the unhealthy state due to the error reported by Readiness probe Readiness probe failed: dial tcp 10.244.3.8:4000: connect: connection refused . And the operators kept waiting it and hanged.

Root Cause
After the port of tidb is changed in the configuration, the readiness probe still tries to connect tidb using the default port set in statefulset causing connection failure.

[csuzhangxc]

TiDB Operator doesn't support changing the port after the TiDB cluster is deployed, and a workaround may be to create an additional service with a customized port.