Deleting master pod from one shard causes master updates in other shards

[4n4nd]

We had a cluster deployed:

[root@platformtester-k8-master-1-a00061-4b2c09f7b009c1b8 scripts]# kubectl-rc
  POD NAME                             IP               NODE         ID                                        ZONE             USED MEMORY  MAX MEMORY  KEYS  SLOTS
  + rediscluster-node-for-redis-cjtgr  192.168.172.70   10.15.11.85  edb9266a55c8a2806cb860724984dd5431d7aae0  platform-zone-B  3.00M        2.80G             6554-9830
  | rediscluster-node-for-redis-bzlgd  192.168.193.99   10.15.11.82  a91df8f7488bec237b41ffc851f0a1fadb24cbd7  platform-zone-A  5.18M        2.80G           
  | rediscluster-node-for-redis-zksts  192.168.62.44    10.15.11.84  fe7a82302fe6f39f771c11d1e67211ea1008966d  platform-zone    2.95M        2.80G           
  + rediscluster-node-for-redis-fct4j  192.168.62.41    10.15.11.84  dfe280003c3a1e09310e6ae1ecf38c50331549b9  platform-zone    5.40M        2.80G             0-3276
  | rediscluster-node-for-redis-2pcfl  192.168.81.119   10.15.11.83  091f077dcb45c6dce7f56e1ab662af0edcf66bc7  platform-zone-A  2.90M        2.80G           
  | rediscluster-node-for-redis-d6zr4  192.168.172.117  10.15.11.85  3dd335dd3d359aff5c868e0ece5336edbcf4e255  platform-zone-B  3.94M        2.80G           
  + rediscluster-node-for-redis-jhd9k  192.168.81.118   10.15.11.83  ccafee5678d33943c2d60003a3f55ac413ddf193  platform-zone-A  8.22M        2.80G             3277-6553
  | rediscluster-node-for-redis-8wsfq  192.168.62.43    10.15.11.84  5c43965a7f444284d19af0e76f26fba01b8a7844  platform-zone    3.14M        2.80G           
  | rediscluster-node-for-redis-xwnpb  192.168.57.109   10.15.11.87  6522856556acb5fe7b34c2d7d498a88c22f5757e  platform-zone-C  3.79M        2.80G           
  + rediscluster-node-for-redis-p2fdg  192.168.57.103   10.15.11.87  24a87a775d54adfe2069fc20368704800707d3c2  platform-zone-C  10.95M       2.80G             13108-16383
  | rediscluster-node-for-redis-cz2rv  192.168.193.100  10.15.11.82  f24a67e5090cec59b0301b3934773d5975eb6619  platform-zone-A  3.26M        2.80G           
  | rediscluster-node-for-redis-d9qw9  192.168.81.117   10.15.11.83  cc82bc2e2cffefeaeb34983b224e602505e5c163  platform-zone-A  3.05M        2.80G           
  + rediscluster-node-for-redis-p8l5q  192.168.193.97   10.15.11.82  ff537ede00a7c508ae4d322c83a45edf6b99570a  platform-zone-A  5.74M        2.80G             9831-13107
  | rediscluster-node-for-redis-fcq8r  192.168.172.127  10.15.11.85  7763b26883837808d9a8550af4cfc85a3938b7e1  platform-zone-B  2.94M        2.80G           
  | rediscluster-node-for-redis-nqwpj  192.168.57.110   10.15.11.87  35275718f7c3fb032f439855505a88f3e6d458ca  platform-zone-C  3.45M        2.80G           

  NAME            NAMESPACE          PODS      OPS STATUS  REDIS STATUS  NB PRIMARY  REPLICATION  ZONE SKEW
  node-for-redis  fed-redis-cluster  15/15/15  ClusterOK   OK            5/5         2-2/2        1/2/BALANCED

And we were performing some resiliency tests on it. So, we deleted a master pod for one of the shards and check how it recovers.

[root@platformtester-k8-master-1-a00061-4b2c09f7b009c1b8 scripts]# kc delete pod rediscluster-node-for-redis-jhd9k 
pod "rediscluster-node-for-redis-jhd9k" deleted

The cluster was successfully able to recover, and a replica pod replaced the deleted master pod.

[root@platformtester-k8-master-1-a00061-4b2c09f7b009c1b8 scripts]# kubectl-rc
  POD NAME                             IP               NODE         ID                                        ZONE             USED MEMORY  MAX MEMORY  KEYS        SLOTS
  + rediscluster-node-for-redis-2pcfl  192.168.81.119   10.15.11.83  091f077dcb45c6dce7f56e1ab662af0edcf66bc7  platform-zone-A  635.45M      2.80G       db0=143194  0-3276
  | rediscluster-node-for-redis-d6zr4  192.168.172.117  10.15.11.85  3dd335dd3d359aff5c868e0ece5336edbcf4e255  platform-zone-B  633.02M      2.80G       db0=143194
  | rediscluster-node-for-redis-fct4j  192.168.62.41    10.15.11.84  dfe280003c3a1e09310e6ae1ecf38c50331549b9  platform-zone    637.19M      2.80G       db0=143194
  + rediscluster-node-for-redis-cjtgr  192.168.172.70   10.15.11.85  edb9266a55c8a2806cb860724984dd5431d7aae0  platform-zone-B  599.32M      2.80G       db0=143255  6554-9830
  | rediscluster-node-for-redis-bzlgd  192.168.193.99   10.15.11.82  a91df8f7488bec237b41ffc851f0a1fadb24cbd7  platform-zone-A  599.25M      2.80G       db0=143255
  | rediscluster-node-for-redis-zksts  192.168.62.44    10.15.11.84  fe7a82302fe6f39f771c11d1e67211ea1008966d  platform-zone    599.58M      2.80G       db0=143255
  + rediscluster-node-for-redis-d9qw9  192.168.81.117   10.15.11.83  cc82bc2e2cffefeaeb34983b224e602505e5c163  platform-zone-A  597.94M      2.80G       db0=142787  13108-16383
  | rediscluster-node-for-redis-cz2rv  192.168.193.100  10.15.11.82  f24a67e5090cec59b0301b3934773d5975eb6619  platform-zone-A  597.33M      2.80G       db0=142787
  | rediscluster-node-for-redis-p2fdg  192.168.57.103   10.15.11.87  24a87a775d54adfe2069fc20368704800707d3c2  platform-zone-C  597.23M      2.80G       db0=142787
  + rediscluster-node-for-redis-p8l5q  192.168.193.97   10.15.11.82  ff537ede00a7c508ae4d322c83a45edf6b99570a  platform-zone-A  602.21M      2.80G       db0=143685  9831-13107
  | rediscluster-node-for-redis-fcq8r  192.168.172.127  10.15.11.85  7763b26883837808d9a8550af4cfc85a3938b7e1  platform-zone-B  602.91M      2.80G       db0=143685
  | rediscluster-node-for-redis-nqwpj  192.168.57.110   10.15.11.87  35275718f7c3fb032f439855505a88f3e6d458ca  platform-zone-C  602.56M      2.80G       db0=143685
  + rediscluster-node-for-redis-xwnpb  192.168.57.109   10.15.11.87  6522856556acb5fe7b34c2d7d498a88c22f5757e  platform-zone-C  596.71M      2.80G       db0=142831  3277-6553
  | rediscluster-node-for-redis-8wsfq  192.168.62.43    10.15.11.84  5c43965a7f444284d19af0e76f26fba01b8a7844  platform-zone    596.57M      2.80G       db0=142831
  | rediscluster-node-for-redis-9nxs4  192.168.81.120   10.15.11.83  d83782fb30b062810acffedde93ea6304de95e3d  platform-zone-A  601.99M      2.80G       db0=142831

  NAME            NAMESPACE          PODS      OPS STATUS  REDIS STATUS  NB PRIMARY  REPLICATION  ZONE SKEW
  node-for-redis  fed-redis-cluster  15/15/15  ClusterOK   OK            5/5         2-2/2        2/1/BALANCED

But if you check, the masters for other shards changed as well. This issue does not occur every time. Is this expected behavior?

[cin]

I can't say that I've ever looked at this logic, but that behavior is NOT expected. Will look into it. My guess is we have a bug in the logic that keeps zones in balance and it overcorrected.

[4n4nd]

Could you point me to this logic in the code? Maybe I can try and help.

[cin]

The replica to primary promotion logic is here. Will try to reproduce this tomorrow.

[cin]

I need to test this some more but I was able to reproduce it fairly regularly w/the draft PR. I was unable to reproduce it when using the latest tag however. I was testing this locally on my laptop w/kind. I'm going to test in a real cluster and see what happens. I may have introduced a bug when altering the replica placement logic.