everyone.
I have an issue with testing Redis Sentinel setup on a microk8s cluster.
We have a microk8s cluster and use Redis Sentinel setup for high availability. There is one Redis master and 2 replicas, one per node. Each of the three Redis pods has a sentinel container.
We try to simulate the situation when a node goes down by stopping microk8s on the node, that runs the Redis master. (microk8s stop) But the sentinels do not elect a new master, but wait for the old one to recover, and the replicas connect to it again. The timeouts are quite low, 1 sec.
It works perfectly fine, when just the pod with the Redis master is killer. And we just found out, that it also works when the whole VM of the node in question is shut down. Just not when only microk8s is stopped on the VM for testing.
Any ideas what the difference is between shutting down k8s on the VM and shutting down the VM?