How to model a StatefulSet with regional PVCs?

Question

We have a three-node GKE cluster. One node per zone.

We have a StatefulSet with three replicas. Each with their own PVC.

Now the PVCs on GKE use a regional storage class, but we see that the volumes only ever exist in two of the three zones.

What we fear can happen during a deployment or maintenance window is, two of the StatefulSet pods being killed and then being schedule on a node where the volume cannot be attached to due to regional/zonal constraints.

What is the best practice here to set this up correctly?

Answer 1

google regional persistent disk will choose only one primary and secondary zone, and usage is restricted to only those zones for its lifetime. it can't be used in more than two zones.

https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd

If using a regional cluster, you can leave allowedTopologies unspecified. If you do this, when you create a Pod that consumes a PersistentVolumeClaim which uses this StorageClass a regional persistent disk is provisioned with two zones. One zone is the same as the zone that the Pod is scheduled in. The other zone is randomly picked from the zones available to the cluster.