We're currently running a 4 node geode cluster (two servers, two stand-alone locators) in our production and prod-like environments and a 2 node cluster in our dev environments (one server, on locator).
We've changed the following settings from their defaults as we found the the two member environments would crash daily and they helped.
Servers and locators: gemfire.member-timeout=600000 (default 5000)
Locators only: -gemfire.member-weight=15 (default 3)
We've since discovered that in the case of a 4 member cluster, the gemfire.member-timeout=600000 causes failover to be delayed by exactly 10 minutes when one of the two servers is taken offline.
So we've restored all settings to their defaults and so far 4 node geode clusters look good but the defaults don't work at all for 2 member clusters; members lose each other after a couple of hours and both the server and the locator has to be restarted before connectivity is restored.
What are some settings we can play with to get reliable comms in a two member server/locator cluster?