I have three nodes in cluster.
Whole cluster is set up using Galera Manager.
For unknown reasons I got errors like below:
2024-02-12 08:50:36 | error.log | 2024-02-12 8:50:34 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
2024-02-12 08:50:36 | error.log | at ./gcomm/src/pc.cpp:connect():160
2024-02-12 08:50:36 | error.log | 2024-02-12 8:50:34 0 [ERROR] WSREP: ./gcs/src/gcs_core.cpp:gcs_core_open():221: Failed to open backend connection: -110 (Connection timed out)
2024-02-12 08:50:37 | error.log | 2024-02-12 8:50:35 0 [ERROR] WSREP: ./gcs/src/gcs.cpp:gcs_open():1674: Failed to open channel 'BAST_NEW' at 'gcomm://[ip list]': -110 (Connection timed out)
2024-02-12 08:50:37 | error.log | 2024-02-12 8:50:35 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2024-02-12 08:50:37 | error.log | 2024-02-12 8:50:35 0 [ERROR] WSREP: wsrep::connect(gcomm://[ip list]) failed: 7
2024-02-12 08:50:37 | error.log | 2024-02-12 8:50:35 0 [ERROR] Aborting
- All nodes are in same VPN
- Traffic in VPN is open
- They can ping themselves
- It looks like network problem BUT it works fine around one month without problems.
- No changes in firewall made since production run
After restarting VM (node) problem is still present and mysql is down.
Any idea what might have happened and how to restore communication?
Thanks in advance!