docker services, choose a preferable node to run or rearrange all services if leader comes down

Question

I have 2 swarm nodes and I whish that in case one node shut down, the other one rearrange all services to itself. Right now I have one leader(manager) and one worker, and it works perfectly if the worker goes down, because leader rearranges all services to itself. My problem here is when leader goes down and no one assumes services within it. I already tried with two managers, but didn't works.

So I am thinking about to let all my services in the worker node so if leader node goes down there is no problem at all and if worker node goes down, leader node would rearrange all services to itself. I tried with

deploy:
  placement:
    constraints:
      - "node.role!=manager"

But it also does not works, because it will never instance this service in a manager node.

So I would like to ask if there is any way to make those two nodes to rearrange all services to itself in case other goes down?! or There is an way to configure a service to "preferably" be deployed in one specific node if that node is available otherwise be deployed in any other node?

Answer 1

The rub of it is, you need 3 nodes, all managers. It is not a good idea, even with a 2 node swarm, to make 2 nodes managers as docker swarm uses the raft protocol for manager quorum, and this protocol requires a clear majority. With two manager nodes, if either node goes down, the remaining manager node only represents 50% of the swarm managers and so will not represent the swarm until qorum is restored.

Once you have 3 nodes - all managers - the swarm will tolerate any single nodes failure and move tasks to the other two nodes.

Don't bother with 4 manager nodes - they dont provide extra protection from single node failures, and don't protect from two node failures as, again, only 2 out 4 does not represet more than 50%, to survive 2 node failures you want 5 managers.