How to deploy system dependent services with docker stack?

Question

I must deploy a service service1 (with username/image) into three different systems (node1, node2 & node3). Due to the architectural differences in the systems, I built the username/image for the all devices and tagged it as username/image:node1, username/image:node2 and username/image:node3 (for node1, node2 and node3 respectively). However, I created the following compose.yml file to deploy the service into the cluster:

version: '3'

services:
  service1:
    image: username/image:node${nodeID}
    deploy:
      replicas: 6
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.with_imgTag==node${imgTag}
      restart_policy:
        condition: on-failure

First, I set labels to the nodes as:

docker node update --label-add with_imgTag=node1 node1
docker node update --label-add with_imgTag=node2 node2
docker node update --label-add with_imgTag=node3 node3

and then deploy service1 with the following command:

for ((c=1; c<=3; c++)) do nodeID=$c imgTag=$c docker stack deploy -c compose.yml stack-4-service-1; done

In the stack-4-service-1, the docker stack deploy updates (the previous service or at least the running one) with a new one into any node (as the restart_policy is specified). I want to deploy the service1 where a node is not hosting service1 at the moment (by specifying max_replicas_per_node). How can I do that? - Thanks in advance!

Answer 1

A service has a 1:1 relationship with its image so if you want to use "replicas" and "max_replicas_per_node" to manage how a service is scheduled on nodes you need to have a single image.

If by architectural differences you mean cpu or OS architecture then build a multiplatform image so you can use a single image ref. If you mean different configurations, then use mounts or volumes unique to each node to carry the per-node differences.

In the case that volumes and mounts can't be used, you could build one image with all N variants and use environment variables that use service template parameters to control behaviour in your startup script.

e.g.

services:
  service1:
    image: user/image:latest
    environment:
      THIS_NODE: "{{.Node.Hostname}}"
    command:
    - /bin/bash
    - -c
    - |
      case $$THIS_NODE in
        node1)
          app /data/node1
          ;;
        node2)
          app /data/node2
          ;;
      esac