How to configure docker to expose an Erlang node?

Question

I wrote a simple docker image which starts up an Erlang node (rebar3 release, console launch mode). It starts fine and lets me ping the node from within the container. However, I can't get erl shell to ping it from the host — it simply returns pang and nothing is logged in the dockerized console.

The Dockerfile just starts the node, it doesn't do anything more interesting.

---

Checklist

• Cookie is set and matches
• sname is set on both nodes
• Docker node is reachable from other container nodes
• I refer to the docker node using full sname (tried nodename@localhost, nodename@machinename and [email protected])
• epmd port is exposed (tried without it as well)

---

What could I have forgotten to make it work?

Answer 1

Disclaimer: This answer is valid for Linux systems. Also, I think that the first and last options are the easiest

There are several ways to do so, but before, let's set some key ideas:

1. The following script is run inside docker simulating a node, it just waits for a node connection, prints it and terminates.

SCRIPT_RUN_IN_DOCKER='ok = net_kernel:monitor_nodes(true), fun F() -> receive {nodeup, N} -> io:format("Connected to ~p~n", [N]), init:stop() end end().'

2. In order for the distribution protocol to succeed, not only the node needs to be reached at the name used for the ping, the whole node name must match.

3. Erlang's -node can be used with IP, it will be used extensively in the following commands

Options

Now, let's get with the options (All the commands are to be run in different terminals)

Docker: Host network namespace

When starting docker in host's network namespace (--net=host), there's no difference with running both outside of docker (for network purposes). It's the easiest way to connect both nodes using plain docker.

• -name (ip):

   $> docker run --net=host erlang erl -noinput -name [email protected] -setcookie cookie -eval $SCRIPT_RUN_IN_DOCKER
   Connected to '[email protected]'
  
   $> erl -noinput -name [email protected] -setcookie cookie -eval "net_adm:ping('[email protected]'), init:stop()."
  

• -sname with @localhost:

   $> docker run --net=host erlang erl -noinput -sname foo@localhost -setcookie cookie -eval $SCRIPT_RUN_IN_DOCKER
   Connected to bar@localhost
  
   $> erl -noinput -sname bar@localhost -setcookie cookie -eval "net_adm:ping('foo@localhost'), init:stop()."
  

• -sname with @$(hostname -f):

   $> docker run --net=host erlang erl -noinput -sname foo -setcookie cookie -eval $SCRIPT_RUN_IN_DOCKER
   Connected to 'bar@amazing-hostname'
  
   $> erl -noinput -sname bar -setcookie cookie -eval "net_adm:ping('foo@$(hostname -f)'), init:stop()."
  

Docker: Using docker's default bridge (docker0)

By default, docker starts the containers in its own bridge, and these ips can be reached without need to expose any port.

ip a show docker0 lists 172.17.0.1/16 for my machine, and erlang listens in 172.17.0.2 (shown in docker inspect <container>)

• -name (ip):

   $> docker run erlang erl -noinput -name [email protected] -setcookie cookie -eval $SCRIPT_RUN_IN_DOCKER
   Connected to bar@baz
  
   $> erl -noinput -name bar@baz -setcookie cookie -eval "net_adm:ping('[email protected]'), init:stop()."
  

• -sname (fake name resolving to container ip):

   # The trick here is to have exactly the same node name for the destination, otherwise the distribution protocol won't work.
   # We can achieve the custom DNS resolution in linux by editing /etc/hosts
  
   $> tail -n 1 /etc/hosts
   172.17.0.2 erlang_in_docker
  
   $> docker run erlang erl -noinput -name foo@erlang_in_docker -setcookie cookie -eval $SCRIPT_RUN_IN_DOCKER
   Connected to 'bar@amazing-hostname'
  
   $> erl -noinput -sname bar -setcookie cookie -eval "net_adm:ping('foo@erlang_in_docker'), init:stop()."
  

Docker: Using some other docker bridge

Just create the new network and repeat the previous steps, using the ips from the new network

docker network create erlang_docker_network
docker inspect erlang_docker_network

Docker: Exposing ports with two EPMDs

When exposing ports you have to juggle ports and ips because the EPMD ports must be the same.

In this case you are going to have two epmds, one for the host and other for the container (EPMD rejects name requests from non-local peers), listening in the same port number.

The trick here is (ab)using the 127.0.0.* ips that point all to localhost to simulate different nodes. Note the flag to set the distribution port, as mentioned by @legoscia

• -name (ip):

   $> epmd -address 127.0.0.1
  
   $> docker run -p 127.0.0.2:4369:4369/tcp -p 127.0.0.2:9000:9000/tcp erlang erl -noinput -name [email protected] -setcookie cookie -kernel inet_dist_listen_min 9000 -kernel inet_dist_listen_max 9000 -eval $SCRIPT_RUN_IN_DOCKER
   Connected to bar@baz
  
   $> erl -noinput -name bar@baz -setcookie cookie -eval "net_adm:ping('[email protected]'), init:stop()."
  

• -sname (fake name resolving to 127.0.0.2)

   And here we need again the DNS resolution provided by /etc/hosts
  
   $> tail -n 1 /etc/hosts
   127.0.0.2 erlang_in_docker
  
   $> epmd -address 127.0.0.1
  
   $> docker run -p 127.0.0.2:4369:4369/tcp -p 127.0.0.2:9000:9000/tcp erlang erl -noinput -name foo@erlang_in_docker -setcookie cookie -kernel inet_dist_listen_min 9000 -kernel inet_dist_listen_max 9000 -eval $SCRIPT_RUN_IN_DOCKER
   Connected to bar@baz
  
   $> erl -noinput -sname bar@baz -setcookie cookie -eval "net_adm:ping('foo@erlang_in_docker'), init:stop()."
  

Docker-compose

docker-compose allows you to easily set up multi-container systems. With it, you don't need to create/inspect networks.

Given the following docker-compose.yaml:

version: '3.3'

services:
  node:
    image: "erlang"
    command:
      - erl
      - -noinput
      - -sname
      - foo
      - -setcookie
      - cookie
      - -eval
      - ${SCRIPT_RUN_IN_DOCKER} # Needs to be exported
    hostname: node

  operator:
    image: "erlang"
    command:
      - erl
      - -noinput
      - -sname
      - baz
      - -setcookie
      - cookie
      - -eval
      - "net_adm:ping('foo@node'), init:stop()."
    hostname: operator

If you run the following docker-compose run commands, you'll see the results:

$> docker-compose up node
Creating network "tmp_default" with the default driver
Creating tmp_node_1 ... done
Attaching to tmp_node_1
node_1      | Connected to baz@operator
tmp_node_1 exited with code 0

$> docker-compose run operator