mDNS (Avahi) working TO RPis but not between them

Question

Avahi/mDNS is running by default on recent versions of Raspian. Great. Very convenient to just ssh [email protected].

I am doing development on a Mac and operating a local network of headless Raspberry Pis. Up until now, I was able to use mDNS to access the Pis, and the Pis used mDNS to connect to each other.

Today, I shifted the RPis to a private local network by setting them up on a wireless router unconnected to the internet. Once I join the private network, I am still able to access them via mDNS:

% ssh [email protected]
Linux scheduler 5.10.63-v7l+ #1459 SMP Wed Oct 6 16:41:57 BST 2021 armv7l

Last login: Mon Aug  1 09:07:43 2022
pi@scheduler:~ $ 

and

wes@macbook % ssh [email protected]
Linux crossing 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l

Last login: Mon Aug  1 09:07:46 2022
pi@crossing:~ $

But when they try to access each other, I get some results I don't understand:

pi@scheduler:~ $ ping crossing.local
PING crossing.local (10.0.0.1) 56(84) bytes of data.
From 192.168.0.1 (192.168.0.1) icmp_seq=1 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=2 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=3 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=4 Destination Net Unreachable

Here's what Avahi reports:

pi@scheduler:~ $ service avahi-daemon status
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
   Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2022-08-01 09:07:37 PDT; 41min ago
 Main PID: 388 (avahi-daemon)
   Status: "avahi-daemon 0.7 starting up."
    Tasks: 2 (limit: 1438)
   CGroup: /system.slice/avahi-daemon.service
           ├─388 avahi-daemon: running [scheduler.local]
           └─414 avahi-daemon: chroot helper

Aug 01 09:08:08 scheduler avahi-daemon[388]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 169.
Aug 01 09:08:08 scheduler avahi-daemon[388]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.
Aug 01 09:48:29 scheduler avahi-daemon[388]: Files changed, reloading.
Aug 01 09:48:29 scheduler avahi-daemon[388]: No service file found in /etc/avahi/services.

Here's my hosts and hostname files:

pi@scheduler:~ $ cat /etc/hosts
127.0.0.1   localhost
::1     localhost ip6-localhost ip6-loopback
ff02::1     ip6-allnodes
ff02::2     ip6-allrouters

127.0.1.1   scheduler
pi@scheduler:~ $ cat /etc/hostname 
scheduler

What does avahi say about it? Let's see:

pi@brs-scheduler:~ $ avahi-resolve --name brs-crossing.local -4
brs-crossing.local  192.168.0.214
pi@brs-scheduler:~ $ ifconfig | grep "inet 192"
        inet 192.168.0.109  netmask 255.255.255.0  broadcast 192.168.0.255
pi@brs-scheduler:~ $ ping brs-crossing.local
PING brs-crossing.local (10.0.0.1) 56(84) bytes of data.
From 192.168.0.1 (192.168.0.1) icmp_seq=1 Destination Net Unreachable

So for some reason, on this private network, mDNS is resolving correctly, but ping and ssh don't resolve properly?

What am I missing?

Answer 1

Unsurprisingly, since the pis worked fine on the local net and stopped working on a private net with a new router, it had to do with the configuration of the new router not mDNS.

mDNS was working fine:

pi@scheduler:~ $ avahi-resolve --name crossing.local -4
crossing.local  192.168.0.214

The new router on the private net had two operating modes "router" and "access point." In "router" mode, the router was pushing a DNS nameserver IP to clients which was somehow hosing ping and ssh and other services, despite mDNS working okay.

pi@scheduler:~ $ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 192.168.0.1

Once the router was placed in "access point" mode, and DHCP was turned on manually, everything worked.

Obscure problem. Obscure solution.