backblaze seems to ignore path mtu - self build debian router has problems with too big packages

Question

The root cause of my problem seems to be too big packages:

17:55:29.577781 eth0  In  IP (tos 0x0, ttl 55, id 62820, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                         
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], cksum 0x9bde (correct), seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231592886 ecr 1643349940], length 1448                                    
17:55:29.577833 eth0  Out IP (tos 0xc0, ttl 64, id 59040, offset 0, flags [none], proto ICMP (1), length 576)                                                                                                      
    #.#.#.# > 45.11.37.254: ICMP #.#.#.# unreachable - need to frag (mtu 1450), length 556                                                                                                           
        IP (tos 0x0, ttl 55, id 62820, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                                           
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231592886 ecr 1643349940], length 1448                                                            
17:55:29.798696 eth0  In  IP (tos 0x0, ttl 55, id 62821, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                         
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], cksum 0x9ba6 (correct), seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231592942 ecr 1643349940], length 1448                                    
17:55:29.798749 eth0  Out IP (tos 0xc0, ttl 64, id 59074, offset 0, flags [none], proto ICMP (1), length 576)                                                                                                      
    #.#.#.# > 45.11.37.254: ICMP #.#.#.# unreachable - need to frag (mtu 1450), length 556                                                                                                           
        IP (tos 0x0, ttl 55, id 62821, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                                           
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231592942 ecr 1643349940], length 1448
17:55:30.254893 eth0  In  IP (tos 0x0, ttl 55, id 62822, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                         
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], cksum 0x9b34 (correct), seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231593056 ecr 1643349940], length 1448                                    
17:55:30.255241 eth0  Out IP (tos 0xc0, ttl 64, id 59143, offset 0, flags [none], proto ICMP (1), length 576)                                                      
    #.#.#.# > 45.11.37.254: ICMP #.#.#.# unreachable - need to frag (mtu 1450), length 556                                                                                                           
        IP (tos 0x0, ttl 55, id 62822, offset 0, flags [DF], proto TCP (6), length 1500)                                                                                                                           
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231593056 ecr 1643349940], length 1448
17:55:31.152224 eth0  In  IP (tos 0x0, ttl 55, id 62823, offset 0, flags [DF], proto TCP (6), length 1500)
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], cksum 0x9a54 (correct), seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231593280 ecr 1643349940], length 1448
17:55:31.152275 eth0  Out IP (tos 0xc0, ttl 64, id 59260, offset 0, flags [none], proto ICMP (1), length 576)
    #.#.#.# > 45.11.37.254: ICMP #.#.#.# unreachable - need to frag (mtu 1450), length 556
        IP (tos 0x0, ttl 55, id 62823, offset 0, flags [DF], proto TCP (6), length 1500)
    45.11.37.254.443 > #.#.#.#.58708: Flags [.], seq 134:1582, ack 518, win 235, options [nop,nop,TS val 2231593280 ecr 1643349940], length 1448

But the fragmented packages never hitting the podman/host if I see it correctly.

I'll now try to force fragmentation and defragmentation. I think backblaze isn't reacting to icmp path mtu.

---

How can I debug a situation where partially data is dropped for some hosts?

• host run curl https://www.google.com - working
• container run curl https://www.google.com - working
• host run curl https://s3.eu-central-003.backblazeb2.com/ - working
• container run curl https://s3.eu-central-003.backblazeb2.com/ - not working
  • SSL hello from server is received but no data afterwards
• host run nc -z s3.eu-central-003.backblazeb2.com 443 - working
• container run nc -z s3.eu-central-003.backblazeb2.com 443 - working

More details can be found here: https://github.com/containers/podman/issues/16021

Answer 1

I found another root cause and solution: podman sets as default mtu 1.500. This leads to a mss(https://en.wikipedia.org/wiki/Maximum_segment_size) of also around 1.500. This of cause is more than the 1.450 which the router and host can talk. After changing the mtu via ifconfig {interface} mtu 1450 up the mss was correctly configured. The request comes through the network and everything works.

---

The question is a deadlock question :D

• Backblaze sends packages with 1.500 MTU
• Debian send icmp response
• Backblaze resends package with 1.500 MUT

After my knowledge this problem is only solvable on server-side and not client side.

I'll evaluate multiple options:

• http-proxy
• public ipv4 for all server
• socks-proxy