Selenium DNS error when DNS works absolutely fine

218 Views Asked by At

I very much apologize if this is not quite a stack overflow question, but Selenium seems to be acting rather weirdly. I am using geckodriver with Firefox, and during startup, I get a weird DNS exception. My DNS works just fine, and after the DNS exception Selenium starts normally, and I can perform any request I like. But it takes like six seconds to start, and this seems to be because of the DNS exception. This is getting annoying as I am repeatedly testing new code, and this delay will be very impractical once I start actually automating the code (unless I reuse the same drivers).

I am using a python venv to contain the packages. I'm also using pypy 7.3.13 if that bears any relevance at all.

Here is how I'm starting Selenium:

...
    opts = Options()
    opts.page_load_strategy = 'normal'

    # This works for headless.
    # Note that headless is False in the context of this question.
    if headless == True:
        environ[ 'MOZ_HEADLESS' ] = '1'

    # the following two lines were an attempt to solve this problem by manually 
    # specifying a working DNS server. The same thing happened as if I removed them...
    opts.set_preference( 'network.trr.mode', 2 )
    opts.set_preference( 'network.trr.uri', 'https://mozilla.cloudflare-dns.com/dns-query' )

    # verbose is True in the context of the question
    if verbose == True:
        print( opts.to_capabilities() )

    driver = webdriver.Firefox( options = opts )
...

Here is the selenium log:

Selenium Manager binary found at: /home/the_bananae/.local/lib/scraping/lib/pypy3.10/site-packages/selenium/webdriver/common/linux/selenium-manager
Executing process: /home/the_bananae/.local/lib/scraping/lib/pypy3.10/site-packages/selenium/webdriver/common/linux/selenium-manager --browser firefox --debug --output json
Found geckodriver 0.33.0 in PATH: /home/the_bananae/.local/bin/geckodriver
firefox detected at /usr/bin/firefox
Running command: /usr/bin/firefox -v
Output: "Mozilla Firefox 120.0.1"
Detected browser: firefox 120.0.1

Exception managing firefox: error sending request for url (https://github.com/mozilla/geckodriver/releases/latest): error trying to connect: dns error: failed to lookup address information: Try again
# ^^^ this ^^^

Driver path: /home/the_bananae/.local/bin/geckodriver
Browser path: /usr/bin/firefox
Using driver at: /home/the_bananae/.local/bin/geckodriver
Started executable: `/home/the_bananae/.local/bin/geckodriver` in a child process with pid: 158408

And if it helps, here's cat /etc/os-release:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

And here's uname -a:

Linux whitebanana 6.6.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Nov 2023 00:37:40 +0000 x86_64 GNU/Linux

(yes, my user is the_bananae on the host whitebanana. don't judge)

And here's ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:18:95:3a:00:41 brd ff:ff:ff:ff:ff:ff
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:e2:3c:37:d5:b6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.63/24 scope global dynamic noprefixroute wlan0
       valid_lft 84560sec preferred_lft 84560sec
    inet6 fe80::96e2:3cff:fe37:d5b6/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

As you can see, I'm connected to wifi; I use iwd to this end, and DNS resolution is handled by systemd-resolved. Just for completeness, here's the latest systemd-resolved log:

-- Boot 01457dc11d9f44fe8af8ad983c66374f --
Dec 07 13:32:20 whitebanana systemd[1]: Starting Network Name Resolution...
Dec 07 13:32:20 whitebanana systemd-resolved[551]: Positive Trust Anchors:
Dec 07 13:32:20 whitebanana systemd-resolved[551]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 07 13:32:20 whitebanana systemd-resolved[551]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 07 13:32:20 whitebanana systemd-resolved[551]: Using system hostname 'whitebanana'.
Dec 07 13:32:20 whitebanana systemd[1]: Started Network Name Resolution.
Dec 07 13:33:20 whitebanana systemd-resolved[551]: Switching to fallback DNS server 1.1.1.1#cloudflare-dns.com.
Dec 07 13:35:09 whitebanana systemd-resolved[551]: wlan0: Bus client set DNS server list to: 208.67.222.222, 208.67.220.220
Dec 07 13:35:10 whitebanana systemd-resolved[551]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 208.67.222.222.
Dec 07 22:48:31 whitebanana systemd-resolved[551]: Clock change detected. Flushing caches.
Dec 07 22:48:31 whitebanana systemd-resolved[551]: Switching to fallback DNS server 1.1.1.1#cloudflare-dns.com.
Dec 07 22:48:33 whitebanana systemd-resolved[551]: wlan0: Bus client set DNS server list to: 8.8.8.8, 4.2.2.2
Dec 08 13:30:11 whitebanana systemd-resolved[551]: Clock change detected. Flushing caches.
Dec 08 13:30:11 whitebanana systemd-resolved[551]: Switching to fallback DNS server 1.1.1.1#cloudflare-dns.com.
Dec 08 13:40:01 whitebanana systemd-resolved[551]: wlan0: Bus client set DNS server list to: 192.168.1.1
Dec 08 13:40:01 whitebanana systemd-resolved[551]: wlan0: Bus client set search domain list to: mynetworksettings.com

There seems to be some fishy business with it switching the DNS servers, but my DNS normally works just fine. It is also not switching DNS servers while I'm starting the selenium client, and this just seems to be for initialization.

I've also looked at this and this issue on the Selenium Git, but I have found the answers unsatisfactory. After all, my DNS works. Is this a quirk of the venv?

I have also tried specifying a manual DNS server using these options (network.trr.uri and network.trr.mode specifically) and I had no luck with that either. This does not seem to be a DNS error really, it seems that the local venv configuration is bugged somehow.

Thanks to those that can help with such a specific, weird problem.

0

There are 0 best solutions below