How to try all servers in dns using libcurl?

Question

I need to regularly and randomly test with Linux/C++/libcurl the responses of several servers that are available through a single DNS name, such as

$ host example.com
n1.example.com 1.2.3.4
n2.example.com 1.2.3.5
n3.example.com 1.2.3.6

The list changes. When I try https://example.com libcurl always uses the same IP for the span of the TTL, and I cannot switch to the next host. There is CURLOPT_DNS_CACHE_TIMEOUT setopt, but setting it to zero does not help - even if I fully recreate easycurl object I still get the same IP. Therefore, this does not help: curl - How to set up TTL for dns cache & How to clear the curl cache

I can of course manually resolve DNS names and iterate, but are there any options? Polling randomly is okay. I see curl uses c-ares. Is there a way to clean up the cache there and will it help?

Answer 1

I cannot do exactly what I need with curl without doing a resolve by myself, but there are findings for the others to share with:

First of all, as a well-written TCP client, curl will try the hosts from the DNS list from top to bottom until a successful connection is made. Since then it will use that host even if it returns some higher level error (such as SSL error or HTTP 500). This is good for all major cases.

Curl command line of newer curl versions has --retry and --retry-all-errors - but there are no such things in libcurl, unfortunately. The feature is being enhanced right now, and there is no release yet as of 2021-07-14 that will enumerate all DNS hosts until there is one that returns HTTP 200. Instead, the released curl versions (I tried 7.76 and 7.77) will always do retries with the same host. But the nightly build (2021-07-14) does enumerate all DNS hosts. Here is how it behaves for two retries and three inexisting hosts (note, the retries will happen if any host returns HTTP 5xx):

$ ./src/curl http://nohost.sureno --trace - --retry 2 --retry-all-errors
    == Info:   Trying 192.168.1.112:80...
    == Info: connect to 192.168.1.112 port 80 failed: No route to host
    == Info:   Trying 192.168.1.113:80...
    == Info: connect to 192.168.1.113 port 80 failed: No route to host
    == Info:   Trying 192.168.1.114:80...
    == Info: connect to 192.168.1.114 port 80 failed: No route to host
    == Info: Failed to connect to nohost.sureno port 80 after 9210 ms: No route to host
    == Info: Closing connection 0
    curl: (7) Failed to connect to nohost.sureno port 80 after 9210 ms: No route to host
    Warning: Problem (retrying all errors). Will retry in 1 seconds. 2 retries
    Warning: left.
    == Info: Hostname nohost.sureno was found in DNS cache
    == Info:   Trying 192.168.1.112:80...
    == Info: connect to 192.168.1.112 port 80 failed: No route to host
    == Info:   Trying 192.168.1.113:80...
    == Info: connect to 192.168.1.113 port 80 failed: No route to host
    == Info:   Trying 192.168.1.114:80...
    == Info: connect to 192.168.1.114 port 80 failed: No route to host
    == Info: Failed to connect to nohost.sureno port 80 after 9206 ms: No route to host
    == Info: Closing connection 1
    curl: (7) Failed to connect to nohost.sureno port 80 after 9206 ms: No route to host
    Warning: Problem (retrying all errors). Will retry in 2 seconds. 1 retries

This behavior can be very helpful for the users of libcurl, but unfortunately, these retry flags presently have no mapping to curl_easy_setopt. And as a result, if you give --libcurl to the command line you will not see any retry-related code