wget force retry until there is a connection

Question

I have a bash script that uses wget running on a 3g device. I want the script to pause until I get a response so I have setup my code like this:

wget -t 0 -T 15 --retry-connrefused www.example.com

The problem is that when there is no connection at all (3g drops for a few seconds) the DNS can't resolve the host and wget stops retrying.

Is there a way to force retry until the connection comes back? I know I can write a loop but I want to know if there is a setting in wget I can use. If there is not what the best variable to build the loop on?

More details:

With the code I wrote, wget will retry but if the device has no Internet connection at all (for example I pull out the 3g dongle from the device) it will stop retrying telling me it can't resolve the host address. It sends that after the 15 seconds defined by -T 15

wget: unable to resolve host address

Answer 1

This loop should do this:

while true;do
wget -T 15 -c http://example.com && break
done

---

How it works:

1. In case there is no network connection, the while loop will not break and it will run the wget command continuously and keep printing error message.
2. As soon as it gets connected to the Internet, wget starts resolving the host and getting the files.
3. Now if the connection is lost or some other error occurs, default retry (don't use 0 or inf i.e unlimited retry, use limited value) of wget will retry to get the files until timeout of 15 seconds reached. After 15 seconds the wget command will fail and print error output and thus the while loop won't break. So it will again reach in a state where there is no connection or such and keep printing error message.
4. Again as soon as it gets connected or the error is resolved, wget starts resolving the host and getting the files. These steps (1-4) continue as long as the files are not downloaded completely.
5. This wget command uses -c option, i.e resume option. So every instances of wget will start (downloading) from where it left off.
6. When the files get downloaded completely and the wget command succeeds, the loop will break.

Answer 2

I would not use a loop. You could end up with multiple downloads. Rather invent smarter way to check the network connection. Try pinging some sites beforehand and issue wget only if they succeed. If they fail the script could wait and try the test again. Ping can do quiet tests and use counts and deadlines for returns.

Answer 3

Here is a script that you can use to resolve your problem

FILENAME=$1
DOWNURL=$2

wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
FILESIZE=$(stat -c%s "$FILENAME")

while [ $FILESIZE \< 1000 ]; do
    sleep 3
    wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
    FILESIZE=$(stat -c%s "$FILENAME")
done

The script forces the download to continue until it finishes. The 1000 could be changed to fit whatever size file you are downloading.

Answer 4

If your version of wget supports it, you should use --retry-on-host-error to retry name resolution failures.

From the man page for wget:

‘--retry-on-host-error’

Consider host errors, such as “Temporary failure in name resolution”, as non-fatal, transient errors.

source: https://www.gnu.org/software/wget/manual/wget.html