Downloading Multiple Files via HTTP/HTTPS

[Trevor]

We've looked at how to download a single file from the web using wget:

            $  wget  http://domain.com/somefile

                           or

            $  wget  https://domain.com/somefile

No biggee!  Peace of cake!

What if we had multiple files that we wanted to download, using wget.  No problem!  We can simply load up a file, with the URLs that point to each file to be downloaded, use the -i option, and we're done.  Let's look at a simple example:

Here are a couple of URLs that point to two different files:

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/pdf/interactively_installing_rhe...

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/pdf/interactively_installing_rhe...

For this example, I will assume that those two URLs are placed in a file named "peedeefs".  The following command will download the two files that those URLs point to, into the current directory:

$  wget  -i  peedeefs

The simple magic that makes this multi-file download happen is the -i option.

This won't get me a Pulitzer Prize, but I think it's pretty cool!

[Blue_bird]

Nice..! Thanks for sharing..!

[Trevor]

Thanks for viewing!!!!

[Chetan_Tiwary_]

@Trevor so you want to take on the mighty wget ? Hang on tight :

1. Restart a download that was interrupted by wget :

wget ‐‐continue example.com/big.file.iso

2. Fetch a file only when the server’s copy is newer :

wget ‐‐continue ‐‐timestamping wordpress.org/latest.zip

3. mirror a site  at a throttled rate with randomized pauses :

wget ‐‐limit-rate=20k ‐‐wait=60 ‐‐random-wait --mirror example.com

what is happening here :

1. Mirror example.com entirely, fetching every linked page and asset.

2. Limit bandwidth to 20 KiB/s, preventing saturation of your network or the server’s.

3. Wait 60 seconds between requests, then randomize the actual delay between half and one-and-a-half times that value (30–90 s), which avoids hammering the server with uniform intervals.

[Trevor]

Chetan, I knew you were going to take this to another level, and so I strapped myself in - shoulder and waist!  I knew that merely hanging on tight was not going to secure me in the seat!!!

Thank you for another elevating contribution!!!