Wget 1.13.4 ( Lucid 528-005 )

[chrome307]

* Updated package - Lucid 528-005 uses 1.12



Homepage:

http://www.gnu.org/software/wget/

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including:

• Can resume aborted downloads, using REST and RANGE
  Can use filename wild cards and recursively mirror directories
  NLS-based message files for many different languages
  Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  Runs on most UNIX-like operating systems as well as Microsoft Windows
  Supports HTTP proxies
  Supports HTTP cookies
  Supports persistent HTTP connections
  Unattended / background operation
  Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring
  GNU Wget is distributed under the GNU General Public License.

NB wget-1.13.4-i686 is the binary file, the other two are not required unless you want the documentation and native language support (non-english)

[RetroTechGuy]

And how about a sample batch file (I don't recall what version of wget I'm using, but I doubt there have been any significant changes)

I have a file called something like "get50" (as in get 50 MB, then exit). Create a file, cut and paste the following text into it.

Code: Select all

wget --mirror --recursive --continue --proxy=on --convert-links --quota=50M --timeout=5 --tries=30 --input-file=urls.txt --output-file=get.log

What I do from the command line is:

Code: Select all

source ./get50

Looking at the line, you can see the quota adjust (adjust to your preferred level).

You insert a stack of URLS in a file called "urls.txt", which must sit in the local directory. Note: if the files in the urls.txt are already "gotten", wget skips and then continues on the next file/url. No limit to the number of URLS in the file (that I'm aware of...).

The log file is also written to the local directory.

How the 50 MB limit works under wget is: it "gets" files until the download total exceeds 50 MB, and then continues until the current file is complete.

So, you could start the above on a typical Puppy ISO (~100MB), and it would complete a total of 1 file (50MB or 1 file, whichever is greater).

Keep in mind that not all sites tolerate "robots", so wget may not be welcome at all of them.

In years past, I would often do something of the nature (command line only, no GUI):

Code: Select all

source ./get50 ; poweroff

Which would complete the operation, then perform a machine shutdown.

[chrome307]