Wget: want date of downloaded file to be current date

[hoven]

When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. I understand that's the best practice, but I don't like it that way.

Alternatively, Curl keeps the current date by default. However, I have a problem with Curl because it fails to download files completely from some servers. Curl often cuts off with about one percent of the file still to go.

Can either of these two behaviors be modified? I would be very happy to use Curl if the incomplete file situation could be resolved.

[Flash]

I don't really know anything about wget except that If you enter wget --help in a console, wget will list all of its options. Maybe one of them will work for you.
I never even heard of Curl before now. I entered curl --help in a console and found that it has lots of options too.

[L18L]

When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. ...

touch the file
more info:

Code: Select all

touch --help
man touch

[hoven]

I don't really know anything about wget except that If you enter wget --help in a console, wget will list all of its options. Maybe one of them will work for you.

I've tried reading through the many options but I can't find one that does what I want. The timestamping options are for website mirroring.

Curl will give the current time and date by default, but it's the download breaking before the end that's the problem. Maybe it's an incompatibility with the servers in question.

[hoven]

When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. ...

touch the file

Well if touching is your thing

I'm aware of touch, but I don't want to do it for hundreds of files every time I download.

[npierce]

When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server.

Actually, when using wget, both the time the file was last modified and the time that you downloaded it are saved. One is the mtime (modification of file content) and one is the ctime (change of file status).

The ctime is available with:

Code: Select all

ls -lc

Will that fulfill your needs?

If not, read on . . .

Personally, I like having both timestamps available to me. But there are certain drawbacks to depending upon the ctime to store the download time:

• If you copy the file, the ctime won't go with it. The new copy's ctime will have the time that it was created.
  
  If you change the files owner, group, or permissions, or if you add or remove a hard link to the file, the ctime is changed to the current time.

(Usually, neither of those are problems for me because I usually just leave the file where I've downloaded it to, never change its status, and backup using mkisofs, which (if I remember correctly) copies each file's existing ctime into the directory of the .iso image.)

But if any of those drawbacks would be a problem for you, or if you have other reasons, you could try a script like this to copy the ctime to the mtime:

Code: Select all

#!/bin/bash
for FILENAME in "$@" ; do
CTIME="`stat -c %z "$FILENAME"`"
touch -m -d "$CTIME" "$FILENAME"
done

The script takes a list of filenames for parameters, which can include glob patterns. So, if you name it, for instance, "ctime_to_mtime", you can do this:

Code: Select all

ctime_to_mtime example.pdf

or

Code: Select all

ctime_to_mtime example1.pdf example2.pdf example3.pdf

or

Code: Select all

ctime_to_mtime *.pdf *.txt

or

Code: Select all

ctime_to_mtime downloads/20120924/*

(Of course you need to be careful when you use glob patterns. If you are in the wrong directory, or make a typo in the glob pattern, you could accidentally change the mtimes on a whole bunch of unintended files.)

(And you would need to be careful not to run this twice on the same file. Since running it changes ctime, running it again on the same file would overwrite mtime with the time you previously ran it.)

[Karl Godt]

I don't use curl or wget much . Both have a continue option :

curl :
-C/--continue-at <offset>
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be
skipped counted from the beginning of the source file before it
is transferred to the destination. If used with uploads, the
ftp server command SIZE will not be used by curl.

Use "-C -" to tell curl to automatically find out where/how to
resume the transfer. It then uses the given output/input files
to figure that out.

If this option is used several times, the last one will be used.

wget is lesser c :

-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by
another program. For instance:

wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

wget has also, which i don't know how they work :
-N
--timestamping
Turn on time-stamping.

*

Lastly that i had wget running was to download patches for bash like :
history | grep wget
wget -r -l1 -np -R .sig -N -nc ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/
which kept the date for the files from the server (-N) .

and wget -c for a partially downloaded windows-8-test.iso of more than 3GB .