When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. I understand that's the best practice, but I don't like it that way.
Alternatively, Curl keeps the current date by default. However, I have a problem with Curl because it fails to download files completely from some servers. Curl often cuts off with about one percent of the file still to go.
Can either of these two behaviors be modified? I would be very happy to use Curl if the incomplete file situation could be resolved.
Wget: want date of downloaded file to be current date
- L18L
- Posts: 3479
- Joined: Sat 19 Jun 2010, 18:56
- Location: www.eussenheim.de/
Re: Wget:want date of downloaded file to be current date
touch the filehoven wrote:When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. ...
more info:
Code: Select all
touch --help
man touch
I've tried reading through the many options but I can't find one that does what I want. The timestamping options are for website mirroring.Flash wrote:I don't really know anything about wget except that If you enter wget --help in a console, wget will list all of its options. Maybe one of them will work for you.
Curl will give the current time and date by default, but it's the download breaking before the end that's the problem. Maybe it's an incompatibility with the servers in question.
Re: Wget:want date of downloaded file to be current date
Well if touching is your thingL18L wrote:touch the filehoven wrote:When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server. ...
I'm aware of touch, but I don't want to do it for hundreds of files every time I download.
Actually, when using wget, both the time the file was last modified and the time that you downloaded it are saved. One is the mtime (modification of file content) and one is the ctime (change of file status).hoven wrote:When I use Wget it makes the date of downloaded files to be the last-modified date from the remote server.
The ctime is available with:
Code: Select all
ls -lc
If not, read on . . .
Personally, I like having both timestamps available to me. But there are certain drawbacks to depending upon the ctime to store the download time:
- If you copy the file, the ctime won't go with it. The new copy's ctime will have the time that it was created.
If you change the files owner, group, or permissions, or if you add or remove a hard link to the file, the ctime is changed to the current time.
But if any of those drawbacks would be a problem for you, or if you have other reasons, you could try a script like this to copy the ctime to the mtime:
Code: Select all
#!/bin/bash
for FILENAME in "$@" ; do
CTIME="`stat -c %z "$FILENAME"`"
touch -m -d "$CTIME" "$FILENAME"
done
Code: Select all
ctime_to_mtime example.pdf
Code: Select all
ctime_to_mtime example1.pdf example2.pdf example3.pdf
Code: Select all
ctime_to_mtime *.pdf *.txt
Code: Select all
ctime_to_mtime downloads/20120924/*
(And you would need to be careful not to run this twice on the same file. Since running it changes ctime, running it again on the same file would overwrite mtime with the time you previously ran it.)
I don't use curl or wget much . Both have a continue option :
curl :
-C/--continue-at <offset>
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be
skipped counted from the beginning of the source file before it
is transferred to the destination. If used with uploads, the
ftp server command SIZE will not be used by curl.
Use "-C -" to tell curl to automatically find out where/how to
resume the transfer. It then uses the given output/input files
to figure that out.
If this option is used several times, the last one will be used.
wget is lesser c :
-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by
another program. For instance:
wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
wget has also, which i don't know how they work :
-N
--timestamping
Turn on time-stamping.
*
Lastly that i had wget running was to download patches for bash like :
history | grep wget
wget -r -l1 -np -R .sig -N -nc ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/
which kept the date for the files from the server (-N) .
and wget -c for a partially downloaded windows-8-test.iso of more than 3GB .
curl :
-C/--continue-at <offset>
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be
skipped counted from the beginning of the source file before it
is transferred to the destination. If used with uploads, the
ftp server command SIZE will not be used by curl.
Use "-C -" to tell curl to automatically find out where/how to
resume the transfer. It then uses the given output/input files
to figure that out.
If this option is used several times, the last one will be used.
wget is lesser c :
-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by
another program. For instance:
wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
wget has also, which i don't know how they work :
-N
--timestamping
Turn on time-stamping.
*
Lastly that i had wget running was to download patches for bash like :
history | grep wget
wget -r -l1 -np -R .sig -N -nc ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/
which kept the date for the files from the server (-N) .
and wget -c for a partially downloaded windows-8-test.iso of more than 3GB .