The time now is Wed 25 Apr 2018, 14:44
All times are UTC - 4 |
Author |
Message |
linuxcbon
Joined: 09 Aug 2007 Posts: 1165
|
Posted: Sun 30 Jul 2017, 08:24 Post subject:
|
|
Hi all,
I have a remark : isnt zeroing the whole stick more reliable ?
Until now, your results depend on the file system and maybe other factors...
Last edited by linuxcbon on Mon 31 Jul 2017, 14:54; edited 1 time in total
|
Back to top
|
|
 |
linuxcbon
Joined: 09 Aug 2007 Posts: 1165
|
Posted: Sun 30 Jul 2017, 08:27 Post subject:
|
|
Ah, I found a much faster and reliable method
dd an iso to the usb :
Code: | # dd if=xerus64-8.3.iso of=/dev/sdb
856064+0 records in
856064+0 records out
438304768 bytes (438 MB) copied, 191,186 s, 2,3 MB/s |
It gives the same results as zeroing the whole stick, but is much faster .
And it doesnt depend on the file system.
NOTE : DONT USE dd ON A USB STICK WITH IMPORTANT DATA !!!
ALWAYS BACKUP YOUR DATA !!!!
Last edited by linuxcbon on Mon 31 Jul 2017, 14:55; edited 1 time in total
|
Back to top
|
|
 |
BarryK
Puppy Master

Joined: 09 May 2005 Posts: 8529 Location: Perth, Western Australia
|
Posted: Sun 30 Jul 2017, 19:49 Post subject:
|
|
linuxcbon wrote: | Hi all,
I have a remark : isnt zeroing the whole stick more reliable ?
Until now, your results depend on the file system and maybe other factors...
Why not just do a # dd if=/dev/zero of=/dev/sdb
I dont know how long it takes with huge sized stick. Mine is 2GB so it is done quickly , and I got :
Code: | # dd if=/dev/zero of=/dev/sdb
dd: writing to `/dev/sdb': No space left on device
4137985+0 records in
4137984+0 records out
2118647808 bytes (2,1 GB) copied, 823,927 s, 2,6 MB/s |
It is a very crappy USB 2.0 stick.  |
You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".
Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.
Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"
And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache.
_________________ http://bkhome.org/news/
Last edited by BarryK on Sun 30 Jul 2017, 20:23; edited 1 time in total
|
Back to top
|
|
 |
tallboy

Joined: 21 Sep 2010 Posts: 907 Location: Oslo, Norway
|
Posted: Sun 30 Jul 2017, 20:17 Post subject:
|
|
BarryK wrote: | Here are more tests:
http://barryk.org/news/?viewDetailed=00631
The question that was asked whether type of filesystem makes a difference, it does! |
Ouch, that was a surprisingly big difference between fat32, ext2 and ext4! Thank you for testing, Barry! I use ext2, and I usually only do back up to the flash sticks. When I'm finished with a session and do the backups, I am really not very concerned about the upload speed, my files are usually not that big.
linuxcbon wrote: | Ah, I found a much faster and reliable method
dd an iso to the usb : |
OK, let's do a test:
Code: | # dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s |
It's slower!! This is the same flash stick that I tested to a speed of 3.2MB/s using the first method
tallboy
_________________ True freedom is a live Puppy on a multisession CD/DVD.
|
Back to top
|
|
 |
scsijon
Joined: 23 May 2007 Posts: 1313 Location: the australian mallee
|
Posted: Sun 30 Jul 2017, 22:10 Post subject:
|
|
BarryK wrote: | And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache. |
And i'm wondering if this is a real point factor in measuring and the results ocurring, especially if there is or isn't a cache to start with. Is it the cache speed and size capacity affecting your measuring, rather than the overall performance of the flash drive? Maybe different blocksizes would result in different speeds with the cache being filled before real memory writeout is completed.
I'd like to see some re-testing with and without this setting. I also wonder if there is someway to find out how much cache is in there and what it's specs are as i'm sure it would also affect the report.
I's also like to see what happens if you did it to a so called 'unformatted' drive partition where it's writing 'raw'. That's suppose to be the fastest for a Hard drive(, but easiest to corrupt,) so needs software to support the partition table, as we use to do with some of our largest Netware Servers.
|
Back to top
|
|
 |
tallboy

Joined: 21 Sep 2010 Posts: 907 Location: Oslo, Norway
|
Posted: Sun 30 Jul 2017, 22:42 Post subject:
|
|
Uh, uh!
I must have made a mistake, I need some help here!
The last test, from post above:
Code: | # dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s |
I wrote /dev/sdb1 instead of /mnt/sdb1.
I tried to correct the test, but;
Code: | # dd if=custom-puppy.iso of=/mnt/sdb1/dummyfile
dd: opening `/mnt/sdb1/dummyfile': No space left on device | What?
This is strange, to say it mildly, it is still my SanDisk Cruzer Edge 32 Gb that used to have 15 Gb free space, now it has 6 empty directories -still with names - and one empty file.
Code: | # df -h /mnt/sdb1/
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 16T 16T 0 100% /mnt/sdb1 | Wow, 16Terabyte?? One more try:
Code: | # df /mnt/sdb1/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 17179387340 17179387340 0 100% /mnt/sdb1 | That must be a new world record!
Code: | # ls -Ahov /dev/sdb1
brw-rw---- 1 root 8, 17 2017-07-31 02:28 /dev/sdb1 |
Code: | # ls -Ahov /mnt/sdb1
total 40K
drwxr-xr-x 3 root 4.0K 2017-04-27 22:55 dirname
drwxr-xr-x 17 root 4.0K 2016-05-04 18:39 dirname
drwx------ 2 root 16K 2015-05-15 05:34 lost+found
drwxr-xr-x 21 root 4.0K 2017-07-13 06:51 dirname
drwxr-xr-x 3 root 4.0K 2017-05-29 13:56 dirname
-rw-r--r-- 1 root 3.3K 2015-05-17 06:08 filename.txt
drwxr-xr-x 10 root 4.0K 2016-08-10 03:23 dirname |
I changed the names of directories and files just for the forum, they are all empty.
What the hell happened?
What should I do now? Can I run fsck to find some lost files? Or umount all?
tallboy
Description |
|
Filesize |
43.62 KB |
Viewed |
283 Time(s) |

|
_________________ True freedom is a live Puppy on a multisession CD/DVD.
|
Back to top
|
|
 |
BarryK
Puppy Master

Joined: 09 May 2005 Posts: 8529 Location: Perth, Western Australia
|
Posted: Mon 31 Jul 2017, 07:09 Post subject:
|
|
tallboy wrote: | Uh, uh!
I must have made a mistake, I need some help here!
The last test, from post above:
Code: | # dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s |
|
That will have destroyed the content of partition sdb1.
You will have to reformat it with ext4 or whatever.
_________________ http://bkhome.org/news/
|
Back to top
|
|
 |
tallboy

Joined: 21 Sep 2010 Posts: 907 Location: Oslo, Norway
|
Posted: Mon 31 Jul 2017, 07:21 Post subject:
|
|
Thank you, Barry. I feared it was somthing like that. The stick was only one of two backup sticks, so there's no catastrophy, but anyway...
Do you now why the names of the emptydirectories still are intact? Why wasn't it wiped clean?
tallboy
_________________ True freedom is a live Puppy on a multisession CD/DVD.
|
Back to top
|
|
 |
tallboy

Joined: 21 Sep 2010 Posts: 907 Location: Oslo, Norway
|
Posted: Mon 31 Jul 2017, 07:32 Post subject:
|
|
Ha, ha, it's not wiped clean!
But the directory names just hanged there in the previous terminal window., they are gone now.
I unmounted all partitions, and remounted it, and I now have 230Mb of Puppy on my SanDisk!
But, when I hower the mouse over the flash icon, it says filesystem iso9660, size 29.1 Gb!
When I ask with df:
Code: | # df -h /mnt/sdb1/
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 230M 230M 0 100% /mnt/sdb1 |
Darn, I lost almost 16 Tb!
tallboy
_________________ True freedom is a live Puppy on a multisession CD/DVD.
|
Back to top
|
|
 |
rcrsn51

Joined: 05 Sep 2006 Posts: 11889 Location: Stratford, Ontario
|
Posted: Mon 31 Jul 2017, 07:52 Post subject:
|
|
Something to consider: For each person who uses dd incorrectly, has a disaster and reports it, there will be ten people who don't report it.
Last edited by rcrsn51 on Mon 31 Jul 2017, 08:35; edited 1 time in total
|
Back to top
|
|
 |
tallboy

Joined: 21 Sep 2010 Posts: 907 Location: Oslo, Norway
|
Posted: Mon 31 Jul 2017, 08:00 Post subject:
|
|
Haha, I guess you are right about that, rcrsn51. I have done some incredibly stupid things with Linux over the years, and almost all could have been avoided if I just had read the manual first!
I'm running off a live-CD, so I guess I'll restart the Puppy, to be on the safe side. I'll wipe the stick clean now, but I think it's better to format it from a fresh Puppy!
It's one of the advantages of a live Puppy, so you see I have learnt something from all my mistakes!
tallboy
_________________ True freedom is a live Puppy on a multisession CD/DVD.
|
Back to top
|
|
 |
linuxcbon
Joined: 09 Aug 2007 Posts: 1165
|
Posted: Mon 31 Jul 2017, 14:56 Post subject:
|
|
BarryK wrote: | You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".
Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.
Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"
And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache. |
I dont know where you get all that from, and if this is important or mandatory for the testing, I will look at the dd manual.
But one thing is sure : your method is not reliable as it depends on the filesystem.
|
Back to top
|
|
 |
linuxcbon
Joined: 09 Aug 2007 Posts: 1165
|
Posted: Mon 31 Jul 2017, 14:57 Post subject:
|
|
tallboy wrote: | Haha, I guess you are right about that, rcrsn51. I have done some incredibly stupid things with Linux over the years, and almost all could have been avoided if I just had read the manual first!
I'm running off a live-CD, so I guess I'll restart the Puppy, to be on the safe side. I'll wipe the stick clean now, but I think it's better to format it from a fresh Puppy!
It's one of the advantages of a live Puppy, so you see I have learnt something from all my mistakes!
tallboy |
You didnt read the command correctly, it is sdb and not sdb1.
I have updated my message with this :
NOTE : DONT USE dd ON A USB STICK WITH IMPORTANT DATA !!!
ALWAYS BACKUP YOUR DATA !!!!
|
Back to top
|
|
 |
linuxcbon
Joined: 09 Aug 2007 Posts: 1165
|
Posted: Mon 31 Jul 2017, 17:40 Post subject:
|
|
BarryK wrote: | You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".
Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.
Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"
And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache. |
Ok I have read some dd manual and other pages :
- ‘fdatasync’ "Synchronize output data just before finishing. This forces a physical write of output data. "
---> it is the same as a "sync" after the dd command.
- ‘bs=bytes’ "Set both input and output block sizes to bytes."
‘count=n’ "Copy n ‘ibs’-byte blocks from the input file,"
---> It is mostly used for writing many small files , as in bs=1M count=1024 , which writes 1024 files of 1Mb.
You dont need it if you write one big ISO file.
- "oflag=direct" "Use direct I/O for data, avoiding the buffer cache. Note that the kernel may impose restrictions on read or write buffer sizes. For example, with an ext4 destination file system and a Linux-based kernel, using ‘oflag=direct’ will cause writes to fail with EINVAL if the output buffer size is not a multiple of 512. "
---> So it is not recommended or needed.
A plain and simple "dd if=puppy.iso of=/dev/sdb ; sync" is enough.
I noticed that many people prefer to test the write speed like this, on a mounted filesystem, so it depends on ext4, ext2, fat etc. :
# dd if=/dev/zero of=/mnt/sdb1/testfile bs=1M count=1024
# sync
# rm /mnt/sdb1/testfile
|
Back to top
|
|
 |
BarryK
Puppy Master

Joined: 09 May 2005 Posts: 8529 Location: Perth, Western Australia
|
Posted: Mon 31 Jul 2017, 22:44 Post subject:
|
|
linuxcbon wrote: |
A plain and simple "dd if=puppy.iso of=/dev/sdb ; sync" is enough. |
NO, it is not, dd will complete before flushing all data to the drive.
Also, writing to the filesystem on the drive is precisely what I want to test. Not raw writing to the drive.
scsijon,
Yes, "oflag=direct" does make a difference, I tested on one flash stick, and got a slower transfer rate with it. So, it did appear to be doing what the docs state, bypassing the cache in the drive.
I did that test before my original post to this thread and blog.
_________________ http://bkhome.org/news/
|
Back to top
|
|
 |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|