Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 20 Oct 2018, 10:42
All times are UTC - 4

 Forum index » Advanced Topics » Hardware
USB Flash Drive Quality
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 5 of 6 [82 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6 Next
Author Message
linuxcbon

Joined: 09 Aug 2007
Posts: 1197
PostPosted: Sun 30 Jul 2017, 08:24    Post subject:  
Hi all,
I have a remark : isnt zeroing the whole stick more reliable ?
Until now, your results depend on the file system and maybe other factors...
Last edited by linuxcbon on Mon 31 Jul 2017, 14:54; edited 1 time in total
Back to top
View user's profile Send private message 
linuxcbon

Joined: 09 Aug 2007
Posts: 1197
PostPosted: Sun 30 Jul 2017, 08:27    Post subject:  
Ah, I found a much faster and reliable method Very Happy
dd an iso to the usb :

Code:
# dd if=xerus64-8.3.iso of=/dev/sdb
856064+0 records in
856064+0 records out
438304768 bytes (438 MB) copied, 191,186 s, 2,3 MB/s


It gives the same results as zeroing the whole stick, but is much faster . Very Happy
And it doesnt depend on the file system.

NOTE : DONT USE dd ON A USB STICK WITH IMPORTANT DATA !!!
ALWAYS BACKUP YOUR DATA !!!!
Last edited by linuxcbon on Mon 31 Jul 2017, 14:55; edited 1 time in total
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 8702
Location: Perth, Western Australia
PostPosted: Sun 30 Jul 2017, 19:49    Post subject:  
linuxcbon wrote:
Hi all,

I have a remark : isnt zeroing the whole stick more reliable ?

Until now, your results depend on the file system and maybe other factors...

Why not just do a # dd if=/dev/zero of=/dev/sdb
I dont know how long it takes with huge sized stick. Mine is 2GB so it is done quickly , and I got :
Code:
# dd if=/dev/zero  of=/dev/sdb
dd: writing to `/dev/sdb': No space left on device
4137985+0 records in
4137984+0 records out
2118647808 bytes (2,1 GB) copied, 823,927 s, 2,6 MB/s

It is a very crappy USB 2.0 stick. Shocked


You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".

Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.

Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"

And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache.
_________________
http://bkhome.org/news/

Last edited by BarryK on Sun 30 Jul 2017, 20:23; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
tallboy


Joined: 21 Sep 2010
Posts: 1026
Location: Oslo, Norway
PostPosted: Sun 30 Jul 2017, 20:17    Post subject:  
BarryK wrote:
Here are more tests:

http://barryk.org/news/?viewDetailed=00631

The question that was asked whether type of filesystem makes a difference, it does!


Ouch, that was a surprisingly big difference between fat32, ext2 and ext4! Thank you for testing, Barry! I use ext2, and I usually only do back up to the flash sticks. When I'm finished with a session and do the backups, I am really not very concerned about the upload speed, my files are usually not that big. Very Happy

linuxcbon wrote:
Ah, I found a much faster and reliable method
dd an iso to the usb :

OK, let's do a test:
Code:
# dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s


It's slower!! This is the same flash stick that I tested to a speed of 3.2MB/s using the first method Confused

tallboy
_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
scsijon

Joined: 23 May 2007
Posts: 1362
Location: the australian mallee
PostPosted: Sun 30 Jul 2017, 22:10    Post subject:  
BarryK wrote:
And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache.


And i'm wondering if this is a real point factor in measuring and the results ocurring, especially if there is or isn't a cache to start with. Is it the cache speed and size capacity affecting your measuring, rather than the overall performance of the flash drive? Maybe different blocksizes would result in different speeds with the cache being filled before real memory writeout is completed.

I'd like to see some re-testing with and without this setting. I also wonder if there is someway to find out how much cache is in there and what it's specs are as i'm sure it would also affect the report.

I's also like to see what happens if you did it to a so called 'unformatted' drive partition where it's writing 'raw'. That's suppose to be the fastest for a Hard drive(, but easiest to corrupt,) so needs software to support the partition table, as we use to do with some of our largest Netware Servers.
Back to top
View user's profile Send private message Visit poster's website 
tallboy


Joined: 21 Sep 2010
Posts: 1026
Location: Oslo, Norway
PostPosted: Sun 30 Jul 2017, 22:42    Post subject:  
Uh, uh!
I must have made a mistake, I need some help here!
The last test, from post above:
Code:
# dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s

I wrote /dev/sdb1 instead of /mnt/sdb1.
I tried to correct the test, but;
Code:
# dd if=custom-puppy.iso of=/mnt/sdb1/dummyfile
dd: opening `/mnt/sdb1/dummyfile': No space left on device
What?
This is strange, to say it mildly, it is still my SanDisk Cruzer Edge 32 Gb that used to have 15 Gb free space, now it has 6 empty directories -still with names - and one empty file.
Code:
# df -h /mnt/sdb1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              16T   16T     0 100% /mnt/sdb1
Wow, 16Terabyte?? One more try:
Code:
# df /mnt/sdb1/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdb1            17179387340 17179387340         0 100% /mnt/sdb1
That must be a new world record!
Code:
# ls -Ahov /dev/sdb1
brw-rw---- 1 root 8, 17 2017-07-31 02:28 /dev/sdb1

Code:
# ls -Ahov /mnt/sdb1
total 40K
drwxr-xr-x  3 root 4.0K 2017-04-27 22:55 dirname
drwxr-xr-x 17 root 4.0K 2016-05-04 18:39 dirname
drwx------  2 root  16K 2015-05-15 05:34 lost+found
drwxr-xr-x 21 root 4.0K 2017-07-13 06:51 dirname
drwxr-xr-x  3 root 4.0K 2017-05-29 13:56 dirname
-rw-r--r--  1 root 3.3K 2015-05-17 06:08 filename.txt
drwxr-xr-x 10 root 4.0K 2016-08-10 03:23 dirname

I changed the names of directories and files just for the forum, they are all empty.
What the hell happened?
What should I do now? Can I run fsck to find some lost files? Or umount all?

tallboy
16gig-flash.jpg
 Description   
 Filesize   43.62 KB
 Viewed   291 Time(s)

16gig-flash.jpg

_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 8702
Location: Perth, Western Australia
PostPosted: Mon 31 Jul 2017, 07:09    Post subject:  
tallboy wrote:
Uh, uh!
I must have made a mistake, I need some help here!
The last test, from post above:
Code:
# dd if=custom-puppy.iso of=/dev/sdb1
470984+0 records in
470984+0 records out
241143808 bytes (241 MB) copied, 87.0704 s, 2.8 MB/s


That will have destroyed the content of partition sdb1.
You will have to reformat it with ext4 or whatever.
_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
tallboy


Joined: 21 Sep 2010
Posts: 1026
Location: Oslo, Norway
PostPosted: Mon 31 Jul 2017, 07:21    Post subject:  
Thank you, Barry. I feared it was somthing like that. The stick was only one of two backup sticks, so there's no catastrophy, but anyway...
Do you now why the names of the emptydirectories still are intact? Why wasn't it wiped clean?

tallboy
_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
tallboy


Joined: 21 Sep 2010
Posts: 1026
Location: Oslo, Norway
PostPosted: Mon 31 Jul 2017, 07:32    Post subject:  
Ha, ha, it's not wiped clean!
But the directory names just hanged there in the previous terminal window., they are gone now.
I unmounted all partitions, and remounted it, and I now have 230Mb of Puppy on my SanDisk!
But, when I hower the mouse over the flash icon, it says filesystem iso9660, size 29.1 Gb!
When I ask with df:
Code:
# df -h /mnt/sdb1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             230M  230M     0 100% /mnt/sdb1


Darn, I lost almost 16 Tb! Laughing

tallboy
_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
rcrsn51


Joined: 05 Sep 2006
Posts: 12324
Location: Stratford, Ontario
PostPosted: Mon 31 Jul 2017, 07:52    Post subject:  
Something to consider: For each person who uses dd incorrectly, has a disaster and reports it, there will be ten people who don't report it.
Last edited by rcrsn51 on Mon 31 Jul 2017, 08:35; edited 1 time in total
Back to top
View user's profile Send private message 
tallboy


Joined: 21 Sep 2010
Posts: 1026
Location: Oslo, Norway
PostPosted: Mon 31 Jul 2017, 08:00    Post subject:  
Haha, I guess you are right about that, rcrsn51. I have done some incredibly stupid things with Linux over the years, and almost all could have been avoided if I just had read the manual first! Very Happy

I'm running off a live-CD, so I guess I'll restart the Puppy, to be on the safe side. I'll wipe the stick clean now, but I think it's better to format it from a fresh Puppy!
It's one of the advantages of a live Puppy, so you see I have learnt something from all my mistakes! Laughing

tallboy
_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
linuxcbon

Joined: 09 Aug 2007
Posts: 1197
PostPosted: Mon 31 Jul 2017, 14:56    Post subject:  
BarryK wrote:
You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".

Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.

Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"

And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache.


I dont know where you get all that from, and if this is important or mandatory for the testing, I will look at the dd manual. Smile
But one thing is sure : your method is not reliable as it depends on the filesystem.
Back to top
View user's profile Send private message 
linuxcbon

Joined: 09 Aug 2007
Posts: 1197
PostPosted: Mon 31 Jul 2017, 14:57    Post subject:  
tallboy wrote:
Haha, I guess you are right about that, rcrsn51. I have done some incredibly stupid things with Linux over the years, and almost all could have been avoided if I just had read the manual first! Very Happy

I'm running off a live-CD, so I guess I'll restart the Puppy, to be on the safe side. I'll wipe the stick clean now, but I think it's better to format it from a fresh Puppy!
It's one of the advantages of a live Puppy, so you see I have learnt something from all my mistakes! Laughing

tallboy


You didnt read the command correctly, it is sdb and not sdb1.

I have updated my message with this :
NOTE : DONT USE dd ON A USB STICK WITH IMPORTANT DATA !!!
ALWAYS BACKUP YOUR DATA !!!! Surprised
Back to top
View user's profile Send private message 
linuxcbon

Joined: 09 Aug 2007
Posts: 1197
PostPosted: Mon 31 Jul 2017, 17:40    Post subject:  
BarryK wrote:
You will at least need to add "conv=fdatasync", or if using busybox dd that doesn't work, have to use "conv=fsync".

Without it, dd will finish before completely flushing data to the drive, and that could be a lot of data.

Also recommend that you specify a large block size, as the flash technology prefers it, say "bs=1M"

And while you are at it, append "oflag=direct", which, I am not sure, but think that it may bypass the flash drive cache.


Ok I have read some dd manual and other pages :

- ‘fdatasync’ "Synchronize output data just before finishing. This forces a physical write of output data. "
---> it is the same as a "sync" after the dd command.

- ‘bs=bytes’ "Set both input and output block sizes to bytes."
‘count=n’ "Copy n ‘ibs’-byte blocks from the input file,"
---> It is mostly used for writing many small files , as in bs=1M count=1024 , which writes 1024 files of 1Mb.
You dont need it if you write one big ISO file.

- "oflag=direct" "Use direct I/O for data, avoiding the buffer cache. Note that the kernel may impose restrictions on read or write buffer sizes. For example, with an ext4 destination file system and a Linux-based kernel, using ‘oflag=direct’ will cause writes to fail with EINVAL if the output buffer size is not a multiple of 512. "
---> So it is not recommended or needed.

A plain and simple "dd if=puppy.iso of=/dev/sdb ; sync" is enough.

I noticed that many people prefer to test the write speed like this, on a mounted filesystem, so it depends on ext4, ext2, fat etc. :
# dd if=/dev/zero of=/mnt/sdb1/testfile bs=1M count=1024
# sync
# rm /mnt/sdb1/testfile
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 8702
Location: Perth, Western Australia
PostPosted: Mon 31 Jul 2017, 22:44    Post subject:  
linuxcbon wrote:

A plain and simple "dd if=puppy.iso of=/dev/sdb ; sync" is enough.


NO, it is not, dd will complete before flushing all data to the drive.

Also, writing to the filesystem on the drive is precisely what I want to test. Not raw writing to the drive.

scsijon,
Yes, "oflag=direct" does make a difference, I tested on one flash stick, and got a slower transfer rate with it. So, it did appear to be doing what the docs state, bypassing the cache in the drive.
I did that test before my original post to this thread and blog.
_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 5 of 6 [82 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Hardware
Jump to:  


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.0796s ][ Queries: 14 (0.0144s) ][ GZIP on ]