Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 15 Dec 2017, 18:04
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
The UDF format takes little overhead space on flash drive.
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [15 Posts]  
Author Message
musher0


Joined: 04 Jan 2009
Posts: 11277
Location: Gatineau (Qc), Canada

PostPosted: Wed 01 Nov 2017, 23:04    Post subject:  The UDF format takes little overhead space on flash drive.
Subject description: A comparison table of overheads for the UDF, ext2, ext3, ext4, f2fs and vfat formats
 

Hello all.

As the title says.

Those interested in formatting their USB thumb-drives using the UDF format
may also be interested in reading this Superuser forum thread:
https://superuser.com/questions/39942/using-udf-on-a-usb-flash-drive

Please find attached a comparison table. I took the trouble of formatting the
same USB thumb-drive in UDF, ext? and f2fs. The results are attached.

~~~~~~~~~~~~~~~~~~~~~~

-- NOTES --

The one-liner used was:
Code:
df -h -T | head -1;df -h -T | grep sdd

The UDF format was achieved using the format-udf.sh script mentioned in
the above-mentioned thread.

I had to compile the required xxd utility since we do not have it on Puppy.
(Also attached.)

The mkudffs utility is Ubuntu's.

The Puppy used being xenial32-706 with stemsee's kernel 4.1.2

No formatting was involved to get the overhead data for the vfat format, since
off-the-shelf USB thumb-drives come formatted in vfat, as you may know.

For the other formats, gparted_shell was used.

~~~~~~~~~~~~~~~~~~~~~~

Comments welcome.

BFN
Comparaison-UDF.png
 Description   
 Filesize   150.92 KB
 Viewed   400 Time(s)

Comparaison-UDF.png

xxd-2012-01-19_exec.pet
Description 
pet

 Download 
Filename  xxd-2012-01-19_exec.pet 
Filesize  10.3 KB 
Downloaded  19 Time(s) 

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
ETP


Joined: 19 Oct 2010
Posts: 1031
Location: UK

PostPosted: Thu 02 Nov 2017, 04:09    Post subject: The UDF format takes little overhead space on flash drive.  

Hi musher0,

Thanks for doing these comparitive tests.

I gather that this thread is really also about the max file size that each each format can handle.
You may wish to add an extra column to the table to show that.
Performance is probably a matter for another thread.

The referenced thread at superuser.com is over 8 years old - shades of Pelo here Shocked Laughing

GParted now has the same level of support for UDF as it does for f2fs. That level is just about adequate.

https://gparted.org/features.php

udftools is available via PPM (ubuntu-xenial-universe)

_________________
Regards ETP



Kennels
Back to top
View user's profile Send private message 
belham2

Joined: 15 Aug 2016
Posts: 1362

PostPosted: Thu 02 Nov 2017, 05:26    Post subject: Re: The UDF format takes little overhead space on flash drive.
Subject description: A comparison table of overheads for the UDF, ext2, ext3, ext4, f2fs and vfat formats
 

musher0 wrote:
Hello all.

As the title says.

Those interested in formatting their USB thumb-drives using the UDF format
may also be interested in reading this Superuser forum thread:
https://superuser.com/questions/39942/using-udf-on-a-usb-flash-drive

Please find attached a comparison table. I took the trouble of formatting the
same USB thumb-drive in UDF, ext? and f2fs. The results are attached.

~~~~~~~~~~~~~~~~~~~~~~

-- NOTES --

The one-liner used was:
Code:
df -h -T | head -1;df -h -T | grep sdd

The UDF format was achieved using the format-udf.sh script mentioned in
the above-mentioned thread.

I had to compile the required xxd utility since we do not have it on Puppy.
(Also attached.)

The mkudffs utility is Ubuntu's.

The Puppy used being xenial32-706 with stemsee's kernel 4.1.2

No formatting was involved to get the overhead data for the vfat format, since
off-the-shelf USB thumb-drives come formatted in vfat, as you may know.

For the other formats, gparted_shell was used.

~~~~~~~~~~~~~~~~~~~~~~

Comments welcome.

BFN



Hi Musher,

This thread really interests me Wink

Reason is that I have a lot of old 2.0 USB thumbdrives that have 128MB capacity (which in reality usually means 120-121MB max). Either from my own stash, the extended family's stash, or people just give them to me because they no longer want them because they think they are useless. And, in the spirit of making them not 'useless', I 'frugal' install pups to them, all with a modern browser (Palemoon 27.5.1), updated Sylapheed, openssl updated, redshift, gnome player and a few other things (the puppy scripts we all use and stuff like mtpaint) updated. (Using recently created themes and icons helps create the impression of newness---like a new dress on a beautfiul woman Very Happy )

As a base for doing all this, I use mainly Thinslacko, which is Micko's great creation from a few years ago, based on Slacko 5.3.3. Or I use Unicornpup6.0, which is Phil's great creation from 2014.

When I'm finished, and I am usually right at the 120-121MB max of these drives, I then give them and/or show them to people (and keep a few for myself, of course) and I have never failed to get a look of amazement on their faces as they watch an OS--with a modern day browser, email, video player & puppy's wealth of small script programs---boot up on a 128MB old usb stick on their fat, piggy many, many gigabyte Windows 10 machines! (I really, really miss Micko & Phil and other developers keeping this tradition alive of releasing stripped, down base Puppies like ThinSlacko5.3.3 and Unicornpup6.0 that we can build from. Using woof-CE, or having the skills and/or knowledge, to make a 'stripped' down OS is a major, major pain in the you-know-what Crying or Very sad )

Anyhow, my question is this; over the past years I spent a lot of time re-formatting these 128MB sticks. I quickly learned that vfat provided the smallest footprint, giving me much needed extra space to do the things I need to do. So, to me your graph of results in almost amazing. What I mean is you used 16GB! A 16GB thumbdrive in your test above, can I ask how it is possible that when you formatted it, it is showing that "vfat" only used 8.0KB?!!! On my very, very small 128MB thumbdrives, vfat (fat16) takes up way more than that using Gparted from any various modern day puppy to format/partition it.

So, how is or was that possible? Do you have a special Gparted program? A special formatting process? Or what?? I need all the room on my thumbdrives I can get, and I am usually bumping right up next to the previously mentioned 120-121MB maximum that advertised old 128MB 2.0 thumbdrives gives me.

Thanks for any reply/hints/tips/etc.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2636

PostPosted: Thu 02 Nov 2017, 06:02    Post subject:  

I'm also a little skeptical about all of the ext* having the same usage figures. Real ext2 should be much smaller as it has no journal. This may have to do with later kernels using a single driver for all three variants. I'm not trying to criticize, just a healthy skepticism.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 11277
Location: Gatineau (Qc), Canada

PostPosted: Thu 02 Nov 2017, 11:11    Post subject:  

amigo wrote:
I'm also a little skeptical about all of the ext* having the same usage figures. Real ext2 should be much smaller as it has no journal. This may have to do with later kernels using a single driver for all three variants. I'm not trying to criticize, just a healthy skepticism.

Hello amigo.

Please remember that there is nothing on that drive except the format overhead : no files!!!
As a starting point, all ext? formats occupying 37 Mg on the disk? Sounds right to me,
they're the same family of formats. I didn't tamper with he figures.

I thought of doing the comparison again with a mostly full dirve. Maybe later. Then we'd
see the differences brought about by the journaled formats.

BFN.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 11277
Location: Gatineau (Qc), Canada

PostPosted: Thu 02 Nov 2017, 11:35    Post subject:  

@belham2

About the vfat format line in the above table, I didn't have to format the drive. As I
said, that is the default reading for an off-the-shelf flash drive you buy in any store.

I bought the 16 Gb NeXt stick at The Source, brought it home, plugged it in my
portable, mounted it, and did a reading with:
Code:
df -h -T | head -1;df -h -T | grep sdd
Period.

You're a Neanderthal, eh? Very Happy I'm a Cro-Magnon myself and I never saw a 128 Mb
Flash drive! Lowest capacity I ever saw in a Flash drive is 512 Mb, +/- 15 years ago.

As to producing trimmed down Puppies one could put on a 128 Mb flash drive in this
day and age, everybody is for virtue... But last Puppy I saw below 100 Mb's and
offering "an app per function" was o1micko's Slacko 5.3.3.

I think that nowadays, even a CLI-only Puppy would be larger than 100 Mb's. But if
you feel up to the challenge, go right ahead!

BFN.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 11277
Location: Gatineau (Qc), Canada

PostPosted: Thu 02 Nov 2017, 12:06    Post subject: Re: The UDF format takes little overhead space on flash drive.  

ETP wrote:
Hi musher0,

Thanks for doing these comparitive tests.
You're welcome!

ETP wrote:
I gather that this thread is really also about the max file size that each each format can handle.(...)
Nope. Strictly overhead.

The question of file size has been discussed in many forums. As a reminder,
only vfat is limited to a maximum of 2 Gb's per file. As to the higher limit,
it remains virtual, IMO: I don't think our Puppyists will ever have to deal with
files that big. If interested, they may want to do a search, starting here:
https://en.wikipedia.org/wiki/Comparison_of_file_systems
(the "Limits" section, +/- at the middle of the page)

ETP wrote:
GParted now has the same level of support for UDF as it does for f2fs. That level is just about adequate.

https://gparted.org/features.php
(... Un-funny part of your message left out! Wink ...)

udftools is available via PPM (ubuntu-xenial-universe)
Not on this xenialPup. UDF is not in the list of formats offered by GParted. (Please see attached capture.)
Perhaps our Woof-CE guys did not include it at compile time?

Besides, the format-udf.sh script mentioned in the OP is probably superior in this
case, because dedicated to formatting Flash drives.

As to obtaining the udftools through PPM, as I have been telling everyone on this
board for more than a year, without anyone paying attention:

My PPM works for a week and then
the downloading section fails,
on any of my Puppies.


I am probably not the only one having this problem, but people are too shy to say
it out loud. Else, why would there be at this time on this board two replacement
projects for the PPM, one by mistfire and one by sc0ttman?

Which is why I use pkgs.org a lot. And recommend it.

BFN.
No-UDF_in_this_list.jpg
 Description   
 Filesize   131.69 KB
 Viewed   283 Time(s)

No-UDF_in_this_list.jpg


_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
8Geee


Joined: 12 May 2008
Posts: 1300
Location: N.E. USA

PostPosted: Mon 20 Nov 2017, 17:10    Post subject:  

Probably the PPM needs frequent updates on the end-user side?
I suspect that given all of the extensions/apps used. My slackware stuff needs the occasional update, but I'm using a fairly simple pup with all of the shares/servers removed.

Regards
8Geee

_________________
Linux user #498913

Good God!, by the stars in the sky we are lost!
And into the breach we got tossed!
And the world is comin' on fast! --Florence Welch
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 11277
Location: Gatineau (Qc), Canada

PostPosted: Mon 20 Nov 2017, 19:12    Post subject:  

8Geee wrote:
Probably the PPM needs frequent updates on the end-user side?
I suspect that given all of the extensions/apps used. My slackware stuff needs the occasional update, but I'm using a fairly simple pup with all of the shares/servers removed.

Regards
8Geee

mistfire offered a lifebuoy in the form of his puppy-get:
http://murga-linux.com/puppy/viewtopic.php?p=970741&sort=lastpost#970741

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
Sailor Enceladus

Joined: 22 Feb 2016
Posts: 1327

PostPosted: Mon 20 Nov 2017, 20:29    Post subject:  

Interesting. When I made a 23.66GB ext4 partition at the end of my hard drive with GParted it said 500MB or 800MB was "used" right off the bat. Whatever the number was, it was quite higher than I expected (0... or similar Very Happy ). I guess putting it into perspective it was only 2-3% though. I'll have to try it again to see the exact number it said.
Back to top
View user's profile Send private message 
Peterm321

Joined: 29 Jan 2009
Posts: 386

PostPosted: Wed 22 Nov 2017, 16:00    Post subject:  

My understanding of VFAT is that has two FATS consisting of block mappings that have an entry for each cluster. Maximum clustersize = 128KB.

The thing about it is that it starts small but every file added to a VFAT system needs entries for all of its clusters. The FATs therefore expand as the disk fills.

Ext4 creates more space initially, but the size of the reserved sectors remains static as the disk fills. It might be possible to to create a smaller filesystem by setting a smaller inode count to mkfs.ext4. The default for a typical 16GB USB drive is many thousands of indodes. I use Ext4 on USB drives typically with 8192 inodes which is plenty for storing photos, media or zip file backups.

Ext4 also supports extents, which saves entries for large files but unlike some other extent supporting filesystems journalling can be disabled with Ext4, advisable in fact with USB drives.
Back to top
View user's profile Send private message 
Lassar

Joined: 08 Jul 2014
Posts: 174

PostPosted: Fri 24 Nov 2017, 18:03    Post subject: What do you think about the f2fs format?  

What do you think about the f2fs format?

I have so much trouble with file system corruption on flash drives.

Do you really think that UDF is better then f2fs.

I am looking for a file format, that you can boot from.

I have a flash drive would not boot. I had to partition it to two partitions, and boot from the second partition.

Just tried rufus, formated it to UDF on one partition.

It boots just fine.
Back to top
View user's profile Send private message 
step

Joined: 04 May 2012
Posts: 954

PostPosted: Sun 26 Nov 2017, 18:10    Post subject: Re: What do you think about the f2fs format?  

Lassar wrote:
What do you think about the f2fs format?

I have so much trouble with file system corruption on flash drives.

I am looking for a file format, that you can boot from.

I have used f2fs on bootable flash drives for a couple of years, and I was very happy with it. I never had an issue. Then I upgraded the kernel IIRC to a 4.9.x version (x64) and I started seeing entire directories and their contents mysteriously disappear after a reboot. I ended up skipping the kernel upgrades for a while. Now I don't use f2fs on flash drives anymore, not so much because of those issues, but because some devs recommend ext4 even for flash. I'm told recent kernels mount ext4 with options that "play well" for flash drive wear. I haven't verify this statement with targeted tests.

_________________
Fatdog64-710|+Packages|Kodi|Findnrun|+forum|gtkmenuplus
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2636

PostPosted: Mon 27 Nov 2017, 05:27    Post subject:  

UDF does no wear-leveling or other flash-friendly operations. For bootablility on the maximum number of machine types and ages, FAT is the best choice.
A reliable method for setting up a bootable flash, would be to have a FAT partition for boot only, using a separate partition with a better filesystem, ext4, f2fs or whatever.
The FAT could be pretty small, holding only the kernel and initrd needed to then access the main file system on the other partition. The FAT would not even need to be mounted by the running system or if so, the mounted read-only.
Back to top
View user's profile Send private message 
Peterm321

Joined: 29 Jan 2009
Posts: 386

PostPosted: Mon 27 Nov 2017, 14:38    Post subject:  

Wear levelling should be implemented in the firmware, I would be surprised if the firmware levelling would be switched off for Ext4.

I have also tried f2fs. My experience is that it chokes on charsets with ascii codes > 127 and this caused me problems though otherwise f2fs works well on the flash drives where I still use it. The charset issue I had may have resolved in other perhaps more recent kernels.

Recent f2ftools also have a decent checking utilty something that was lacking in earlier versions.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [15 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0684s ][ Queries: 15 (0.0060s) ][ GZIP on ]