The UDF format takes little overhead space on flash drive.

[musher0]

Hello all.

As the title says.

Those interested in formatting their USB thumb-drives using the UDF format
may also be interested in reading this Superuser forum thread:
https://superuser.com/questions/39942/u ... lash-drive

Please find attached a comparison table. I took the trouble of formatting the
same USB thumb-drive in UDF, ext? and f2fs. The results are attached.

~~~~~~~~~~~~~~~~~~~~~~

-- NOTES --

The one-liner used was:

Code: Select all

df -h -T | head -1;df -h -T | grep sdd

The UDF format was achieved using the format-udf.sh script mentioned in
the above-mentioned thread.

I had to compile the required xxd utility since we do not have it on Puppy.
(Also attached.)

The mkudffs utility is Ubuntu's.

The Puppy used being xenial32-706 with stemsee's kernel 4.1.2

No formatting was involved to get the overhead data for the vfat format, since
off-the-shelf USB thumb-drives come formatted in vfat, as you may know.

For the other formats, gparted_shell was used.

~~~~~~~~~~~~~~~~~~~~~~

Comments welcome.

BFN

[ETP]

Hi musher0,

Thanks for doing these comparitive tests.

I gather that this thread is really also about the max file size that each each format can handle.
You may wish to add an extra column to the table to show that.
Performance is probably a matter for another thread.

The referenced thread at superuser.com is over 8 years old - shades of Pelo here

GParted now has the same level of support for UDF as it does for f2fs. That level is just about adequate.

https://gparted.org/features.php

udftools is available via PPM (ubuntu-xenial-universe)

[belham2]

Hello all.

As the title says.

Those interested in formatting their USB thumb-drives using the UDF format
may also be interested in reading this Superuser forum thread:
https://superuser.com/questions/39942/u ... lash-drive

Please find attached a comparison table. I took the trouble of formatting the
same USB thumb-drive in UDF, ext? and f2fs. The results are attached.

~~~~~~~~~~~~~~~~~~~~~~

-- NOTES --

The one-liner used was:

Code: Select all

df -h -T | head -1;df -h -T | grep sdd

The UDF format was achieved using the format-udf.sh script mentioned in
the above-mentioned thread.

I had to compile the required xxd utility since we do not have it on Puppy.
(Also attached.)

The mkudffs utility is Ubuntu's.

The Puppy used being xenial32-706 with stemsee's kernel 4.1.2

No formatting was involved to get the overhead data for the vfat format, since
off-the-shelf USB thumb-drives come formatted in vfat, as you may know.

For the other formats, gparted_shell was used.

~~~~~~~~~~~~~~~~~~~~~~

Comments welcome.

BFN

Hi Musher,

This thread really interests me

Reason is that I have a lot of old 2.0 USB thumbdrives that have 128MB capacity (which in reality usually means 120-121MB max). Either from my own stash, the extended family's stash, or people just give them to me because they no longer want them because they think they are useless. And, in the spirit of making them not 'useless', I 'frugal' install pups to them, all with a modern browser (Palemoon 27.5.1), updated Sylapheed, openssl updated, redshift, gnome player and a few other things (the puppy scripts we all use and stuff like mtpaint) updated. (Using recently created themes and icons helps create the impression of newness---like a new dress on a beautfiul woman )

As a base for doing all this, I use mainly Thinslacko, which is Micko's great creation from a few years ago, based on Slacko 5.3.3. Or I use Unicornpup6.0, which is Phil's great creation from 2014.

When I'm finished, and I am usually right at the 120-121MB max of these drives, I then give them and/or show them to people (and keep a few for myself, of course) and I have never failed to get a look of amazement on their faces as they watch an OS--with a modern day browser, email, video player & puppy's wealth of small script programs---boot up on a 128MB old usb stick on their fat, piggy many, many gigabyte Windows 10 machines! (I really, really miss Micko & Phil and other developers keeping this tradition alive of releasing stripped, down base Puppies like ThinSlacko5.3.3 and Unicornpup6.0 that we can build from. Using woof-CE, or having the skills and/or knowledge, to make a 'stripped' down OS is a major, major pain in the you-know-what )

Anyhow, my question is this; over the past years I spent a lot of time re-formatting these 128MB sticks. I quickly learned that vfat provided the smallest footprint, giving me much needed extra space to do the things I need to do. So, to me your graph of results in almost amazing. What I mean is you used 16GB! A 16GB thumbdrive in your test above, can I ask how it is possible that when you formatted it, it is showing that "vfat" only used 8.0KB?!!! On my very, very small 128MB thumbdrives, vfat (fat16) takes up way more than that using Gparted from any various modern day puppy to format/partition it.

So, how is or was that possible? Do you have a special Gparted program? A special formatting process? Or what?? I need all the room on my thumbdrives I can get, and I am usually bumping right up next to the previously mentioned 120-121MB maximum that advertised old 128MB 2.0 thumbdrives gives me.

Thanks for any reply/hints/tips/etc.

[amigo]

I'm also a little skeptical about all of the ext* having the same usage figures. Real ext2 should be much smaller as it has no journal. This may have to do with later kernels using a single driver for all three variants. I'm not trying to criticize, just a healthy skepticism.

[musher0]

I'm also a little skeptical about all of the ext* having the same usage figures. Real ext2 should be much smaller as it has no journal. This may have to do with later kernels using a single driver for all three variants. I'm not trying to criticize, just a healthy skepticism.

Hello amigo.

Please remember that there is nothing on that drive except the format overhead : no files!!!
As a starting point, all ext? formats occupying 37 Mg on the disk? Sounds right to me,
they're the same family of formats. I didn't tamper with he figures.

I thought of doing the comparison again with a mostly full dirve. Maybe later. Then we'd
see the differences brought about by the journaled formats.

BFN.

[musher0]

@belham2

About the vfat format line in the above table, I didn't have to format the drive. As I
said, that is the default reading for an off-the-shelf flash drive you buy in any store.

I bought the 16 Gb NeXt stick at The Source, brought it home, plugged it in my
portable, mounted it, and did a reading with:

Code: Select all

df -h -T | head -1;df -h -T | grep sdd

Period.

You're a Neanderthal, eh? I'm a Cro-Magnon myself and I never saw a 128 Mb
Flash drive! Lowest capacity I ever saw in a Flash drive is 512 Mb, +/- 15 years ago.

As to producing trimmed down Puppies one could put on a 128 Mb flash drive in this
day and age, everybody is for virtue... But last Puppy I saw below 100 Mb's and
offering "an app per function" was o1micko's Slacko 5.3.3.

I think that nowadays, even a CLI-only Puppy would be larger than 100 Mb's. But if
you feel up to the challenge, go right ahead!

BFN.

[musher0]

Hi musher0,

Thanks for doing these comparitive tests.

You're welcome!

I gather that this thread is really also about the max file size that each each format can handle.(...)

Nope. Strictly overhead.

The question of file size has been discussed in many forums. As a reminder,
only vfat is limited to a maximum of 2 Gb's per file. As to the higher limit,
it remains virtual, IMO: I don't think our Puppyists will ever have to deal with
files that big. If interested, they may want to do a search, starting here:
https://en.wikipedia.org/wiki/Compariso ... le_systems
(the "Limits" section, +/- at the middle of the page)

GParted now has the same level of support for UDF as it does for f2fs. That level is just about adequate.

https://gparted.org/features.php
(... Un-funny part of your message left out! ...)

udftools is available via PPM (ubuntu-xenial-universe)

Not on this xenialPup. UDF is not in the list of formats offered by GParted. (Please see attached capture.)
Perhaps our Woof-CE guys did not include it at compile time?

Besides, the format-udf.sh script mentioned in the OP is probably superior in this
case, because dedicated to formatting Flash drives.

As to obtaining the udftools through PPM, as I have been telling everyone on this
board for more than a year, without anyone paying attention:

My PPM works for a week and then
the downloading section fails,
on any of my Puppies.

I am probably not the only one having this problem, but people are too shy to say
it out loud. Else, why would there be at this time on this board two replacement
projects for the PPM, one by mistfire and one by sc0ttman?

Which is why I use pkgs.org a lot. And recommend it.

BFN.

[8Geee]

Probably the PPM needs frequent updates on the end-user side?
I suspect that given all of the extensions/apps used. My slackware stuff needs the occasional update, but I'm using a fairly simple pup with all of the shares/servers removed.

Regards
8Geee

[musher0]

Probably the PPM needs frequent updates on the end-user side?
I suspect that given all of the extensions/apps used. My slackware stuff needs the occasional update, but I'm using a fairly simple pup with all of the shares/servers removed.

Regards
8Geee

mistfire offered a lifebuoy in the form of his puppy-get:
http://murga-linux.com/puppy/viewtopic. ... ost#970741

[Sailor Enceladus]

Interesting. When I made a 23.66GB ext4 partition at the end of my hard drive with GParted it said 500MB or 800MB was "used" right off the bat. Whatever the number was, it was quite higher than I expected (0... or similar ). I guess putting it into perspective it was only 2-3% though. I'll have to try it again to see the exact number it said.

[Peterm321]

My understanding of VFAT is that has two FATS consisting of block mappings that have an entry for each cluster. Maximum clustersize = 128KB.

The thing about it is that it starts small but every file added to a VFAT system needs entries for all of its clusters. The FATs therefore expand as the disk fills.

Ext4 creates more space initially, but the size of the reserved sectors remains static as the disk fills. It might be possible to to create a smaller filesystem by setting a smaller inode count to mkfs.ext4. The default for a typical 16GB USB drive is many thousands of indodes. I use Ext4 on USB drives typically with 8192 inodes which is plenty for storing photos, media or zip file backups.

Ext4 also supports extents, which saves entries for large files but unlike some other extent supporting filesystems journalling can be disabled with Ext4, advisable in fact with USB drives.

[Lassar]

What do you think about the f2fs format?

I have so much trouble with file system corruption on flash drives.

Do you really think that UDF is better then f2fs.

I am looking for a file format, that you can boot from.

I have a flash drive would not boot. I had to partition it to two partitions, and boot from the second partition.

Just tried rufus, formated it to UDF on one partition.

It boots just fine.

[step]

What do you think about the f2fs format?

I have so much trouble with file system corruption on flash drives.

I am looking for a file format, that you can boot from.

I have used f2fs on bootable flash drives for a couple of years, and I was very happy with it. I never had an issue. Then I upgraded the kernel IIRC to a 4.9.x version (x64) and I started seeing entire directories and their contents mysteriously disappear after a reboot. I ended up skipping the kernel upgrades for a while. Now I don't use f2fs on flash drives anymore, not so much because of those issues, but because some devs recommend ext4 even for flash. I'm told recent kernels mount ext4 with options that "play well" for flash drive wear. I haven't verify this statement with targeted tests.

[amigo]

UDF does no wear-leveling or other flash-friendly operations. For bootablility on the maximum number of machine types and ages, FAT is the best choice.
A reliable method for setting up a bootable flash, would be to have a FAT partition for boot only, using a separate partition with a better filesystem, ext4, f2fs or whatever.
The FAT could be pretty small, holding only the kernel and initrd needed to then access the main file system on the other partition. The FAT would not even need to be mounted by the running system or if so, the mounted read-only.

[Peterm321]

Wear levelling should be implemented in the firmware, I would be surprised if the firmware levelling would be switched off for Ext4.

I have also tried f2fs. My experience is that it chokes on charsets with ascii codes > 127 and this caused me problems though otherwise f2fs works well on the flash drives where I still use it. The charset issue I had may have resolved in other perhaps more recent kernels.

Recent f2ftools also have a decent checking utilty something that was lacking in earlier versions.