How to stuff 7% more material into your sfs's.

[musher0]

Hello all.

This mksquashfs trick is not well know, but it should, IMO.

By comparison with the usual line,

Code: Select all

mksquashfs Directory/ Directory.sfs -noappend -all-root -comp xz

we compress about 7% more if we use this one:

Code: Select all

mksquashfs Directory/ Directory.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%

# Source:
https://techoverflow.net/blog/2013/01/1 ... rom-folder

I tried it on the various sfs's of the DpupStretch7. For ex., the main puppy
sfs went down from 190 Mb to 176 Mb by compressing with the new parms.

AFAICT, they take a bit longer to create, but are perhaps a bit snappier to
unpack to RAM when you load your Puppy.

BFN.

[nic007]

Is that the maximum compression? Takes longer to compress but does seem a bit quicker to uncompress. Suppose it may be wothwhile if you are only going to compress once. I got my Racy base sfs down to 109MB instead of 116MB (I think this was with another parameter I used before). Normal xz compression is 121MB, so about a ten percent difference. Not bad

[musher0]

Is that the maximum compression? (...)

I don't know! Worth investigating, certainly.
BFN.

[nic007]

Is that the maximum compression? (...)

I don't know! Worth investigating, certainly.
BFN.

It's actually not the best. Somehow I got the compression of my tahr base sfs down to 174MB (sometime ago, can't remember what I did). Your method makes it 179MB.

[drunkjedi]

Do xz compressed sfs work on all pups? Older pups?
Just asking....
I generally don't use any options with mksquashfs.
But my sfs are not distributed widely, mostly my personal use.

[musher0]

Do xz compressed sfs work on all pups? Older pups?
Just asking....
I generally don't use any options with mksquashfs.
But my sfs are not distributed widely, mostly my personal use.

Only with pups that have xz, I suppose.

[nic007]

All puppys with kernel 3 and upwards. Don't know if it's possible to install those modules in older puppys but it definitely does not work by default for older puppys.

[greengeek]

Thanks musher0 - won't get a chance to try this just yet but well worth knowing about. I use manual mksquashfs frequently. If there is a space saving to be had I am keen to learn it.

cheers!

[greengeek]

Well, i decided i just had to try this so here is some comparison information:

I normally use the following command to compress my personal sfs directory:

Code: Select all

mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs

and I compared the performance of the resulting sfs against the output of the following code using your new syntax:

Code: Select all

mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%

I have attached a pic comparing the size comparison of the two methods. Your method gives a size of 351MB where my normal method gives a size of 454MB.

That is a huge saving.

The question for me is how does that affect boot times? Is it better or worse? The comparison for me is that my normal method boots (from end of bootloader to functional desktop) in 1 minute exactly. Your version boots in 1 minute 21 seconds. Hmmm - I was hoping for a miracle but they are hard to find. At least there is some advantage for me in using your method - I can fit my special puppy on a smaller storage device.

However, maybe there is some variation of your method that might work well for me to reduce my boot times?

My results will not be typical - these tests are done with my non-standard derivative of Slacko 5.6 where I have reversed the normal puppy sfs and the zdrv sfs (which is my personal sfs).

Others may have different results with a standard pup.

I am quite excited by the prospect of shrinking both size and boot times. Maybe Slacko lacks the utilities to make the best of this method. I will have to do more experimentation.

[nic007]

Well, i decided i just had to try this so here is some comparison information:

I normally use the following command to compress my personal sfs directory:

Code: Select all

mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs

and I compared the performance of the resulting sfs against the output of the following code using your new syntax:

Code: Select all

mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%

I have attached a pic comparing the size comparison of the two methods. Your method gives a size of 351MB where my normal method gives a size of 454MB.

That is a huge saving.

The question for me is how does that affect boot times? Is it better or worse? The comparison for me is that my normal method boots (from end of bootloader to functional desktop) in 1 minute exactly. Your version boots in 1 minute 21 seconds. Hmmm - I was hoping for a miracle but they are hard to find. At least there is some advantage for me in using your method - I can fit my special puppy on a smaller storage device.

However, maybe there is some variation of your method that might work well for me to reduce my boot times?

My results will not be typical - these tests are done with my non-standard derivative of Slacko 5.6 where I have reversed the normal puppy sfs and the zdrv sfs (which is my personal sfs).

Others may have different results with a standard pup.

I am quite excited by the prospect of shrinking both size and boot times. Maybe Slacko lacks the utilities to make the best of this method. I will have to do more experimentation.

Your first compression method is gzip, second is xz, so completely different compression tools. This thread is about using the best xz compression setting. As for zip vs xz. Zip is faster (for compression not so much for uncompression) but results in a much bigger file as you have noticed. If you use the xz default setting, compression will be faster than the xz compression with the extra parameters resulting in a bigger file(10% at most) but will still result in a massive savings from gzip. compression.

[perdido]

Would the block size used (in the OP's command line it is 1MB) be affected by kernel limitations imposed by physical memory?

In other words, if I have a computer with 768MB of RAM is that going to limit my maximum block size request when using mksquashfs
to something under 768MB since the kmalloc maximum size is limited to physical memory?

I don't think kmalloc limitations imposed by physical memory can be increased using a swap file, or can it?

EDIT LOL - Just reread my post, please disregard gotta go drink some coffee!

.

[musher0]

Thanks for the methodical test, greengeek.

[8Geee]

This is rather interesting stuff.
My questions:

1.) What do those parameter deltas actually do?
if something is not root (is that by priviledge or location?)... what happens?


2.) Is block-size dependant upon file-size? I notice my 152Mb sfs shows a block-size of 9 bytes.

Regards
8Geee

[greengeek]

Your first compression method is gzip, second is xz, so completely different compression tools.

Thanks for the heads up. I haven't followed the increasing usage of xz so this is new to me.

This thread is about using the best xz compression setting.

Well maybe not only xz? The title potentially suggests any method for stuffing more data into a smaller sfs.

If you use the xz default setting, compression will be faster than the xz compression with the extra parameters resulting in a bigger file(10% at most) but will still result in a massive savings from gzip. compression.

Thanks - i will try to do some testing with the default xz syntax.

[musher0]

This is rather interesting stuff.
My questions:

1.) What do those parameter deltas actually do?
if something is not root (is that by priviledge or location?)... what happens?


2.) Is block-size dependant upon file-size? I notice my 152Mb sfs shows a block-size of 9 bytes.

Regards
8Geee

Hi, 8Geee.

Good to hear from you.

I do not have the answers to your questions. I must confess I am only a
parrot repeating some info, not a program analyst!

I am sure someone knowledgeable on this board can provide answers.

BFN.

[rufwoof]

a bit longer to create, but are perhaps a bit snappier to
unpack to RAM when you load your Puppy

LZ4 compresses less tightly (larger size), but decompresses much quicker than both gzip and xz. It's a trade off between disk IO speed and processing (decompression) and ram transfer speed. LZ4 when using multi-cores can throw out data at ram transfer speeds (extremely fast).

Reading more disk IO/decompress quickly can be quicker than less disk IO (tighter compression)/slower output (decompression).

Also bear in mind that once a program has been read in once during a session, most of that will remain memory bound (cached), such that subsequent re-runs of the same program/libs will already be in memory (cache). Especially in the case of puppy that might be say 500MB uncompressed in total, running on a PC that perhaps has 2GB or ram. Rather than loading the entire pup into ram at startup (slower bootup), just loading files as and when required will load less (gparted for instance might never even be used during a session). Slower to initially load a program (such as gparted if invoked during a session) than having it already available in memory, but no difference for subsequent re-runs of the program.

Generally not loading puppy into ram at startup, using lz4 compressed sfs's is the better overall choice IMO. Maximising xz compression will tend to see a slower system by comparison whilst IO is more predominant, but thereafter the two tend to compare anyway (once cached).

As a example I run Debian frugally with a save partition (rather than a save file/folder) and comparing having all of that inside a sfs to having all of the filesystem in the save partition (non compressed) sees very little difference between the two. For me, having all of the data in the save partition leaving a empty main sfs means that I can also boot that as though it was a full install. There is a noticeable difference between booting full versus frugal however as with frugal all changes are being recorded in memory, not back on disk (I personally like frugal as you can just reboot without saving to undo all changes made during the session i.e. I like to keep the main system pristine/factory-fresh - I only ever tend to boot as though full when I apply updates, and then immediately reboot back to frugal again afterwards).

[greengeek]

LZ4 compresses less tightly (larger size), but decompresses much quicker than both gzip and xz.

Any chance you could suggest a good default syntax to build an sfs from a directory using LZ4? (and do you have any idea if LZ4 is understood by Slacko?)

[nic007]

Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Results compressing Racy base sfs:
gzip = 142MB
-comp xz (default setting) = 121MB
-comp xz -b 1048576 -Xdict-size 100% = 109MB
-comp xz -b 1048576 -Xbcj x86 = 105MB

Compression slowish, decompression time not affected. Good if you are only going to compress once or seldomly.

[musher0]

Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Results compressing Racy base sfs:
gzip = 142MB
-comp xz (default setting) = 121MB
-comp xz -b 1048576 -Xdict-size 100% = 109MB
-comp xz -b 1048576 -Xbcj x86 = 105MB

Compression slowish, decompression time not affected. Good if you are only going to compress once or seldomly.

Hi nic007.

Your memory came back, I see!!!

"Oh, sh?t." That's a 35% space savings over gz. Wow.

Bravo nic007.

[greengeek]

Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Thanks for the parameters. Using this gave me the following comparison for my base sfs:

454MB using gz method

351MB using Mushero xz syntax

335MB using nic007 xz syntax

Unfortunately both of the xz methods create an issue for me when the system tries to run my firewall setup script in /root/Startup so I assume I must have a delay in unpacking something at a critical moment.

Not a big issue and probably something I can find a way around (I think it is due to the fact I use xdotool to generate a keyboard action during the firewall script. Bad practice really).

The other issue for me is that the xz methods both take 30 seconds longer than gz to boot my machine but then i am working with quite a large sfs so maybe not a problem for others.

The significant space saving does make the xz syntaxes useful for making best use of storage space.