Question about unsquashfs/mksquashfs

Using applications, configuring, problems
Post Reply
Message
Author
belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

Question about unsquashfs/mksquashfs

#1 Post by belham2 »

Hi all,

Have a question that has confused me for a long time, and I'm sure there is a simple answer (or something I'm not understanding) but I haven't thought of it or come across it yet.

The question is thus: any main puppy_save_####.sfs file.

Why is it, if you take any main puppy_save.sfs (from any pup ISO), and just do the following, you'll end with quite significant different size .sfs files:

1) Example: click on puppy-slacko_5.7 ISO to open it in Rox.

2) Note the exact MB size of the puppy_slacko_5.7.sfs that is made with compression "xz"

3) copy the puppy_slacko_5.7.sfs to an empty folder on any partition

3) open a terminal in that folder and run 'unsquashfs puppy_slacko_5.7.sfs'

4) see the "squashfs-root" folder it makes

5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/. 'new.fs' -all-root -noappend -comp xz"

6) then note the MB size of the this new.sfs


Should it NOT be the exact same size as when you started? I have run this test on at least 12-15 (and more, given the number of 'frugal' installs I've run over the years) and every single time, your 'new.sfs' will be significantly larger than the original .sfs you started with from the ISO. And I am talking anywhere from 10% to sometimes 20% larger.

Question is: Why? Why is not the exact same sized compressed xz .sfs file made??

Is there something in my string of terminal command appendages that I am running that I shouldn't, or that maybe I am not running/adding and I should?

Thanks for any help/feedback about not understanding this (please note, if it helps, it hasn't mattered if the hard drive/ssd/usb partition(s) have been formatted ext 2/3/4, the results are still the same).



[Edit---in writing the message, simply forgot the "." after the "squashfs-root-/" .........see Greengeek's comment below]
Last edited by belham2 on Thu 28 Sep 2017, 08:24, edited 1 time in total.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

Re: Question about unsquashfs/mksquashfs

#2 Post by greengeek »

belham2 wrote:5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
I can't offer any info re xz compression - but is it just a typo that the "." is missing from "squashfs-root/." or does that make some sort of difference?

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

Re: Question about unsquashfs/mksquashfs

#3 Post by belham2 »

greengeek wrote:
belham2 wrote:5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
I can't offer any info re xz compression - but is it just a typo that the "." is missing from "squashfs-root/." or does that make some sort of difference?

Hi Greengeek,

Fingers were a flying & typing, and the brain was not keeping up. Translation: just simply forgot it :wink:

User avatar
fredx181
Posts: 4448
Joined: Wed 11 Dec 2013, 12:37
Location: holland

#4 Post by fredx181 »

Hi belham2

To get information only about original sfs you can use the -s option

Code: Select all

unsquashfs -s puppy_slacko_5.7.sfs
And it may be something like this

Code: Select all

Compression xz
	Dictionary size 524288
	Filters selected: x86
Block size 524288
So, for example, to set the same block size for the new sfs

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86
Still maybe not the exact same size, could depend on other options too.

EDIT: Also added "-Xbcj x86" to above example, don't know if it makes any difference in size though.
For info: blocksize can be set also to 1M (double of 524288) , which gives more compression, smaller file (but booting and running will probably be a little slower then)

Fred
Last edited by fredx181 on Thu 28 Sep 2017, 09:21, edited 3 times in total.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

Re: Question about unsquashfs/mksquashfs

#5 Post by greengeek »

belham2 wrote:5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
And presumably that applies to the "new.fs" spelling as well? (If I use that syntax I do get a larger file created...)

Do you get the same results if you start with an xz compressed sfs that you created yourself? If not then maybe the pup developer created the original pup with different xz parameters (or more efficient xz utility) than is available to you?

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

#6 Post by belham2 »

@greengeek-------you're killing me here. :wink: Did you (or still do?) proof Queen Mum's releases after she did them, and busted her chops for any digressions?

@Fred----Thanks! Very useful info, I am trying it with a few pups to see if it makes it better and/or a difference.




P.S. My name is 'Belham2', and I hereby absolve myself of all spelling, punctuation, grammar, mis-typo errorssssssss----hahahha--- along with any idiocy I concoct, in this thread and/or messages. This shall also apply to all future threads/posts. :D

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#7 Post by nic007 »

xz has different compression settings, you need to use the exact same options to get the same compressed size. For example: when using the default xz compression setting, I can compress the Racy sfs to 120 MB. When using the maximum setting, I can compress it to 105MB. Pre-kernel 3 puppys do not support xz compression.

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

#8 Post by belham2 »

nic007 wrote:xz has different compression settings, you need to use the exact same options to get the same compressed size. For example: when using the default xz compression setting, I can compress the Racy sfs to 120 MB. When using the maximum setting, I can compress it to 105MB. Pre-kernel 3 puppys do not support xz compression.
Hi nic007!

Ok, that is totally new to me (as was Fred's post).

Can I ask: is there a way (in terminal or otherwise) I can find out what compression was used in a released puppy's ISO main .sfs file? I mean, when ya click on the ISO, you can see the puupy_#####.sfs directly, but is there anyway to know what was used "cz" -wise? Should I just assume it was the max? And if that is true, why is not my setting (as I entered it above in earlier message) not using the max?? What has to happen in the terminal commands for max to be used? Something like what Fred wrote?:

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86
would be less than

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b 1048576 -Xbcj x86
???

What the heck is "max" when using 'xz'? Can I just write:

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b maximum (or max) -Xbcj x86

What is confusing me is that I did like Fred suggested above in his post

Code: Select all

unsquashfs -s puppy_slacko_5.7.sfs
I found the exact that was used. Yet despite entering that exact same in terminal, and mksquashfs-ing, the puppy-slacko_5.7.sfs came out a "BIGGER' size than what was in the ISO---despite nothing being added to the .sfs. So...... :?.....I'm lost at what I'm doing wrong.

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

#9 Post by belham2 »

Ok, wanted to followup in case anyone else was curious about this thread, and wondered how to control "xz" compression used via the terminal.

This site explains it clearly;
https://www.rootusers.com/13-simple-xz-examples/

Basically, you are looking at using a number system (between '-0' and '-9'. The level of compression applied to a file using xz can be specified as a value between 0 (less compression) and 9 (best compression). Default running 'xz' in terminal with no flags uses the default of "6".

There is also an "extreme mode" that can be utilized. The -e or --extreme flag can be specified to use more CPU when encoding to increase the compression ratio, however this will take more time. And there is also a -T or --threads flag where you can specify the number of worker cpu core threads to use, potentially increasing xz performance.


Thanks again t all who replied here. I am successfully now doing unsqaushfs and mksquashfs that is governed by the media (size) I am installing too, and also whether I want to consider that media's speed perfromance based on the 'xz' compression used.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#10 Post by rufwoof »

In a terminal run mksquashfs and it will show the compression choices it supports. Those might not match what is supported by the kernel i.e. if you make a puppy.sfs using a compression method that the kernel doesn't support then it wont boot.

In the absence of both mksquashfs and kernel support for lz4 I usually go with lzo level 1 compression myself for throughput (speed). Larger filesize but quick to decompress (and compress). For tightest compression (smallest file) a high compression xz tends to do best, but throughput is slower.

mksquashfs somedir somefile.sfs -comp lzo -Xcompression-level 1

User avatar
fredx181
Posts: 4448
Joined: Wed 11 Dec 2013, 12:37
Location: holland

#11 Post by fredx181 »

belham2 wrote:What is confusing me is that I did like Fred suggested above in his post
Code:
unsquashfs -s puppy_slacko_5.7.sfs


I found the exact that was used. Yet despite entering that exact same in terminal, and mksquashfs-ing, the puppy-slacko_5.7.sfs came out a "BIGGER' size than what was in the ISO---despite nothing being added to the .sfs. So...... Confused.....I'm lost at what I'm doing wrong.
I was curious about that also and found that using: -Xbcj x86 , makes smaller size, so got the same size as original (having blocksize 524288) using:

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86
But it could be that using: -Xdict-size is also important (however resulting in the same size as above command for me)

Code: Select all

mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xdict-size 524288 -Xbcj x86
EDIT: About "maximum", AFAIK maximum for xz is setting blocksize: -b 1048576 (or 1M)

Fred

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#12 Post by nic007 »


Jose A. Senna
Posts: 43
Joined: Tue 20 Dec 2016, 23:16

#13 Post by Jose A. Senna »

Is there an option to make unsquashfs display the size the file(s) shall have after
decompression? I could not find it with unsquashfs --help

mostly_lurking
Posts: 328
Joined: Wed 25 Jun 2014, 20:31

#14 Post by mostly_lurking »

Jose A. Senna wrote:Is there an option to make unsquashfs display the size the file(s) shall have after decompression? I could not find it with unsquashfs --help
I haven't found any either, but you can mount the SFS file (by clicking on it in ROX Filer; click again later to unmount) and then check the size of the directory where it is mounted.

For example, I've mounted my devx.sfs. The mount point is /mnt/+root+wary_devx_511.sfs. When I right-click that directory and use ROX's "Count" option, it tells me that the disk usage is 435MB, while choosing "Properties" tells me the size of the compressed file, 106MB. You can also run that folder through a graphical disk mapping tool like GdMap, or use a command like du in a terminal:

Code: Select all

# du -sbh /mnt/+root+wary_devx_511.sfs
435M	/mnt/+root+wary_devx_511.sfs
(Reference: How do I get the size of a directory on the command line?)

This will tell you the SFS content's actual size, which might be different from the space that it takes up on disk when it's uncompressed. When I actually unsquash that SFS, "Count" still says the resulting folder is 435MB big, but "Properties", which seems to show the size on disk, is now up to 493MB. (Difference between actual file size and size on disk.)

So it seems that the above file size checks can give you a basic idea of how big the SFS's contents may be, but you won't know exactly how much disk space they will occupy until you unsquash them.

Jose A. Senna
Posts: 43
Joined: Tue 20 Dec 2016, 23:16

#15 Post by Jose A. Senna »

The ROX 'mount' then 'count' method worked.
Not ideal, but much better than nothing.
The 'du' command in terminal (with exactly the
same options as in your example) showed only
the squashed file size.
Thank you.

Post Reply