Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 20 Oct 2017, 07:00
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Question about unsquashfs/mksquashfs
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [15 Posts]  
Author Message
belham2

Joined: 15 Aug 2016
Posts: 1156

PostPosted: Thu 28 Sep 2017, 04:03    Post subject:  Question about unsquashfs/mksquashfs  

Hi all,

Have a question that has confused me for a long time, and I'm sure there is a simple answer (or something I'm not understanding) but I haven't thought of it or come across it yet.

The question is thus: any main puppy_save_####.sfs file.

Why is it, if you take any main puppy_save.sfs (from any pup ISO), and just do the following, you'll end with quite significant different size .sfs files:

1) Example: click on puppy-slacko_5.7 ISO to open it in Rox.

2) Note the exact MB size of the puppy_slacko_5.7.sfs that is made with compression "xz"

3) copy the puppy_slacko_5.7.sfs to an empty folder on any partition

3) open a terminal in that folder and run 'unsquashfs puppy_slacko_5.7.sfs'

4) see the "squashfs-root" folder it makes

5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/. 'new.fs' -all-root -noappend -comp xz"

6) then note the MB size of the this new.sfs


Should it NOT be the exact same size as when you started? I have run this test on at least 12-15 (and more, given the number of 'frugal' installs I've run over the years) and every single time, your 'new.sfs' will be significantly larger than the original .sfs you started with from the ISO. And I am talking anywhere from 10% to sometimes 20% larger.

Question is: Why? Why is not the exact same sized compressed xz .sfs file made??

Is there something in my string of terminal command appendages that I am running that I shouldn't, or that maybe I am not running/adding and I should?

Thanks for any help/feedback about not understanding this (please note, if it helps, it hasn't mattered if the hard drive/ssd/usb partition(s) have been formatted ext 2/3/4, the results are still the same).



[Edit---in writing the message, simply forgot the "." after the "squashfs-root-/" .........see Greengeek's comment below]

Last edited by belham2 on Thu 28 Sep 2017, 04:24; edited 1 time in total
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4659
Location: Republic of Novo Zelande

PostPosted: Thu 28 Sep 2017, 04:19    Post subject: Re: Question about unsquashfs/mksquashfs  

belham2 wrote:
5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
I can't offer any info re xz compression - but is it just a typo that the "." is missing from "squashfs-root/." or does that make some sort of difference?
Back to top
View user's profile Send private message 
belham2

Joined: 15 Aug 2016
Posts: 1156

PostPosted: Thu 28 Sep 2017, 04:25    Post subject: Re: Question about unsquashfs/mksquashfs  

greengeek wrote:
belham2 wrote:
5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
I can't offer any info re xz compression - but is it just a typo that the "." is missing from "squashfs-root/." or does that make some sort of difference?



Hi Greengeek,

Fingers were a flying & typing, and the brain was not keeping up. Translation: just simply forgot it Wink
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 2517
Location: holland

PostPosted: Thu 28 Sep 2017, 04:52    Post subject:  

Hi belham2

To get information only about original sfs you can use the -s option
Code:
unsquashfs -s puppy_slacko_5.7.sfs

And it may be something like this
Code:
Compression xz
   Dictionary size 524288
   Filters selected: x86
Block size 524288

So, for example, to set the same block size for the new sfs
Code:
mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86


Still maybe not the exact same size, could depend on other options too.

EDIT: Also added "-Xbcj x86" to above example, don't know if it makes any difference in size though.
For info: blocksize can be set also to 1M (double of 524288) , which gives more compression, smaller file (but booting and running will probably be a little slower then)

Fred

Last edited by fredx181 on Thu 28 Sep 2017, 05:21; edited 3 times in total
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4659
Location: Republic of Novo Zelande

PostPosted: Thu 28 Sep 2017, 04:52    Post subject: Re: Question about unsquashfs/mksquashfs  

belham2 wrote:
5) turn right around, in same terminal, and run: 'mksquashfs squashfs-root/ 'new.fs' -all-root -noappend -comp xz"
And presumably that applies to the "new.fs" spelling as well? (If I use that syntax I do get a larger file created...)

Do you get the same results if you start with an xz compressed sfs that you created yourself? If not then maybe the pup developer created the original pup with different xz parameters (or more efficient xz utility) than is available to you?
Back to top
View user's profile Send private message 
belham2

Joined: 15 Aug 2016
Posts: 1156

PostPosted: Thu 28 Sep 2017, 05:31    Post subject:  

@greengeek-------you're killing me here. Wink Did you (or still do?) proof Queen Mum's releases after she did them, and busted her chops for any digressions?

@Fred----Thanks! Very useful info, I am trying it with a few pups to see if it makes it better and/or a difference.




P.S. My name is 'Belham2', and I hereby absolve myself of all spelling, punctuation, grammar, mis-typo errorssssssss----hahahha--- along with any idiocy I concoct, in this thread and/or messages. This shall also apply to all future threads/posts. Very Happy
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2181
Location: Cradle of Humankind

PostPosted: Thu 28 Sep 2017, 06:21    Post subject:  

xz has different compression settings, you need to use the exact same options to get the same compressed size. For example: when using the default xz compression setting, I can compress the Racy sfs to 120 MB. When using the maximum setting, I can compress it to 105MB. Pre-kernel 3 puppys do not support xz compression.
Back to top
View user's profile Send private message 
belham2

Joined: 15 Aug 2016
Posts: 1156

PostPosted: Thu 28 Sep 2017, 10:24    Post subject:  

nic007 wrote:
xz has different compression settings, you need to use the exact same options to get the same compressed size. For example: when using the default xz compression setting, I can compress the Racy sfs to 120 MB. When using the maximum setting, I can compress it to 105MB. Pre-kernel 3 puppys do not support xz compression.


Hi nic007!

Ok, that is totally new to me (as was Fred's post).

Can I ask: is there a way (in terminal or otherwise) I can find out what compression was used in a released puppy's ISO main .sfs file? I mean, when ya click on the ISO, you can see the puupy_#####.sfs directly, but is there anyway to know what was used "cz" -wise? Should I just assume it was the max? And if that is true, why is not my setting (as I entered it above in earlier message) not using the max?? What has to happen in the terminal commands for max to be used? Something like what Fred wrote?:

Code:

mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86


would be less than

Code:

mksquashfs squashfs-root new.sfs -comp xz -b 1048576 -Xbcj x86


???

What the heck is "max" when using 'xz'? Can I just write:

Code:

mksquashfs squashfs-root new.sfs -comp xz -b maximum (or max) -Xbcj x86



What is confusing me is that I did like Fred suggested above in his post
Code:
unsquashfs -s puppy_slacko_5.7.sfs


I found the exact that was used. Yet despite entering that exact same in terminal, and mksquashfs-ing, the puppy-slacko_5.7.sfs came out a "BIGGER' size than what was in the ISO---despite nothing being added to the .sfs. So...... Confused.....I'm lost at what I'm doing wrong.
Back to top
View user's profile Send private message 
belham2

Joined: 15 Aug 2016
Posts: 1156

PostPosted: Thu 28 Sep 2017, 12:29    Post subject:  

Ok, wanted to followup in case anyone else was curious about this thread, and wondered how to control "xz" compression used via the terminal.

This site explains it clearly;
https://www.rootusers.com/13-simple-xz-examples/

Basically, you are looking at using a number system (between '-0' and '-9'. The level of compression applied to a file using xz can be specified as a value between 0 (less compression) and 9 (best compression). Default running 'xz' in terminal with no flags uses the default of "6".

There is also an "extreme mode" that can be utilized. The -e or --extreme flag can be specified to use more CPU when encoding to increase the compression ratio, however this will take more time. And there is also a -T or --threads flag where you can specify the number of worker cpu core threads to use, potentially increasing xz performance.


Thanks again t all who replied here. I am successfully now doing unsqaushfs and mksquashfs that is governed by the media (size) I am installing too, and also whether I want to consider that media's speed perfromance based on the 'xz' compression used.
Back to top
View user's profile Send private message 
rufwoof

Joined: 24 Feb 2014
Posts: 2144

PostPosted: Thu 28 Sep 2017, 13:15    Post subject:  

In a terminal run mksquashfs and it will show the compression choices it supports. Those might not match what is supported by the kernel i.e. if you make a puppy.sfs using a compression method that the kernel doesn't support then it wont boot.

In the absence of both mksquashfs and kernel support for lz4 I usually go with lzo level 1 compression myself for throughput (speed). Larger filesize but quick to decompress (and compress). For tightest compression (smallest file) a high compression xz tends to do best, but throughput is slower.

mksquashfs somedir somefile.sfs -comp lzo -Xcompression-level 1
Back to top
View user's profile Send private message 
fredx181


Joined: 11 Dec 2013
Posts: 2517
Location: holland

PostPosted: Thu 28 Sep 2017, 13:44    Post subject:  

belham2 wrote:
What is confusing me is that I did like Fred suggested above in his post
Code:
unsquashfs -s puppy_slacko_5.7.sfs


I found the exact that was used. Yet despite entering that exact same in terminal, and mksquashfs-ing, the puppy-slacko_5.7.sfs came out a "BIGGER' size than what was in the ISO---despite nothing being added to the .sfs. So...... Confused.....I'm lost at what I'm doing wrong.


I was curious about that also and found that using: -Xbcj x86 , makes smaller size, so got the same size as original (having blocksize 524288) using:
Code:
mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xbcj x86

But it could be that using: -Xdict-size is also important (however resulting in the same size as above command for me)
Code:
mksquashfs squashfs-root new.sfs -comp xz -b 524288 -Xdict-size 524288 -Xbcj x86


EDIT: About "maximum", AFAIK maximum for xz is setting blocksize: -b 1048576 (or 1M)

Fred
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2181
Location: Cradle of Humankind

PostPosted: Thu 28 Sep 2017, 18:55    Post subject:  

Check this thread: http://www.murga-linux.com/puppy/viewtopic.php?t=110035
Back to top
View user's profile Send private message 
Jose A. Senna

Joined: 20 Dec 2016
Posts: 30

PostPosted: Tue 10 Oct 2017, 20:49    Post subject:
Subject description: another question about unsquashfs
 

Is there an option to make unsquashfs display the size the file(s) shall have after
decompression? I could not find it with unsquashfs --help
Back to top
View user's profile Send private message 
mostly_lurking

Joined: 25 Jun 2014
Posts: 218

PostPosted: Wed 11 Oct 2017, 12:46    Post subject:  

Jose A. Senna wrote:
Is there an option to make unsquashfs display the size the file(s) shall have after decompression? I could not find it with unsquashfs --help

I haven't found any either, but you can mount the SFS file (by clicking on it in ROX Filer; click again later to unmount) and then check the size of the directory where it is mounted.

For example, I've mounted my devx.sfs. The mount point is /mnt/+root+wary_devx_511.sfs. When I right-click that directory and use ROX's "Count" option, it tells me that the disk usage is 435MB, while choosing "Properties" tells me the size of the compressed file, 106MB. You can also run that folder through a graphical disk mapping tool like GdMap, or use a command like du in a terminal:
Code:
# du -sbh /mnt/+root+wary_devx_511.sfs
435M   /mnt/+root+wary_devx_511.sfs
(Reference: How do I get the size of a directory on the command line?)

This will tell you the SFS content's actual size, which might be different from the space that it takes up on disk when it's uncompressed. When I actually unsquash that SFS, "Count" still says the resulting folder is 435MB big, but "Properties", which seems to show the size on disk, is now up to 493MB. (Difference between actual file size and size on disk.)

So it seems that the above file size checks can give you a basic idea of how big the SFS's contents may be, but you won't know exactly how much disk space they will occupy until you unsquash them.
Back to top
View user's profile Send private message 
Jose A. Senna

Joined: 20 Dec 2016
Posts: 30

PostPosted: Thu 12 Oct 2017, 19:19    Post subject:
Subject description: Tried your solution
 

The ROX 'mount' then 'count' method worked.
Not ideal, but much better than nothing.
The 'du' command in terminal (with exactly the
same options as in your example) showed only
the squashed file size.
Thank you.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [15 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0655s ][ Queries: 14 (0.0041s) ][ GZIP on ]