lzop - most efficient compressor for big files

Filemanagers, partitioning tools, etc.
Post Reply
Message
Author
musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

lzop - most efficient compressor for big files

#1 Post by musher0 »

(@Flash: please move this thread if it should go elsewhere. Thanks.)
~~~~~~~~~
Hello,

the subject came up in another thread, but I think the Puppy incarnation
should go here.

Compiled from source, stripped with

Code: Select all

strip --strip-unneeded lzop
Pet file contains man1 and html docs at /usr/share/man and
/usr/share/doc respectively.
Executable in /bin replaces the busybox lzop provided in the Puppy.

The reason for this *.pet archive is that the busybox lzop destroys the
archived file automatically after the lzop archive is created.
The real lzop keeps the original file, and this is what we want when
backing-up pupsave files.

Recommendation: use with SFR's excellent PackIt / UExtract
archiving scripts.


Caveat: erase or rename the OOTB Puppy lzop link in /bin BEFORE
installing the attached *.pet archive.


Try to enjoy sadly. :shock: (Just playing with words...) :) BFN.

musher0
Attachments
lzop-1.03-i386.pet
Erase or rename the Puppy lzop link in /bin
BEFORE installing this *.pet archive.
(53.07 KiB) Downloaded 501 times
Last edited by musher0 on Thu 16 Apr 2015, 21:47, edited 1 time in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
SFR
Posts: 1800
Joined: Wed 26 Oct 2011, 21:52

#2 Post by SFR »

Hey Musher0

Just a tip: you can avoid the deletion while using busybox's lzop by enabling '-c' option (Write to stdout) and redirecting the output:

Code: Select all

lzop -c somefile > somefile.lzo
I have used the above method in (recommended by you, thanks! :) ) PackIt/UExtract, so input files are safe, whether you use busybox's or full lzop.

Greetings!
[color=red][size=75][O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource[/size][/color]
[b][color=green]Omnia mea mecum porto.[/color][/b]

gcmartin

#3 Post by gcmartin »

I kinda see what is shared by Musher0 and alluded to by SFR. When this was originally done, we had 486s, 5-686s, with 64/128 RAM. The size difference versus the benefits (including "most efficient archiver for big files", others mentioned and etc.) is worth a look for today's PUPs.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#4 Post by musher0 »

@SFR: Thanks for the information.

@all:
The capture below should make clear all the features that the "busybox
lzop" neglected to incorporate from the original lzop.

Don't get me going on busybox: in short, you can always trust busybox to
bring a genuine utility this close to being useless. (The notable exception
being ash.)

BFN.

musher0
Attachments
lzop-compare_2015-04-13.jpg
(74.29 KiB) Downloaded 535 times
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#5 Post by rufwoof »

If you Frugal boot puppy pfix=ram,nocopy then a level 1 lzop compressed puppy, something like (assuming your version of mksquashfs supports lzop)

mksquashfs <puppydir> puppy_tarh_6.0.5.sfs -noappend -comp lzo -Xcompression-level 1

compresses relatively quickly, but not as tightly as other alternative compressors.

However the compression is still reasonable (perhaps halves the file size) and decompresses very quickly. Generally the time it takes to data transfer around half the filesize and decompress that is comparable to data transfering the full non compressed file from disk. i.e. no difference (generally) in times to use either the lzop compressed or non compressed.

If you use the nocopy boot option then parts of puppy are read from disk as and when required, but being relatively small once in memory often remain in memory (paged in) for the entire session. Booting without the nocopy option transfers the entire puppy into ram, but that includes many files (content) that might not be used often. gparted for instance. Booting with nocopy results in only the files you (system) uses - and boots a lot quicker.

The nocopy option however does mean that the HDD needs to remain mounted during a session.

Of the possible choices of frugal booting, I prefer storing puppy sfs on hdd and the concept of using pfix=ram,nocopy booting. I don't use a savefile/folder except when I create one to preserve changes/updates, that I then copy into puppy sfs (and delete the savefolder) i.e. extract (unsquashfs) the puppy, copy -rf the savefolder content into that extracted puppy, and then reform the puppy (mksquashfs). That way I have control over when puppy changes, and at all other times its a read only system (with data/docs (and portable firefox/chrome) etc all stored outside of puppy space).

lzop IMO is the #1 best choice for such frugal booted pups.
Last edited by rufwoof on Sun 03 Jan 2016, 18:28, edited 1 time in total.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#6 Post by musher0 »

Thanks for sharing your experience notes, rufwoof.
Would it be the same booting from CD/DVD ?

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#7 Post by rufwoof »

I don't know (and I no longer have a working CD/DVD so I can't test it). I suspect that it might be quicker to read around half as much data off CD and decompress in ram (i.e. lzop'd) than read the full amount directly off CD. But it might be better in the case of a CD boot to simply copy the entire pup in one go (at bootup) assuming there's sufficient ram to do so. Otherwise its multiple CD search for file, data transfer ..etc per session (and the CD has to remain in its drive). Once paged in (in ram), re-reads might not occur on a PC with copious amounts of ram, but of course if a file that hasn't already been loaded is invoked then it would have to read that from the CD.

For a HDD based frugal with nocopy, booting is very quick (15 seconds on the 15 year old single core Celeron D that I'm currently using (timed from when initrd starts (after BIOS) to when the puppy gui desktop first shows). More so if you also comment out the Updating section in /etc/rc.d/rc.sysinit. And from my perpective subsequent running isn't noticeably slower than if the pup was fully transferred into ram at bootup.

I'd suggest perhaps using a full copy if CD frugal, nocopy for a HDD frugal. Giving grub/puppy helpful pointers as to locations of files also helps, such as including pmedia=idehd, psubdir, PDEV1. i.e. mine looks like :

title TAHR605-K4.3.2-NV
rootnoverify (hd0,2)
kernel /TAHR605-K4.3.2-NV/vmlinuz pfix=ram,nocopy pmedia=idehd PDEV1=sda3 psubdir=TAHR605-K4.3.2-NV
initrd /TAHR605-K4.3.2-NV/initrd.gz
boot
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#8 Post by rufwoof »

errrh!

Not so good. pfix=ram i.e. with copy works great. With pfix=ram,nocopy things start falling over. NTFS partition getting mounted as read only. After running a big mksquashfs having a desktop that when icons are clicked nothing happens, command prompt indicating gettext errors in xwin ... weird.

Easy fix. Revert to copying puppy sfs totally into ram at bootup and just tolerate the extra 15 second longer boot time (my pup is around 1.5GB non compressed, 700MB lzop -1 compressed - as it includes all firmware, modules, nvidia, Libre, Skype ...etc.).

I guess that's one of the issues with having such a flexible puppy system. Sometimes what works one way wont work another way. Fatdog (and Quirky) seek to address that by having a more focussed choice.

EDIT : DOH! (brain fart)

With the nocopy in effect I remastered to the live running version. When copied (just pfix=ram) that's not a issue because the new version is on disk and the live version is in ram. With pfix=ram,nocopy and remastering to a live sfs ... it (obviously) causes problems. !!!!
Last edited by rufwoof on Tue 05 Jan 2016, 11:27, edited 1 time in total.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#9 Post by musher0 »

Thanks for the additional info, rufwoof. It's interesting, but I meant the compression part
through mksquashfs. Sorry for having been unclear in my question before.

I'm currently building a pup and would like to try the lzop compression on the sfs to
gain speed at bootup, as you mention.

What version of mksquashfs are you using that allows lzop compression? I looked at
my version, and there are only the gzip and xz choices for compression.

Thanks in advance.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#10 Post by rufwoof »

musher0 wrote:Thanks for the additional info, rufwoof. It's interesting, but I meant the compression part
through mksquashfs. Sorry for having been unclear in my question before.

I'm currently building a pup and would like to try the lzop compression on the sfs to
gain speed at bootup, as you mention.

What version of mksquashfs are you using that allows lzop compression? I looked at
my version, and there are only the gzip and xz choices for compression.

Thanks in advance.
Its one I compiled myself. Specifically wanted lz4 compression support at the time, but used a source with both lz4 and lzop. I just named it mksquashfs5 and symlinked mksquashfs to that. Ditto for unsquashfs. Also compiled/grabbed stand alones of lz4 (can't recall from where) and lzop's (horses mouth : http://www.lzop.org/). IIRC the squashfs source was from here http://sourceforge.net/projects/squashf ... uashfs4.3/.

That's why I also went for emsee's 4.3.2 kernel as the standard Tahr kernel is limited (can only boot/sfs load gz or xz sfs's).

Size limits posting a single sfs, attached are /usr and /lib separately. I think that's all the files/libs required ... and maybe more (don't overwrite the libs you already have)
Attachments
lib.sfs.gz
fake .gz
(124 KiB) Downloaded 400 times
usr.sfs.gz
fake .gz
(204 KiB) Downloaded 350 times
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#11 Post by musher0 »

Hi.

Thanks for the files.

You wrote:
> That's why I also went for emsee's 4.3.2 kernel as the standard Tahr
kernel is limited (can only boot/sfs load gz or xz sfs's).


Same for udf, as I discovered.

Ahah. The cat is coming out of the bag, is she? :) That line sounds like a
bit much for my old head just now. Can you elaborate a bit?

It's beginning to sound like some great Linux contents editors in high
places are deciding what us regular mortal Linuxians can use and not use.

Thanks.

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#12 Post by musher0 »

BTW, my remake of pemasu's UpupPrecise-3.8.3.1 is described
here and here.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#13 Post by rufwoof »

musher0 wrote:You wrote:
> That's why I also went for emsee's 4.3.2 kernel as the standard Tahr
kernel is limited (can only boot/sfs load gz or xz sfs's).


Same for udf, as I discovered.

Ahah. The cat is coming out of the bag, is she? :) That line sounds like a
bit much for my old head just now. Can you elaborate a bit?
Don't fully understand it myself. In Tahr standard with those mksquashfs and unsquashfs you can make/squash and extract/unsquash using lzo or lz4, however Tahr isn't able to view the content/install (sfs load) those sfs's i.e. the kernel was compiled with a different configuration than the configuration used to compile the 4.3.2 kernel (as made by Stemsee) that can view/install those sfs's - that I'm using with Puppy Tahr 6.0.5.

All to do with how the DOTConfig (kernel compile) was configured when the kernel was compiled. You can see each pup's kernel compile configuration (DOTConfig) in /etc/modules/DOTconfig-xxxx In mine (emsee 4.3.2) there is for instance some lines containing

CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y

i.e. all set to y (yes) for those choices. Other DOTConfig's (pups) might have some of those as m (modules) or n (no) or even commented out. I don't understand such configuation other than a very casual awareness.

To fully use lzo or lz4 ... or whatever, you need mksquashfs and unsquashfs that supports those choices of compression AND a kernel that's compiled to support those choices of compressed squashed file systems (SFS's). mksquashfs creates a filesystem that's compressed - but the operating system (kernel) can only read/load that filesystem if it has been built to do so.

At least that's my interpretation - but that could be wrong!
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#14 Post by musher0 »

Excellent explanation, rufwoof. It's much clearer in my mind now. Many thanks.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#15 Post by rufwoof »

Having used frugal (to HDD) of Tahr 6.0.5 with emsee's 4.3.2 kernel and lz4 compressed initrd, puppy, zdrv and adrv sfs's (I store my 'savefolder' as a read only version in adrv), and booting pfix=ram,nocopy for a while now, its been running like lightning. Boots and runs exceptionally quick. Loading other sfs's created using lz4 compression and they load/run quick also.

Daft when a 15+ year old single core runs quicker than the latest Windows running on a modern multi-core. And having a read only based pup, configured as you like it, means that its more resilient.

The main factor for me is that whilst lzop -1 and lz4 high compression are somewhat comparable for filesize. lz4 has the edge on the decompression side, and its scalable - more cores - faster. Whilst lzop I believe is tied to a single core. Apparently multi-core lz4 can run at speeds approaching ram speeds, so if its 2:1 compressed data being decompressed that's somewhat like virtually doubling up the amount of ram.

Oddly, not reading all of puppy into ram at bootup (booting pfix=ram,nocopy) seems to generally run much quicker (entire session) than if puppy is read into ram at bootup (just pfix=ram alone). Perhaps only copying files into ram as/when needed means that only small parts of puppy are actually being loaded into ram as there's a lot in each puppy that more often doesn't actually get used each session (gparted for instance) - leaving more available for what actually is being run.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

Post Reply