PPM_links-15.4 - hacked PPM files supporting petx

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

PPM_links-15.4 - hacked PPM files supporting petx

#1 Post by gyro »

The current version is at http://www.murga-linux.com/puppy/viewto ... 75#1038775

The attached file 'ppm_links-15.tar' contains files that patch the Puppy Package Manager so that it behaves as outlined below.
It contains 5 files:

'./usr/local/petget/installpkg.sh'

1. Respects all symbolic links to directories in the local file system.
So on my xenialpup '/usr/lib/i386-linux-gnu/libassuan.so.0.7.2' is written as '/usr/lib/libassuan.so.0.7.2',
'/tmp/x.txt' is written as '/initrd/mnt/zram0/tmp/x.txt'.

2. Ignores 'PUPMODE' and writes all files to the '/' file system. Same for "frugal install" and "full install".
So doesn't muck around with whiteout files or stack layers.

3. Extracts all source formats to a directory in /tmp.
Produces a list of the contents of this directory as the '.files' file.
Processes each entry in the '.files' file.

'./usr/local/petget/removepreview.sh'

1. Processes each entry in the '.files' file, in reverse order, so that files get removed before the containing directory.
If a processed directory is empty in the local file system, it is removed.

2. Removes files from the '/' file system, respecting any symbolic links to directories in the local file system.

3. After removing a file/directory it checks if this actiion produced a whiteout file in the RW directory, directly.
If it's an "aufs" stack, any whiteout file is removed from the RW directory, directly.
If it's an "overlayfs" stack, the full real path to the whiteout file is written to an "exclude file" so that the file will be excluded from the RW directory during the next reboot.
If it's a "full install", no whiteout files should be detected.

'./usr/local/mi-utils/dir2list'
A CLI utility to echo a list of directories and files contained in a directory.
Run without parameters to see usage help.
The included 'installpkg.sh' redirects the output of this utility to the '.files' file.

'./usr/local/mi-utils/save-add'
A CLI utility to write specified files from a specified directory to the '/' file system.
Run without parameters to see usage help.
The included 'installpkg.sh' calls this utility to do the actual file writing and directory creation.

'./usr/local/mi-utils/save-remove'
A CLI utility to remove specified files from the '/' file system.
Run without parameters to see usage help.
The included 'removepreview.sh' calls this utility to do the actual removing.

To use:
Either, extract somewhere and then copy the 5 files to their appropriate locations.
or

Code: Select all

tar xf ppm_links-15.tar -C /
Note1:
While the second method is simpler, I don't recommend doing this on your production Puppy.

Note 2:
I have not provided a '.pet' file, since this is clobbering the actual file executed to install a '.pet'.

gyro
Attachments
ppm_links-15.tar
Extract contents to use.
(12.62 KiB) Downloaded 250 times
Last edited by gyro on Sun 06 Oct 2019, 12:05, edited 2 times in total.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

PPM_links-15.1.tar - hacked versions of some PPM files

#2 Post by gyro »

Fixes an issue when used with an "overlayfs" stack.

Changes:
The attached file 'ppm_links-15-1.tar' contains an extra file, './usr/local/mi-utils/save-exclude'.
The file './usr/local/petget/removepreview.sh' is modified.
The file './usr/local/mi-utils/save-remove' is modified.

'./usr/local/mi-utils/save-exclude'
A CLI utility to exclude specified files from the '/' file system.
Run without parameters to see usage help.
The included 'removepreview.sh' calls this utility to get rid of files in an "overlayfs" stack.

The updated './usr/local/petget/removepreview.sh',
now calls './usr/local/mi-utils/save-exclude' for an "overlayfs" stack,
but './usr/local/mi-utils/save-remove', otherwise.

'./usr/local/mi-utils/save-remove'
Now removes files from only an "aufs" stack, (or "full install").

The acid test is to install a '.pet' containing '/etc/rc.d/rc.shutdown', no problems anywhere.
But uninstalling this pet with 'ppm_links.15' in an "overlayfs" stack is a problem,
at shutdown the whiteout file for '/etc/rc.d/rc.shutdown' is still in place, no 'rc.shutdown', big problem.

This is all because directly modifying any layer in an "overlayfs" stack is forbidden.
While 'save-remove' can directly remove the whiteout files from the RW layer of an "aufs" stack.

To use:
Either, extract somewhere and then copy the 6 files to their appropriate locations.
or

Code: Select all

tar xf ppm_links-15-1.tar -C /
Note1:
While the second method is simpler, I don't recommend doing this on your production Puppy.

Note 2:
I have not provided a '.pet' file, since this is clobbering the actual file executed to install a '.pet'.

gyro
Attachments
ppm_links-15.1.tar
Extract to use.
(12.77 KiB) Downloaded 254 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

PPM_links-15.4 - hacked PPM files supporting petx

#3 Post by gyro »

This release is all about 'petx', although it still performs as per previous versions.

ppm_petx-15.4.tar:
contains hacked Puppy utility files to support 'petx'.
ppm_petx-15.4.sfs:
contains the 'petx' CLI utility, and sould be loaded as an extra-sfs.

What is 'petx'?
Here is the 'petx' usage message:
--------------------------------
petx v1
Support utility for an extended 'pet' package format,
Stored in a '.pet.sfs' file.

Filename:
<package name>.pet.sfs
'./downloads/pmcputemp-0.63a.pet.sfs'

Internal format:
'data' - a directory containg the actual files to be installed.
'meta' - a directory containg 'pet.specs', 'pinstall.sh', 'md5sum.txt', ...

Usage:
petx -h
or
petx pet2 <filename of '.pet' file>
petx 2pet <filename of '.pet.sfs' file>
petx dir2 <package name directory> - with 'petx' internal format
petx 2dir <filename of '.pet.sfs' file>
petx chk <filename of '.pet.sfs' file>
petx ls <filename of '.pet.sfs' file>
--------------------------------

A 'petx' file can be installed as either a '.pet' or a '.sfs'

An easy way to start is to load ppm_petx-15.4.sfs with 'sfs_load',
and then, in a console, convert a currrent '.pet' to 'petx' with a command like:

Code: Select all

petx pet2 /path/to/top_prog-0.3.pet
then click on 'top_prog-0.3.pet.sfs' and select 'View contents', to look inside the '.sfs' and observe the contents.

The 'petx' format includes an internal 'meta/md5sum.txt' which can be checked with command like:

Code: Select all

 petx chk ./top_prog-0.3.pet.sfs
To try using a 'petx' file, you can install the contents of ppm_petx-15.4.tar with:

Code: Select all

tar xf ppm_links-15-1.tar -C /
or extract it somewhere and copy the files into place.
Now click on the 'top_prog-0.3.pet.sfs' file, the hacked 'filemnt' should show a dialog where you can, view the contents, install as a '.sfs. ar install as '.pet'.
With the included hacked PPM files and hacked 'sfs_load', all these options should work.

Note1:
The 'petx dir2' facility does not include code to generate a new 'pet.specs' file, like it should.
The only way to produce a 'petx' file using the current version of the 'petx' utility is to
generate a normal '.pet' file using 'dir2pet' and then convert it using 'petx pet2'

Note2:
The 'petx ls' facility lists the files contained in the internal 'data/' directory, not all the files in the 'petx' file.

Note3:
The 'petx 2dir' facility is really just an 'unsquashfs' but to a directory with the name of the package.

Note4:
The 'petx dir2' produces a new 'meta/md5sum.txt', in case any files in the package directory were changed.
So it is preferred over 'mksquashfs' for this functionality.

gyro
Attachments
ppm_petx-15.4.tar
simply extract
(36.45 KiB) Downloaded 154 times
ppm_petx-15.4.sfs.gz
gunzip to produce '.sfs' file
(2.36 KiB) Downloaded 177 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#4 Post by gyro »

Using 'petx' file as though it were a normal '.sfs' file:
After mounting the file at '/mount/point/dir',
add '/mount/point/dir/data' to the stack,
instead of adding '/mount/point/dir' to the stack.

gyro

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#5 Post by sc0ttman »

I'd be interested in making the petx format work by default in Pkg...

Pkg can already install SFS files as though they were normal .pet packages, so internal petx support would probably be a good idea..
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#6 Post by s243a »

So one could use this if they wanted to test a package but didn't necessarily want to save it to the savefile or savefolder?

Also does it overwrite any files in /usr/local/petget, or are these all alternative files?
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#7 Post by s243a »

s243a wrote:Also does it overwrite any files in /usr/local/petget, or are these all alternative files?
Related to this. Aside from the memory advantages of directly writing to the savefile/savefolder, the ability to write directly to the savefile/savefolder allows one to make changes to packages that need to be changed all at once (e.g. glibc) and would break the system if they were changed incrementally by writing each file separately to "/".

For these reason, I think it is good if the petx files are separate files from the existing package manager files or if not separate then it would be good if there were some configuration options that one could change to get the old behaviour.

BTW does writing directly to "/" fix some bug in the existing package manager?
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#8 Post by gyro »

sc0ttman wrote:I'd be interested in making the petx format work by default in Pkg...

Pkg can already install SFS files as though they were normal .pet packages, so internal petx support would probably be a good idea..
If you look inside the '/usr/local/petget/installpkg.sh' file from 'ppm_petx-15.4.tar' you will see how easy it is to use the 'petx' format internally,
('.pet' seems to be the worst).

Of course, once 'petx' is the internal format, a 'petx' file just needs to be mounted.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#9 Post by gyro »

s243a wrote:So one could use this if they wanted to test a package but didn't necessarily want to save it to the savefile or savefolder?

Also does it overwrite any files in /usr/local/petget, or are these all alternative files?
If you do a "tar tf" of 'ppm_petx-15.4.tar' you will see that it overwrites a number of currrent Puppy files, including a number in '/usr/local/petget'.

The concept is to be a replacement for the current.
So this makes no difference to the process of 'testing' a package.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#10 Post by gyro »

@s243a,

Why do I write directly to '/'?

1. It makes the code a whole lot simpler.
The process is the same for both a frugal-install, and a full-install.
The process is the same for a stack that supports direct changing of a lower layer (e.g. aufs) and a stack that does not support direct changing of any layer (e.g. overlayfs).

2. In the past PPM has had problems with symbolic links that point to directories.
When focusing on just a single layer, it is all too easy to forget that such symbolic links might exist in a lower layer.
By writing to '/', (i.e. the stack), any such links are always visible.

3. The only technical advantage I see for writing directly to a particular layer is when Puppy is running with an odd pupmode.
Then writing to the stack will write into RAM and then be copied into the save layer via 'snapmergepuppy' at some later time.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

extra-sfs files and 'meta' files

#11 Post by gyro »

An interesting "advantage" of using 'petx' files as 'extra-sfs' files is the capability of each 'extra-sfs' to have a'pinstall.sh', a 'puninstall.sh', and a 'pet.specs' stored in the 'meta' directory.
Because these files have to to accessed via the sfs's mount point, they never confusingly appear in the stack, and they never conflict with those of another sfs file.

An example:
Say we have a 'petx' file 'Dukto-6.0_1.pet.sfs'.
During the boot process it gets mounted at '/initrd/pup_sfs6',
and the directory '/initrd/pup_sfs6/data' gets appended to the stack.
Then the file '/initrd/pup_sfs6/data/usr/bin/dukto' will appear in the stack as '/usr/bin/dukto'.
But the file '/initrd/pup_sfs6/meta/pet.specs' will not appear in the stack, it will only be accessible as '/initrd/pup_sfs6/meta/pet.specs'.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#12 Post by gyro »

I have attached 'ppm_petx-15.5.tar' and 'ppm_petx-15.5.sfs'.

This release is purely to fix a bug in the '/usr/local/petget/installpkg.sh' script.

Symptoms in PPM:
If a package is installed that does not include a 'pet.specs', it appears to fail, even though the package is in fact installed and the '.files' files are present in the 'root/.packages' directory.
There is no entry for the package in '/root/.packages/user-installed-packages', only a blank line.
The '/usr/local/petget/installpkg.sh' script failed to put any information into the $DB_ENTRY variable.

Since this script resides in 'ppm_petx-15.5.tar', there is no need to download 'ppm_petx-15.5.sfs' if you already have 'ppm_petx-15.4.sfs'.

gyro
Attachments
ppm_petx-15.5.tar
simply extract
(36.45 KiB) Downloaded 132 times
ppm_petx-15.5.sfs.gz
gunzip to produce .sfs file
(2.36 KiB) Downloaded 133 times

Post Reply