Possibilities based on working with SFS files in pup-431

This is where threads concerning the development of the next version of Puppy live.
Post Reply
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Possibilities based on working with SFS files in pup-431

#1 Post by gyro »

NOTE: all this stuff has been done on a frugal install of pup-431.

1) Myth - dynamic loading and unloading of SFS files dosen't work with aufs2 in pup-431 - busted!

True, the "load_sfs" script in common use at the time of the release of 431 did not work. But the problem lay with the script, not aufs2.
The "append" and "del" functions of aufs still work fine with the new stripped down version of aufs.
I have modified versions of "load_sfs" and "unload_sfs" that work in pup-431.


2) When an sfs file is removed, sometimes not all files disappear, and if you delete them, the old files don't reappear.

One of the useful things about using sfs files, is that when one is added, all the files just appear in the filesystem, no decompressimg or copying reuired. Then when the sfs file is removed, the files just disappear.
Unless, of course, a file in the sfs has been 'modified'. The new version of the file does not disappear because it is not in the sfs, but in the pup_rw layer.
An example, lets say you installed a "firefox-3.0.1.sfs" some time ago, and allowed it to automatically update to firefox 3.0.17. Now you remove the "firefox-3.0.1.sfs" and replace it with "firefox-3.5.7.sfs". The problem is that all the updates of firefox-3.0.1 to firefox-3.0.17 are still in the pup_rw layer, and 'cover' any firefox-3.5.7 files with the same filename. Hence you now have stuffed up firefox.
This is when aufs can seem to be a pain. It's clear what needs to be done, the offending files in the pup_rw layer, need to be deleted. But normal deletion won't work at this stage, since aufs will create ".wh." files so that the 'covered' files in the new sfs appear to be deleted.

An apparrently simple solution is to remove the new sfs, delete the offending files, and then add the new sfs again. This will most likely work, provided that none of the files have the same name as a file in any still active sfs files, in which case aufs will create a ".wh." file, and that file will disappear.
A reliable solution, is to boot into a different copy of puppy, i.e. one that uses a different pupsave file, mount the pupsave file containing the offending files as a loop device, and then delete the offending files. This works because the files in the pupsave are now mounted in a unique path within the filesystem.

I now have a couple of scripts that can safely remove offending files from the current pup_rw layer (e.g. pupsave.2fs). "rmfile-rw.sh" removes a single file or directory. "rmlist-rw.sh" removes all the files listed in the text file provides as a parameter. Both these have been developed on a PUPMODE=13 system, so they remove files from both pup_rw and pup_ro1 if necessary. Currently, the aufs cache gets fixed, but not the rox-filer cache, so it might be necessary to refresh an open rod-filer window.
In the firefox example above, simply running "rmfile-rw.sh /usr/local/firefox" would fix the problem on my machine, since I install firefox in "/usr/local/firefox/".


3) Modifying "init" so that pup-431.sfs gets loaded at the bottom of the aufs stack.

One of the problems with the current way of loading sfs files in Puppy, is that they can't overwrite any thing in pup-431.sfs. A typical example would be a "firefox.sfs" wanting to overwrite "/usr/local/bin/defaultbrowser".
One solution is to modify "init" to seperate pup-431.sfs from the pupsave stuff when defining the aufs stack. This means pup-431.sfs can be loaded last instead of straight underneath the pupsave stuff (pup_rw). This means that a "firefox.sfs" can now simply 'cover' "/usr/local/defaultbrowser".
Unfortunately this produced a number of nasty side-effects. The problem is that none of our current crop of sfs files have been only been tested while being loaded under pup-431.sfs, so when they are loaded above pup-431.sfs some unwanted things appear. A simple example is devx_431.sfs, this contains the normal "man" program, which usually gets 'covered' by the puppy specific "man" in pup-431.sfs, not a problem. But with devx_431.sfs above pup-431.sfs, the useful "man" in pup-431.sfs gets 'covered' by the useless "man" in devx_431.sfs.
So doing this could result in strange thins happening until all sfs files where revised. Perhaps a corageous course.

I have implemented a possible compromise, a modified pup-431 that introduces a new sfs file that I have called "zad_431.sfs" which gets loaded at the bottom of the aufs stack. The idea is that this sfs file contains all the stuff that currently resides in pup-431.sfs, that folk would want to overwrite. And the slightly smaller pup-431.sfs still gets loaded above other sfs files, as is the current situation.
This sfs file contains all the application decouplers, i.e. all the "/usr/local/default*" scripts + /"usr/bin/xterm". It currently also contains all the "/usr/share/applications/*.desktop" files.
So a "forefox.sfs" can 'cover' "/usr/local/defaultbrowser" but devx_431.sfs will not 'cover' "man".

A possible extension of this concept would be to divide each puppy release into two sfs files, a system sfs (pup-431.sfs) and an applications sfs (apps-431.sfs). Then folk could create alternate applications sfs's. Or simply load the system sfs high and the applications sfs low.

Another possibility which I haven't tried yet, would be to introduce another sfs file called pat-431.sfs.
This sfs file, if it exists, contains patches to pup-431.sfs, and is always loaded above pup-431.sfs.


4) Manipulating sfs files.

Confession: My current 'production' puppy 431 system, runs in a .pet free state.

I have a sctipt called "mixfs.sh" that can combine several sfs files, or directories containing 'filesystems' into a single sfs.
If I want to use a .pet I extract it to a directory and then add the name of that directory to the list of directories to be included in my single "my-apps-sfs4.sfs".
I use the same mechanism to directly patch pup-431.sfs and devx_431.sfs.

Why?
a) When running Puppy from the SD card of my EeePC, it was very very slow in saving pup_save.2fs to the SD card. So I began to consider how I might minimise the size of the pupsave file.
b) I realised that most of any software package is inherantly read only, so it is a waste of valuable read/write space to install it there, it would be more appropriately stored in an sfs file.
c) Since I discovered Puppy's propensity for defecating in it's own nest by not cleanly unmounting pupsave files in a frugal install; I simply don't trust pupsave to be reliable, and I want to be able to discard pupsave with a minimum of inconvenience.


I have attached a ".tar.gz" file containing the mentioned scripts, and a ".gz" file containing zad_431.sfs

gyro
Attachments
gyro-sfs-scripts.tar.gz
The scripts I use as mentioned above.
(2.76 KiB) Downloaded 1736 times
zad_431.sfs.gz
An example of suggested extra sfs for puppy releases.
(11.75 KiB) Downloaded 1620 times

tlchost
Posts: 2057
Joined: Sun 05 Aug 2007, 23:26
Location: Baltimore, Maryland USA
Contact:

#2 Post by tlchost »

Sounds interesting....do your scripts overcome the low number of SFS files that 4.3.1 is able to load at startup?

Thanks
Thom

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#3 Post by gyro »

tlchost wrote:Sounds interesting....do your scripts overcome the low number of SFS files that 4.3.1 is able to load at startup?
No.
My memory is that other folk on the forum have addressed this issue.

gyro

nooby
Posts: 10369
Joined: Sun 29 Jun 2008, 19:05
Location: SwedenEurope

#4 Post by nooby »

Sounds as a good thing you have going there.

doesn't the other devs have to know about it. Would it be too "much" to send a pm to them to look at this thread.

I would next puppy to have these features to load and unload sfs files.
I use Google Search on Puppy Forum
not an ideal solution though

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

How to combine

#5 Post by emil »

goingnuts wrote OTF SFS Loader which can dynamically load sfs files.
It is in the pupngo thread
Until now it just works with aufs1 (till puppy 4.2). I think your script can make it work also for newer puppies.
Great ... :D

Post Reply