HowTo use sfs and changes folder with a full Linux install

[wiak]

Note that this HowTo is intended to take away some of the mystery of how Linux distros can use overlayfs mount option (or aufs can be used with minor tweak to what follows) to implement load-sfs and changes/persistence facilities. In particular I wish to show how such facility can be used with a full install despite some myths going around that such facility is only useful for frugal installs.

Moreover, this HowTo is purposively written to show the actual commands required - I strongly believe that first principles are more important to understanding than a polished implementation, which this isn't! Rather this HowTo just provides a simple example, which hopefully aids actual understanding. It is intended to show you how simple the 'layering' methodology really is (independent distro makers aren't actually performing magic - they just happen to have learned these and other tricks), though I certainly won't be explaining the individual steps (please google and use command man pages for that).

Having said all that, I'm not saying this is for beginners, and since most users on Puppy are using frugal installs and aufs rather than full installs and overlayfs, this howto will maybe only be of interest to a few in its current form. Easy as I say to modify for aufs and/or frugal install situations as long as you understand the principles of operation illustrated in this example (or make a test full install of Void Linux and try it... as shown!).

Scripters can of course easily then implement all of this with a user friendly gui frontend... I'm using overlayfs, but can be easily changed to use aufs instead (I will show the minor alterations for that later along with same for frugal installs, though both these alterations are trivial really. I used overlayfs because it has been officially adopted for Linux kernel and aufs wasn't available by default on the system I used). In practice, a distro maker would likely implement these commands in the boot scripts (be it inside some init script or inside initrd/initramfs followed by a switch_root) but I hope to here show that these facilities can be used simply and independently of such grand schemes, or in addition to them (not that appropriately modifying initramfs, for example, is actually very difficult anyway - I might cover that later but want to avoid any confusing extra complexity for now).

The following shows how to take a full install of a Linux distribution and by means of an overlayfs of three layers (lower, middle, upper) be able to do the following:

1. Temporarily load and use an sfs file without needing to extract it.
2. Create a writable changes folder without affecting the underlying root filesystem (so that any changes made can be rolled back afterwards).
3. Successfully achieve the above two items without having to modify any part of the boot system (such as initramfs - though for more permanent use, that could be done later) and with no need to reboot the system to use the loaded sfs (which can easily be unloaded afterwards etc...).

The following should work with any full installed Linux (and alternatively could be arranged to work with frugal installs though I won't go into that possibility at the moment). As it happens I was using a full install of 64bit Void Linux which I had installed to the /dev/sda7 ext4 formatted partition I have on my computer. The sfs file I used in my tests was from tahrpup64 repo: gimp&mypaint-x86_64.sfs; though I renamed it to replace the annoying & with an underscore instead.

The directory 'lower' has the underlying root filesystem of Void Linux mounted readonly (via -o bind,ro) to it.
The directory 'middle' has the gimp_mypaint sfs loop mounted to it (readonly).
The directory 'upper' is writable and ends up containing any changes made whilst running the resulting merged filesystem in a chroot.

To get everything going, I simply cut and pasted the following into a terminal on my full-installed Void Linux system though any similar full-installed Linux system would do (assuming it has overlayfs included by kernel etc):

Code: Select all

mkdir -p /overlaystuff
cd /overlaystuff
mkdir -p lower middle upper work merged
mount -o bind,ro / lower
mount /root/Downloads/gimp_mypaint-x86_64.sfs middle
mount -t overlay -o lowerdir=middle:lower,upperdir=upper,workdir=work none merged
mount --bind /proc merged/proc
mount --bind /sys merged/sys
mount --bind /dev merged/dev
mount --bind /tmp merged/tmp
mount -t devpts devpts merged/dev/pts
chroot merged

Note that prior to the command 'chroot merged', you may in some circumstances need to copy /etc/resolv.conf to merged/etc/resolv.conf but that is not necessary in my above setup since resolv.conf in overlayfs layer 'lower' is simply a readonly mirror of the original in underlying void root filesystem.

After the above, you can, for example, then start gimp (from the loaded sfs) by entering:

Code: Select all

gimp &

End result of above is that gimp from the sfs is running successfully on the system. Once finished you can close gimp and enter 'exit' in the terminal to exit from the chroot and then clean up the mounts with:

Code: Select all

umount merged/proc && umount merged/sys && umount merged/dev/pts && umount merged/dev && umount merged/tmp && umount merged && umount lower && umount middle

followed, if you wish, by:
cd /
and remove the /overlaystuff directory to start afresh.

Note that it is also trivial to backup that 'upper' (changes folder) for re-use, or make the whole process a bit more automatic and user friendly with some scripting with yad and/or gtkdialog or whatever...

As I said, it is pretty much trivial to also open up initramfs and implement the above in there along with a switch_root so that the overlay system becomes the actual root filesystem in practice. However, I'm not wanting that additional complexity at the moment since I want this HowTo to show how simple using an overlayfs for these 'tricks' is, without actually needing any initramfs or init modifications (init and initramfs being somewhat system implementation dependent anyway).

wiak

More to come... (maybe, if I find time for that, but busy now...)
EDIT: for version that uses aufs rather than overlayfs see below post:
http://murga-linux.com/puppy/viewtopic. ... 96#1025496
The aufs code described there works with Void Linux host but not in Puppy or Dogs without some changes.

[wiak]

reserved

[wiak]

reserved

[wiak]

reserved

[rufwoof]

Might be of interest, if you substitute pflask for chroot https://github.com/ghedo/pflask

pflask can also create other types of mount points, have a look at the manpage for more information.

Additionally, using the --ephemeral option it's possible to tell pflask to discard any change applied to the root filesystem once the container terminates:

$ sudo pflask --chroot=/path/to/rootfs --ephemeral -- /sbin/init

This can be used for example for a build environment, where dependencies can be installed at every run on a clean rootfs, without the need to recreate the rootfs every time.

Haven't tried it, but presumably if you chroot / with ephemeral then all changes would be lost/not-saved.

Also/alternatively I believe you can use unionfs-fuse to mount sfs's in a full install. Along the lines of http://murga-linux.com/puppy/viewtopic. ... 16#1025216

[wiak]

I had earlier been playing with aufs so accidentally got lower and middle the wrong way round (since aufs syntax is the opposite way round to overlayfs for specifying lower layers order); have fixed first post now (realized my error as I woke up this morning...). That is:

I previously put:

Code: Select all

mount -t overlay -o lowerdir=lower:middle,upperdir=upper,workdir=work none merged

but with overlayfs (unlike aufs), lowerdir is done in highest layer of that first order, so have now correctly changed first post to:

Code: Select all

mount -t overlay -o lowerdir=middle:lower,upperdir=upper,workdir=work none merged

Of course if you really want the sfs file in lowest layer you use the order of lower:middle... The difference in practice is that when files are same in the sfs (such as some libs) as in the readonly mounted root filesystem you might want to put the sfs layer under the root filesystem layer to not take precedence over the corresponding files in the root filesystem (e.g. because the root filesystem contains newer version of libs than same ones in sfs).

The rule is that the files in the higher of the two layers take precedence.

I'll show how to use aufs instead later today probably (just a code line change really) and how to use it with Puppy with aufs (even say if you've just done a pfix=ram boot).

EDIT: No, not with Puppy (or one of the Debian/Ubuntu Dogs) with aufs; seems can't use an aufs layer (such as Puppy root filesystem) as one of the merging aufs layers (after binding Puppy aufs root-filesystem to lower. So the following slight change WON'T work since Puppy root filesystem will result in an aufs mount in lower, which is not allowed (okay, I believe, if overlayfs used to build a Puppy or Dog though):

Code: Select all

mkdir -p /mnt/sda5/overlaystuff
cd /mnt/sda5/overlaystuff
mkdir -p lower middle upper merged
mount -o bind,ro / /mnt/sda5/lower
mount /mnt/sda5/gimp_mypaint-x86_64.sfs middle
mount -t aufs -o br=lower=ro:middle:upper=rw none merged

As far as I recall you could do it if Puppy (or Dog) was using overlayfs for its underlying system construction since it is allowed with overlayfs that the lower layers can themselves be previously constructed overlayfs layers (if you understand what I mean... i.e. nested layers are allowed with overlayfs). Maybe nested layers are possible with aufs too, but I'd have to read up on that:

EDIT2: I've since read:
https://github.com/concourse/concourse/issues/1045

The critical flaw with aufs and overlay for our use case is that they do not nest. Within a container with an aufs filesystem, you cannot then create more aufs filesystems with an aufs directory as lower directory. The same is true for overlay.

So looks like aufs can't use nested aufs layer. However, I think above quote is wrong for overlayfs; I've used nested overlayfs layers with overlayfs. Which, at last, would be an advantage of overlayfs over using aufs... I'll find a quote... EDIT3: Here it is:

https://www.kernel.org/doc/Documentatio ... rlayfs.txt

The lower filesystem can be any filesystem supported by Linux and does
not need to be writable. The lower filesystem can even be another
overlayfs.

So overlayfs wins in this case. For Puppy example I need to thus find a Puppy that uses overlayfs instead of aufs. I believe gyro produced one of these - I'll search. But that alas won't help me provide simple Puppy example using its normal aufs mode of operation... oh well maybe I'll think of something else for that aufs case.

wiak

[wiak]

Might be of interest, if you substitute pflask for chroot https://github.com/ghedo/pflask

pflask can also create other types of mount points, have a look at the manpage for more information.

Additionally, using the --ephemeral option it's possible to tell pflask to discard any change applied to the root filesystem once the container terminates:

$ sudo pflask --chroot=/path/to/rootfs --ephemeral -- /sbin/init

This can be used for example for a build environment, where dependencies can be installed at every run on a clean rootfs, without the need to recreate the rootfs every time.

Haven't tried it, but presumably if you chroot / with ephemeral then all changes would be lost/not-saved.

Also/alternatively I believe you can use unionfs-fuse to mount sfs's in a full install. Along the lines of http://murga-linux.com/puppy/viewtopic. ... 16#1025216

Yes, these are interesting alternatives

wiak

[watchdog]

Your code can work also with a chroot jail instead of a full linux install. I have used a folder where is the extracted sfs of precise instead of a full linux install. I did some customizations also from rufwoof's post about xephyr and containers.

[gyro]

For Puppy example I need to thus find a Puppy that uses overlayfs instead of aufs. I believe gyro produced one of these - I'll search.

Indeed I have, and I'm still working on it, http://www.murga-linux.com/puppy/viewtopic.php?t=116059.

I hope to explore your idea sometime.

One of the limitations of overlayfs, is the inability to change the makeup of the layers of the stack, so no 'Load-Sfs on the fly'.
So, maybe it's possible to emulate adding an extra ro layer by making a new stack that contains the old stack and doing a chroot to it.
Although there might be some problems with the rw layer.

I curently stack a ro stack as the lower part of a rw stack.
But I've never tried to use a rw stack as a lower directory.

The other idea I had was for 'init' to chroot to the created stack and then 'rc.shutdown' could finish with an "exit", thus returning to the original 'init' and the original 'initrd.gz', and actually 'umount' filesystems.

So, if you have any bright ideas??

gyro

[wiak]

The other idea I had was for 'init' to chroot to the created stack and then 'rc.shutdown' could finish with an "exit", thus returning to the original 'init' and the original 'initrd.gz', and actually 'umount' filesystems.

Actually, in my own usage I normally do chroot into the stacks I'm experimenting with and exit out then umount the bind filesystem's at the end (though I tend to often be booted into either XenialDog64 or lxde Linux Void. However for past week Ive also been booting via a little initramfs of my own whose init ends up switch_root into external 'init' where I chroot in and out of nested overlayfs stscks Im playing with.

However most of my silent efforts last few days involve building modular build boot system for weedog with first flavour based on Void Linux packages along with its native xbps package manager (which is a terrific package manager by the way, which truly matches dpkg/apt in its accurate flexibility - have that working in a very small commandline booting version of weedog now, and via xbps can easily add X, window managers and so on and be able to easily script the build process.

I'm hoping to modularise also the init/initramfd selection capability so users/developers can use alternative layering schemes whether based on aufs or my preference of using overlayfs (or later perhaps even using alternative symlink method similar to how tinycorelinux works, but overlayfs works so easily/well nowadays I'm not really convinced tinycore's many loop-mounted symlink system is worth bothering about (and overlayfs preferred since kernel adopted; I do also like to add container functionality, maybe via pflask mechanism later, but these are just extra frills and my main weedog concern is to make it truly simple to understand and use for forum members who are not guru-like but want to develop their own systems too. I'm not into producing a final polished distro sine I have no personal interest in that careful polishing work and there are others here who are much more interested and thus better at that than I ever would be.


I'm in no hurry to publish anything yet though and first release (called WeeDog FirstRib will be as basic, unsophisticated, and simple as it sounds, hopefully. That much is already basically functioning, and with xpbs package manager, but I'm too busy playing/experimenting with it to even package a pre-alpha version as yet, but that much should appear within s couple of weeks since my partner is off on business trip do I need something to keep me busy...

EDIT: Once I've completed a Void flavour I plan to investigate alternatives such as Debian itself since I believe I can make particularly wee dogs using this method. I don't want a resulting complicated build-script like woofCE; that is not user friendly for most forum members. Rather I'd have similar small easy to understand buildscripts one for each flavour rather than add complexity to a single multi-flavour buildscript mechanism. But anyway, I'm finding Void such a good repo/package-management system that may end up being all I'm interested to slot in myself (but since the scripts will be simple at least others should find it easy to make their alternatives for experimenting with other package managers in the FirstRib...).

I'm not trying to compete directly with either tinycorelinux nor slitaz in install size. Void linux repo + package management offers more functionality in its bang for backs but still, I'm hoping that FirstRib will be under 50 MB in download size and the simplest basic X gui capable WeeDog under 100 MB, but I'm just guessing because will depend on the dependencies Void package manager installs (but should be better/leaner than similar system built via Debian repos I expect). Main thing is not the size however (users can add as many extras - window managers - browsers etc - as they want), rather it is the simplicity of the system so that it is easily added to and understood (unlike pretty much all these other ones out there, whose system scripts tend to be actually pretty complicated). Of course those who want to polish the end result will bloat the scripts and gain lots of functionality user-friendliness as a result - but I will leave that to them!...

Main issue I have is how much firmware support to include - that's what really bloats Void - their firmware support is huge. I am building WeeDog for my own use so I intend only including the firmware/modules I actually need for my own build but that won't be sufficient for other user's systems. However, I think my target 'audience' would be for forum members who are able to predetermine which firmware/module support their own systems need; I would provide instructions how to modify their copy of WeeDog to work with these addon firmware items (though I might also provide a small and large firmware sfs addon, though I wouldn't be using that personally myself - I don't want bloat firmware included on my own install - just want what my system actually needs...). First release of FirstRib probably won't include firmware extras at all... so some may be disappointed it won't work on their systems out of the box - well, that's what forum support is for and simpler the system the easier it is to explain modifications/additions anyway and more can be learned easily from it IMO. Puppy is too complicated under the hood and woof-CE mirrors that and is even worse since trying to cater for alternative upstream repos (whereas DebianDog uses native Debian Package manager, which is a more guaranteed approach for an almost the same distro install size - but WeeDog will start from an even more basic beginning, so easy to understand and grow from the biscuits we feed it later).

Wiak

[rufwoof]

Fatdog 8 supports btrfs (cat /proc/filesystems), it also has the btrfs tools (btrfs-progs) already installed by default - so seemingly all ready to go with respect to btrfs.

As a outline of what btrfs is/does

Btrfs is the next-generation Linux filesystem all cram-full of advanced features designed for maximum data protection and massive scalability such as copy-on-write, storage pools, checksums, support for 16, count 'em, 16-exabyte filesystems, journaling, online grow and shrink, and space-efficient live snapshots. If you're accustomed to using LVM and RAID to
manage your data storage, Btrfs can replace these.

A snapshot is a copy of a Btrfs subvolume at a particular point in time. It's many times faster than making a traditional backup, and incurs no downtime. You can make snapshots of a filesystem whenever you want, and then quickly roll back to any of them.

I believe that btrfs can be the lower filesystem in a overlayfs (???). So when you can quickly/easily change the content of that when using btrfs (switch to a different snapshot) that could be like loading/unloading sfs.

Seems to me like btrfs, squashfs and overlayfs are pretty much becoming established as part of the core Linux kernel and as such is the direction in which effort should be expended?

With btrfs you can create sub volumes - which somewhat act as though they were their own individual volume/partition. When a snapshot is created it takes up little space, in effect is just pointers. As/when files are changed the Copy on Write action creates a new copy of that changed file content. So with few changes snapshots actually contain very little actual data and are quick to start/copy ...etc. Same sort of thing as a Puppy save area in many respects.

Typically for Puppy use we might boot a 'clean' version, save data outside of that (on a separate device/partition), use it and not save, so the next reboot is clean. Or if we do want to tweak things, we might boot, make the changes, and then save a new version of our 'clean' version that we subsequently use to boot. btrfs can replicate that activity using snapshots. But requires its own btrfs format partition. However we can also set up btrfs inside a file filesystem, similar to Puppy save files, but that instead are btrfs format.

Code: Select all

truncate -s1G 1GB.img  # Space allocation of a 1GB file
ld=$(losetup --show --find 1GB.img)
echo "$ld"

You will now have a loop device (for example, /dev/loop2) that can be treated as a block device. So format that to btrfs

Code: Select all

mkfs -t btrfs "$ld" 

and mount it

Code: Select all

mkdir -p /mnt/ld
mount "$ld" /mnt/ld

When you're all done, tidy up (remove everything)

Code: Select all

umount /mnt/ld
losetup -d "$ld"
rm 1GB.img

If you wanted to create a partition table on the block device, include the

Code: Select all

--partscan 

flag on the losetup command, which will create the associated devices such as /dev/loop2p1

.... which means that btrfs could potentially be used in a similar vein to puppy save files, that can be stored/used from other formats such as NTFS.

I haven't tried using a btrfs file filesystem, more usually with a actual btrfs partition you'd use something like the following. Which would need to be modified/revised to merge that with the above.

The following assumes a actual btrfs partition on sdb2

Code: Select all

mkdir -p /mnt/btrfs
mount /dev/sdb2 /mnt/btrfs

create a btrfs sub volume using

Code: Select all

btrfs subvolume create /mnt/btrfs/vol1

list its ID using

Code: Select all

btrfs subvolume list /mnt/btrfs

which might return something like ...

ID 257 gen 7 top level 5 path vol1

and umount that top level view

Code: Select all

umount /mnt/btrfs

and mount the sub volume instead using

Code: Select all

mount -o subvol=vol1 /dev/sdb2 /mnt/btrfs

... or ...

Code: Select all

mount -o subvolid=257 /dev/sdb2 /mnt/btrfs

To create a snapshot of that ....

Code: Select all

umount /mnt/btrfs
mount /dev/sdb2 /mnt/btrfs
btrfs subvolume snapshot /mnt/btrfs/vol1 /mnt/btrfs/vol1-snapshot
umount /mnt/btrfs

Now you can mount either the original version vol1

Code: Select all

mount -o subvol=vol1 /dev/sdb2 /mnt/btrfs

or its snapshot vol1-snapshot

Code: Select all

mount -o subvol=vol1-snapshot /dev/sdb2 /mnt/btrfs

I guess instead of the /dev/sdb2 used in the above, that substituting the $ld device such as /dev/loop2 that we created in the first part, would do the trick for when using a btrfs file filesystem, but I haven't actually tried that myself.

EDIT: Have tried it now (i.e. I created the loop device as outlined earlier and if that's /dev/loop2 then use that /dev/loop2 in place of /dev/sdb2 in the above btrfs code) and it seemingly worked OK (btrfs in a file filesystem, with sub-volumes created/mounted and snapshots created etc., and where the btrfs file filesystem was created/stored on a ext3 partition).

[rufwoof]

Main issue I have is how much firmware support to include - that's what really bloats Void - their firmware support is huge

The OpenBSD install process is a basic cli script where you either have a full CD/DVD to install from, or you can use a basic network to install via, and where you must have a hard wired ethernet connection, i.e. its initial firmware is just enough to support a text terminal, keyboard and ethernet. Then you enter your locale ...etc and it pulls down/installs things from the net (or off CD/DVD) and the very last part of installation is that it runs fw_update ... that seemingly scans for what hardware you actually have and pulls down the firmware for that. Later, if you add a new device ... then run fw_update again yourself and it will scan the system and pull down the firmware as required.

Having just enough firmware initially to run a Xvesa display, keyboard and ethernet is considerably smaller than having a wide range of kit supported from the offset. Also there's way less firmware installed in the final version as well, just enough for what actual kit you're using. Perhaps a practice to follow.

For portability you'd have to also include a process to net connect and pull down firmware for whatever alternative/different kit you were booting/using.

[wiak]

Having just enough firmware initially to run a Xvesa display, keyboard and ethernet is considerably smaller than having a wide range of kit supported from the offset. Also there's way less firmware installed in the final version as well, just enough for what actual kit you're using. Perhaps a practice to follow.

Yes, what you describe is similar to what I'm basically aiming towards (eventually). The first of my releases (and to a large extent, hopefully, any and all) will be very simple however, and do next to nothing, but include modular addition of a package manager with minimum dependencies. That's why I've called it FirstRib (of my main distro objective called WeeDog).

WeeDog, which will nevertheless also be very basic since polishing it up is not a priority for me. Actually I'm likely to make various flavours of both, since, as I said somewhere long ago, I like the idea of having modular system including various possible package managers as modules and various initramfs modules for different overlay save persistence effects.

What I personally want to avoid in my scheme is the use of complex system scripts to provide that sort of versatility. I find that complex scripts, such as used in Puppy, Slitaz and TinyCore, make it difficult for most potential contributors to understand, so tends to create a developer elite, obscuring via high tech complex logic detail the simplicity of underlying operations. I'd rather provide something anyone with only basic scripting ability can understand and enjoy further developing.

Rather therefore than using individual scripts containing tons of logic to effect that flexibility, I intend providing simple systems, one for each flavour, each using very simple config scripts that result in their different flavours. That means also that I'm not intended producing a woof-CE like do-all-flavours complex multi-script build system. That alienates users in my opinion, so to create instead some do-very-little simple systems that can be added to modularly I intend also making multiple simple build systems - basically single scripts, one for each flavour. There will of course be much in common between each of these scripts, but I will purposively resist merging these scripts via complex selection logic (others may later produce a one-script to rule them all, but whilst that merging can be easily done that, I stress, is not my own intention or personal philisophy).

Hopefully, though the end result will not produce polished distro, it will allow involvement of many aspiring developers to improve for their own uses. So my objective is more fun and educational rather than competing with the existing Pups and Dogs and Debians etc of the world.

I'm using Void Linux package manager as a starter because Void repo is truly independent from the likes of Debian (as is its package manager and other system components).

I've also read up on btrf filesystem whilst originally investigating overlayfs, and it certainly becomes an attractive addition and, as you say, a possible way of effecting sfs-loading in a convenient manner. I will probably be leaving that issue of how to achieve top and bottom sfs-loading for a later time but will keep a close eye on the work of yourself and others in your implementations of that.

wiak

[rufwoof]

Just for the record - overlayfs documentation indicates it supports up to 128 overlays, but in practice that includes the changes (upperdir), top and lowerdir, so that actual number is 125 that can be layered within lowerdir

A quick test indicates however that you can combine multiple layered sets as part of lowerdir ... which extends the number considerably

i.e. if a and b folders are mount points for files a.sfs and b.sfs and we ...

mount -t overlay overlay -o lowerdir=a:b,upperdir=changes,workdir=w top1

and then ...
mount -t overlay overlay -o lowerdir=x:y:top1,upperdir=changes2,workdir=w2 top2

Then conceptually we have potentially 125x125 possible layers for lowerdir.

Repeat for a third time and its 125x125x125.

Haven't actually tested that though

Could be used as a form of version control. Open a file stored in a sfs, edit that file; Make a sfs of the changes folder (upperdir) and add that changes sfs to the set (lowerdir) when the 'file' is next mounted. Resulting in many sfs's each with their own image of how the file was changed over time and where you can 'roll back' to any past prior version.

Just remember that lowerdir works right to left (leftmost is the highest image, rightmost is the deepest/lowest).

lowerdir=ac,upperdir=changes

<highest layer>
changes
a
b
c
<deepest layer>

... so in this context, the most recent mksquashfs of the changes folder would be mounted and that mount point inserted as the first item immediately after lowerdir=

[rufwoof]

Overlayfs observation. upperdir and workdir must be on the same filesystem. You cannot use a upperdir that is within a encfs folder - must be a real folder, however you can use /tmp and you can use a file filesystem.

I've set up a hdd based FILES/DATA folder where DATA is encfs encrypted. Mounting that and storing changes (upperdir) and work folders under /tmp, and top (visible level) mounted as /root/top, and multiple sfs's (saves) stored within the DATA folder (so encrypted when DATA is encfs closed). A new sfs is added to the set - a mksquashfs copy of the changes folder, whenever a 'save' action is triggered (and the changes folder cleared once the sfs has been created). With lowerdir=sfs1:sfs2:....sfs110 all loaded (prior 110 saves) it continues to work OK. But that is near the upper limit without layering layers (overlayfs has a 125 limit for lowerdir= folders, but can have layered layers as I outlined in a earlier post).

With that layered data setup I can roll back to (or inspect) any earlier version of data content, a form of time-machine (for data). It's also quick to backup as a rsync of FILES (top level folder) simply syncs the differences in encrypted .DATA folder content (i.e. additional sfs's (saves) made since the last rsync).

With changes (upperdir) and work both set to use /tmp, visible (top) layer mounted under /root and all the sfs's stored in encfs encrypted /FILES/DATA folder, that obscures clear text data (encrypted content when closed i.e. if the laptop is lost/stolen), and leaves no residual footprints (all runs in ram/encrypted swap, so cannot have deleted files recovered/seen via forensics).

The 110 sfs's was just a test (had to create additional loop devices to support up to 125 sfs's being mounted). In practice after 100 or so saves it would be better to simply merge all of the past saves (sfs's) into one and set that as sfs1 in the next mount (just a single sfs of all changes to data) ... and store the original 100 odd sfs's from which that was created as historic/archive files.

That setup/arrangement will work both in frugal and full Puppy install cases. It's really just applying sfs layering using overlayfs against data only (I tend to not use save against OS/Puppy once its set up the way I like it - just reboot the exact same working/configured Puppy (Fatdog) time after time).

As changes could be large in some cases (such as moving additional music/video files into my DATA/music folder), I allocate a large encrypted swap (26GB), and I've set sfs compression when a 'save' is run (mksquashfs of the changes folder) to use lz4 compression - which tends to be relatively quick at both compressing and decompressing, even though that's not the tightest of compressors (xz tends to produce smaller compressed file sizes, but is rather slow at both compressing and decompressing).

Running 'losetup -l' or 'mount' commands when that is loaded with 100+ sfs's does look odd, as it throws back loads of lines (losetup) or a massively long command line (mount) But it works, which is the main thing.