Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#181 Post by gyro »

peebee wrote: see that "/usr/sbin/overlay-setup-frugal" is in overlay_mods.sfs but do I install the .sfs and then run it or do I extract it and run it??
The most reliable way to do it in a "normal" puppy is to sfs_load the "overlay_mods.sfs" from the tar file, thus making the new utilities available, but not modifying the system.
Then the process I outlined in my previous post should work.

Sorry about the confusion, my bad.
I think the new process is documented in the post announcing the first version to include delta's.
But I should fix the first post to make things clear.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#182 Post by gyro »

I have updated the first post to include current usage information.
gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#183 Post by peebee »

gyro wrote:I have updated the first post to include current usage information.
gyro
Thanks @gyro - now works fine.....
Attachments
Screenshot.png
(228.38 KiB) Downloaded 784 times
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#184 Post by gyro »

peebee wrote:Thanks @gyro - now works fine.....
Great. Thanks for testing.
gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#185 Post by peebee »

Hi @gyro

Have you had any thoughts on how an experimental woof-ce build might be done to include overlayfs?

Presumably the overlay-mods.sfs would be converted to a .pet and included in the DISTRO_PKG_SPECS ??

and then overlay-setup-frugal would need to be run via
EXTRA_COMMANDS
in
_00build.conf ??

?? or not??

Cheers
peebee
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#186 Post by gyro »

@peebee,

All my current methods for producing an "overlay_init" Puppy work outside woof-ce.
"overlay-setup_frugal" converts a normal frugal install to an "overlay_init" frugal install.
"overlay-make-iso" converts a normal iso to an "overlay_init" iso. Unfortunatley this requires the "overlay" module to be loaded, as it merges directories by creating an "overlayfs" stack.

One method of integrating "overlay_init" into woof-ce is to actually include it all into a normal Puppy, with the alternative init being activated by a boot parameter at run time.
I'm not convinced that "bloating" a normal Puppy with all the extra files from "overlay_init" is the best way to proceed.

"overlay_init" consists of an alternative initird.gz (smaller), a number of files that replace existing files in Puppy, and quite a few new files with no equivalent in normal Puppy.

My original idea was that woof-ce would be capable of producing an alternative "initrd.gz" just as it can produce different Puppies.
So, similar to a woof-ce user choosing to build a "upupbb" or "slacko" Puppy, they would be able to choose to build an "aufs" or an "overlay" initrd.gz.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#187 Post by gyro »

I have uploaded a delta for upupbb-18.05+8.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.

This is just a maintenance release for a newer upupbb, and to fix a couple of minor bugs.

gyro

watchdog
Posts: 2021
Joined: Fri 28 Sep 2012, 18:04
Location: Italy

#188 Post by watchdog »

gyro wrote:I have uploaded a delta for upupbb-18.05+8.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
I just did a new frugal install with istructions from here and peebee. All working well for my basic needs. Excuse my ignorance: no multiple savefolders support at boot?

Later: help files in the iso require to be rewrited...

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#189 Post by gyro »

watchdog wrote:I just did a new frugal install with istructions from here and peebee. All working well for my basic needs. Excuse my ignorance: no multiple savefolders support at boot?
Yes there is no such support.
It was something I left till later, but now I find I don't miss it.
I have 2 directories, "pupsaves" (where my savefolders normally reside) and "pupsavesAlt". When I want to try a possibly "dangerous" pet, I run "Saveconfig"->"Folder"->"Select" and select "pupsavesAlt".
On reboot I'm now using a copy of my savefolder in "pupsavesAlt".
When I have finished, I run "Saveconfig"->"Folder"->"Default" to go back to my normal savefolder, which results in a "Cancel", "Replace" or "Keep" dialog, referring to my normal savefolder in "pupsaves".
If the test was successful, I select "Replace". If the test failed, I select "Keep".
No, it's not the same as multi-savefolder support in "init", but unfortunately the result is that real multi-savefolder is currently a low priority.
watchdog wrote:Later: help files in the iso require to be rewrited...
Yes, again you are correct. And the answer is similar, there always seems to be something more urgent.

PS. Thanks for testing.

gyro

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#190 Post by rockedge »

I have set up a Puppy Linux Bionic18.05+8 using the OverlayFs and the instructions from this thread.

Code: Select all

title BB18.05+8-OL (sdb1/BB18.05+8-OL)
  uuid 8a8ea99d-a1b0-4c43-b1a0-d4ce5c9c7dfa
  kernel /BB18.05+8-OL/vmlinuz   psubdir=BB18.05+8-OL pdrv=sdb1
  initrd /BB18.05+8-OL/initrd.gz 
I used overlay_init_upupbb-0.9.tar.
the OS booted cleanly and upupbb is running great.
DELL INSPIRON E1505
Attachments
upupbb1805_overlay.png
(136.39 KiB) Downloaded 477 times

User avatar
Wyk72
Posts: 18
Joined: Tue 01 Sep 2009, 12:55

UpupBB 18.05+8 + OverlayFS init + kernel 4.17.10 + BTRFS

#191 Post by Wyk72 »

After a whole afternoon studying this (and other) threads on the subject, I managed to assemble (i.e. hack) a working upupbb 18.05+8 on an USB stick formatted with BTRFS.

- I used the info provided in this thread to install upupBB 18.05+8 with overlayFS on a (ext4 formatted) USB stick.

- I compiled the latest stable kernel (4.17.10) with the stemsee script (nubuild.sh) provided in this thread:

http://www.murga-linux.com/puppy/viewtopic.php?t=96723

- I used the change-kernel script provided here by gyro, to install the 4.17.10 vmlinuz/ zdrv sfs (modules)

- I formatted a new USB stick with BTRFS using gparted. Set the "boot" flag.

- booted from another system, copied the whole content of the previous USB stick to the BTRFS formatted one

- Downloaded extlinux from PPM, installed it onto the BTRFS formatted USB stick (mbr too)

- Edited the extlinux.conf with proper UUID values, i.e.:

Default puppy
display boot.msg
prompt 1
timeout 50

F1 boot.msg
F2 help.msg
F3 help2.msg

label puppy
kernel vmlinuz
append initrd=initrd.gz uuid=799f112b-800a-4291-b03f-74e999ab75eb pupsfs=799f112b-800a-4291-b03f-74e999ab75eb

- modified the upupbbBOOTspecs file in the root (/) directory of the BTRFS stick with the correct UUID

And that was it :)

I am testing it now, it looks fine for the most part. I am looking for a way to enable ZSTD file compression, it must be the initial "Mount" command in the init script by gyro.
Attachments
Screenshot.jpg
(230.07 KiB) Downloaded 327 times

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#192 Post by rockedge »

After a whole afternoon studying this (and other) threads on the subject, I managed to assemble (i.e. hack) a working upupbb 18.05+8 on an USB stick formatted with BTRFS.

nice! :D

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#193 Post by gyro »

@rockedge,
Thanks for testing, good to see it works for you.

I'm just about to upgrade my overlay_init upupbb to upupbb-18.05+10.
I usually do this by simply replacing adrv, fdrv, puppy and ydrv sfs files with those from peebee's new iso.
(Replacing the zdrv just results in "rc.update" doing a whole lot of unnecessary stuff.)

Later: Upgrade to upupbb-18.05+10 worked fine.

gyro
Last edited by gyro on Thu 02 Aug 2018, 19:47, edited 2 times in total.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: UpupBB 18.05+8 + OverlayFS init + kernel 4.17.10 + BTRFS

#194 Post by gyro »

Wyk72 wrote:I am testing it now, it looks fine for the most part.
Hmm.., I've never tried running Puppy on BTRFS, glad it works. Thanks for testing.
Wyk72 wrote:I am looking for a way to enable ZSTD file compression, it must be the initial "Mount" command in the init script by gyro.
I'm not sure how you expect Puppy to use ZSTD file compression.
If it is supposed to be used for the sfs files, then this depends on the squashfs support within Puppy, not my init script.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#195 Post by gyro »

I like to place unique labels on my partitions, so my grub4dos entry for overlay_init upupbb is:

Code: Select all

title Puppy upupbb 18.05 (sdb2/upupbb)
  uuid 0db94719-cdf1-44b7-9766-23db62fb85a5
  kernel /puppy/upupbb/vmlinuz intel_pstate=disable net.ifnames=0 psfspart=Work psfsdir=/puppy/upupbb
  initrd /puppy/upupbb/initrd.gz
Note the "psfspart=" parameter.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: UpupBB 18.05+8 + OverlayFS init + kernel 4.17.10 + BTRFS

#196 Post by gyro »

Wyk72 wrote:After a whole afternoon studying this (and other) threads on the subject, I managed to assemble (i.e. hack) a working upupbb 18.05+8 on an USB stick formatted with BTRFS.
Does "blkid" report it as "TYPE=btrfs" or "TYPE=BTRFS"?
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#197 Post by gyro »

Any remaining files for this project have been moved to http://www.mediafire.com/folder/inbkdjmie89hm/overlay

gyro

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

Re: Another re-write of the "init" script - using OverlayFs?

#198 Post by rufwoof »

gyro wrote:The results of my reading research are:

1) OverlayFs is much simpler that aufs, it seems to lack any ability to modify an existing stack.
If this proves to be true in practice, then the stack would have to be created once in "init", and then remain unchanged for the session.
"sfs_load" would be reduced to being an sfs manager whose results did not take effect until after a reboot.
Utilities that execute sfs files, by loading them into the aufs stack, executing the program inside and then unloading them would go.
There might also be some challanges with pupmode=13, since what happens when a directory that is part of an overlay is modified directly is undefined. So "snapmergepuppy" might have to go.
You can't modify it, but you can re-layer it.

Create a overlay ...
mkdir sfs changes work top
mount -t overlay overlay -o lowerdir=sfs,upperdir=changes,workdir=work top

ls -la top ... returns nothing
echo hello >top/hello.txt ... and top now contains a single file (as does changes).

mount fd64.sfs sfs
and create a overlay of the current top folder with that sfs included (again using top as the mountpoint for that overlay)
mount -t overlay overlay -o lowerdir=top:sfs,upperdir=changes,workdir=work top

and ls top will show the sfs content together with the hello.txt file we created earlier. changes will have just the single file (hello.txt). Note here that sfs is below top, if you wanted the sfs to take precedence (be on top of top), then it would be ...lowerdir=sfs:top....

umount top ... and the sfs is gone, we're back to the first level top folder content again. Note however that if any changes were made whilst the sfs was loaded then it will complain about missing files. A option might be to use a changes2 folder for changes made whilst the sfs was loaded.

A problem however is that if you're IN the top folder then you wont see the changes under that parent set of inodes, you have to cd out of it and then back into it to see the changes.

Loading and unloading is however sequential. umount top ... unloads the sfs and has you back to the first layer again. So for instance you wouldn't be able to sfs load a, sfs load b, sfs unload a .. but instead have to accept sfs load a, sfs load b, umount b, umount a i.e back out of sfs's in the reverse order of what they were loaded.

In the case of Puppy, where layering is set up in initrd and then switch rooted, I suspect you may instead have to resort to using chroot and have a mechanism to cd out of that and then back in again in order to see changes (loaded sfs's). But if you do that, then might as well just ditch the current overlay altogether and back in initrd again just reform a new overlayfs that includes the loaded (or excludes unloaded) sfs's.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#199 Post by gyro »

@rufwoof,

Yes, there might be ways to keep on building new stacks, but it's complicated.

1. The rw directory needs to remain the rw directory, i.e. the top of the old stack needs to be the top of the new stack.

2. When changing from one stack to another, we are remounting /.

I have thought of simply mounting new stack on top of old stack, linux supports mounting to the same mount point again.
But???

There may be a messy way to do it, but for now it's in my too hard basket.

But be my guest, play with it.
If you get to some apparently working code, let me know.

gyro

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#200 Post by rufwoof »

I have a small Bulldog based boot system that basically boots to initramfs cli, with wifi net connected ...etc. That also includes screen (terminal multiplex (multiple screens)).

Booting that and then starting screen, and starting a new screen within that (so access to two cli's), and (in screen 0) running ...

mkdir initramfs changes work top
mount -o bind,ro / initramfs
mount -t overlay overlay -o lowerdir=initramfs,upperdir=changes,workdir=work top

Has one screen (screen 0) running the initramfs in a Puppy like layered manner (but using overlayfs), and the other screen (screen 1) remaining in the initramfs

I can flip between the those two screens.

What's occurred here is that we've chrooted to / using overlayfs, so all changes in that instance don't actually make changes to the system, they're just recorded in the /changes folder (layered system).

chroot doesn't map mounted folders, so in that chrooted session (screen 0) that doesn't actually 'see' the content of top, work or changes folder contents. screen 1 however can see, and change the content. For example using screen 1 to create a new file /changes/root/hello.txt results in screen 0 (the chrooted session) seeing that file appear, its in effected added into top (/ in the chrooted session). So I could use screen 1 to make snapshot copies of the /changes folder at any time (save), or add new things to it, such as extracting a sfs ...etc.

Alternatively rather than running two screens, a background task could be left running and monitor for the creation of any 'actions'. Perhaps a special folder where any files created in that folder by the chroot session (screen 0) are picked up by the initramfs session (screen 1) and actioned, such as making a snapshot (sfs) of the current changes folder content, or replacing that with another content.

You can mount multiple layers as the lowerdir, lowerdir=sfs:initramfs ... type syntax; OR perhaps changes (upperdir=) could be a stack, that can be rearranged as/when desired. Just has to be writable, as that is where all changes for the /top folder (top view (screen 0)) are recorded. At least I think that is the case (I need to do some more testing). If so, then that changes stack could (might) be restacked/rearranged as/when desired, in whatever layering arrangement you desire.

EDIT : Apparently NOT. Doesn't like having a layered filesystem as the upperdir (changes folder content).
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

Post Reply