Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 29 Mar 2020, 05:40
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Another re-write of the "init" script - using OverlayFs?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 14 of 14 [206 Posts]   Goto page: Previous 1, 2, 3, ..., 12, 13, 14
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Fri 03 Aug 2018, 03:22    Post subject: Re: UpupBB 18.05+8 + OverlayFS init + kernel 4.17.10 + BTRFS
Subject description: it works!
 

Wyk72 wrote:
After a whole afternoon studying this (and other) threads on the subject, I managed to assemble (i.e. hack) a working upupbb 18.05+8 on an USB stick formatted with BTRFS.
Does "blkid" report it as "TYPE=btrfs" or "TYPE=BTRFS"?
gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Tue 23 Apr 2019, 15:39    Post subject:  

Any remaining files for this project have been moved to http://www.mediafire.com/folder/inbkdjmie89hm/overlay

gyro
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3719

PostPosted: Wed 23 Oct 2019, 17:28    Post subject: Re: Another re-write of the "init" script - using OverlayFs?
Subject description: Release of overlay_init-0.9 - files moved to "mediafire"
 

gyro wrote:
The results of my reading research are:

1) OverlayFs is much simpler that aufs, it seems to lack any ability to modify an existing stack.
If this proves to be true in practice, then the stack would have to be created once in "init", and then remain unchanged for the session.
"sfs_load" would be reduced to being an sfs manager whose results did not take effect until after a reboot.
Utilities that execute sfs files, by loading them into the aufs stack, executing the program inside and then unloading them would go.
There might also be some challanges with pupmode=13, since what happens when a directory that is part of an overlay is modified directly is undefined. So "snapmergepuppy" might have to go.

You can't modify it, but you can re-layer it.

Create a overlay ...
mkdir sfs changes work top
mount -t overlay overlay -o lowerdir=sfs,upperdir=changes,workdir=work top

ls -la top ... returns nothing
echo hello >top/hello.txt ... and top now contains a single file (as does changes).

mount fd64.sfs sfs
and create a overlay of the current top folder with that sfs included (again using top as the mountpoint for that overlay)
mount -t overlay overlay -o lowerdir=top:sfs,upperdir=changes,workdir=work top

and ls top will show the sfs content together with the hello.txt file we created earlier. changes will have just the single file (hello.txt). Note here that sfs is below top, if you wanted the sfs to take precedence (be on top of top), then it would be ...lowerdir=sfs:top....

umount top ... and the sfs is gone, we're back to the first level top folder content again. Note however that if any changes were made whilst the sfs was loaded then it will complain about missing files. A option might be to use a changes2 folder for changes made whilst the sfs was loaded.

A problem however is that if you're IN the top folder then you wont see the changes under that parent set of inodes, you have to cd out of it and then back into it to see the changes.

Loading and unloading is however sequential. umount top ... unloads the sfs and has you back to the first layer again. So for instance you wouldn't be able to sfs load a, sfs load b, sfs unload a .. but instead have to accept sfs load a, sfs load b, umount b, umount a i.e back out of sfs's in the reverse order of what they were loaded.

In the case of Puppy, where layering is set up in initrd and then switch rooted, I suspect you may instead have to resort to using chroot and have a mechanism to cd out of that and then back in again in order to see changes (loaded sfs's). But if you do that, then might as well just ditch the current overlay altogether and back in initrd again just reform a new overlayfs that includes the loaded (or excludes unloaded) sfs's.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Wed 23 Oct 2019, 18:50    Post subject:  

@rufwoof,

Yes, there might be ways to keep on building new stacks, but it's complicated.

1. The rw directory needs to remain the rw directory, i.e. the top of the old stack needs to be the top of the new stack.

2. When changing from one stack to another, we are remounting /.

I have thought of simply mounting new stack on top of old stack, linux supports mounting to the same mount point again.
But???

There may be a messy way to do it, but for now it's in my too hard basket.

But be my guest, play with it.
If you get to some apparently working code, let me know.

gyro
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3719

PostPosted: Thu 05 Dec 2019, 12:12    Post subject:  

I have a small Bulldog based boot system that basically boots to initramfs cli, with wifi net connected ...etc. That also includes screen (terminal multiplex (multiple screens)).

Booting that and then starting screen, and starting a new screen within that (so access to two cli's), and (in screen 0) running ...

mkdir initramfs changes work top
mount -o bind,ro / initramfs
mount -t overlay overlay -o lowerdir=initramfs,upperdir=changes,workdir=work top

Has one screen (screen 0) running the initramfs in a Puppy like layered manner (but using overlayfs), and the other screen (screen 1) remaining in the initramfs

I can flip between the those two screens.

What's occurred here is that we've chrooted to / using overlayfs, so all changes in that instance don't actually make changes to the system, they're just recorded in the /changes folder (layered system).

chroot doesn't map mounted folders, so in that chrooted session (screen 0) that doesn't actually 'see' the content of top, work or changes folder contents. screen 1 however can see, and change the content. For example using screen 1 to create a new file /changes/root/hello.txt results in screen 0 (the chrooted session) seeing that file appear, its in effected added into top (/ in the chrooted session). So I could use screen 1 to make snapshot copies of the /changes folder at any time (save), or add new things to it, such as extracting a sfs ...etc.

Alternatively rather than running two screens, a background task could be left running and monitor for the creation of any 'actions'. Perhaps a special folder where any files created in that folder by the chroot session (screen 0) are picked up by the initramfs session (screen 1) and actioned, such as making a snapshot (sfs) of the current changes folder content, or replacing that with another content.

You can mount multiple layers as the lowerdir, lowerdir=sfs:initramfs ... type syntax; OR perhaps changes (upperdir=) could be a stack, that can be rearranged as/when desired. Just has to be writable, as that is where all changes for the /top folder (top view (screen 0)) are recorded. At least I think that is the case (I need to do some more testing). If so, then that changes stack could (might) be restacked/rearranged as/when desired, in whatever layering arrangement you desire.

EDIT : Apparently NOT. Doesn't like having a layered filesystem as the upperdir (changes folder content).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Sat 07 Dec 2019, 06:01    Post subject:  

@ruwoof,
When you say "chrooted", are you referring to the line,
"mount -o bind,ro / initramfs"?

I still can't quite get my head around the relationship between "top" and "/", since "initramfs" is the lowerdir of the stack.

I also would have thought you should be creating "/top/root/hello.txt" not "/changes/root/hello.txt".

Would this approach allow for the replacement of the stack with a different stack, i.e. containing a different sfs file in it's lowerdir list?

gyro

Last edited by gyro on Sat 07 Dec 2019, 06:31; edited 1 time in total
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Sat 07 Dec 2019, 06:14    Post subject:  

This project has become about experimenting with a lot more things than just overayfs, so I've morphed it into "Minimal Init Overlay (mio)",
at http://www.murga-linux.com/puppy/viewtopic.php?t=116059.

In the "mio" project, the overlayfs stuff is pretty much the same as here, but the other stuff is moving on.

There will be no more releases in this project.
Even if advancements are made with the overlayfs stuff, it will be released as a new version of the "mio" project.

Sorry, I meant to make this statement when I started the "mio" project.

gyro
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3719

PostPosted: Sat 07 Dec 2019, 08:00    Post subject:  

gyro wrote:
@ruwoof,
When you say "chrooted", are you referring to the line,
"mount -o bind,ro / initramfs"?

mkdir initramfs and then mount -o bind,ro / initramfs has the current / system also reflected in intramfs/ ... excepting any mounted folders, as though the root system had been shifted over/into the initramfs folder. So when we chroot to that then its pretty much the same, not too dissimilar to doing a chroot /
Quote:
I still can't quite get my head around the relationship between "top" and "/", since "initramfs" is the lowerdir of the stack.

.. but we're not actually doing just a chroot / we're creating a stack/layered sytem, where / (initramfs folder) is at the bottom (lowerdir) in that stack. top is the top folder (the visible layer), changes is the folder where changes are stored (like whiteout files in aufs, but using overlayfs style instead), work is just a (empty) folder for overlayfs to work.
Quote:
I also would have thought you should be creating "/top/root/hello.txt" not "/changes/root/hello.txt".

Either really. Creating a file in the top (visible layer), adds that file to the changes folder (and vice-versa).
Quote:
Would this approach allow for the replacement of the stack with a different stack, i.e. containing a different sfs file in it's lowerdir list?

If you exit the chroot, then you're back into the initramfs system, so yes you could create a entirely different stack set and chroot into that new stack. Maybe leaving the previous stack as before, or deleting it. If left as before then you'd have the option to again exit the chroot and revert back to that former stack. Best if the new stack has different changes/work folders as otherwise the new stack wouldn't work (work folder not empty) and/or the changes folder content for the original stack might have files/folders in it that conflicted with the correct operation/use of the new stack.

Unlike switch-root that in effect wipes out the initramfs content, with chroot the initramfs is still there. You could set capsh restrictions (provided built in) to prevent exiting out of a chroot to make it impossible to get back to the initramfs cli if that was desired, but equally if you block exiting out of a chroot then neither can you chroot into anything else.

In some ways, setting up and running (chroot'ing into) a stack, and then exiting out again, is a bit like having done a reboot back into the initramfs level, but without having had to go back through BIOS bootup. The record of changes made whilst in the stack are also available, so you could chroot back into the exact same stack and carry on as before. Or with the same stack re-established after a clean boot, along with a copy of the changes folder content loaded into the changes folder prior to doing the chroot, then the changes would have persisted across reboots. Fundamentally whilst you can't dynamically add/remove sfs's from overlayfs as you can with aufs, you can in effect do the equivalent of a quick reboot (exit chroot), reform the stack with additional (or removed) sfs's and chroot into that. But that does mean a fresh desktop whenever a sfs is loaded/unloaded, as though the desktop had just been restarted (couldn't for example have programs left running, or windows open ...etc. when a sfs was to be loaded/unloaded (as we exit the chroot)). However if you have a cli still running in the initramfs system then that could extract a sfs directly into the changes folder whilst the chroot system was still running, and that would be as though the sfs content had been added to the chrooted system. Or the initramfs cli could wipe out and replace the entire content of the changes folder content, which again the chrooted system would see those changes as having occured.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Sun 08 Dec 2019, 07:34    Post subject:  

@ruwoof,

So, theoretically we could have something a bit like "Restart graphical server" in "Puppy Log Out", called "Restart Stack".
But would we end up performig most of "rc.shutdown" then "init" then "rc.sysinit", anyway.
Which is why for now I'm living with doing a "reboot" to change the composition of the stack.

Just to be clear, are you doing a chroot to top?

I find your approach interesting, since I have been wondering if there are ways of running a Puppy other than by using switch-root.
But frustratingly, I need to leave this "rabbit hole" for a later date.

gyro
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3719

PostPosted: Sun 08 Dec 2019, 08:11    Post subject:  

Yes I chroot into top. That's just the naming convention that stuck with me. Typically I do mkdir sfs changes work top ... and then load (mount) a sfs to sfs, know that work is just required by overlayfs, changes has all changes recorded in it and top is the top 'visible' layer and the syntax is

mount -t overlay overlay -o lowerdir=sfs,upperdir=changes,workdir=work top
chroot top

If the main system is set up for such, then you can just chroot top as above and then run startx without having to go through rc. In my cutdown Bulldog (Fatdogs initramfs cli) I have a complete cli/tui system - that wifi net connects, has full OpenSSH (rather than dropbear), mc ...etc. I can and often do just use that boot (ssh into hashbang where tmux is running with irc, email ..etc. all as left when I was last logged in). Using that I could just mount the main fatdog sfs and chroot into it and run startx and have the gui desktop appear, with any changes being stored in the initramfs's changes folder (that could be pre-loaded from a saved version such as unsquashing a sfs of a prior changes folder mksquashfs. Or drop out of that (exit the chroot), set up a entirely different stack perhaps with another Puppy main sfs and chroot into that.

The main 'cost', is that unlike with aufs where you can just load a sfs at any point during a session with other windows open etc, with that method you do have to exit/restart - similar to exiting X and restarting X.

If you set up capabilities (capsh) then systems can be started with selected capabilities dropped. Along the lines of what EasyOS does. Bearing in mind that if you drop chroot capabilities then once you've chrooted into the main system (gui) then that's trapped, can't chroot out of itself (nor chroot into anything else). However if in initramfs you're running screen/tmux or even another session (ctrl-alt-Fn), then that can end (kill) that chroot'd gui session (in effect running cli in one terminal session, where that terminal session is running gui; running initramfs cli in another terminal session).

Fundamentally using chroot instead of switch-root.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1736
Location: Brisbane, Australia

PostPosted: Sun 08 Dec 2019, 09:13    Post subject:  

@ruwoof,

Just one more question:
Have you tried making "changes" and "work", directories on a mounted Linux partition? (Thus providing persistence)

Then I'm going to have to leave this, until I have some time to produce some actual code that tries to do this, in a Puppy environment.

gyro
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 14 of 14 [206 Posts]   Goto page: Previous 1, 2, 3, ..., 12, 13, 14
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0857s ][ Queries: 12 (0.0295s) ][ GZIP on ]