Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: Announcing a new version

#31 Post by gyro »

Moose On The Loose wrote:Has the loading of an SFS on the fly been made possible yet?
No, I have still not found a way to modify an existing overlay stack, but testing of this is not yet complete.
Moose On The Loose wrote:If not perhaps a "fast reboot" could be added that doesn't reboot but instead pivots off the overlayed file system makes changes and goes back onto the modified overlayed system. This would be a lot faster than a complete reboot.
Or could try building a new "fixed" stack and pivot root to that.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#32 Post by Rangan Masti »

Wish you success guys.

User avatar
Moose On The Loose
Posts: 965
Joined: Thu 24 Feb 2011, 14:54

Re: Announcing a new version

#33 Post by Moose On The Loose »

gyro wrote:
Moose On The Loose wrote:Has the loading of an SFS on the fly been made possible yet?
No, I have still not found a way to modify an existing overlay stack, but testing of this is not yet complete.
Moose On The Loose wrote:If not perhaps a "fast reboot" could be added that doesn't reboot but instead pivots off the overlayed file system makes changes and goes back onto the modified overlayed system. This would be a lot faster than a complete reboot.
Or could try building a new "fixed" stack and pivot root to that.

gyro
Yes, if a modified stack can be done, it seems like a faster way to get to a changed system. It may take some trickery to make the changes up to that point stay but not override the new SFS.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#34 Post by gyro »

When I tried to change the list of ro directories in an overlay with a "remount", there were no error messages, but the overlay remained unchanged. So this seems to be a dead-end.

But I was able to create a new overlay using some directories from the old overlay (including the rw dir, "upperdir" and "workdir") + a new ro directory. Then unmounting the old overlay and moving the new overlay to the original mount point. This worked.
Although this was just using play directories, not the actual / overlay.
So there is still some hope of being able to create a new overlay at /pup_new and doing some sort of pivot root to /pup_new, (again).

Unfortunately, it is probably safer to start by returning to the ancient puppy method of building the whole stack in "init" and not modifying it at all.

I've uploaded a new version of the "init" and "initrd.gz".
This one creates an ro overlay containing just the sfs's at /pup_sfss, and after sorting out what directory to use as an rw directory, creates the full overlay on /pup_new, using /pup_sfss as the "lowerdir".
This restores the logic of dealing with the sfs's first, then sorting out the save layer.
This also means that in the running system /initrd/pup_sfss contains the files of all the sfs's.

This version is also the first one that writes a PUPSTATE where the variables that are written are fully compatible with current puppies.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#35 Post by Rangan Masti »

Hi Good morning, where to find the new initrd uploaded.for testing?
Thank you and have a great day!

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#36 Post by gyro »

Rangan Masti wrote:where to find the new initrd uploaded.for testing?
The url's of all the files I've uploaded for this project are now provided in the 1st post.
But here's the one for the "initrd.gz" any way, http://www.fishprogs.software/puppy/initrd/initrd.gz.
Note: If the kernel in your puppy does not support "overlayfs", "init" will fail.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#37 Post by gyro »

I've had no success with trying to modify an overlay that is mounted as /, so I'm moving on to try some save layer techniques.

I've uploaded http://www.fishprogs.software/puppy/ini ... verlay.sfs, to help with testing. It currently only includes "shutdowconfig" but in the next version it will also include "rc.shutdown".
And I've added some information on how to use the files, to the first post.

This version can boot automatically to pupmode=12 using a savefolder.
Just boot with boot parameters similar to the following:

Code: Select all

pdrv=sdc2:/slacko/ ydrv=:ydrv_overlay.sfs pfix=nocopy
If you already have a ydrv but no adrv you could specify "adrv=:ydrv_overlay.sfs" instead.
It should boot as pupmode=5 for the first boot. On first shutdown there should be no "shutdownconfig".
If your frugal install directory is on a Linux partition, a reboot should result in pupmode=12.

There is no logical reason why the mount point of a savefile could not be used as the rw directory of the overlay instead of a savefolder.

Because there is no way to modify an overlay mounted as /, and modifying any ro directory in an overlay leads to an "undefined" result, pupmode=13 simply cannot work. So I'm working on a pseudo pupmode=13.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#38 Post by Rangan Masti »

Hi, good afternoon. Testing initrd with my compiled kernel 4.12.0-rc6 with
built-in overlayfs.Thank you for the link last sent. Have a good day.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#39 Post by gyro »

Rangan Masti wrote:Hi, good afternoon. Testing initrd with my compiled kernel 4.12.0-rc6 with
built-in overlayfs.Thank you for the link last sent. Have a good day.
You're welcome, and thanks for testing.
Just for information, overlayfs support can be builtin or a module; if it's a module, the "init" script will "insmod" it.

Just one question, does your kernel have aufs support?
This should work without aufs support, but the puppy kernel's I use for testing all include aufs support. And it's on my todo list to test it using a kernel that doesn't have aufs support. So if it works with your kernel and it doesn't have aufs support, I can cross that test off my list.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: Announcing a new version

#40 Post by gyro »

Moose On The Loose wrote:Yes, if a modified stack can be done, it seems like a faster way to get to a changed system. It may take some trickery to make the changes up to that point stay but not override the new SFS.
There is no problem creating a new overlay stack that includes the current rw directory and a different set of ro directories. The problem is getting that new overlay stack to replace the current /, in a running system.
So, I'm putting this into the too hard basket for now.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#41 Post by Rangan Masti »

Hi good evening. Thank you for the response. My compiled kernel has no aufs module in it.

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#42 Post by peebee »

gyro wrote:It should boot as pupmode=5 for the first boot. On first shutdown there should be no "shutdownconfig".
If your frugal install directory is on a Linux partition, a reboot should result in pupmode=12.
This didn't happen for me....on 1st shutdown the message "Session not saved" flashed up and on reboot it was still pupmode=5.

I renamed the ydrv-overlay to a full ydrv name....it installed correctly....there was no shutdownconfig

Boot:
title Devtest (sda3/devtest)
uuid 95a633e7-6fab-431d-9fa8-550812ab03f8
kernel /devtest/vmlinuz pmedia=atahd pdrv=sda3 psubdir=devtest pfix=fsck
initrd /devtest/initrd.gz
Attachments
Screenshot.png
(75.66 KiB) Downloaded 527 times
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#43 Post by gyro »

Rangan Masti wrote:Hi good evening. Thank you for the response. My compiled kernel has no aufs module in it.
Ok, thanks.
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#44 Post by gyro »

peebee wrote:I renamed the ydrv-overlay to a full ydrv name....it installed correctly....there was no shutdownconfig
Well you had to rename the ydrive, because there is no "ydrv=" parameter in your boot entry to override the default filename.

But you should get pupmode=12.
Where does the symbolic link "/initrd/pup_rw" point to?
It should point to an "/mnt/sda3/devtest/LxPupScsave" directory.

I have this working on LxPupSc 17.06.22.

Edit:
Ah, ah, you also need a "pfix=nocopy" instead of "pfix=fsck" on your boot entry.
Without the "pfix=nocopy" it remains at pupmde=5, all in ram.
With the "pfix=nocopy", since the partition is going to remain mounted anyway, it tries for pupmode=12.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#45 Post by peebee »

gyro wrote:Edit: Ah, ah, you also need a "pfix=nocopy" instead of "pfix=fsck" on your boot entry.
Hi gyro

Is this a temporary fix or a design decision for how the new init will always work?

Putting pfix=nocopy does not seem intuitive to me - I want Puppy loaded into ram to maintain the speed advantages.....I've never used nocopy so I'm vaguely remembering that that's what it means....

Thanks
peebee
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#46 Post by gyro »

peebee wrote:Is this a temporary fix or a design decision for how the new init will always work?

Putting pfix=nocopy does not seem intuitive to me - I want Puppy loaded into ram to maintain the speed advantages.....I've never used nocopy so I'm vaguely remembering that that's what it means..
Sorry, I should make it clear what is actualy part of the feature under test.
The use of pfix=nocopy as the trigger is just a testing convenience, not a new way for puppy to operate.
The possible feature is, that we can have normal pupmode=12 with overlayfs and without any shutdownconfig.
In the next version, pupmode=12 will be triggered by the existence of a save directory, either empty or populated.

Note:
I've never been a fan of the concept of users being presented with shutdownconfig on first shutdown.
I am pursuing the idea of having a puppy that has persistence, without any shutdownconfig. Although there would still need to be a save layer configuration utility for those who like to fiddle.

For the record, "pfix=nocopy" means "don't copy the sfs's to ram".

Gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Updated files

#47 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains modified versions of "shutdownconfig", "rc.update", and "rc.shutdown".

Changes in this 30 June version:

1. The pupmode=12 is now triggered by the existence of a save folder.
If you create a new directory apporpriately named as a save folder immediately after a fresh frugal install, it will boot in pupmode=5 except for using the save folder as it's rw directory.
On reboot it will come up as a normal pupmode=12, (no shutdownconfig involved).

2. There's a new pupmode=21 (5+16), this is automatic persistence following a normal pupmode=5 boot.
Simply do a fresh frugal install, ensure that the boot parameters include ones similar to either "pdrv=sdc2:/puppy/slacko/" or "pdev1=sdc2 psubdir=/puppy/slacko".
Booting will produce the usual first boot.
A reboot should not produce a first boot, but it should contain the changes you made during first boot.
Make further configuration changes and reboot, the changes should be retained.
When you are finished making changes, add a "pfix=nosave" boot parameter.
Now, changes made during a session should be ignored.
This facility should work on any filesystem, I've tested it on ext4 and vfat.

How does this pupmode=21 work:
On shutdown, the contents of the "pup_rw" in ram, are written to an sfs file.
In "init", the contents of this sfs file are copied into "pup_rw" before the main overlay is created.

It all begins with a limitation of overlayfs:
Can't modify any read-only diretory in the stack while it is mounted, (result is "undefined").
This means that "snapmergepuppy" can't do it's thing, in either the running system, or during shutdown, since the main overlay stack is still mounted.
So if we're not running pupmode=12, about the only thing we can do during shutdown, is take a copy of the "pup_rw" in ram, and sort it out in "init" before the main overlay stack is mounted.
The copy of "pup_rw" is an sfs because it can safely store all linux files on any filesystem, and it can be read directly.
pupmode=21 is the simplist thing that "init" can do with this copy of the previous "pup_rw".
The next thing on the todo list is to produce a pupode=29 (13+16), that will be a pseudo pupmode=13.

Note1: Booting with "pfix=ram", will always produce a first boot, ignoring any saved stuff.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#48 Post by peebee »

Looking good.....

Code: Select all

Pup-SysInfo Hardware Report (Summary), Fri 30 Jun 2017

Linux Kernel: 4.11.8-lxpup-32-pae (i686)
Kernel Version: #1 SMP Fri Jun 30 18:06:51 +08 2017
PAE Enabled: Yes

Kernel Command Line:
pmedia=atahd pdrv=sda3 psubdir=devtest pfix=fsck

Distro: LxPup-Sc 17.06.26
Desktop Panel: lxpanel 0.9.3
Window Manager: Openbox 3.6.1
Desktop Start: xwin startlxde

Boot File System: ext4
Boot Media: atahd

PUPMODE=21
PUPSFS=sda3,ext4,/devtest/puppy_LxPupSc_17.06.26.sfs
PUPSAVE=
Booted - rebooted - made wifi connection - rebooted - wifi came up :)
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#49 Post by gyro »

peebee wrote:Booted - rebooted - made wifi connection - rebooted - wifi came up :)
Great.
Thanks for testing.
gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#50 Post by Rangan Masti »

Hi, good evening. Testing your new initrd.All the BEST. :lol:

Post Reply