Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#46 Post by gyro »

peebee wrote:Is this a temporary fix or a design decision for how the new init will always work?

Putting pfix=nocopy does not seem intuitive to me - I want Puppy loaded into ram to maintain the speed advantages.....I've never used nocopy so I'm vaguely remembering that that's what it means..
Sorry, I should make it clear what is actualy part of the feature under test.
The use of pfix=nocopy as the trigger is just a testing convenience, not a new way for puppy to operate.
The possible feature is, that we can have normal pupmode=12 with overlayfs and without any shutdownconfig.
In the next version, pupmode=12 will be triggered by the existence of a save directory, either empty or populated.

Note:
I've never been a fan of the concept of users being presented with shutdownconfig on first shutdown.
I am pursuing the idea of having a puppy that has persistence, without any shutdownconfig. Although there would still need to be a save layer configuration utility for those who like to fiddle.

For the record, "pfix=nocopy" means "don't copy the sfs's to ram".

Gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Updated files

#47 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains modified versions of "shutdownconfig", "rc.update", and "rc.shutdown".

Changes in this 30 June version:

1. The pupmode=12 is now triggered by the existence of a save folder.
If you create a new directory apporpriately named as a save folder immediately after a fresh frugal install, it will boot in pupmode=5 except for using the save folder as it's rw directory.
On reboot it will come up as a normal pupmode=12, (no shutdownconfig involved).

2. There's a new pupmode=21 (5+16), this is automatic persistence following a normal pupmode=5 boot.
Simply do a fresh frugal install, ensure that the boot parameters include ones similar to either "pdrv=sdc2:/puppy/slacko/" or "pdev1=sdc2 psubdir=/puppy/slacko".
Booting will produce the usual first boot.
A reboot should not produce a first boot, but it should contain the changes you made during first boot.
Make further configuration changes and reboot, the changes should be retained.
When you are finished making changes, add a "pfix=nosave" boot parameter.
Now, changes made during a session should be ignored.
This facility should work on any filesystem, I've tested it on ext4 and vfat.

How does this pupmode=21 work:
On shutdown, the contents of the "pup_rw" in ram, are written to an sfs file.
In "init", the contents of this sfs file are copied into "pup_rw" before the main overlay is created.

It all begins with a limitation of overlayfs:
Can't modify any read-only diretory in the stack while it is mounted, (result is "undefined").
This means that "snapmergepuppy" can't do it's thing, in either the running system, or during shutdown, since the main overlay stack is still mounted.
So if we're not running pupmode=12, about the only thing we can do during shutdown, is take a copy of the "pup_rw" in ram, and sort it out in "init" before the main overlay stack is mounted.
The copy of "pup_rw" is an sfs because it can safely store all linux files on any filesystem, and it can be read directly.
pupmode=21 is the simplist thing that "init" can do with this copy of the previous "pup_rw".
The next thing on the todo list is to produce a pupode=29 (13+16), that will be a pseudo pupmode=13.

Note1: Booting with "pfix=ram", will always produce a first boot, ignoring any saved stuff.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#48 Post by peebee »

Looking good.....

Code: Select all

Pup-SysInfo Hardware Report (Summary), Fri 30 Jun 2017

Linux Kernel: 4.11.8-lxpup-32-pae (i686)
Kernel Version: #1 SMP Fri Jun 30 18:06:51 +08 2017
PAE Enabled: Yes

Kernel Command Line:
pmedia=atahd pdrv=sda3 psubdir=devtest pfix=fsck

Distro: LxPup-Sc 17.06.26
Desktop Panel: lxpanel 0.9.3
Window Manager: Openbox 3.6.1
Desktop Start: xwin startlxde

Boot File System: ext4
Boot Media: atahd

PUPMODE=21
PUPSFS=sda3,ext4,/devtest/puppy_LxPupSc_17.06.26.sfs
PUPSAVE=
Booted - rebooted - made wifi connection - rebooted - wifi came up :)
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#49 Post by gyro »

peebee wrote:Booted - rebooted - made wifi connection - rebooted - wifi came up :)
Great.
Thanks for testing.
gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#50 Post by Rangan Masti »

Hi, good evening. Testing your new initrd.All the BEST. :lol:

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#51 Post by gyro »

@Rangan Masti,
Thanks.
gyro

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#52 Post by Keef »

I've tried this with the current 32bit Slacko (RC3). It works as advertised (well once I read the instructions properly), but as well as the manually created save directory, there is a 'slackowork' directory. Inside that is another empty directory called 'work''.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#53 Post by gyro »

Keef wrote:but as well as the manually created save directory, there is a 'slackowork' directory. Inside that is another empty directory called 'work''.
Yep, that's a directoty required by overlayfs.
If you "cat /proc/mounts" and look at the line for the overlay mounted on /, you will see the "workdir" mentioned.
Thanks for testing.

PS: Have you tried not creating a save directory?
Just do a clean frugal install and change stuff, and reboot, and change stuff, and reboot.
You should notice that there is persistence, even when running completely in ram, pupmode=21. This is a facility we don't have in current puppy.

gyro

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#54 Post by Keef »

I'd rather stupidly forgot to rename the ydrv :oops: I was wandering why I was being prompted to save when I shutdown. Which I didn't do, but I did get persistence if I made savefolder myself. Need to pay more attention.

Having done what I should have done in the first place, yes it does save changes without any intervention from me. Posting from the install now, after first reboot.

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#55 Post by Keef »

I've been experimenting with loading sfs files. First used a small sfs for the Alacritty terminal. Get the warning that Pupmode 21 is experimental, but it works ok. Then tried the devx. Took quite a while to load (compared to the normal situation), but it still worked correctly.
Both stay loaded after reboot.
The most noticable thing is that with a large sfs like the devx installed, saving takes a lot longer. I assume that is because of the time it takes the slackosave.ram.sfs to be built.
No option with sfs_load to uninstall the sfs, but I suppose this could be done manually - I haven't tried yet.
[EDIT] Scratch that - it eventually dawned on my why that is not possible.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#56 Post by gyro »

@Keef,
I'm surprised loading extra sfs's did anything, no aufs stack.
And I haven't done any overlayfs support for extra sfs's.

If you were really desperate to load the devx, you could rename the zdrv to the fdrv and rename the devx to zdrv, or some similar fiddle.
Note: if you try doing anything like this, don't load the devx as an adrv or ydrv, it should always go under the main puppy...sfs.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Announcing a new version

#57 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since this version of "ydrv_overlay.sfs" contains an updated version of "rc.shutdown", and the command line utilities "lsoverlay" and "showoverlay".

Changes in this 3 July version:

1. Contains a new pupmode=29 a pseudo pupmode=13 that uses tmpfs for the rw directory and the save folder is the top ro directory. I'm not using pupmode=13 for this, so as not to confuse the running puppy, e.g. "snapmergepuppy".
2. Includes "pfix=rosave" to control use of pupmode=29 instead of pupmode=12, and "pfix=nosave" to stop saving during shutdown.

During first boot, any empty save folder is ignored, the boot proceeds as a normal pupmode=5 boot, all in ram.
During second boot:
If the first boot included "pfix=nosave" another first boot will be performed.
Else if there is no save folder present it will boot pupmode=21, all in ram, but remembering changes made in the previous session.
Else if a save folder exists and "pfix=rosave" is specified it will boot pupmode=29.
Else if a save folder exists, it will boot pupmode=12.

During any boot, if the conditions for one of pupmode=29, pupmode=12 or pupmode=21 are not met, it will do a normal pupmode=5 boot.

Since with overlayfs, there can be no writing to any ro directory while the main overlay is mounted, "snapmergepuppy" type processing is done by "init".
As per the previous version the only saving done in "rc.shutdown", is to create an sfs file on disk, from the "pup_rw" in ram, unless "pfix=nosave" has been specified.

"pfix=nosave" has no effect on pupmode=12, because all writing has already been done before shutdown.

Overlayfs documentation says that a directory on any filesystem supported by Linux can be used as a read-only directory.
So theoretically pupmode=29 should be able to use a save folder any any filesystem, if the "snapmergepuppy" type code in "init" was careful about what files were written to disk.
I tried this with a directory on a "vfat" partition and the overlay mount failed.
I tried this with a directory on an "ntfs" partition and the overlay mount worked.
But the result was quite annoying, in ROX filer, a single click on a text document appeared to do nothing, since the file had "execute permission" set, ROX filer executed it.
I then abandoned this concept, the pupmode=29 implemented here only works on Linux filesystems.

Running "lsoverlay" in a console will produce a list of the directories in the "/" overlay, in the order top to bottom.
"showoverlay" goes one step further and unwinds any directories that are mountpoints for loop devices to show the mounted files.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#58 Post by peebee »

sfs loading next....??

Code: Select all

# lsoverlay 
/initrd/mnt/tmpfs/pup_rw
/initrd/pup_a
/initrd/pup_y
/initrd/pup_ro2
/initrd/pup_f
/initrd/pup_z
# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 181.1M  1 loop /initrd/pup_ro2
loop1    7:1    0  20.6M  1 loop /initrd/pup_a
loop2    7:2    0    28K  1 loop /initrd/pup_y
loop3    7:3    0  15.6M  1 loop /initrd/pup_f
loop4    7:4    0  22.9M  1 loop /initrd/pup_z
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#59 Post by gyro »

peebee wrote:sfs loading next....??
Yes.
I'm currently thinking of an "init" that automatically loads any extra sfs's found in the frugal install directory.
gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#60 Post by Rangan Masti »

Hi, Good morning.
May be Light House Puppy init could be just an example!!!
wish you good success.

Rangan

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#61 Post by gyro »

Rangan Masti wrote:May be Light House Puppy init could be just an example!!!
Thanks, but on further thought, I've decided not to go this way. It's simple and obvious when setting it up, but there might be some difficulties when trying to remove a live sfs from the running system.
So I've implemented one that uses a list, which unfortunately then needs a utility to maintain that list. But more on that when I get to the point of releasing it.

Note: Many of the things I try, are ideas I've seen in the forum over the years. Many of these have never been incorporated into official puppy. e.g. using sfs files as a save mechanism.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#62 Post by Rangan Masti »

Hi, thank you for the reponse. Do not give up!!. Try further. Wish you a good success!!

Rangan

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Updated version

#63 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains a new utility "extra_sfs", and the command line utilities "file2initrd" and "initrd2file".

The significant change in this 18 July version is, extra sfs support.

The "extra_sfs" utility is a replacement for "sfs_load", in the sense that it masks "sfs_load" and takes its place in the menu.
But it does not load or unload any sfs files.
It simply maintains a list of extra sfs files that are loaded by "init" into the "pup_sfss" overlay, before the main overlay is created.

If "init" fails to load any extra sfs file, an appropriate message is written to "/initrd/tmp/bootinit.log", but the file remains in the list.
You can then either use "extra_sfs" to remove the failing entry, or fix the problem so that it will load successfully next boot.

The list is stored as a variable called "XTRA_SFS" in the "BOOT_SPECS" file.
This file can reside in the frugal install directory or in "initrd.gz", but it must reside in the same place as "DISTRO_SPECS".
So if you use the "file2initrd" utility to move "DISTRO_SPECS" into "initrd.gz" then you should do the same with any "BOOT_SPECS".
Also if you move these files into "initrd.gz", it will confuse "extra_sfs" if copies still exist in the frugal install directory. So either delete them, or rename them.
It's easier to just leave these files in the frugal install directory.

Note on DISTRO_SPECS:
This file defines the puppy you are booting, so the same "initrd.gz" can boot many different puppies provided it has access to the appropriate DISTRO_SPECS.
So, having this file outside "initrd.gz" means the same "initrd.gz" file can be tested with many puppies without change, I've booted both 64bit and 32bit puppies with the same "initrd.gz".
The limitation of the current "initrd.gz" is that the puppies kernel must support overlayfs.

gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#64 Post by Rangan Masti »

Hi, good work !
--Testing!!. Have a nice evening.

Rangan

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#65 Post by peebee »

Looking good here - although I did have an oddity when I did a reboot - I got the saving message but this was followed by a dropout to the terminal prompt and neither reboot nor poweroff actually worked - had to press the power off button....

Code: Select all

# lsoverlay
/initrd/mnt/tmpfs/pup_rw
/initrd/pup_a
/initrd/pup_y
/initrd/pup_ro2
/initrd/pup_f
/initrd/pup_z
/initrd/pup_5
Attachments
Screenshot.png
(147.39 KiB) Downloaded 281 times
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

Post Reply