Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#41 Post by Rangan Masti »

Hi good evening. Thank you for the response. My compiled kernel has no aufs module in it.

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#42 Post by peebee »

gyro wrote:It should boot as pupmode=5 for the first boot. On first shutdown there should be no "shutdownconfig".
If your frugal install directory is on a Linux partition, a reboot should result in pupmode=12.
This didn't happen for me....on 1st shutdown the message "Session not saved" flashed up and on reboot it was still pupmode=5.

I renamed the ydrv-overlay to a full ydrv name....it installed correctly....there was no shutdownconfig

Boot:
title Devtest (sda3/devtest)
uuid 95a633e7-6fab-431d-9fa8-550812ab03f8
kernel /devtest/vmlinuz pmedia=atahd pdrv=sda3 psubdir=devtest pfix=fsck
initrd /devtest/initrd.gz
Attachments
Screenshot.png
(75.66 KiB) Downloaded 527 times
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#43 Post by gyro »

Rangan Masti wrote:Hi good evening. Thank you for the response. My compiled kernel has no aufs module in it.
Ok, thanks.
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#44 Post by gyro »

peebee wrote:I renamed the ydrv-overlay to a full ydrv name....it installed correctly....there was no shutdownconfig
Well you had to rename the ydrive, because there is no "ydrv=" parameter in your boot entry to override the default filename.

But you should get pupmode=12.
Where does the symbolic link "/initrd/pup_rw" point to?
It should point to an "/mnt/sda3/devtest/LxPupScsave" directory.

I have this working on LxPupSc 17.06.22.

Edit:
Ah, ah, you also need a "pfix=nocopy" instead of "pfix=fsck" on your boot entry.
Without the "pfix=nocopy" it remains at pupmde=5, all in ram.
With the "pfix=nocopy", since the partition is going to remain mounted anyway, it tries for pupmode=12.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#45 Post by peebee »

gyro wrote:Edit: Ah, ah, you also need a "pfix=nocopy" instead of "pfix=fsck" on your boot entry.
Hi gyro

Is this a temporary fix or a design decision for how the new init will always work?

Putting pfix=nocopy does not seem intuitive to me - I want Puppy loaded into ram to maintain the speed advantages.....I've never used nocopy so I'm vaguely remembering that that's what it means....

Thanks
peebee
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#46 Post by gyro »

peebee wrote:Is this a temporary fix or a design decision for how the new init will always work?

Putting pfix=nocopy does not seem intuitive to me - I want Puppy loaded into ram to maintain the speed advantages.....I've never used nocopy so I'm vaguely remembering that that's what it means..
Sorry, I should make it clear what is actualy part of the feature under test.
The use of pfix=nocopy as the trigger is just a testing convenience, not a new way for puppy to operate.
The possible feature is, that we can have normal pupmode=12 with overlayfs and without any shutdownconfig.
In the next version, pupmode=12 will be triggered by the existence of a save directory, either empty or populated.

Note:
I've never been a fan of the concept of users being presented with shutdownconfig on first shutdown.
I am pursuing the idea of having a puppy that has persistence, without any shutdownconfig. Although there would still need to be a save layer configuration utility for those who like to fiddle.

For the record, "pfix=nocopy" means "don't copy the sfs's to ram".

Gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Updated files

#47 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains modified versions of "shutdownconfig", "rc.update", and "rc.shutdown".

Changes in this 30 June version:

1. The pupmode=12 is now triggered by the existence of a save folder.
If you create a new directory apporpriately named as a save folder immediately after a fresh frugal install, it will boot in pupmode=5 except for using the save folder as it's rw directory.
On reboot it will come up as a normal pupmode=12, (no shutdownconfig involved).

2. There's a new pupmode=21 (5+16), this is automatic persistence following a normal pupmode=5 boot.
Simply do a fresh frugal install, ensure that the boot parameters include ones similar to either "pdrv=sdc2:/puppy/slacko/" or "pdev1=sdc2 psubdir=/puppy/slacko".
Booting will produce the usual first boot.
A reboot should not produce a first boot, but it should contain the changes you made during first boot.
Make further configuration changes and reboot, the changes should be retained.
When you are finished making changes, add a "pfix=nosave" boot parameter.
Now, changes made during a session should be ignored.
This facility should work on any filesystem, I've tested it on ext4 and vfat.

How does this pupmode=21 work:
On shutdown, the contents of the "pup_rw" in ram, are written to an sfs file.
In "init", the contents of this sfs file are copied into "pup_rw" before the main overlay is created.

It all begins with a limitation of overlayfs:
Can't modify any read-only diretory in the stack while it is mounted, (result is "undefined").
This means that "snapmergepuppy" can't do it's thing, in either the running system, or during shutdown, since the main overlay stack is still mounted.
So if we're not running pupmode=12, about the only thing we can do during shutdown, is take a copy of the "pup_rw" in ram, and sort it out in "init" before the main overlay stack is mounted.
The copy of "pup_rw" is an sfs because it can safely store all linux files on any filesystem, and it can be read directly.
pupmode=21 is the simplist thing that "init" can do with this copy of the previous "pup_rw".
The next thing on the todo list is to produce a pupode=29 (13+16), that will be a pseudo pupmode=13.

Note1: Booting with "pfix=ram", will always produce a first boot, ignoring any saved stuff.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#48 Post by peebee »

Looking good.....

Code: Select all

Pup-SysInfo Hardware Report (Summary), Fri 30 Jun 2017

Linux Kernel: 4.11.8-lxpup-32-pae (i686)
Kernel Version: #1 SMP Fri Jun 30 18:06:51 +08 2017
PAE Enabled: Yes

Kernel Command Line:
pmedia=atahd pdrv=sda3 psubdir=devtest pfix=fsck

Distro: LxPup-Sc 17.06.26
Desktop Panel: lxpanel 0.9.3
Window Manager: Openbox 3.6.1
Desktop Start: xwin startlxde

Boot File System: ext4
Boot Media: atahd

PUPMODE=21
PUPSFS=sda3,ext4,/devtest/puppy_LxPupSc_17.06.26.sfs
PUPSAVE=
Booted - rebooted - made wifi connection - rebooted - wifi came up :)
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#49 Post by gyro »

peebee wrote:Booted - rebooted - made wifi connection - rebooted - wifi came up :)
Great.
Thanks for testing.
gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#50 Post by Rangan Masti »

Hi, good evening. Testing your new initrd.All the BEST. :lol:

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#51 Post by gyro »

@Rangan Masti,
Thanks.
gyro

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#52 Post by Keef »

I've tried this with the current 32bit Slacko (RC3). It works as advertised (well once I read the instructions properly), but as well as the manually created save directory, there is a 'slackowork' directory. Inside that is another empty directory called 'work''.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#53 Post by gyro »

Keef wrote:but as well as the manually created save directory, there is a 'slackowork' directory. Inside that is another empty directory called 'work''.
Yep, that's a directoty required by overlayfs.
If you "cat /proc/mounts" and look at the line for the overlay mounted on /, you will see the "workdir" mentioned.
Thanks for testing.

PS: Have you tried not creating a save directory?
Just do a clean frugal install and change stuff, and reboot, and change stuff, and reboot.
You should notice that there is persistence, even when running completely in ram, pupmode=21. This is a facility we don't have in current puppy.

gyro

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#54 Post by Keef »

I'd rather stupidly forgot to rename the ydrv :oops: I was wandering why I was being prompted to save when I shutdown. Which I didn't do, but I did get persistence if I made savefolder myself. Need to pay more attention.

Having done what I should have done in the first place, yes it does save changes without any intervention from me. Posting from the install now, after first reboot.

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#55 Post by Keef »

I've been experimenting with loading sfs files. First used a small sfs for the Alacritty terminal. Get the warning that Pupmode 21 is experimental, but it works ok. Then tried the devx. Took quite a while to load (compared to the normal situation), but it still worked correctly.
Both stay loaded after reboot.
The most noticable thing is that with a large sfs like the devx installed, saving takes a lot longer. I assume that is because of the time it takes the slackosave.ram.sfs to be built.
No option with sfs_load to uninstall the sfs, but I suppose this could be done manually - I haven't tried yet.
[EDIT] Scratch that - it eventually dawned on my why that is not possible.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#56 Post by gyro »

@Keef,
I'm surprised loading extra sfs's did anything, no aufs stack.
And I haven't done any overlayfs support for extra sfs's.

If you were really desperate to load the devx, you could rename the zdrv to the fdrv and rename the devx to zdrv, or some similar fiddle.
Note: if you try doing anything like this, don't load the devx as an adrv or ydrv, it should always go under the main puppy...sfs.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Announcing a new version

#57 Post by gyro »

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/ini ... verlay.sfs.
Both need to be downloaded, since this version of "ydrv_overlay.sfs" contains an updated version of "rc.shutdown", and the command line utilities "lsoverlay" and "showoverlay".

Changes in this 3 July version:

1. Contains a new pupmode=29 a pseudo pupmode=13 that uses tmpfs for the rw directory and the save folder is the top ro directory. I'm not using pupmode=13 for this, so as not to confuse the running puppy, e.g. "snapmergepuppy".
2. Includes "pfix=rosave" to control use of pupmode=29 instead of pupmode=12, and "pfix=nosave" to stop saving during shutdown.

During first boot, any empty save folder is ignored, the boot proceeds as a normal pupmode=5 boot, all in ram.
During second boot:
If the first boot included "pfix=nosave" another first boot will be performed.
Else if there is no save folder present it will boot pupmode=21, all in ram, but remembering changes made in the previous session.
Else if a save folder exists and "pfix=rosave" is specified it will boot pupmode=29.
Else if a save folder exists, it will boot pupmode=12.

During any boot, if the conditions for one of pupmode=29, pupmode=12 or pupmode=21 are not met, it will do a normal pupmode=5 boot.

Since with overlayfs, there can be no writing to any ro directory while the main overlay is mounted, "snapmergepuppy" type processing is done by "init".
As per the previous version the only saving done in "rc.shutdown", is to create an sfs file on disk, from the "pup_rw" in ram, unless "pfix=nosave" has been specified.

"pfix=nosave" has no effect on pupmode=12, because all writing has already been done before shutdown.

Overlayfs documentation says that a directory on any filesystem supported by Linux can be used as a read-only directory.
So theoretically pupmode=29 should be able to use a save folder any any filesystem, if the "snapmergepuppy" type code in "init" was careful about what files were written to disk.
I tried this with a directory on a "vfat" partition and the overlay mount failed.
I tried this with a directory on an "ntfs" partition and the overlay mount worked.
But the result was quite annoying, in ROX filer, a single click on a text document appeared to do nothing, since the file had "execute permission" set, ROX filer executed it.
I then abandoned this concept, the pupmode=29 implemented here only works on Linux filesystems.

Running "lsoverlay" in a console will produce a list of the directories in the "/" overlay, in the order top to bottom.
"showoverlay" goes one step further and unwinds any directories that are mountpoints for loop devices to show the mounted files.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#58 Post by peebee »

sfs loading next....??

Code: Select all

# lsoverlay 
/initrd/mnt/tmpfs/pup_rw
/initrd/pup_a
/initrd/pup_y
/initrd/pup_ro2
/initrd/pup_f
/initrd/pup_z
# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 181.1M  1 loop /initrd/pup_ro2
loop1    7:1    0  20.6M  1 loop /initrd/pup_a
loop2    7:2    0    28K  1 loop /initrd/pup_y
loop3    7:3    0  15.6M  1 loop /initrd/pup_f
loop4    7:4    0  22.9M  1 loop /initrd/pup_z
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#59 Post by gyro »

peebee wrote:sfs loading next....??
Yes.
I'm currently thinking of an "init" that automatically loads any extra sfs's found in the frugal install directory.
gyro

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#60 Post by Rangan Masti »

Hi, Good morning.
May be Light House Puppy init could be just an example!!!
wish you good success.

Rangan

Post Reply