Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 21 Aug 2017, 02:58
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Another re-write of the "init" script - using OverlayFs?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 5 of 7 [93 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Sat 15 Jul 2017, 06:56    Post subject:  

Rangan Masti wrote:
May be Light House Puppy init could be just an example!!!
Thanks, but on further thought, I've decided not to go this way. It's simple and obvious when setting it up, but there might be some difficulties when trying to remove a live sfs from the running system.
So I've implemented one that uses a list, which unfortunately then needs a utility to maintain that list. But more on that when I get to the point of releasing it.

Note: Many of the things I try, are ideas I've seen in the forum over the years. Many of these have never been incorporated into official puppy. e.g. using sfs files as a save mechanism.

gyro
Back to top
View user's profile Send private message 
Rangan Masti

Joined: 01 Jan 2013
Posts: 30
Location: Germany, Berlin

PostPosted: Sat 15 Jul 2017, 07:50    Post subject:    

Hi, thank you for the reponse. Do not give up!!. Try further. Wish you a good success!!

Rangan
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Mon 17 Jul 2017, 11:35    Post subject: Updated version  

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/initrd/ydrv_overlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains a new utility "extra_sfs", and the command line utilities "file2initrd" and "initrd2file".

The significant change in this 18 July version is, extra sfs support.

The "extra_sfs" utility is a replacement for "sfs_load", in the sense that it masks "sfs_load" and takes its place in the menu.
But it does not load or unload any sfs files.
It simply maintains a list of extra sfs files that are loaded by "init" into the "pup_sfss" overlay, before the main overlay is created.

If "init" fails to load any extra sfs file, an appropriate message is written to "/initrd/tmp/bootinit.log", but the file remains in the list.
You can then either use "extra_sfs" to remove the failing entry, or fix the problem so that it will load successfully next boot.

The list is stored as a variable called "XTRA_SFS" in the "BOOT_SPECS" file.
This file can reside in the frugal install directory or in "initrd.gz", but it must reside in the same place as "DISTRO_SPECS".
So if you use the "file2initrd" utility to move "DISTRO_SPECS" into "initrd.gz" then you should do the same with any "BOOT_SPECS".
Also if you move these files into "initrd.gz", it will confuse "extra_sfs" if copies still exist in the frugal install directory. So either delete them, or rename them.
It's easier to just leave these files in the frugal install directory.

Note on DISTRO_SPECS:
This file defines the puppy you are booting, so the same "initrd.gz" can boot many different puppies provided it has access to the appropriate DISTRO_SPECS.
So, having this file outside "initrd.gz" means the same "initrd.gz" file can be tested with many puppies without change, I've booted both 64bit and 32bit puppies with the same "initrd.gz".
The limitation of the current "initrd.gz" is that the puppies kernel must support overlayfs.

gyro
Back to top
View user's profile Send private message 
Rangan Masti

Joined: 01 Jan 2013
Posts: 30
Location: Germany, Berlin

PostPosted: Mon 17 Jul 2017, 15:40    Post subject:    

Hi, good work !
--Testing!!. Have a nice evening.

Rangan
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2807
Location: Worcestershire, UK

PostPosted: Mon 17 Jul 2017, 18:30    Post subject:  

Looking good here - although I did have an oddity when I did a reboot - I got the saving message but this was followed by a dropout to the terminal prompt and neither reboot nor poweroff actually worked - had to press the power off button....
Code:
# lsoverlay
/initrd/mnt/tmpfs/pup_rw
/initrd/pup_a
/initrd/pup_y
/initrd/pup_ro2
/initrd/pup_f
/initrd/pup_z
/initrd/pup_5
Screenshot.png
 Description   
 Filesize   147.39 KB
 Viewed   235 Time(s)

Screenshot.png


_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Mon 17 Jul 2017, 18:46    Post subject:  

peebee wrote:
Looking good here - although I did have an oddity when I did a reboot - I got the saving message but this was followed by a dropout to the terminal prompt and neither reboot nor poweroff actually worked - had to press the power off button....
Hmm.., I've never seen "rc.shutdown" dropout to a terminal.
Did you try Ctrl+Alt+Delete?
Does it do it every shutdown?
What pupmode are you running? (see /etc/rc.d/PUPSTATE.)

Edit: Never mind the pupmode question, it looks like you are running in pupmode=21.

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2807
Location: Worcestershire, UK

PostPosted: Tue 18 Jul 2017, 06:29    Post subject:  

gyro wrote:
peebee wrote:
Looking good here - although I did have an oddity when I did a reboot - I got the saving message but this was followed by a dropout to the terminal prompt and neither reboot nor poweroff actually worked - had to press the power off button....
Hmm.., I've never seen "rc.shutdown" dropout to a terminal.
Did you try Ctrl+Alt+Delete?
Does it do it every shutdown?
What pupmode are you running? (see /etc/rc.d/PUPSTATE.)

Edit: Never mind the pupmode question, it looks like you are running in pupmode=21.

gyro

Ignore me - it was just an oddity - am unable to replicate/repeat the problem....

One quirk/feature I notice is that the frugal install partition is not mounted after boot which is what you would expect with the aufs initrd.....so for example extra sfs manager does not work until the partition is mounted.....

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Tue 18 Jul 2017, 07:02    Post subject:  

peebee wrote:
Ignore me - it was just an oddity - am unable to replicate/repeat the problem....
I also have been unable to replicate.
I just tried it with a fresh frugal install of LxPupSc-17.07.23T.iso.
peebee wrote:
One quirk/feature I notice is that the frugal install partition is not mounted after boot which is what you would expect with the aufs initrd.....so for example extra sfs manager does not work until the partition is mounted.....
Pupmode=21 is an extension of pupmode=5, everything is in ram, even extra sfs's. So no partitions are mounted by default in the running system.
"extra_sfs" needs direct access to files in the frugal install directory, so it won't run until the partition is mounted manually, in pupmode=5 and pupmode=21.
Code could be added to "extra_sfs" to mount the partition if required, after all "rc.shutdown" has to do just that so it can write the save sfs. I'll think about it.
I tend to avoid this issue by creating an empty save folder, thus switching to pupmode=12 on the next boot.

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2807
Location: Worcestershire, UK

PostPosted: Tue 18 Jul 2017, 07:39    Post subject:  

gyro wrote:
I tend to avoid this issue by creating an empty save folder, thus switching to pupmode=12 on the next boot.

gyro

Ahhhh - hadn't twigged you could / should do that.....

If I do....

what is the ....work directory for?

what is the relationship of the save folder to the ...save.sfs.bak?

Thanks
peebee

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Tue 18 Jul 2017, 08:16    Post subject:  

@peebee,
Before I answer your questions there is just 1 thing:
Adding
Code:
CONFIG_TMPFS_XATTR=y
to your kernel config should improve reliability when tmpfs is used as the rw layer of the main overlay, i.e. pupmode=5,21,29.
I noticed in my recent LxPupSc test that in dmesg, overlayfs complained that the upper fs does not support xattr.
peebee wrote:
what is the ....work directory for?
It's required by overlayfs for it's own internal use.
In pupmode=5,21,29 it's there as "/initrd/mnt/tmpfs/pup_wk".
peebee wrote:
what is the relationship of the save folder to the ...save.sfs.bak
When "init" finds an empty save folder, it copies the content of ...save.ram.sfs to the save folder and then uses that as the rw layer directly. This provides a seamless transition from pupmode=21 to pupmode=12.
After this all changes are written directly to the save folder. No new ...save.ram.sfs is produced and ...save.ram.sfs.bak is ignored.
So ...save.ram.sfs.bak is the sfs that was used to populate the initialy empty save folder.

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2807
Location: Worcestershire, UK

PostPosted: Tue 18 Jul 2017, 11:31    Post subject:  

gyro wrote:
Adding
Code:
CONFIG_TMPFS_XATTR=y
to your kernel config

It's required by overlayfs for it's own internal use.

...save.ram.sfs.bak is ignored.

Thanks....

config changed for next kernel build....

does ....work have to be in the frugal install directory? perhaps it can be in /tmp or deleted at shutdown??

perhaps ....save.sfs.bak can also be deleted when no longer needed?

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Tue 18 Jul 2017, 12:20    Post subject:  

peebee wrote:
config changed for next kernel build....
Thanks, I look forward to that.

peebee wrote:
does ....work have to be in the frugal install directory? perhaps it can be in /tmp or deleted at shutdown??
According to the overlayfs documentation, it has to be in the same file system as the rw directory.
It seemed like a simple idea to always put it beside the rw directory.
It's required, why should it not be obvious?

peebee wrote:
perhaps ....save.sfs.bak can also be deleted when no longer needed?
The ...save.ram.sfs.bak file could be deleted at any time. It's just some backup, it's never actually used.
It's not even necessary to create it. It simply provides an extra recovery option if things go pear-shape. This is still a testing environment.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Fri 21 Jul 2017, 10:57    Post subject:  

peebee wrote:
config changed for next kernel build....
I just tried a fresh frugal install of LxPupSc 10.07.25, and it's 4.12.3 kernel seems to have fixed the issue.
gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1400
Location: Brisbane, Australia

PostPosted: Fri 21 Jul 2017, 14:06    Post subject: A reflection on where it's at  

So what have we learnt from this project?

Issues:

1. It has been reported on the net that there is a problem with using tmpfs as the upper fs for an overlay.
This is of concern because this init script does this for pupmode=5, pupmode=21 and pupmode=29.
Apparently this problem is related to a lack of extended attribute support in tmpfs, which can be fixed by compiling the kernel with "CONFIG_TMPFS_XATTR=y".
Fortunately the kernels in slacko 6.9.9.9 and LxPupSc 17.07.25 meet this requirement. Unfortunately the released kernel in xenialpup 7.0.8.1 does not.
The issue can be avoided by running in pupmoode=12.
Note: I have never experienced any problem, although I did notice recent kernels did complain with "overlayfs: uppper fs does not support xattr" in dmesg.

2. Compared to aufs, overlayfs has some significant limitations, i.e. can't add or delete directories in a stack, can't directly modify the contents of a directory in a stack.
This init script works within these limitations.

Things we can do:

1. We can boot and run normal woof-ce puppies using overlayfs for the stack.
The upside is, overlayfs is supposed to be very efficient, and is supported directly in the kernel, (no aufs patch needed).
The downside is, "sfs_load" is gone, the stack has to be completely created by init.
We can still have extra sfs's, it's just that it takes a reboot, to actually load or unload an sfs.

2. The same initrd.gz can be used to boot many different puppies.
I introduced the facility for init to use a DISTRO_SPECS from outside the initrd.gz to simplify testing.
But it has demonstrated that, just as we can move huge-kernels from one puppy to another, it's possible to move initrd.gz's from one puppy to another.
So I can take this initrd.gz, push my "TIME_ZONE" file into it, and expect to be able to boot any woof-ce puppy with it, after extracting the puppies DISTRO_SPECS from the release initrd.gz.

3. We can easily specify things like extra sfs's before first boot, using a BOOT_SPECS file, beside the DISTRO_SPECS file.
(And any specified extra sfs's will be loaded by first boot.)

4. We can have a puppy that has automatic persistence, without shutdownconfig.
Of course pupmode=21 has significant limitations, i.e. it all has to fit back into the tmpfs on next boot.
So it would not be recommended for use as a "work" puppy.

Other new things:

1. Enable save folder by simply creating an appropriately named directory.
If done before first boot, e.g. as part of installation, it will be ignored until first reboot.

2. "pfix=nosave" boot parameter, disables automatic saving at shutdown in pupmode=5, pupmode=21 and pupmode=29, i.e. when /initrd/mnt/tmpfs/pup_rw is the rw directory of the main overlay.

3. "pfix=rosave" boot parameter, use a read-only save layer, i.e. choose pupmode=29 instead of pupmode=12.
Pupmode=29 is a replacement for pupmode=13, but I didn't want to confuse puppy utilities like snapmergepuppy.

4. "pfix=nobs" boot parameter, disable processing of BOOT_SPECS file.

5. "pfix=nofsckp" boot parameter, disable fsck of each partition before first mount.
The previous "pfix=fsckp" processing is now the default.

gyro
Back to top
View user's profile Send private message 
woodenshoe-wi

Joined: 28 Jul 2017
Posts: 6
Location: Wisconsin

PostPosted: Sat 29 Jul 2017, 00:33    Post subject: RE: A reflection on where it's at  

I just wanted to let you know that after replacing the 386 binaries in the initrd.gz with ARM binaries I was able to use your init script on my Pi2 (1GB RAM) with only two small modifications.

Code:

--- original_init   2017-07-17 09:25:51.000000000 -0500
+++ edited_init   2017-07-19 13:02:38.000000000 -0500
@@ -574,6 +574,12 @@
 [ -d /mnt/tmpfs ] || mkdir -p /mnt/tmpfs
 mount -t tmpfs tmpfs /mnt/tmpfs
 
+#raspberry pi, load squashfs kernel module
+[ -d /mnt/boot] || mkdir -p /mnt/boot
+mount /dev/mmcblk0p1 /mnt/boot
+insmod "/mnt/boot/modules/$KERNELVER/squashfs.ko"
+umount /mnt/boot
+
 #mount pdrv
 mount_sfs "$PDRV_SPEC" "pdrv"
 [ $? -eq 0 ] || fatal_error "$mount_sfs_RET"
@@ -615,7 +621,9 @@
    fi
 fi
 ZDRV_MNTPT=''
-sortout_sfs_spec "$DISTRO_ZDRVSFS" "$ZDRV"
+#sortout_sfs_spec "$DISTRO_ZDRVSFS" "$ZDRV"
+#raspberry pi, need to support two different zdrvs for armv6 and armv7 kernels
+sortout_sfs_spec "zdrv_raspup_$(uname -r).sfs" "mmcblk0p1:/zdrv_raspup_$(uname -r).sfs"
 if [ "$sortout_sfs_spec_RET" ];then
    ZDRV_SPEC="$sortout_sfs_spec_RET"
    mount_sfs "$ZDRV_SPEC" "zdrv"


Is this re-write of the init script going to eventually be merged with the init in woof-CE, or is it a separate experiment?

I'm not sure if the Pi1/Pi0 has enough RAM (512MB) to run completely in RAM, but it opens the possibility to running frugally in a FAT formatted SD card without reformatting/writing an image file to it.

I am planning to (eventually) figure out how to cross compile a kernel with AUFS support, but it is nice to already be able to play with the pre-compiled kernels.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 5 of 7 [93 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0622s ][ Queries: 15 (0.0059s) ][ GZIP on ]