Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 23 Nov 2017, 22:15
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Another re-write of the "init" script - using OverlayFs?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 6 of 7 [96 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Sat 29 Jul 2017, 23:16    Post subject: Re: RE: A reflection on where it's at  

woodenshoe-wi wrote:
I just wanted to let you know that after replacing the 386 binaries in the initrd.gz with ARM binaries I was able to use your init script on my Pi2 (1GB RAM) with only two small modifications.
Thanks.
The latest version contains even less binaries.
woodenshoe-wi wrote:
Is this re-write of the init script going to eventually be merged with the init in woof-CE, or is it a separate experiment?
I don't know.
When I get to the end, i.e. an "initrd.gz" I can use for my every day puppy, including all my bright ideas that worked, we will have 3 options.
1. Replace the current "init" in woof-ce with one based on this one.
2. Port some of the stuff into the current woof-ce "init".
3. Leave it is as a stand alone "initrd.gz".
Whatever happens, since I can boot "off-the-rack" puppies with it, I intend to use it with my every day woof-ce based puppy.

gyro

Last edited by gyro on Sat 29 Jul 2017, 23:22; edited 1 time in total
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Sat 29 Jul 2017, 23:20    Post subject: Updated version  

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/initrd/ydrv_overlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains an updated "extra_sfs", updated "initmodules" and new bootspecs utilities.

Changes in this 30 July version:

1. "extra_sfs" now works in puppy with no mounted partitions, i.e. pupmode=5 and pupmode=21.
It will automatically mount the partition containing puppy...sfs so sfs files can be selected, and then umount it on exit.
If you want to include sfs files from other partitions you will have to manually mount these partitions before running "extra_sfs".

2. "initmodules" now works. It stores a "PIMOD=" parameter directly in BOOT_SPECS.

3. Both "extra_sfs" and "initmodules" use the new BOOT_SPECS utilities, "bootspecs2tmpbs", "variable2tmpbs", and "tmpbs2bootspecs". These are included in
"bootspecs2tmpbs" retrives BOOT_SPECS, from inside the initrd.gz if necessary, and stores it in /tmp.
"variable2tmpbs" is called by utilities like "initmodules" to store their data in BOOT_SPECS.
Code:
variable2tmpbs -q PIMOD "$MODFILELIST"
"variable2tmpbs" calls "bootspecs2tmpbs" if necessary.
"tmpbs2bootspecs" restores BOOT_SPECS to it's nornal place, inside initrd.gz if necessary. It's automatically run by rc.shutdown.
Running any of these utilities with a "-h" parameter will display some usage information.

4. The "init" script now uses "time of last modification as seconds since Epoch" to specify each sfs in BOOTCONFIG.
A deficiency in the current system is that if an sfs is replaced by one with the same name but differrent contents, rc.update will not see the difference.
rc.update does not care abouit the names of the sfs's, it just checks if the provided information is different from the previous one. e.g. replace the adrv with one containing a different set of applications.

5. The "init" script has some improvements in the messages it produces. The output from fsck is now sent to "/initrd/tmp/bootinit.log" as well as the console.
It's also been trimmed by removing some unused messages.

6. The "initrd.gz" has been further trimmed by removing further unused binaries, so it is a smaller download.

Edit: Sorry I forgot to mention that the ydrv_overlay.sfs now includes a 32bit yad. I got tired of "extra_sfs"->"Add" not opening in the correct directory with older versions of yad.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Sun 30 Jul 2017, 00:37    Post subject: overlayfs, tmpfs, and the kernel  

overlayfs, tmpfs, and the kernel:

I have read reports on the net that overlayfs is unreliable, and in particular when using tmpfs as the top (read-write) directory.
I have also read reports on the net that tmpfs is ok if it is configured in the kernel with xattr support.

My experience with this "initrd.gz":
I have had no reliability problems in the many, many boots of the 4 different puppies I have used in testing.
I am aware of only 1 system hang using tmpfs as the top directory, but that was using a kernel that was not configured with xattr support for tmpfs.

Unfortunately the only puppy containing a kernel configured with both the following:
Code:
CONFIG_OVERLAY_FS=m
CONFIG_TMPFS_XATTR=y
is LxPupSc 17.07.25.
So I've done a bit of kernel compiling using the kernel-kit from woof-ce.
I compiled an appropriately configured 4.9.39 32bit PAE kernel in xenialpup 7.0.8.1.
I'm currently successfully using this kernel to test this "initrd.gz" on xenialpup 7.0.8.1 and tahr 6.0.6.
I'm testing slacko 6.9.9.9 in pupmode=12 only, thus avoiding using tmpfs as the top directory. Slacko's kernel does not have xattr support for tmpfs.
I have not tried using the LxPupSc 4.12.3 kernel with slacko.

Note: This kernel package does not contain any firmware, you need to supply an fdrv.
I use http://www.fishprogs.software/puppy/firmware-sfs/linux_firmware_20170627.sfs.

Downloads:
http://www.fishprogs.software/puppy/initrd//huge-4.9.39-gyro-pae.tar.bz2
http://www.fishprogs.software/puppy/initrd//huge-4.9.39-gyro-pae.tar.bz2.md5.txt
http://www.fishprogs.software/puppy/initrd//huge-4.9.39-gyro-pae.tar.bz2.sha256.txt

gyro
Back to top
View user's profile Send private message 
woodenshoe-wi

Joined: 28 Jul 2017
Posts: 7
Location: Wisconsin

PostPosted: Sun 30 Jul 2017, 15:13    Post subject:  

I will have to modify you new version for ARM use and try it out Very Happy

I added yad to DISTRO_PKGS_SPECS-raspbian-stretch so I don't need to install it but I need the ARM version of course.

After realizing that I forgot to copy the right version kernel to the SD card Embarassed I have my Pi1 booting from a SD card that still has its factory FAT formatting, no repartitioning, no ext2 filesystem, no need to write an image file to it... So I am excited about that. A big thanks for your work on overlayfs support!

Two feature requests, or maybe I should try to implement this in an ARM fork of your init,

- A raspberry pi does not have a bootloader that offers any type of menu, so having the ability to choose which installation to boot using a config file to make a boot menu would be nice.

- A pi1/pi0 only has 512MB RAM and can't load any sfs files to RAM anyway, so the SD card will have to remain mounted, but on boot if I understand your init script correctly the entire content of raspupsave.ram.sfs gets copied to RAM. Having support for a pupsave.2fs file would get around this limitation but on a SD card or USB stick it should have a tmpfs overlay which means snapmergepuppy which means AUFS... I did manage to modify kernel-kit enough to get it to cross compile a kernel yesterday, but I haven't tested if it actually works.

I don't know if you want to add AUFS support to your OverlayFs based init, add OverlayFs support to the AUFS based woof-ce init, or if it would be best to have a separate init for ARM use that would have parts from both init scripts. I would like to always be able to boot even if the kernel doesn't have AUFS support, but when trying to keep RAM usage down AUFS could be helpful.
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Mon 31 Jul 2017, 07:04    Post subject:  

woodenshoe-wi wrote:
Two feature requests, or maybe I should try to implement this in an ARM fork of your init,

- A raspberry pi does not have a bootloader that offers any type of menu, so having the ability to choose which installation to boot using a config file to make a boot menu would be nice.
Sorry, I won't be doing this since it is pi specific.
The focus of this new init at the moment is firmly on booting an ordinary PC using a directory on a Linux file system for a save folder. I haven't even started on support for booting from CD yet.
woodenshoe-wi wrote:
- A pi1/pi0 only has 512MB RAM and can't load any sfs files to RAM anyway, so the SD card will have to remain mounted, but on boot if I understand your init script correctly the entire content of raspupsave.ram.sfs gets copied to RAM. Having support for a pupsave.2fs file would get around this limitation but on a SD card or USB stick it should have a tmpfs overlay which means snapmergepuppy which means AUFS... I did manage to modify kernel-kit enough to get it to cross compile a kernel yesterday, but I haven't tested if it actually works.
A savefile doesn't need aufs. There's no reason that this overlayfs init should not support a savefile. The top directory (the rw one), has to be on a Linux file system, this can be provided in many ways, the simplest of these is to use a directory on a Linux partition.
But it could also be provided by mounting the Linux filesystem inside a savefile and using it's mountpoint as the top directory. This is what puppy has been doing for years using aufs and uniionfs.
From my perspective it's just a whole lot of extra code that's required to get to the same point. We already know how to do this.
So, savefile support could be added to this init using overlayfs, but I won't be even considering it until after all the fun of testing new possibilites is finished, and I've still got a couple of those.
woodenshoe-wi wrote:
I don't know if you want to add AUFS support to your OverlayFs based init, add OverlayFs support to the AUFS based woof-ce init, or if it would be best to have a separate init for ARM use that would have parts from both init scripts. I would like to always be able to boot even if the kernel doesn't have AUFS support, but when trying to keep RAM usage down AUFS could be helpful.
I have thought of a hybrid overlayfs/aufs init, but for a rather different reason.
Overlayfs is supposed to be very efficient, so it might have better performance with lots of layers, and it supports creating read-only overlays with no read-write layer, (aufs can't do that), but it doesn't support changing the layers in the stack. So I have thought of loading all the sfs's into a read-only overlay and then using this as the bottom layer in a 2 layer aufs stack. Then we could still use "sfs_load on the fly".
Although I doubt that this will come to fruition since one of the drivers behind exploring overlayfs is the possibility that the day will come when patching the kernel with aufs will no longer work.
On the other hand, overlayfs is in the kernel, so we are not putting ourselves at any more risk by using a hybrid than we are by using aufs only.

gyro
Back to top
View user's profile Send private message 
woodenshoe-wi

Joined: 28 Jul 2017
Posts: 7
Location: Wisconsin

PostPosted: Mon 31 Jul 2017, 15:52    Post subject:  

Ok, then it sounds like the best idea is for me to work on a separate ARM init.

Raspberry Pis don't normally boot from CD, so if it would be ok with you I would like to start with your init and then take it in my own direction.
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Mon 31 Jul 2017, 18:18    Post subject:  

@woodenshoe-wi

A few things:

1. Using a savefile rather than a savefolder does NOT necessitate running in pupmode=13(29 in this init) with the rw layer being in tmpfs and the savefile as the top of the ro stack.
Once a savefile is mounted it's mount point can be used in a normal pupmode=12 with the save directory as the top layer i.e. the rw directory of the stack.
This is true in the older init's and in this init.

2. All the kernel's I use are produced by the woorf-ce kernel kit, so they include support for aufs. This init simply chooses not to use it.

3. I have a mantra for myself:
"Windows filesystems for Windows, Linux filesystems for Linux."
It might be a lot easier if you just re-parttion your SD card to have a Linux partition on it and use the "psave=" boot parmeter to direct the savefolder to it.

4. Once I publish things (make them available on the net), I accept that folk have the capability of copying and using them in any way they see fit.
And I publish the stuff I produce in the chance that it may be useful to someone else in some way.
So, if you want to produce your own init based on my code, go for it.
Although you may want to wait for a little while yet till the dust settles. There are a couple of new things I intend to try, that could significantly restructure the current code.
I hope it works well for you.

gyro
Back to top
View user's profile Send private message 
woodenshoe-wi

Joined: 28 Jul 2017
Posts: 7
Location: Wisconsin

PostPosted: Mon 31 Jul 2017, 23:03    Post subject:  

Thanks. I'll gladly wait a while, there are plenty of other things that need working on to get them the way I want.

The reason I want to have pupmode=13 is to minimize writing to the SD card or USB stick. A Raspberry Pi doesn't usually have a hard drive, but a save file on a hard drive can be pupmode=12 and doesn't need snapmergepuppy.

I tried the kernel I cross compiled with kernel-kit and it boots so I can experiment with aufs, but I want to always be able to boot even if the kernel is one from Raspbian that doesn't have aufs.

I agree that NTFS in particular has no place in a linux only environment, but the Raspberry Pi boot loader needs the first partition of the SD card to be FAT. Currently the raspup woof-ce build makes an image file which is a full install in a second Linux partition. I think the only option to back up the SD card in Windows is to make an image of the whole SD card, being able to copy just a pupsave.2fs or pupsave.ram.sfs would be easier.

It helps a lot to be able to borrow from the work that others have done, but it is also easier to try things when you don't have to worry about breaking some functionality that you don't use but someone else needs, so I think I would be more comfortable working on a separate ARM init than making changes to the woof-ce init.
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Mon 07 Aug 2017, 05:09    Post subject: Updated version  

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/initrd/ydrv_overlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" now contains an updated "extra_sfs", updated bootspecs utilities, and new "saveconfig" utility.

Changes in this 7 August version:

1. "init" now supports a new "PMODE" variable.
After first boot, puppy continues to reboot into pupmode=21, until PMODE is set in BOOT_SPECS.
This is done using a utility called "saveconfig". It's in the "Setup" menu.
Based on the answers provided it will set PMODE, PROSAVE and PSAVE, as necessary, in BOOT_SPECS, and create a savefolder.
PMODE is meant to specify only even pupmodes. PROSAVE='yes' upgrades the even pupmode to an odd pupmode.

2. The bootspecs utilities have been updated to include a "tmpbs2variable" utility that echos the value of a specified variable in BOOT_SPECS.
This means that calling uitlities can retrieve and set variables in BOOT_SPECS without regard for the underlying BOOT_SPECS facility.
They have also been updated to fully support the same location priorities as implemented in "init". i.e. if DISTRO_SPECS is in initrd.gz then BOOT_SPECS has to be there also, otherwise it's in the frugal install directory.

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 3014
Location: Worcestershire, UK

PostPosted: Mon 07 Aug 2017, 11:32    Post subject:  

Hi gyro

Tried the new version....clicked on the new menu item Saveconfig and a save folder was created as expected but nothing was written in to it on reboot....BOOT_SPECS was not created....

Also tried to install an .sfs - selected it via Extra-SFS Manager but nothing happened - the .sfs wasn't added to the list but no error message appeared to suggest why not.....

The new implementation does not seem as intuitive (to me) as the previous version....do I have to know about PUPMODES ?? PUPSTATE is:
Code:
PUPMODE=21
PDEV1='sda3'
DEV1FS='ext4'
PUPSFS='sda3,ext4,/devtest/puppy_LxPupSc_17.08.21.sfs'
PUPSAVE=''
PMEDIA='atahd'
#these directories are overlayfs layers in /initrd...
RW_LAYER=''
SAVE_LAYER=''
PUP_LAYER='/pup_ro2'
#The partition that has the tahrsave file is mounted here...
PUP_HOME=''
#(in /initrd) ...note, /mnt/home is a link to it.
#this file has extra kernel drivers and firmware...
ZDRV='sda3,ext4,/devtest/zdrv_LxPupSc_17.08.21.sfs'
FDRV='sda3,ext4,/devtest/fdrv_LxPupSc_17.08.21.sfs'
ADRV='sda3,ext4,/devtest/adrv_LxPupSc_17.08.21.sfs'
YDRV='sda3,ext4,/devtest/ydrv_LxPupSc_17.08.21.sfs'
PSUBDIR='/devtest'
PNOSAVE=''
PUNIONFS='overlay'
PSAVEPART='sda3'
PSAVEDIR='/devtest'
not sure where tahrsave comes from!!

Cheers
peebee
Screenshot.png
 Description   
 Filesize   103.35 KB
 Viewed   97 Time(s)

Screenshot.png


_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
Rangan Masti

Joined: 01 Jan 2013
Posts: 34
Location: Germany, Berlin

PostPosted: Mon 07 Aug 2017, 14:48    Post subject:  

Hi, good evening. I just downladed your new initrd with ydrv. I do test this. Have fun in your work!!!.

Rangan
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Tue 08 Aug 2017, 01:47    Post subject:  

@peebee,

The "tahrsave" bit is easy, it's been hardcoded in this init code for quite some time. Fixed for next release.

I'll try another fresh install to see if I can reproduce your problems.

Edit:
It looks like the BOOT_SPECS file is not being created.
Both setting of a savefolder and adding an extra-sfs depend on stuff being stored in BOOT_SPECS.
I assume you did not push DISTRO_SPECS into the initrd.gz.

gyro

Last edited by gyro on Tue 08 Aug 2017, 01:54; edited 1 time in total
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Tue 08 Aug 2017, 01:49    Post subject:  

I forgot to mention a few changes in the 7 August version:

3. "pfix=nosave" is now supported in pupmode=12.
On first boot after setting this, a backup is made of the savefolder to a ".tar.gz" file.
On subsequent boots the contents of the savefolder is repalced with the contents of this backup file.

4. The save to sfs in rc.shutdown now uses the mksquashfs "-ef" parameter to exclude some files.
The idea is to minimise this size of the sfs file, without using a slow compression method.

5. The POVERLAY='yes' parameter has been replaced with a PUNIONFS='overlay' in /etc/rc.d/PUPSTATE
My intention is to patch the woof-ce "init" to add a PUNIONFS='aufs' entry in /etc/rc.d/PUPSTATE
The point here is that utilities can then be easily patched or written to be unionfs aware.

6. A RW_LAYER parameter has also been added to /etc/rc.d/PUPSTATE.
Again, it's my intention to patch woof-ce "init" to do the same thing.
This is to provide a re-direction facility in /etc/rc.d/PUPSTATE that always provides the top(rw) layer of the stack, even in an odd pupmmode situation.
We introduce some future-proofing to our utilites when we use the variables in /etc/rc.d/PUPSTATE, rather than hard coding their current contents.
I think of the PUPSTATE file as a contract between "init" and the running puppy;
"init" has an obligation to produce an accurate, backwards compatible, PUPSTATE that contains all necessary information. It has no obligation to continue to fill those variables with the same content as current.
Programs and scripts in the running puppy have an obligation to trust the contents of the variables in PUPSTATE, and not try to second guess what those contents might be, based on their current content.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Tue 08 Aug 2017, 08:10    Post subject:  

peebee wrote:
Tried the new version....clicked on the new menu item Saveconfig and a save folder was created as expected but nothing was written in to it on reboot....BOOT_SPECS was not created....

Also tried to install an .sfs - selected it via Extra-SFS Manager but nothing happened - the .sfs wasn't added to the list but no error message appeared to suggest why not.....
I've been able to reproduce these, and fix them, so it now works for me.
The problem is that the line to create an empty BOOT_SPECS file if it doesn't exist, is missing from the current version of "bootspecs2tmpbs".

Hmm... my bad, a fairly obvious hole in my testing regime, I've been using a "YDRV=" line in "BOOT_SPECS" to load "ydrv_overlay.sfs", hence even doing a first boot, my testing usually had an existing "BOOT_SPECS".

I'll upload new versions to-night, hopefully. Just one little fix to "init" that needs a little more work.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1407
Location: Brisbane, Australia

PostPosted: Tue 08 Aug 2017, 09:55    Post subject:  

I've uploaded new versions to http://www.fishprogs.software/puppy/initrd/initrd.gz and http://www.fishprogs.software/puppy/initrd/ydrv_overlay.sfs.
Both need to be downloaded, since the "ydrv_overlay.sfs" contains updated bootspecs utilities.

This 8 August version simply fixes the bugs found by peebee.

gyro
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 6 of 7 [96 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1392s ][ Queries: 12 (0.0037s) ][ GZIP on ]