Rationalisation of init

News, happenings
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Rationalisation of init

#1 Post by gyro »

Suppoprt message translation for non-english puppies. (Not my idea, but I agree it needs to be done.)
Support only new puppies, discard old methods.

Puppy might search all over the place for files, but in the end it uses only files found on PDEV1 and SAVEPART, or as specified with boot parameters.

New basic algorithm:
1. Search for only pup...sfs file, establish PDEV1 based on this search. If it's specified with a boot parameter, look only there.
2. Load pup...sfs file.
3. Look for, and load any found, boot parameter specified other system sfs files.
4. Look for still unfound, and load any found, unspecified other system sfs files on PDEV1.
5. Determine SAVEPART.
6. Umount any partitions that are no longer required.
7. "insmod" any initmodules.
8. find and load savefile/savefolder from SAVEPART, move mountpoint of SAVEPART to /mnt/dev_save, if required.
9. Setup aufs stack.

NOTES:
Do not search for "vmlinuz".
Do not use an IDSTRING, use only filenames.
Only the search for pup...sfs actually loops through partitions.
When searching for pup...sfs, stop once it is found.
When searching for pup...sfs, umount any partitions that do not contain pup...sfs.
Usually, no partition should be mounted more than once.

"other system sfs file" means fdrv, zdrv, ydrv, or adrv.
"pup...sfs" means the main puppy sfs filename as specified in DISTRO_SPECS.

An extra possibility:
The aufs stack could be built after step 6 or built gradualy as each sfs is loaded, using a tmpfs as it's rw layer. This would allow step 7 to use "/pup_new/lib/modules", instead of just zdrv.
Then step 9 would have to replace the tmpfs rw layer with the savefile/savefolder for pupmode=12. (I have sucessfully replaced the rw layer in a test stack. The test stack was not "/".)

Other more minor things:
Let "losetup" provide us with an unused loop device, stop tying to control the number.
Possibly refer to pup...sfs as the pdrv, for consistency with other system sfs's. "pup_ro2" -> "pup_p", "dev_ro2" -> "pdrv".

The latest version of the new "init" script can be downloaded from git-hub with the following command:

Code: Select all

svn export https://github.com/puppylinux-woof-CE/initrd_progs/trunk/0initrd/init
Or, go to https://github.com/puppylinux-woof-CE/i ... nitrd/init and right click on the "Raw" button, then select "Save Link As...".

gyro
Last edited by gyro on Fri 15 Jul 2016, 10:24, edited 4 times in total.

jlst

#2 Post by jlst »

This is a very good roadmap, everything is spot on. The translation thing can be saved for the last, it's something very easy to do, in just one day

jlst

#3 Post by jlst »

Hi gyro, are you already working on any of the proposed changes? Just want to know...

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#4 Post by gyro »

Hi jlst,

I am not currently working on implementation code for these ideas. I'm still thinking about the what and how.

As to the how;
Rewriting from scratch is attractive, but daunting.
Incremental updates to the current "init" seems less daunting.
Although I'm not quite sure how to redo the basic algorithm in "incremental updates".
Maybe create a minimal test init that supports only pupmode=5. Then add pupmode=13, then add pupmode=12 (no crypto, no updating).

As to the what, I have a couple of more ideas:
1. Remove any "extra-sfs" procesing from "init". The idea is to completely isolate puppy system sfs's and "extra-sfs". The later to be handled by "sfs_load" or "sfs_install" without using BOOTCONFIG.
2. Introduce a boot configuration file. It could be "etc/LOCAL_SPECS" in "initrd.gz", or a text file beside pup...sfs, or a boot parameter.
This file would conatain variable specs in place of a number of the current boot parameters e.g "fdrv=", and also the "initmodules" variable.
It could be maintained by an application in the running system.
The contents of the config file must be available to "init" well before the savefile/savefolder is loaded, so it has to be somewhere else that can be readily accessed by "init".
3. Updating the encrypted savefile technology, and possibly extending to savefolder, could also be explored.

Edit 20 June 2016:
4. Prune unnecessary code from current "init".

gyro
Last edited by gyro on Sun 19 Jun 2016, 22:08, edited 1 time in total.

jlst

#5 Post by jlst »

I'd suggest that you start by doing the easiest things.. don't look for vmlinuz, don't use IDSTRING, and as you keep editing like there's no tomorrow, then everything will start to fall into place...

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#6 Post by jamesbond »

This is excellent idea gyro.

I especially like this idea: "The aufs stack could be built ... gradualy as each sfs is loaded, using a tmpfs as it's rw layer." It is specially encouraging since you also said "I have sucessfully replaced the rw layer in a test stack. The test stack was not "/".)".

By not tying the aufs layer to some pre-defined layering, you can extend init to load almost anything. By building it gradually you also reduce the chance of crash because the aufs layer is invalid.

If I may give you a suggestion: The main concern in writing init, is that it *should never ever crash*, or just give up because if you crash then the entire system will crash too. For example, the searches may fail. The aufs layer creation may fail (ie mount -t aufs may fail). When they fail, you should *not* attempt to continue with switch_root because you already know that by trying to "exec switch_root" to a mountpoint that does not exist will make switch_root to fail and exit - and then we have the kernel crashing too.

So while for non-critical operations you can ignore error codes; critical operation such as creating the aufs mountpoint must be monitored. If these critical operation fails, you should not attempt to continue but instead print out a message, and drop into an emergency shell (busybox ash) so that the user hopefully can fix the situation. Hopefully :D
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

jlst

#7 Post by jlst »

The only situation i found that init crashes, and therefore a kernel panic occurs is when it fails to setup the layered filesystem... when it's not able to switch root basically.

I wrote about this here (I also provided code to fix that issue):

http://murga-linux.com/puppy/viewtopic.php?t=102754

An extra check before switching root could be test whether /bin/busybox (a very important app for puppy) exists. If /bin/busybox is not there, then we should assume that something is badly broken, therefore must stop.

With the huge kernel, it can bypass many errors that previously led to a kernel panic, it's because it already has many drivers builtin, that means it reaches the desktop even when the zdrv is not found (and therefore no kernel modules are to be found)

jlst

#8 Post by jlst »

When I was testing the XDRV=drive:file stuff... I found that in some cases init would try to load an empty main SFS file (PUPSFS=""), then switch root and *kernel panic*.

CORRUPTED SFS: I always thought slacko64 was really bad, it would not reach the desktop by any means, however the main sfs was loaded.

Some files were missing... how come? I thought. Booting with kernel debug enabled, I finally found the reason: the SFS was corrupted, so I had to download the ISO again.. and everything was fine.

A .md5.txt file should always be there to check main sfs, init should try to use it.. basename.m5.txt

init should use that file to check the main sfs file, if the md5 checksum has changed, then warn the user (red color) that if something goes wrong it's because the main sfs is corrupted.

Or maybe not, how heavy is checking a md5 checksum?
Last edited by jlst on Fri 17 Jun 2016, 19:14, edited 1 time in total.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#9 Post by amigo »

My suggestion is: put every bit of what is under discussion in the initrd -where it belongs. init should be a real init which will never crash and will do what init is supposed to do -run a normal boot process with all the fine options and file locations which have been tried and proved for >40 years.

All that layering should be done before the main system starts -that is all the things that make a totally fragmented collection of real/loop/layer into a seemingly normal hard-drive setup -that should all be done from the initrd.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#10 Post by gyro »

I do appreciate that "init" is not like an application that can simply "exit" on an error.
If it can reasonably carry on it should, but dropout to a shell if it finds a catastrophic situation.

As to checking the puppy sfs, each sfs should be checked that it exists and that it is non zero in size. But, md5 checks for the main puppy sfs, sounds like an idea, but it will have to wait till later.

I'm currrently working on writing code for the new basic algorithm.
When that's done, I'll try to merge it with other stuff from the current "init" to produce a minimal working "init".
Once I get something that actually boots I will make it available.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#11 Post by gyro »

Still trucking.
Unfortunately it's going where I was afraid it would go; lot's of new code that will take a while to test.

@jlst,
Some good news, my current code for "load_onepupdrv" includes the following code:

Code: Select all

 if [ "$DRVFILE" ];then
  [ -s ${ONEDEVMNTPT}${DRVFILE} ] || return #sfs not Ok
 else
  return #no sfs found
 fi
Only integrity test for sfs being non zero size, but a place holder for md5 or other testing if we want to.

gyro

jlst

#12 Post by jlst »

So you're actually doing a major rewrite? How about using the initrd_progs repo for this? It's actually abandoned now, but you might want push all your changes to init there including your mistakes to see how the program evolves and test every 3 days or so...

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#13 Post by gyro »

I want to test the viability of the "new algorithm", so I created a file containing just the code needed to implement it.
Now I'm importing enough code from the current "init", so that it will become a shell script that will execute and drop out to a console.
Once I've got something that runs, I'll make it available.
It will still be missing a lot of stuff, so there will need to be many updates before it's a fully functional "init".
If I use "initrd_progs", it's "init" file will be useless for any other purpose for some time.

Many of the enhancements that have been mentioned could be implemented by relatively managable individual patches to the current "init".
But completely changing the basic algorithm is a large change, and we either "bite the bullet" or put it into the too hard basket.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#14 Post by gyro »

jlst wrote:So you're actually doing a major rewrite?
Not as in 'produce a new "init" from scratch'.
I've written a test script that implents the aufs stack building algorithm and I will test it running as "init". It doesn't goon to finish the boot, it just drops out to a console so I can look at the created aufs stack.
When this works I plan to port this code back into the current "init". This will still be a large change.

The downside of not going the other way, i.e. porting the current "init" into my script, is that the "dead" code in the current "init" will remain.
So, another step in redoing "init", is to prune the current code.
There is code that references modules, that could go. There is code that implements an obsolete use of the zdrv. There may be more.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Test "init" works.

#15 Post by gyro »

I have attached the code of my test minimal "init" script, it works, when run as "init" in "initrd.gz".
It builds the aufs stack gradually, i.e. it creates it with pup...sfs, and then inserts the others appropriately as it finds them.
This stack is like a pupmode=5 with a directory in a tmpfs as it's rw layer.
The script then swaps the rw layer with a "savefolder" on a hd, thus turning it into a pupmode=12 type stack.
Turning a pupmode=5 stack into a pupmode=13 stack should be even easier, it's just the insertion of the savefolder as layer 1.

This script is a long way from being a replacement "init".
For a start, at the end it just drops out to a console. There's also a whole lot of necessary stuff that it does not do.
But the code for loading the sfs's should be quite useful in porting this stuff into "init".

To work, it depends on files being defined as boot parameters, here's the "kernel" line I used in testing, the partitions are specified by the Label's:

Code: Select all

kernel /puppy/tahru/vmlinuz acpi_osi=Linux libata.noacpi=1 pmedia=usbhd psubdir=/puppy/tahru pupsfs=isoSave ydrv=:show.sfs adrv=:/puppy/pcmanfm32.sfs psave=tsave:/saveit pfix=fsck4
Since it doesn't continue onto a complete boot, I used the following script:

Code: Select all

#!/bin/sh
RWBRANCH=/mnt/dev_save/saveit
mount -o remount,prepend:/mnt/tmpfs/pup_rw=rw,mod:${RWBRANCH}=ro,del:${RWBRANCH} /pup_new
exit
to switch the hd directory out of the stack so I could "umount" the partition, before rebooting (ctrl-alt-del).

So, now it's on to a significantly modified "init".

gyro
Attachments
init.test.gz
gunzip to produce "init.test", then merge into "initrd.gz" as "init".
(5.39 KiB) Downloaded 258 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Where to from here?

#16 Post by gyro »

Following my success with testing aufs stacks in the "init" environment, I have a revised plan for updating the actual "init".

1. Cut search_func into 2, savefile/savefolder stuff becomes search_save_func
2. search_func is reduced to decoding all boot parameters and finding pup...sfs, setting PDEV1
if PDRVDEV is set:
if PDRVFN is empty set to PUPXXXSFS
if /mnt/pdrv$PSUBDIR/$PDRVFN exists, set PDEV1
else search for $PSUBDIR/$PUPXXXSFS as diligently as it was for "vmlinux", set PDEV1 if found
if PDEV1 set DEV1FS and other "vmlinux found" stuff
3. Remove all load sfs code from "LOADING PUPPY FILES" section
4. algorithm becomes:
do search_func stuff
load sfs's to pupmode=5 aufs stack
establish SAVEPART
do "insmod" stuff
do search_save_func
load savefile/savefolder
update aufs stack with savefile/savefolder
continue on to switch root
5. Complete change to use only a single tmpfs.
6. Cleanup more possibly reduntant code, e.g. setting the aufs layer variables in beginning of "LOADING PUPPY FILES" section.

As this has now become a coding exercise, when I have a modified "init" I will open a thread in "Cutting edge" for discussion of the code.

To encourage others to contribute, it might be handy to do this as a file on git-hub in a repository to which I, and others have direct access.
I'm not sure how to set that up.
@jlist, I don't have direct access to the initrd_progs repo.
At the moment the only git-hub repositories I have direct access to, are the forks in my git-hub account.
By "direct access" I mean something I can push "commits" to, using git on my local machine.

gyro

jlst

#17 Post by jlst »

Ok gyro, initrd_progs is somewhat restricted to members of the team, those who have write access to all woofce repos, well, everybody else has to open pull requests ( just like i used to).

I'm certainly quick to merge pull requests so that could be the place, i also think you should a member, or at least have write access to that repo, i'll send a message to micko..

Well, this is what I used to do.... add commits then open a pull request every few days with all the commits... while the pull request was open i kept adding commits to document the changes.. initrd_progs is a place to experiment and push anything and that's what i did until it was ready....

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#18 Post by technosaurus »

I would suggest starting from this:
https://github.com/landley/aboriginal/b ... in/init.sh

Its pretty simplified already, but it unnecessarily parses /proc/cmdline - the kernel passes parameters of the format *=* to envp/environ (environment variable in shell) and other parameters without an equal sign to argv ($@ in shell)

In shell you can use a default variable parameter like this:

Code: Select all

${init:-/linuxrc} #run the init set in the env var $init if it is set, otherwise run /linuxrc
or this:

Code: Select all

#defaultinit=/init
defaultinit=/linuxrc
${init:-${defaultinit:-/init}}
#pass init=whatever on the kernel command line
for the kernel arguments without an equal sign:

Code: Select all

while [ "$1" ]; do
case "$1" in
   quiet)QUIET=1;;
   #other cases here
   *): DO NOTHING;;
esac
shift
done
[ "$QUIET" ] || echo now doing step XX
BTW every single line of code involved with searching is bloat
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#19 Post by gyro »

@technosaurus,

I'll look at that "int.sh". I'm still considering how to do it.

Concerning "searching", I tend to agree, so I want to make it quite direct when things are specified.
Unfortunately for mainstream puppy, removing "searching" altogether is probably a bit too radical a step.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#20 Post by gyro »

@jlst,

How about I setup any interested folk as collaborators to my fork of woof-ce?
Then do a pull-request every so often.

gyro

Post Reply