Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 15 Dec 2017, 21:53
All times are UTC - 4
 Forum index » Taking the Puppy out for a walk » Announcements
Rationalisation of init
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 6 [85 Posts]   Goto page: 1, 2, 3, 4, 5, 6 Next
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Tue 14 Jun 2016, 13:29    Post subject:  Rationalisation of init
Subject description: proof of concept test works, now it's on to doing it.
 

Suppoprt message translation for non-english puppies. (Not my idea, but I agree it needs to be done.)
Support only new puppies, discard old methods.

Puppy might search all over the place for files, but in the end it uses only files found on PDEV1 and SAVEPART, or as specified with boot parameters.

New basic algorithm:
1. Search for only pup...sfs file, establish PDEV1 based on this search. If it's specified with a boot parameter, look only there.
2. Load pup...sfs file.
3. Look for, and load any found, boot parameter specified other system sfs files.
4. Look for still unfound, and load any found, unspecified other system sfs files on PDEV1.
5. Determine SAVEPART.
6. Umount any partitions that are no longer required.
7. "insmod" any initmodules.
8. find and load savefile/savefolder from SAVEPART, move mountpoint of SAVEPART to /mnt/dev_save, if required.
9. Setup aufs stack.

NOTES:
Do not search for "vmlinuz".
Do not use an IDSTRING, use only filenames.
Only the search for pup...sfs actually loops through partitions.
When searching for pup...sfs, stop once it is found.
When searching for pup...sfs, umount any partitions that do not contain pup...sfs.
Usually, no partition should be mounted more than once.

"other system sfs file" means fdrv, zdrv, ydrv, or adrv.
"pup...sfs" means the main puppy sfs filename as specified in DISTRO_SPECS.

An extra possibility:
The aufs stack could be built after step 6 or built gradualy as each sfs is loaded, using a tmpfs as it's rw layer. This would allow step 7 to use "/pup_new/lib/modules", instead of just zdrv.
Then step 9 would have to replace the tmpfs rw layer with the savefile/savefolder for pupmode=12. (I have sucessfully replaced the rw layer in a test stack. The test stack was not "/".)

Other more minor things:
Let "losetup" provide us with an unused loop device, stop tying to control the number.
Possibly refer to pup...sfs as the pdrv, for consistency with other system sfs's. "pup_ro2" -> "pup_p", "dev_ro2" -> "pdrv".

The latest version of the new "init" script can be downloaded from git-hub with the following command:
Code:
svn export https://github.com/puppylinux-woof-CE/initrd_progs/trunk/0initrd/init
Or, go to https://github.com/puppylinux-woof-CE/initrd_progs/blob/master/0initrd/init and right click on the "Raw" button, then select "Save Link As...".

gyro

Last edited by gyro on Fri 15 Jul 2016, 06:24; edited 4 times in total
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Tue 14 Jun 2016, 14:25    Post subject:  

This is a very good roadmap, everything is spot on. The translation thing can be saved for the last, it's something very easy to do, in just one day
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Wed 15 Jun 2016, 16:19    Post subject:  

Hi gyro, are you already working on any of the proposed changes? Just want to know...
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Thu 16 Jun 2016, 11:06    Post subject:  

Hi jlst,

I am not currently working on implementation code for these ideas. I'm still thinking about the what and how.

As to the how;
Rewriting from scratch is attractive, but daunting.
Incremental updates to the current "init" seems less daunting.
Although I'm not quite sure how to redo the basic algorithm in "incremental updates".
Maybe create a minimal test init that supports only pupmode=5. Then add pupmode=13, then add pupmode=12 (no crypto, no updating).

As to the what, I have a couple of more ideas:
1. Remove any "extra-sfs" procesing from "init". The idea is to completely isolate puppy system sfs's and "extra-sfs". The later to be handled by "sfs_load" or "sfs_install" without using BOOTCONFIG.
2. Introduce a boot configuration file. It could be "etc/LOCAL_SPECS" in "initrd.gz", or a text file beside pup...sfs, or a boot parameter.
This file would conatain variable specs in place of a number of the current boot parameters e.g "fdrv=", and also the "initmodules" variable.
It could be maintained by an application in the running system.
The contents of the config file must be available to "init" well before the savefile/savefolder is loaded, so it has to be somewhere else that can be readily accessed by "init".
3. Updating the encrypted savefile technology, and possibly extending to savefolder, could also be explored.

Edit 20 June 2016:
4. Prune unnecessary code from current "init".

gyro

Last edited by gyro on Sun 19 Jun 2016, 18:08; edited 1 time in total
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Thu 16 Jun 2016, 14:26    Post subject:  

I'd suggest that you start by doing the easiest things.. don't look for vmlinuz, don't use IDSTRING, and as you keep editing like there's no tomorrow, then everything will start to fall into place...
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 3105
Location: The Blue Marble

PostPosted: Fri 17 Jun 2016, 13:15    Post subject:  

This is excellent idea gyro.

I especially like this idea: "The aufs stack could be built ... gradualy as each sfs is loaded, using a tmpfs as it's rw layer." It is specially encouraging since you also said "I have sucessfully replaced the rw layer in a test stack. The test stack was not "/".)".

By not tying the aufs layer to some pre-defined layering, you can extend init to load almost anything. By building it gradually you also reduce the chance of crash because the aufs layer is invalid.

If I may give you a suggestion: The main concern in writing init, is that it *should never ever crash*, or just give up because if you crash then the entire system will crash too. For example, the searches may fail. The aufs layer creation may fail (ie mount -t aufs may fail). When they fail, you should *not* attempt to continue with switch_root because you already know that by trying to "exec switch_root" to a mountpoint that does not exist will make switch_root to fail and exit - and then we have the kernel crashing too.

So while for non-critical operations you can ignore error codes; critical operation such as creating the aufs mountpoint must be monitored. If these critical operation fails, you should not attempt to continue but instead print out a message, and drop into an emergency shell (busybox ash) so that the user hopefully can fix the situation. Hopefully Very Happy

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread.
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Fri 17 Jun 2016, 13:35    Post subject:  

The only situation i found that init crashes, and therefore a kernel panic occurs is when it fails to setup the layered filesystem... when it's not able to switch root basically.

I wrote about this here (I also provided code to fix that issue):

http://murga-linux.com/puppy/viewtopic.php?t=102754

An extra check before switching root could be test whether /bin/busybox (a very important app for puppy) exists. If /bin/busybox is not there, then we should assume that something is badly broken, therefore must stop.

With the huge kernel, it can bypass many errors that previously led to a kernel panic, it's because it already has many drivers builtin, that means it reaches the desktop even when the zdrv is not found (and therefore no kernel modules are to be found)
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Fri 17 Jun 2016, 14:52    Post subject:  

When I was testing the XDRV=drive:file stuff... I found that in some cases init would try to load an empty main SFS file (PUPSFS=""), then switch root and *kernel panic*.

CORRUPTED SFS: I always thought slacko64 was really bad, it would not reach the desktop by any means, however the main sfs was loaded.

Some files were missing... how come? I thought. Booting with kernel debug enabled, I finally found the reason: the SFS was corrupted, so I had to download the ISO again.. and everything was fine.

A .md5.txt file should always be there to check main sfs, init should try to use it.. basename.m5.txt

init should use that file to check the main sfs file, if the md5 checksum has changed, then warn the user (red color) that if something goes wrong it's because the main sfs is corrupted.

Or maybe not, how heavy is checking a md5 checksum?

Last edited by jlst on Fri 17 Jun 2016, 15:14; edited 1 time in total
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2636

PostPosted: Fri 17 Jun 2016, 15:10    Post subject:  

My suggestion is: put every bit of what is under discussion in the initrd -where it belongs. init should be a real init which will never crash and will do what init is supposed to do -run a normal boot process with all the fine options and file locations which have been tried and proved for >40 years.

All that layering should be done before the main system starts -that is all the things that make a totally fragmented collection of real/loop/layer into a seemingly normal hard-drive setup -that should all be done from the initrd.
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Fri 17 Jun 2016, 17:20    Post subject:  

I do appreciate that "init" is not like an application that can simply "exit" on an error.
If it can reasonably carry on it should, but dropout to a shell if it finds a catastrophic situation.

As to checking the puppy sfs, each sfs should be checked that it exists and that it is non zero in size. But, md5 checks for the main puppy sfs, sounds like an idea, but it will have to wait till later.

I'm currrently working on writing code for the new basic algorithm.
When that's done, I'll try to merge it with other stuff from the current "init" to produce a minimal working "init".
Once I get something that actually boots I will make it available.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Sat 18 Jun 2016, 14:10    Post subject:  

Still trucking.
Unfortunately it's going where I was afraid it would go; lot's of new code that will take a while to test.

@jlst,
Some good news, my current code for "load_onepupdrv" includes the following code:
Code:
 if [ "$DRVFILE" ];then
  [ -s ${ONEDEVMNTPT}${DRVFILE} ] || return #sfs not Ok
 else
  return #no sfs found
 fi
Only integrity test for sfs being non zero size, but a place holder for md5 or other testing if we want to.

gyro
Back to top
View user's profile Send private message 
jlst

Joined: 23 Nov 2012
Posts: 571

PostPosted: Sat 18 Jun 2016, 15:10    Post subject:  

So you're actually doing a major rewrite? How about using the initrd_progs repo for this? It's actually abandoned now, but you might want push all your changes to init there including your mistakes to see how the program evolves and test every 3 days or so...
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Sat 18 Jun 2016, 23:44    Post subject:  

I want to test the viability of the "new algorithm", so I created a file containing just the code needed to implement it.
Now I'm importing enough code from the current "init", so that it will become a shell script that will execute and drop out to a console.
Once I've got something that runs, I'll make it available.
It will still be missing a lot of stuff, so there will need to be many updates before it's a fully functional "init".
If I use "initrd_progs", it's "init" file will be useless for any other purpose for some time.

Many of the enhancements that have been mentioned could be implemented by relatively managable individual patches to the current "init".
But completely changing the basic algorithm is a large change, and we either "bite the bullet" or put it into the too hard basket.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Sun 19 Jun 2016, 18:02    Post subject:  

jlst wrote:
So you're actually doing a major rewrite?
Not as in 'produce a new "init" from scratch'.
I've written a test script that implents the aufs stack building algorithm and I will test it running as "init". It doesn't goon to finish the boot, it just drops out to a console so I can look at the created aufs stack.
When this works I plan to port this code back into the current "init". This will still be a large change.

The downside of not going the other way, i.e. porting the current "init" into my script, is that the "dead" code in the current "init" will remain.
So, another step in redoing "init", is to prune the current code.
There is code that references modules, that could go. There is code that implements an obsolete use of the zdrv. There may be more.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1409
Location: Brisbane, Australia

PostPosted: Mon 20 Jun 2016, 15:39    Post subject: Test "init" works.  

I have attached the code of my test minimal "init" script, it works, when run as "init" in "initrd.gz".
It builds the aufs stack gradually, i.e. it creates it with pup...sfs, and then inserts the others appropriately as it finds them.
This stack is like a pupmode=5 with a directory in a tmpfs as it's rw layer.
The script then swaps the rw layer with a "savefolder" on a hd, thus turning it into a pupmode=12 type stack.
Turning a pupmode=5 stack into a pupmode=13 stack should be even easier, it's just the insertion of the savefolder as layer 1.

This script is a long way from being a replacement "init".
For a start, at the end it just drops out to a console. There's also a whole lot of necessary stuff that it does not do.
But the code for loading the sfs's should be quite useful in porting this stuff into "init".

To work, it depends on files being defined as boot parameters, here's the "kernel" line I used in testing, the partitions are specified by the Label's:
Code:
kernel /puppy/tahru/vmlinuz acpi_osi=Linux libata.noacpi=1 pmedia=usbhd psubdir=/puppy/tahru pupsfs=isoSave ydrv=:show.sfs adrv=:/puppy/pcmanfm32.sfs psave=tsave:/saveit pfix=fsck4

Since it doesn't continue onto a complete boot, I used the following script:
Code:
#!/bin/sh
RWBRANCH=/mnt/dev_save/saveit
mount -o remount,prepend:/mnt/tmpfs/pup_rw=rw,mod:${RWBRANCH}=ro,del:${RWBRANCH} /pup_new
exit
to switch the hd directory out of the stack so I could "umount" the partition, before rebooting (ctrl-alt-del).

So, now it's on to a significantly modified "init".

gyro
init.test.gz
Description  gunzip to produce "init.test", then merge into "initrd.gz" as "init".
gz

 Download 
Filename  init.test.gz 
Filesize  5.39 KB 
Downloaded  60 Time(s) 
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 6 [85 Posts]   Goto page: 1, 2, 3, 4, 5, 6 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Taking the Puppy out for a walk » Announcements
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1420s ][ Queries: 12 (0.0136s) ][ GZIP on ]