Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 21 Oct 2018, 01:34
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Another re-write of the "init" script - using OverlayFs?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 12 of 14 [196 Posts]   Goto page: Previous 1, 2, 3, ..., 10, 11, 12, 13, 14 Next
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Thu 19 Apr 2018, 05:47    Post subject:  

A simple fix to "install2usb", change:
Code:
printf "1\ny\n"| fsck.vfat $FIXFATPART
to
Code:
printf "1\ny\n"| fsck.vfat -r $FIXFATPART
"-r" is the default in recent versions of "fsck.vfat" but not in older versions.
I tested "install2usb" on Tahr 6.0.6 and the vfat partition was not fixed, until I made this change.
The changed code still works in recent versions.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Fri 20 Apr 2018, 01:59    Post subject: Indicating that vmlinuz is in differnt partition  

I'm having second thoughts about how installers like "install2usb" should indicate that "vmlinuz" is on a partition other than the one containing the frugal install directory with the sfs files.
The problem is that while the boot parameter method works fine in the running target Puppy, it does not help a script that's running on a host Puppy, but trying to update the frugal install directory of a target Puppy, e.g. "overlay-setup-frugal".

My alternative would be for the installer to write a file called "pbootlocation.txt" (or "puppyboot.txt") in the frugal install directory alongside the sfs files.
This file would contain something like the following:
Code:
BOOTPART='VFAT'
BOOTDIR='/xenialpup-7.5'
Where 'VFAT' is the label assigned to the vfat partition, and '/xenialpup-7.5' is the sub-directory of that partition containing "vmlinuz".

This approach should work for any script processing the frugal install directory, simplifies the code in the installer, and does not involve changing "init" or "PUPSTATE".

The aim of either approach is to support not only the "install2usb" script, but also setups where the the boot partition is a usb device but the Puppy frugal install directories reside on a different device, e.g. a Linux partition on an internal HD.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Fri 20 Apr 2018, 23:41    Post subject: "vmlinuz" not in frugal install directory issue  

"vmlinuz" not in frugal install directory issue:
There is a simpler way.

"install2usb" doesn't do anything about the the issue.
"change_kernels" looks for "vmlinuz" in the frugal install directory, if it doesn't find it, it opens a "select file" dialog, so the user can locate it.
Silimarly, if it doesn't find a "zdrv...sfs" in the frugal install, it opens a "select file" dialog so the user can locate it.

This approach can even cope with the situation where the user has "specified" the "zdrv...sfs" to be a file with a different name, e.g. "/mnt/sdc7/kernels/kernel_modules-4.9.92.sfs".

But if the frugal install directory is "normal", then things proceed as before.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Wed 25 Apr 2018, 20:44    Post subject:  

I have attached:

"overlay_installers-0.9.sfs", which includes my latest hack of 01micko's "install2usb", and "overlay-setup-frugal".
This "install2usb" does nothing about the "vmlinuz" in a different partition issue.

"change_kernels" the corresponding hack of this script.
If it can't find "vmlinuz" in the frugal install directory and it's a usb device, it also looks in PSUBDIR on the 1 partition of the device.
Otherwise it opens a file selction dialog for the user to specify the location of the appropriate "vmlinuz".

"install2usb9.diff" my patch to "install2usb".

"change_kernels9.diff" my patch to "change_kernels"

gyro
overlay_installers-0.9.sfs.gz
Description  gunzip to produce "overlay_installers-0.9.sfs"
gz

 Download 
Filename  overlay_installers-0.9.sfs.gz 
Filesize  9.86 KB 
Downloaded  92 Time(s) 
change_kernels.gz
Description  gunzip to produce "change_kernels" script
gz

 Download 
Filename  change_kernels.gz 
Filesize  4.53 KB 
Downloaded  95 Time(s) 
install2usb9.diff.gz
Description  gunzip to produce "install2usb9.diff"
gz

 Download 
Filename  install2usb9.diff.gz 
Filesize  2 KB 
Downloaded  91 Time(s) 
change_kernels9.diff.gz
Description  gunzip to produce "change_kernels9.diff"
gz

 Download 
Filename  change_kernels9.diff.gz 
Filesize  2.04 KB 
Downloaded  90 Time(s) 
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Wed 25 Apr 2018, 20:47    Post subject: Release of overlay_init-0.9  

Release of overlay_init-0.9

The version jump to 0.9 is to indicate that this project is almost done.
I don't expect the "init" script to get any more significant changes.
The utilities are now pretty much functionally complete.

Unfortunately this project has become a little muddied by some of the existing Puppy files that it replaces, changing in woof-ce 'testing'.
To ensure that the patched version in this project properly reflects the version in the parent Puppy, there are multiple releases, each for a particular Puppy.
This issue will be resolved when/if I can get compatible versions of these files into woof-ce 'testing', and then they can be removed from "overlay_init-1.0.tar".

Downloads:
for artfulpup-17.11+16 http://www.fishprogs.software/puppy/initrd/overlay_init_artful-0.9.tar.
for LxPupSc-18.04+3T-k64 http://www.fishprogs.software/puppy/initrd/overlay_init_lxpupsc-0.9.tar.
for slacko-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched http://www.fishprogs.software/puppy/initrd/overlay_init_slacko-0.9.tar.
for slacko64-6.9.9.9-uefi_r6048 http://www.fishprogs.software/puppy/initrd/overlay_init_slacko64-0.9.tar.
for Upup_Bionic_Beaver-18.04+7 http://www.fishprogs.software/puppy/initrd/overlay_init_upupbb-0.9.tar.
for xenialpup-7.5-uefi http://www.fishprogs.software/puppy/initrd/overlay_init_xenial-0.9.tar.

Deltas:
for artfulpup-17.11+16.iso http://www.fishprogs.software/puppy/initrd/artfulpup-17.11+16.iso___artfulpup_overlay0.9-17.11+16.iso.delta.
for LxPupSc-18.04+3T-k64.iso http://www.fishprogs.software/puppy/initrd/LxPupSc-18.04+3T-k64.iso___LxPupSc_overlay0.9-18.04+3T-k64.iso.delta.
for slacko-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched.iso http://www.fishprogs.software/puppy/initrd/slacko-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched.iso___slacko_overlay0.9-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched.iso.delta.
for slacko64-6.9.9.9-uefi_r6048.iso http://www.fishprogs.software/puppy/initrd/slacko64-6.9.9.9-uefi_r6048.iso___slacko64_overlay0.9-6.9.9.9-uefi_r6048.iso.delta.
for Upup_Bionic_Beaver-18.04+7.iso http://www.fishprogs.software/puppy/initrd/Upup_Bionic_Beaver-18.04+7.iso___Upup_Bionic_Beaver_overlay0.9-18.04+7.iso.delta.
for xenialpup-7.5-uefi.iso http://www.fishprogs.software/puppy/initrd/xenialpup-7.5-uefi.iso___xenialpup_overlay0.9-7.5-uefi.iso.delta.

Changes:
1. Simplified the "Squash" facility in "Overlay init setup".
This used to consist of a "split" process on shutdown, and a "purge" process after the next boot.
The "split" process has been optimised so that the gain from a "purge" process is not worth the complication, so the "purge" process had been eliminated.
The "Squash" process used to produce an update by "rc.update", this is now skipped.
2. Re-versioned all the untilities to 0.9
3. Removed "/usr/sbin/bootmanager" from most versions because the modification is now part of woof-ce 'testing'.
4. Removed support for "pbootpart" and "pbootdir" boot parameters from "init" script.
5. Renamed "OVERLAY_SPECS" to "initrdBOOTspecs".
6. Fixed bug, multisession DVD not saving when using "Archive" save mechanism.

Usage:
1. Download a delta file for a Puppy iso file that you already have, and use it to produce an "overlay" iso file.
Do a manual frugal install using this "overlay" iso,
or
install it on a usb stick using 01micko's "install2usb", see "overlay_installers-0.9.sfs.gz" in a previous post.
or
if you don't require UEFI boot, use rcrsn51's ISObooter, but answer "n" in response to it's "splash screen" question.
2. Download one of the tar files into a fresh frugal install of the appropriate Puppy,
run "overlay-setup-frugal", see "overlay_installers-0.9.sfs.gz" in a previous post.

Review, what's so interesting about this package:

1. It uses "overlayfs" not "aufs" to build the union stack.
This is suppoosed to be good because "overlayfs" is in the kernel, and so has a future,
"aufs" is not, and is never likely to be, so some predict that it will die.
"overlayfs" is supposed to be efficient, since all reading and writing of files goes directly to the actual files.

2. It contains a new facility called "bootspecs", this consists of a file that creates a mechanism for the running Puppy to communicate with the "init" script.
This means that the "init" script can more truly be said to "only do what it's told".
This facility includes scripts that make it very easy for any utility in the running system, to communicate with "init", e.g. "initmodules".

3. The complicated boot parameters for specifying what system sfs files to load, are gone.
Instead, in the running Puppy, the "Overlay init setup" GUI utility is used to "slelect" the sfs files to load and their order in 2 system sfs lists.
The "ABOVE" list (defult adrv,ydrv) that gets mounted in the stack above the "MAIN" list (default puppy,fdrv,zdrv).
These lists can contain virtually any number of sfs files with any name and with any location.
The sfs lists facility relies on the "bootspecs" facility.

4. While there is support for "psavepart" (or it's alias "psave") and "psavedir" boot parameters, these are only very rarely needed.
The "saveconfig" utility, (accessed via "Overlay init setup") can be run at any time to "select" the save mode and "select" the location where the save is stored.
So you can copy your savefolder to a different location, or change your save mechanism from "Folder" to "Archive".
The "saveconfig" utility is run automatically on first shutdown.
On first shutdown, if your are not certain that you have a suitable Linux partition for your savefolder, simply choose the "Archive" save mechanism.
When you have sorted your Linux partition, run "Overlay init setup", "Saveconfig", choose "Folder" and "Select" a directory in your Linux partition.
Again, the relevant information is communicated to "init" via the "bootspecs" facility.
And when you choose a savefolder, you always get a proper pupmode=12, never a fiddly pupmode=13.

5. To avoid any chance that Puppy will boot a different copy of the same Puppy version,
the "psfspart" boot parameter (or one of it's aliases, "pdev1", "pupsfs", "pdrv") is required.
And if the Puppy sfs files are stored in a sub-directory of the defined partition, "psfsdir" (or it's alias "psubdir") is required.
Note, that it does not care where the "vmlinuz" and "initrd.gz" files reside, these are already loaded into memory before "init" starts.
The only exception is if you are booting an iso or DVD, i.e. with "pmedia=cd" in which case the "iso9660" partition is chosen.

6. The "saveconfig" utility supports the creation and use of a Luks encrypted partition to contain savefolders.
Instead of having a Luks savefile for a single Puppy, you have a real Luks partition, that can be used to contain the savefolders of many Puppies.

7. The "Archive" save mechanism provides a means of having your cake and eating it too,
i.e having Puppy running totally in RAM, but having changes saved on shutdown and restored on boot.
Good for a limited use Puppy on a cheap usb stick.

8. Multisession DVD support has been enhanced to include the facilities provided by "Overlay init setup", including the selection of sfs file.
So, it's possible to "upgrade" a multisession DVD by writing a replacement sfs file to it,
and then using "Overlay init setup" to select the new sfs file instead of the old sfs file in the appropriate list.
Or you can simply select sfs files located anywhere on a HD, or select a savefolder location on a HD.
The "Archive" save mechanism is implemented for a multisession DVD, each archive file is appended to the DVD at shutdown.

9. "Overlay init setup" also contains a facility to specify some of the "pfix=" boot flags.
This includes a "nosave" facility that works with all save mechanisms, including what we are used to calling "pupmode=12".

10. The "Extra-SFS manager" supports the "Extra" sfs file list, that gets loaded into the stack below the "MAIN" list.
Again, you can select almost any sfs file, to be included in this list.
Unfortunately, extra sfs files are not loaded immediately, they are loaded by "init" because with "overlayfs", the stack cannot be modified after it is mounted.

11. Replacing an sfs file with another of the same name, now leads to "rc.update" displaying the dreaded "next boot will be faster" message.
The information passed from "init" to "rc.update" has been enhanced to include a modification time stamp.
The updating mechanism in "rc.update" has also been enhanced to be a lot more selective to do only what the contents of the "changed" sfs file suggests is needed.

12. Data security is enhanced by "init" always preforming an fsck on a partition before it's first mount.
This is the default behaviour, it can be disabled by booting with "pfix=nofsckp".

13. The timezone that applies during "init" and the subsequent running system can now be specified in "initrd.gz".
When an "overlay_init" Puppy is installed to a usb stick via the "install2usb" script, this is done automatically.
It is also done automatically when "overlay-setup-frugal" is used to convert a "normal" Puppy to an "overlay" Puppy.

14. A read-only stack is mounted as "/initrd/pup_sfs", this contains all the sfs's.
So it's easy to determine if a file exists in any of the loaded sfs's, and if so, read it.

Limitations imposed by "overlayfs":

1. Layers can neither be added to, nor removed from, an "overlayfs" stack.
This means the "sfs_load" as we know it, cannot be supported.

2. The directories that makeup the "overlayfs" stack, cannot be modified directly.
This means there can be no "snapmergepuppy", nor can "PPM" install directly tp "pup_ro1".

3. A read/write "overlayfs" stack, requires 2 directories on the same filesystem for it's top layer.
I have implemented this requirement by always having a "work" directory beside the "save" directory.
This means we can't use a mount point as the top directory, it would have to be 2 directories inside the mount point directory.
So, a "savefile" mechanism could not be implemented in the same manner as current.
Also using a partition as a save layer is not possible.

A note on not "searching":

Whenever there is "searching", there is the chance that more than 1 file exists that matches the search critera,
and the "found" file, might not be the one expected.
There is an advantage in avoiding "searching" if at all possible.

When you install a Puppy, you must know exactly where you installed it.
So to avoid any chance of surprises, why not tell "init" exactly where you installed it.
This "init" insists that you do.

When you select where your changed files should be saved, this "saveconfig" tells "init" exactly where it is saved, via "bootspecs".

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Wed 25 Apr 2018, 21:09    Post subject: Cleanup  

Over the next few days, I will cleanup the download site to remove all the old stuff, leaving just the version 0.9 downloads.
gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Mon 30 Apr 2018, 00:45    Post subject: upupbb-18.05.iso  

I have uploaded a delta for upupbb-18.05.iso http://www.fishprogs.software/puppy/initrd/upupbb-18.05.iso___upupbb_overlay0.9-18.05.iso.delta.

And re-uploaded http://www.fishprogs.software/puppy/initrd/overlay_init_upupbb-0.9.tar to match it.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Mon 30 Apr 2018, 07:00    Post subject:  

I have uploaded http://www.fishprogs.software/puppy/initrd/overlay_compat_upupbb-0.2.pet for upupbb-18.05.
gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Mon 21 May 2018, 00:30    Post subject:  

I have attached "overlay_installers-0.9b.sfs".

There is a bug in "install2usb".
The generated "grub.cfg" file for a uefi install is missing a "pmedia=usbflash" in the first "linux" line.
This only makes a difference if you are installing a "normal" puppy as uefi, since "overlay_init" ignores this "pmedia" bootparameter.

gyro
overlay_installers-0.9b.sfs.gz
Description  gunzip to produce "overlay_installers-0.9b.sfs"
gz

 Download 
Filename  overlay_installers-0.9b.sfs.gz 
Filesize  9.87 KB 
Downloaded  76 Time(s) 
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Mon 21 May 2018, 00:41    Post subject:  

Before anyone asks, yes, this bug exists in the "install2usb" included in overlay_init-0.9, it will be fixed in the next release, probably overlay_init-0.9.1.

Work around:
Simply edit the "grub.cfg" file on the usb stick and insert the missing "pmedia=usbflash"

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 3497
Location: Worcestershire, UK

PostPosted: Wed 13 Jun 2018, 09:41    Post subject:  

Hi gyro

Any plans for updates for woof-ce-master builds?

I tried http://www.fishprogs.software/puppy/initrd/overlay_init_lxpupsc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....

Thanks
peebee

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 1220

PostPosted: Wed 13 Jun 2018, 10:04    Post subject:  

peebee wrote:
Hi gyro

Any plans for updates for woof-ce-master builds?

I tried http://www.fishprogs.software/puppy/initrd/overlay_init_lxpupsc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....

Thanks
peebee


I thought that the OverlayFs didn't support all the pupmodes so it should probably be a boot option rather than the only option.
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Sat 23 Jun 2018, 10:07    Post subject: overlay_init-0.9b.tar  

I have uploaded http://www.fishprogs.software/puppy/initrd/overlay_init-0.9b.tar for curent versions of LxPupSc and upupbb.
This is to reflect recent changes to woof-ce that obselete some of my patches.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1551
Location: Brisbane, Australia

PostPosted: Sat 23 Jun 2018, 10:24    Post subject:  

peebee wrote:
Any plans for updates for woof-ce-master builds?
Yes, but only to patch some exsiting utilities to be compatible with both "normal" and "overlay" puppies.
peebee wrote:
I tried http://www.fishprogs.software/puppy/initrd/overlay_init_lxpupsc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....
What process were you running that failed to find DISTRO_SPECS?
The current "overlay-setup-frugal" uses the "DISTRO_SPECS" from the current "initrd.gz", it does not need "DISTRO_SPECS" in the frugal directory.
I did a manual frugal install of LxPupSc-18.06+1M-k64. (vmlinuz, initrd.gz, and all the .sfs files.)
Copied "overlay_init-0.9b.tar" (see previous post) to this frugal directory.
Then ran "overlay-setup-frugal". It ran to completion.
A re-boot using the resultant frugal directory was sucessfull.

gyro
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 3497
Location: Worcestershire, UK

PostPosted: Sat 23 Jun 2018, 13:17    Post subject:  

gyro wrote:
What process were you running that failed to find DISTRO_SPECS?
The current "overlay-setup-frugal" uses the "DISTRO_SPECS" from the current "initrd.gz", it does not need "DISTRO_SPECS" in the frugal directory.
I did a manual frugal install of LxPupSc-18.06+1M-k64. (vmlinuz, initrd.gz, and all the .sfs files.)
Copied "overlay_init-0.9b.tar" (see previous post) to this frugal directory.
Then ran "overlay-setup-frugal". It ran to completion.
A re-boot using the resultant frugal directory was sucessfull.

gyro

Hi gyro

I was following the instructions in post #1 which makes no mention of "overlay-setup-frugal".....and overlay_setup_frugal does not seem to be in "overlay_init-0.9b.tar" which has just 2 components - initrd.overlay.gz and overlay_mods.sfs

Can you expand the instructions in post #1 to say how to find and run "overlay-setup-frugal"?

I see that "/usr/sbin/overlay-setup-frugal" is in overlay_mods.sfs but do I install the .sfs and then run it or do I extract it and run it??

Thanks
peebee
post#1 wrote:
To use:
In a clean frugal install:
1. Add a copy of DISTRO_SPECS to the frugal install directory.
2. Download the "initrd.gz" and "ydrv_overlay.sfs" into the frugal install directory.
3. Boot, with boot parameters similar to the following:

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 12 of 14 [196 Posts]   Goto page: Previous 1, 2, 3, ..., 10, 11, 12, 13, 14 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0985s ][ Queries: 15 (0.0318s) ][ GZIP on ]