Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 21 Oct 2017, 01:13
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Another re-write of the "init" script - using OverlayFs?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 7 [93 Posts]   Goto page: 1, 2, 3, 4, 5, 6, 7 Next
Author Message
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Tue 30 May 2017, 05:37    Post subject:  Another re-write of the "init" script - using OverlayFs?
Subject description: It works, but now to explore the possabilities
 

Edit:
---Current files available for download---

Replacements for standard puppy files:
http://www.fishprogs.software/puppy/initrd/init
http://www.fishprogs.software/puppy/initrd/initrd.gz
http://www.fishprogs.software/puppy/initrd/ydrv_overlay.sfs

Possibly usefuly utilities:
http://www.fishprogs.software/puppy/initrd/file2initrd
http://www.fishprogs.software/puppy/initrd/initrd2file

An example text file:
http://www.fishprogs.software/puppy/initrd/TIME_ZONE

To use:
In a clean frugal install:
1. Add a copy of DISTRO_SPECS to the frugal install directory.
2. Download the "initrd.gz" and "ydrv_overlay.sfs" into the frugal install directory.
3. Boot, with boot parameters similar to the following:
Code:
pdrv=sdc2:/slacko/ ydrv=:ydrv_overlay.sfs
Note: If the kernel in your puppy does not support "overlayfs", "init" will fail.

---Original message---
I'm not sure if this is a warning, a threat or a promise.
I'm working on another rewrite of the "init" script.

Amongst other things, I want to explore some different technologies, one of these is OverlayFs.
Some folk are saying that aufs is dead, that OverlayFs is the way of the future because it is included in the kernel.
This prediction may or may not happen, but I am interested in finding out what it would mean for puppy it we moved to OverlayFs as a replacement for aufs.

The results of my reading research are:

1) OverlayFs is much simpler that aufs, it seems to lack any ability to modify an existing stack.
If this proves to be true in practice, then the stack would have to be created once in "init", and then remain unchanged for the session.
"sfs_load" would be reduced to being an sfs manager whose results did not take effect until after a reboot.
Utilities that execute sfs files, by loading them into the aufs stack, executing the program inside and then unloading them would go.
There might also be some challanges with pupmode=13, since what happens when a directory that is part of an overlay is modified directly is undefined. So "snapmergepuppy" might have to go.

2) There are similarities, a number of directories are combined together to form a stack,
sfs files have to be mounted and their mountpoints used in forming the stack,
it uses whiteout files in the rw layer to hide files in the ro layers that have been "deleted".

3) But it can have read only overlays, and an overlay can consist of more than 1 directory, which could themselves be overlays.
So we could create a puppy-sfs overlay(ro) containing all the puppy system sfs's, and an extra-sfs overlay(ro) and a save-layer overlay(rw) and then mount all these as a puppy overlay.
Or we could build an sfs overlay(ro) that contains all the sfs's in the current session, and combine that with a save-layer directory to produce the puppy overlay. One of the advantages of this approach is that it would be easy to find if a file exists in any of the sfs's, by simply testing it's existence in the directory where the sfs overlay is mounted.
Or we could build an sfs overlay gradually by combining the previous sfs overlay with the directory for this sfs file to produce a new overlay, and so on. The challange with this approach is that each new overlay needs a new directory.

The next thing to do is to try using OverlayFs, so my questions are:
Has anyone tried to use OverlayFs?
Since it is a kernel facility, is it enabled in current Puppy kernels?
Or do I need do a kernel compile before I can try using it?

gyro

Last edited by gyro on Thu 22 Jun 2017, 16:47; edited 4 times in total
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2930
Location: Worcestershire, UK

PostPosted: Tue 30 May 2017, 14:32    Post subject:  

BarryK has posted some blogs about use of overlayfs - e.g.
http://barryk.org/news/?viewDetailed=00499 - search by content for others....

In my /etc/modules/DOTconfig-4.11.2-210517 it says:
Quote:
# CONFIG_OVERLAY_FS is not set

so I would imagine it would need to be set and the kernel re-built.....

Cheers
peebee

_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Wed 31 May 2017, 10:31    Post subject:  

@peebee,
Thanks.

Unfortunately BK confirms the limitations of overlayfs.
Although I think a pupmode=12 should be doable.
A pupmode=5 might be a bit more difficult, might have to replace tmpfs with a zram.
A pupmode=13 would be the biggest challange, probably have to create a new save layer (pup_ro1) on shutdown and have "init" use the new one and delete the old one.
So still some possibilities that could be explored.

Looks like a kernel compile is needed if I want to try this stuff.

gyro
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 8282
Location: Perth, Western Australia

PostPosted: Thu 01 Jun 2017, 09:02    Post subject:  

I am using overlay f.s. for Quirky and Easy. Though, Quirky is a full installation and doesn't need it, except it is used in woofQ, in the '3builddistro' script -- and I presume, still, also in woofCE.

Yes, only the top layer is rw, all layers below are ro, and cannot be written to at all. So, it does pose a challenge for some of those pupmodes.

In Easy, I am using just one mode, the rw layer is a folder in a drive, requiring a partition with Linux f.s. So there is no need for a save-file, nor a snapmergepuppy.
This is a restriction, cannot use a fat or ntfs partition.

_________________
http://barryk.org/news/
Back to top
View user's profile Send private message Visit poster's website 
rufwoof

Joined: 24 Feb 2014
Posts: 2145

PostPosted: Fri 02 Jun 2017, 09:39    Post subject:  

To upgrade temporarily from Debian Jessie frugal (aufs) to Stretch (overlayfs) I had to install aufs.dkms and add a kernel boot parameter of union=aufs After that it frugally booted/ran fine and sfs's can be loaded as normal. Not as ingrained as using in-built overlayfs, but looks to me that you might have to pair overlayfs with btrfs which is still somewhat evolving (less stable).
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Sun 04 Jun 2017, 05:56    Post subject:  

@BarryK,
Thanks for confirming that you have working, somethig that is effectively a pupmode=12 with a savefolder.
We'll see what happens to the rest.

@rufwoof,
Sorry, I don't understand your comment.
I'm not aware of any woof-ce based puppies that use overlays.
And I don't get what suggests that overlays work better on btrfs.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Sun 04 Jun 2017, 06:02    Post subject:  

I guess what I would really like to know is, any information on how to modify an overlay stack, if it's possible?

gyro
Back to top
View user's profile Send private message 
rufwoof

Joined: 24 Feb 2014
Posts: 2145

PostPosted: Sun 04 Jun 2017, 15:57    Post subject:  

gyro wrote:
@rufwoof,
Sorry, I don't understand your comment.
I'm not aware of any woof-ce based puppies that use overlays.
And I don't get what suggests that overlays work better on btrfs.

I thought the thread was to address a new init (puppy) that was based on using overlayfs.

With btrfs you can set things up as though one set of files/folders were the entire partition/volume, but where another set of files/folders were also as though in their own unique volume. i.e. one linux in one subvolume, another in another subvolume, where snapshots and 'backintime' type instances can be managed/used.

Not sure, but I think Barry's easylinux might be using btrfs ???

Haven't really thought it through, but to modify a overlayfs under btrfs it might be possible to set up a overlay that uses a particular subvolume, and then perhaps change that out for another choice of subvolume ... as though the overlayfs had been modified.

I believe a overlayfs can include overlays. Such that a existing overlayfs that was transitioned over to be one of the (read only) source of another overlays might be managed so that it seemed that the currently used (rw) overlayfs had been modified/extended so-to-speak. Combined with COW (copy on write) you might even have copies/versions of changes being applied to otherwise read-only layers if so needed.

The negative side is that there are rumours of a data-eating bug in some arrangements i.e. its perhaps not mature enough yet for use in anything other than relatively straight forward setups, and where using the likes of the above is more inclined to hit such data corruption problems.
Back to top
View user's profile Send private message 
Rangan Masti

Joined: 01 Jan 2013
Posts: 32
Location: Germany, Berlin

PostPosted: Mon 05 Jun 2017, 04:19    Post subject:  

I am interested in this and waiting egerly the init. Wish you good success!!!
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 8282
Location: Perth, Western Australia

PostPosted: Mon 05 Jun 2017, 09:22    Post subject:  

rufwoof wrote:
Not sure, but I think Barry's easylinux might be using btrfs ???


No. The f.s. is ext4.

_________________
http://barryk.org/news/
Back to top
View user's profile Send private message Visit poster's website 
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Mon 05 Jun 2017, 11:33    Post subject:  

I intend to test using ext4, my overlays will consist of simply stacking directories.

I now have a little test bed "init" that supports only pupmode=5, but no overlays yet, still using aufs.
I intend to test a save layer mechanism based on an sfs, first.
Then try doing it again with overlays.

I haven't even got a puppy running that is capable of overlays, yet.

I should be clear, the use of overlays is only one of the new things I'm trying with the new init.

gyro
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1401
Location: Brisbane, Australia

PostPosted: Tue 06 Jun 2017, 06:59    Post subject:  

Just in case anyone is interested, my current test "init" can be downloaded from http://www.fishprogs.software/puppy/initrd/init.
There is also an "initrd.gz" at http://www.fishprogs.software/puppy/initrd/initrd.gz.

The "initrd.gz" does not contain any DISTRO_SPECS, so it can boot many recent woof-ce based puppies.
To test on an existing frugal install, (a testing one, not your main work one).
1. In the frugal install directory, rename the current "initrd.gz" to something else, (as backup).
2. Download the test "initrd.gz" and move it into the frugal install directory.
3. Copy "etc/DISTRO_SPECS" from inside your savefolder into the install directory, (beside "initrd.gz")
4. reboot.

To return to normal, just delete the new "initrd.gz" and rename your original one back to "initrd.gz"

Note1: This is a simple pupmode=5 boot, it will ignore your savefolder. So, don't bother setting up a save on first shutdown, just click on "No".

Note2: It does no searching at all, so you must tell it where the frugal install directory is with boot parameters, e.g. "pdrv=Work:/puppy/slacko/", where "Work" is the label on the ext4 partition that contains my frugal install directory ("/puppy/slacko/").

gyro
Back to top
View user's profile Send private message 
anikin

Joined: 10 May 2012
Posts: 956

PostPosted: Tue 06 Jun 2017, 09:05    Post subject:  

gyro wrote:
...
The next thing to do is to try using OverlayFs, so my questions are:
Has anyone tried to use OverlayFs?
Since it is a kernel facility, is it enabled in current Puppy kernels?
Or do I need do a kernel compile before I can try using it?

gyro

Just have a look at your kernel dotconfig. For example, my kernel has both aufs and overlayfs enabled like this:
Code:
CONFIG_AUFS_FS=m
CONFIG_OVERLAY_FS=m
Now you can choose between the two by adding either
Code:
union=aufs
or
Code:
union=overlay
as a kernel boot parameter.

I have some limited experience using a Debian Live system with overlayfs. As I'm not that much interested in its stacking or overlay (save) capabilities, it runs OK for me. However, it also has a huge drawback - an extremely slow and weird boot process.
Back to top
View user's profile Send private message 
peebee


Joined: 21 Sep 2008
Posts: 2930
Location: Worcestershire, UK

PostPosted: Tue 06 Jun 2017, 15:22    Post subject:  

Followed the instructions in post #956788 including putting DISTRO_SPECS into the frugal install folder - but did not boot - error message that DISTRO_SPECS could not be found.....

menu.lst entry is
Code:
title Devtest (sda3/devtest)
  uuid 95a633e7-6fab-431d-9fa8-550812ab03f8
  kernel /devtest/vmlinuz  pmedia=atahd psubdir=devtest pfix=fsck
  initrd /devtest/initrd.gz
What am I doing wrong?? Does DISTRO_SPECS have to be inside folder /etc???
_________________
LxPup = Puppy + LXDE
Back to top
View user's profile Send private message Visit poster's website 
Keef


Joined: 20 Dec 2007
Posts: 864
Location: Staffordshire

PostPosted: Tue 06 Jun 2017, 15:50    Post subject:  

You need something like
Code:
pdrv=sda2:/tahr/


Also (when using Tahr), the zdrv needs finding (if you need it for anything)
Code:
zdrv=sda2:/tahr/zdrv_tahr_6.0.6.sfs
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 7 [93 Posts]   Goto page: 1, 2, 3, 4, 5, 6, 7 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0853s ][ Queries: 14 (0.0079s) ][ GZIP on ]