Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

overlay init 0.1 instructions from 07/01/18

#101 Post by Rangan Masti »

Hi, good afternoon. I am using the new initrd on both Slacko64 and Slacko32.
Working fine for me sofar!. Tahnk you. Have a nice day.

Rangan.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: overlay init 0.1 instructions from 07/01/18

#102 Post by gyro »

Rangan Masti wrote:I am using the new initrd on both Slacko64 and Slacko32. Working fine for me sofar!.
Thanks for testing and reporting
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#103 Post by gyro »

01micko wrote:IMHO I don't think a delta is the way to go if the default is to check the filesystem integrity and time skew errors are reported.
The idea of a delta to an iso was to make it easier to try.
I need to create a patched .iso to be able to test CD booting, and installers etc. So I thought I might publish a delta of this iso.
01micko wrote:While it is probably important to check for filesystem errors, couldn't this be done in OS proper? Before a save is created or at least after TZ is set by the user?
No, for most reliable results, fsck should be run before the first mount on every boot, i.e. as this init does.
So idealy the time skew issue needs to be solved in init before the first mount.
If a TIME_ZONE file is present in the initrd.gz, this is exactly what this init script does.

The problem is that the current TIME_ZONE stuff works fine in this test setup environment, where the overlay-setup script can add a TIME_ZONE file to the initrd.gz.
But it won't if there is a published iso, which people expect to just boot directly or install with a standard installer. Since no published iso should contain a TIME_ZONE file.

I have no simple solution at this time.

Theoretically we could modify all the Puppy installers to generate a TIME_ZONE file from the current Puppy and add it to the installed initrd.gz.
(Modifying and maintaining all the Puppy installers is the hard thing here.)
This would not do anything for booting from a CD or iso.

Or the running system could signal the selected TZ to the init script.
This does not resolve the issue for first boot, but does for subsequent boots.
This might be the best compromise.

This TIME_ZONE issue is one of the things I'm working on for version 0.2.

@01micko, thanks for testing, and for the pdf.

gyro

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#104 Post by bigpup »

The TIME_ZONE file:
It's important function is to set the TZ variable, hence the local time-zone in "init", before any partitions are mounted. Thus eliminating any complaints from fsck relatiing to times being in the future.
Since, by default, this "init" does an fsck before the first mount of any partition, I became annoyed with frequently seeing these messages.
This usually indicates your computers internal hardware clock is not working correctly and does not keep correct time.

This is one way to get around this problem.
fake-hwclock
http://manpages.ubuntu.com/manpages/tru ... ock.8.html
The things they do not tell you, are usually the clue to solving the problem.
When I was a kid I wanted to be older.... This is not what I expected :shock:
YaPI(any iso installer)

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#105 Post by gyro »

bigpup wrote:This usually indicates your computers internal hardware clock is not working correctly and does not keep correct time.
Sorry, I have to disagree.
The hardware clock is fine, the problem is that the 'init' script does not know what the actual timezone is, while it's busy mounting and unmounting partitions.
The Puppy tradidional way, is for 'init' to set the timezone to some hardcoded value (hence usually wrong). The problem wih this is that the superblock times in a Linux partition can get skewed into the future either when you reboot to this Puppy or when you reboot to another Puppy, depending on which side of your actual timezone the hardcoded timezone is set to.

The concept of the TIME_ZONE file is to allow 'init' to correctly set the timezone, before any mounts or unmounts of partitions.

The good news about all this timezone stuff is that it does not affect the integrity of the data. It's just annoying messages from fsck.

gyro

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#106 Post by 01micko »

gyro wrote:The problem wih this is that the superblock times in a Linux partition can get skewed into the future either when you reboot to this Puppy or when you reboot to another Puppy, depending on which side of your actual timezone the hardcoded timezone is set to.
What about for ISO booting purposes, hard coding to Pacific/Kiritimati? That is the eastern most time zone so therefore anywhere after must be in the past. I notice fsck doesn't care if the time is in the past. I just checked by skipping my clock back an hour (future superblock error is reported) and skipping it forward an hour and no error reported. Of course I have Australia/Queensland set in my TIME_ZONE file.

In other news, I did the deed with a fresh slacko64 install with kernel updated to 4.9.73 with necessary configs. I approached it a bit differently though. I did the fresh frugal and then booted it, set timezone and from within that install performed all of your modifications. Seems to be working quite well. I have a google-chrome sfs loaded running as 'spot' with added support packages added from PPM (gtk+3, updated mozilla-nss etc).

I'll keep this as a daily driver for a while.

Thanks!
Puppy Linux Blog - contact me for access

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#107 Post by peebee »

Tested with:
LxPupSc-18.01+4T-k64
Kernel 4.14.13 64-bit

All seems fine :D
Cheers
peebee

menu.lst=

Code: Select all

title OverlayFS (sda4/overlayfs)
  uuid 9bb32631-a1f9-425f-81bb-9f816347e8fc
  kernel /overlayfs/vmlinuz  pmedia=atahd pdrv=sda4 psubdir=overlayfs
  initrd /overlayfs/initrd.gz

Code: Select all

====> /initrd/tmp/bootinit.log <====

'FATAL' messages may be insignificant.

Doing fsck of /dev/sda4 as ext4 with e2fsck.
e2fsck 1.43.3 (04-Sep-2016)
Linux-3: clean, 303495/5693440 files, 15090214/22761984 blocks
Mounting /dev/sda4 on /mnt/sda4 as ext4.
Using /mnt/sda4/overlayfs/BOOT_SPECS
Using 'puppy_LxPupSc_18.01.sfs' as sfs0 file...
 Copying /mnt/sda4/overlayfs/puppy_LxPupSc_18.01.sfs to /mnt/tmpfs/
 Mounting /mnt/tmpfs/puppy_LxPupSc_18.01.sfs on /pup_sfs0.
Using 'fdrv_LxPupSc_18.01.sfs' as sfs1 file...
 Copying /mnt/sda4/overlayfs/fdrv_LxPupSc_18.01.sfs to /mnt/tmpfs/
 Mounting /mnt/tmpfs/fdrv_LxPupSc_18.01.sfs on /pup_sfs1.
Using 'zdrv_LxPupSc_18.01.sfs' as sfs2 file...
 Copying /mnt/sda4/overlayfs/zdrv_LxPupSc_18.01.sfs to /mnt/tmpfs/
 Mounting /mnt/tmpfs/zdrv_LxPupSc_18.01.sfs on /pup_sfs2.
Using 'overlay_mods.sfs' as sfs3 file...
 Copying /mnt/sda4/overlayfs/overlay_mods.sfs to /mnt/tmpfs/
 Mounting /mnt/tmpfs/overlay_mods.sfs on /pup_sfs3.
Using 'adrv_LxPupSc_18.01.sfs' as sfs4 file...
 Copying /mnt/sda4/overlayfs/adrv_LxPupSc_18.01.sfs to /mnt/tmpfs/
 Mounting /mnt/tmpfs/adrv_LxPupSc_18.01.sfs on /pup_sfs4.
Adding module kernel/fs/overlayfs/overlay.ko
Mounting pup_sfs3:pup_sfs4:pup_sfs0:pup_sfs1:pup_sfs2 on /pup_sfs...
umount_unneeded - keeping /dev/sda4
Booting with PUPMODE=12
Using personal storage /overlayfs/LxPupScsave [sda4].
Mounting LxPupScsave:pup_sfs on /pup_new...
mount -o move /pup_sfs0 /pup_new/initrd/pup_sfs0
mount -o move /pup_sfs1 /pup_new/initrd/pup_sfs1
mount -o move /pup_sfs2 /pup_new/initrd/pup_sfs2
mount -o move /pup_sfs3 /pup_new/initrd/pup_sfs3
mount -o move /pup_sfs4 /pup_new/initrd/pup_sfs4
mount -o move /pup_sfs /pup_new/initrd/pup_sfs
mount -o move /mnt/sda4 /pup_new/initrd/mnt/sda4
mount -o move /mnt/tmpfs /pup_new/initrd/mnt/tmpfs
'/pup_new/initrd/pup_rw' -> '/initrd/mnt/sda4/overlayfs/LxPupScsave'
'/pup_new/tmp' -> '/initrd/mnt/tmpfs/tmp'
Attachments
Screenshot.png
(179.07 KiB) Downloaded 471 times
Last edited by peebee on Sun 14 Jan 2018, 09:36, edited 1 time in total.
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#108 Post by jamesbond »

01micko wrote:
gyro wrote:The problem wih this is that the superblock times in a Linux partition can get skewed into the future either when you reboot to this Puppy or when you reboot to another Puppy, depending on which side of your actual timezone the hardcoded timezone is set to.
I got interested when I read this, so I've done a few tests.

I found out that this superblock problem only manifest itself when your hardware clock is set to localtime - which is the default settings in most puppies, to accomodate dual booting with Windows. If hwclock is set in UTC, all is good.

I don't know how you do it gyro, but this is how I did it in init (I don't have persistence file in initrd like you do): "[ $TZ ] && hwclock -t" where hwclock is busybox applet similar to coreutils' utility of the same name.

This way, I only need to pass boot parameter TZ=EST-10 and all will be right. Of course, again, this is only necessary when running hwclock in localtime.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#109 Post by peebee »

I've made a delta to produce LxPupSc-18.01+4-overlayfs.iso

See: http://murga-linux.com/puppy/viewtopic. ... 329#980329
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

stemsee

#110 Post by stemsee »

I tried this with artful pup. opened the initrd and put in the DISTRO_SPECS. added kernel arguments pdrv and psubdir and everything else was automatic.
Once at desktop loaded wine-2.2.sfs @200mb. the 32bit pae kernel recognised 7.9gb of 16gb.
Attachments
Screenshot(3).png
(117.18 KiB) Downloaded 157 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#111 Post by gyro »

peebee wrote:I've made a delta to produce LxPupSc-18.01+4-overlayfs.iso

See: http://murga-linux.com/puppy/viewtopic. ... 329#980329
Thanks.
I assume you did not inclued a TIME_ZONE in the initrd.gz in this iso.
gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#112 Post by peebee »

gyro wrote:
peebee wrote:I've made a delta to produce LxPupSc-18.01+4-overlayfs.iso

See: http://murga-linux.com/puppy/viewtopic. ... 329#980329
Thanks.
I assume you did not inclued a TIME_ZONE in the initrd.gz in this iso.
gyro
Yes I did - 01Micko's suggestion as mentioned in the link....
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#113 Post by gyro »

Firstly, thanks folk for taking the time to test this.

Re the timezone thing:
1. Skewing only occurs if the hardware clock is set to local time.
2. If you set it to a fixed value you see the skew reported either:
When you mount a partition with the correct timezone that was umounted with the fixed timezone,
or
When you mount a partition with the fixed timezone that was umounted with the corrent timezone.
3. This init seeks to provide facilities so that the timezone can set consistently for all mounts.
4. The TIME_ZONE file idea makes most sense when you are modifying the initrd.gz before first boot, it does not make as much sense when this intrd.gz is already included in an iso.
5. Version 0.2 will include support for a "ptz" boot parameter. e.g. ptz=AEST-10
6. Version 0.2 will also support the possibility of the time-zone in init being set by some utility in the running system, possibly 'Quick Setup'. The skew issue could still occur on first boot.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#114 Post by gyro »

peebee wrote:
gyro wrote:
peebee wrote:I've made a delta to produce LxPupSc-18.01+4-overlayfs.iso

See: http://murga-linux.com/puppy/viewtopic. ... 329#980329
Thanks.
I assume you did not inclued a TIME_ZONE in the initrd.gz in this iso.
gyro
Yes I did - 01Micko's suggestion as mentioned in the link....
Sorry, but I don't think it's a good idea to use the timzone setting facilities in this manner. If you are going to set the timezone it should be to the correct one.
Which is one of the problems when trying to use version 0.1 in an iso.
Unfortunately version 0.1 effectively sets the timezone to UTC in the absense of a TIME_ZONE file.
Version 0.2 will set it to 'XXX12' (west of everwhere), if it is not set in a TIME_ZONE file or ptz boot parameter.
(I need to test the west v east thing again before releasing version 0.2)

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#115 Post by peebee »

gyro wrote:If you are going to set the timezone it should be to the correct one.
There is no "correct" one for a general purpose .iso.....
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#116 Post by gyro »

peebee wrote:
gyro wrote:If you are going to set the timezone it should be to the correct one.
There is no "correct" one for a general purpose .iso.....
Yes, that's what I was trying to say.
The 'init' needs to have a hardcoded default that ensures that no one sees the issue on first fsck on first boot.
Then provide mechanisms for the correct local timezone to be set in 'init'.
This is what I have coded into "overlay_init-0.2". (Just a bit more testing to do)

Then there is no point in having a TIME_ZONE file in the iso.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#117 Post by gyro »

peebee wrote:I've made a delta to produce LxPupSc-18.01+4-overlayfs.iso

See: http://murga-linux.com/puppy/viewtopic. ... 329#980329
I downloaded this and used it for a fresh manual frugal install.
It appeared to work fine.
I then extracted the TIME_ZONE file from the "initrd.gz" and realised that it appeared to work because it didn't work.
The problem is the TIME_ZONE file:

Code: Select all

<+14>-14|Pacific/Kiritimati
The "init" script assigns the characters before the '|' to the TZ variable. The problem is that the "hwclock" in "initrd.gz" cannot cope with this format, so does not set the expected timezone.
I changed the TIME_ZONE file to:

Code: Select all

LINT-14|Pacific/Kiritimati
and re-inserted it into "intrd.gz", then on first boot I got the superblock timezone warnings from fsck.

When I removed the TIME_ZONE file from "initrd.gz" with "initrdDfile", added "TZ=AEST-10" as a boot parameter, and did another first boot, I got no warnings.

The problem with the TIME_ZONE file is mine, "mk-timezone-file" simply accepted this incompatible format.
This is fixed in the next version which will replace:

Code: Select all

<+14>-14
with

Code: Select all

XXX-14
Appart from this TIME_ZONE thing, the iso worked as expected.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Release of overlay_init-0.2

#118 Post by gyro »

Release of overlay_init-0.2

Download from http://www.fishprogs.software/puppy/ini ... it-0.2.tar.

Changes:

1. New timezone handling including a new format for the "TIME_ZONE" file, example:

Code: Select all

P_TZONE='Australia/Queensland'
P_TZ='AEST-10'
So if you have an existing "TIME_ZONE" file you will need to run "mk-timezone-file" again.
The timezone in "init" can be set in any 1 of 3 ways:
Create a "TIME_ZONE" file with "mk-timezone-file" and add it to the "initrd.gz" with "initrd-pup-file".
Add "TZ=AEST-10" as a boot parameter, use "showTZ" to see an appropriate value for your timezone.
Run "set-init-timezone" after the timezone has been set in the running Puppy. The timezone in "init" will then be set for subsequent boots.
In the absence of any specified timezone, "init" uses "TZ=XXX12", the most westerly timezone.

2. All the timezone handling utilities have been fixed to handle the stange format of some TZ specs at the end of some zoneinfo files.

3. New Targeted update for "rc.update", after sfs stack changes.
When changes to the sfs stack are detected, the sfs is checked to see what updates are required, similar to current "sfs_load".
This is done in both the sfs manager utilities, and "rc.update" itself.
Then "rc.update" only performs the appropriate updates.
"BOOTCONFIG" is no longer used, instead "PREVUNIONRECORD" and "LASTUNIONRECORD" are files on their own.

4. Changed the "Archive" save mechanism to use "PUPMODE=37".

5. Renamed some CLI utilities:
bootspecs2tmpbs -> bootspecs-init
tmpbs2bootspecs -> bootspecs-save
tmpbs2variable -> bs-read-var
variable2tmpbs -> bs-write-var
tmpbsDvariable -> bs-delete-var
initrd2file -> initrd-get-file
file2initrd -> initrd-put-file
initrdDfile -> initrd-delete-file

6. Combined "lsoverlay" and "showoverlay", they now both provide the same output.

7. Removed support for "pimod" boot parameter, it's not really needed.

To try this version, use the same procedure as for overlay_init-0.1,
see http://www.murga-linux.com/puppy/viewto ... 6&start=96

Manual update of overlay_init-0.1 install, that might work:
1. Extract the tar file.
2. Replace the current "overlay_mods.sfs".
3. Extract the updated "init" from "initrd.overlay.gz" with "initrd2file" or "initrd-get-file".
4. Store this updated "init" in the currrent "initrd.gz" with "file2initrd" or "initrd-pup_file".
5. If your current "initrd.gz" contains a "TIME_ZONE" file you need to replace it with a new one.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#119 Post by peebee »

New delta for LxPupSc made with overlay_init-0.2....

Timezone not altered during make process - boots with GMT-8....
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#120 Post by gyro »

peebee wrote:New delta for LxPupSc made with overlay_init-0.2....

Timezone not altered during make process - boots with GMT-8....
Thanks, I'll give it a spin.

Later:
It works as expected.
I did a normal manual frugal install from the iso.
Booted, and as expected the timezone was GMT-8.
After doing "Quick Setup", I ran "mk-timezone-file", and added it into the "initrd.gz" with "initrd-put-file".
On first shutdown I chose 'Archive'.
On reboot all was as expected, I did not get a first boot, all the choices from the first "Quick Setup" were still in place.

gyro

Post Reply