RC7 (STABLE) WeeDogLinux Arch 64 now released

A home for all kinds of Puppy related projects
Message
Author
wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#681 Post by wiak »

rufwoof wrote:I boot with changes=RAM, so upper_changes is mounted to /mnt/layers/RAM/upper_changes.
Yes I also prefer to boot with changes=RAM since I don't always want to save what I'm doing (in fact I often don't). So that for me is the mode I'd use for saving out the, in that case, RAM-based upper_changes. I've been too busy myself on the core build script functionality to try saving out these changes. But I have glanced at the various mechanisms rufwoof has been using, and they all look good to me. So thus far that includes a general utility for simply merging together numbered upper_changes layers, or a simple compressed tar archive which gets untarred back in again (different, but a bit like tinycore Linux save archive), or rufwoof's latest .bak file idea, which is quite novel by the look of it.
Sorry, half asleep just now and typing on android phone so probably typos or autocorrect errors.

wiak

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#682 Post by rufwoof »

This is the full version of my /etc/rc.shutdown code. Note how I use lsblk to list available partitions as a aid to identifying the usb, or record the lsblk list before, then prompt to attach a usb, and then lsblk again and pick out the differences as a means to 'automatically' identify the usb. That's because device names are dynamic (random) i.e. a usb on /dev/sdb1 one time might be /dev/sdc1 the next.

Code: Select all

# Default rc.shutdown for void; add your custom commands here.
#
# This is run by runit in stage 3 after the services are stopped
# (see /etc/runit/3).
# Default rc.shutdown for void; add your custom commands here.
#
# This is run by runit in stage 3 after the services are stopped
# (see /etc/runit/3).

clear
echo -n "Do you want to save changes y/n "
read -n1 n
echo
if [ "$n" == "y" ]; then
    bootfrom=$(grep bootfrom /proc/cmdline | awk -F= '{print $2}' | awk '{print $1'})
    cd $bootfrom
    if [ -f 02changes.sfs ]; then
	if [ -f changes.sfs.bak ]; then
	    rm changes.sfs.bak
	fi
	mv 02changes.sfs changes.sfs.bak
	# renaming the sfs on disk has losetup auto renamed as well so changes.sfs.bak now.
	CL=`losetup | grep changes.sfs.bak | awk '{print $1}'`
	mkdir /mnt/CL
	mkdir /mnt/TOP
	mount -t squashfs -o loop $CL /mnt/CL
	mount -t overlay overlay -o lowerdir=/mnt/layers/RAM/upper_changes:/mnt/CL /mnt/TOP
	sync
	mksquashfs /mnt/TOP 02changes.sfs -comp lz4 -Xhc
	sync
	umount -lf /mnt/TOP
	umount /mnt/CL
	sync
	rmdir /mnt/TOP
	rmdir /mnt/CL
    else # no 02changes.sfs being used so we don't have to merge upper_changes with that
	mksquashfs /mnt/layers/RAM/upper_changes 02changes.sfs -comp lz4 -Xhc
    fi
    clear
    while :; do
	echo Here is a list of devices
	lsblk -l | grep part | grep -Ev loop | awk '{print $1}' >/tmp/l1
	cat /tmp/l1
	echo -n "Enter the usb device name i.e sdb1 or attach the usb now and press Enter "
	read D
	if [ -z "$D" ]; then
	    sleep 1
	    lsblk -l | grep part | grep -Ev loop | awk '{print $1}' >/tmp/l2
	    D=$(diff --side-by-side --suppress-common-lines \
             /tmp/l1 /tmp/l2 | awk -F'>' '{print $2}' | sed 's/ //g' | sed $'s/\t//g')
	fi
	[ -f /tmp/l1 ] && rm /tmp/l1
	[ -f /tmp/l1 ] && rm /tmp/l2

	# validate
	[ ! -d /mnt/$D ] && mkdir /mnt/$D
	mount /dev/$D /mnt/$D >/dev/null 2>&1
	if [ -f /mnt/$D/VOID6/initramfs05.gz ]; then
	    break
	else
	    rmdir /mnt/$D
	    clear
	    echo "That doesn't look like a valid boot (save) usb ... trying again"
	fi
    done
    # update [md5sum filesize] recording on usb for intrusion detection
    store02md5 $bootfrom /mnt/$D/VOID6/initramfs05.gz
fi
store02md5 looks like the following. Basically when a new save is made (new 02changes.sfs created) it just writes the concatenated md5sum and filesize for 02changes.sfs to the usb's initramfs (inside), so my bootup process (rdsh1.plug) can measure the current md5sum and filesize checksum for the hdd stored 02changes.sfs and compare it with the one recorded inside initramfs (/02.md5) and flag if that differs (potential intrusion detected). I also have a store01md5 that is very similar, but records the 01firstrib_rootfs.sfs and rdsh1.plug files md5sum/filesize(s) for checking at bootup.

You have to store/compare both the md5sum and filesize as otherwise its relatively trivial to just add on bytes at the end of a modified file until its md5sum is the same as the original files md5sum.

Code: Select all

#!/bin/bash

# $1 where hdd 02changes.sfs is stored i.e. /mnt/sda1/VOID6
# $2 pre mounted location of usb's initramfs05.gz i.e. /mnt/sdb1/VOID6/initramfs05.sfs

_usage () {
    echo Requires two parameters
    echo "$1 where hdd 02changes.sfs is stored i.e. /mnt/sda1/VOID6"
    echo "$2 pre mounted location of usb's initramfs05.gz i.e. /mnt/sdb1/VOID6/initramfs05.gz"
    exit
}

if [ -z $2 ]; then
	_usage
fi

if [ ! -f $2 ]; then
    _usage
fi

if [ ! -f $1/02changes.sfs ]; then
    echo $1/02changes.sfs not found
    _usage
fi

CD=`pwd`
cd /tmp
mkdir extracted
cd extracted
echo Extracting usb initramfs
zcat $2 | cpio -id
cd $1
echo Recording md5sum
# use the same method as in initramfs (rdsh1.plug)
M=`busybox md5sum 02changes.sfs`
S=`busybox stat -c %s 02changes.sfs`
echo -n "$M $S" > /tmp/extracted/02.md5
# { md5sum 02changes.sfs; stat --printf="%s" 02changes.sfs; } | tr "\n" " " >/tmp/extracted/02.md5
cd /tmp/extracted
echo Recreating the initramfs
find . | cpio -o -H newc | gzip >$2
cd ..
echo "Syncing (flushing)"
sync
rm -rf extracted
echo "Done. Don't forget to umount and unplug the usb"
cd $CD
This is pretty much my rdsh1.plug content, except I use a ansi art voidlinu logo in the logon function

Code: Select all

_logo () {
	echo voidlinux
}

SFSLOC=`env | grep bootfrom | awk -F= '{print $2}'`
CD=`pwd`
cd $SFSLOC

_logo
echo "Running Intrusion Detection checks ... please wait "
if [ ! -f 01firstrib_rootfs.sfs ]; then
    echo
    echo No 01firstrib_rootfs.sfs file found
    echo -n "Press Enter "
    read n
else # Don't bother wasting time if there's no recorded md5
    if [ -f /01.md5 ]; then
	M=`md5sum 01firstrib_rootfs.sfs`
	S=`stat -c %s 01firstrib_rootfs.sfs`
	echo -n "$M $S" > /tmp/01.md5
    fi
fi
if [ -f 02changes.sfs ]; then
    if [ -f /02.md5 ]; then
	M=`md5sum 02changes.sfs`
	S=`stat -c %s 02changes.sfs`
	echo -n "$M $S" > /tmp/02.md5
    fi
fi
if [ -f rdsh1.plug ]; then
    if [ -f /rdsh1.plug.md5 ]; then
	M=`md5sum rdsh1.plug`
	S=`stat -c %s rdsh1.plug`
	echo -n "$M $S" > /tmp/rdsh1.plug.md5
    fi
fi

_logo
if [ -f /01.md5 ] && [ -f 01firstrib_rootfs.sfs ]; then
    D01=`diff /01.md5 /tmp/01.md5`
    if [ ! -z "$D01" ]; then
	echo "01firstrib_rootfs.sfs md5 checksum or size does not match"
	echo "That could mean the sfs on HDD may have been tampered with"
	echo -n "Power down and investigate, or Press Enter to continue "
	read n
    else
	echo "01firstrib_rootfs.sfs validated as OK"
    fi
else
    echo "No checksum recording found for 01firstrib_rootfs.sfs"
    echo "Unable to validate, integrity checks not fully operational"
fi
if [ -f /02.md5 ] && [ -f 02changes.sfs ]; then
    D01=`diff /02.md5 /tmp/02.md5`
    if [ ! -z "$D01" ]; then
	echo "02changes.sfs md5 checksum or size does not match"
	echo "That could mean the sfs on HDD may have been tampered with"
	echo -n "Power down and investigate, or Press Enter to continue "
	read n
    else
	echo "02changes.sfs validated as OK"
    fi
else
    if [ -f 02changes.sfs ]; then
	echo "No checksum recording found for 02changes.sfs"
	echo "Unable to validate, integrity checks not fully operational"
    fi
fi

if [ ! -f /01.md5 ]; then
    _logo
    echo "No pre-recorded checksum for 01firstrib_rootfs.sfs"
fi

# check our rdsh1.plug hasn't been tampered with
if [ -f /rdsh1.plug.md5 ] && [ -f rdsh1.plug ]; then
    D01=`diff /rdsh1.plug.md5 /tmp/rdsh1.plug.md5`
    if [ ! -z "$D01" ]; then
	echo "rdsh1.plug md5 checksum or size does not match"
	echo "That could mean the sfs on HDD may have been tampered with"
	echo -n "Power down and investigate, or Press Enter to continue "
	read n
    else
	echo "rdsh1.plug validated as OK"
    fi
else
    if [ -f rdsh1.plug ]; then
	echo "No checksum recording found for rdsh1.plug.sfs"
	echo "Unable to validate, integrity checks not fully operational"
    fi
fi

if [ -f /tmp/rdsh1.plug.md5 ]; then
    rm /tmp/rdsh1.plug.md5
fi
if [ -f /tmp/01.md5 ]; then
	rm /tmp/01.md5
fi
if [ -f /tmp/02.md5 ]; then
	rm /tmp/02.md5
fi
cd $CD
echo
echo -n "Remove USB and Press Enter "
read n
clear
I boot from usb, so mbr, bootloader, vmlinux, initramfs are all on usb, and part way through bootup it prompts to remove the usb. 01firstrib_rootfs.sfs, 02changes.sfs and rdsh1.plug are on hdd, not copied into ram just run from where they are, so only what is actually needed actually gets read (and typically stored in cache), using lz4 decompression (which is very fast). And where those files are validated (md5sum/filesize as above) against the 'key' (recording of md5sum and filesize) stored on the usb.

Really I should move 02changes.sfs up to a higher number filename in case of wanting to load sfs's below that at bootup, perhaps rename it 10changes.sfs instead.

Another benefit of having changes=RAM is that any changes actually made are lost at shutdown/reboot. If upper_changes (changes) were stored on disk then even if deleted/not saved at shutdown, they're still there, just flagged as 'available' blocks ... that could be read (all your browser activities/cache, opened documents content ...etc.). I also load a swap file at bootup, but use encryption for that as well ... for similar reason. Its encrypted with a random session key at each bootup that not even I know, so the content is in effect unreadable without that key.

Presently I set it to use a encrypted swap that is 50% of the size of memory as typically when it does get used it tends to use relatively little unless its a runaway process in which case all of it will tend to get used no matter how much is available. I invoke it in /etc/rc.local with a line of /usr/local/bin/swap-encrypted.sh sda1 (i.e. the script is called swap-encrypted.sh and is stored in /usr/local/bin)

Code: Select all

# Requires xbps-install cryptsetup

T=`swapon -s | grep partition | cut -d' ' -f1`
if [ ! -z $T ]; then
   clear
   echo "Looks like you already have a swap partition activated"
   sleep 3
   exit
fi

# swap file device partition i.e. sdc1 passed on kernel boot cmd line
SF=`cat /proc/cmdline | awk -F'swappartition=' '{print $2}' | awk '{print $1}'`
if [ -z ${SF} ]; then  # if no kernel boot parameter for swappartition= then
	SF=$1          # maybe being run as a script and passed $1 argument
fi
if [ ! -z ${SF} ]; then
	if [ ! -d /mnt/${SF} ]; then
    		mkdir -p /mnt/${SF}
	fi
	mount /dev/${SF} /mnt/${SF}
	C=$?
	if [ $C -eq 0 ] || [ $C -eq 32 ]; then  # 32 if already mounted
		cd /mnt/${SF}
		if [ ! -f swapfile.crypt ]; then
			MEM=$(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE) / (1024 * 2048)))
   			echo "creating $MEM MB swapfile /mnt/${SF}/swapfile.crypt ... please wait"
   			echo "Once created once it can be reused in later sessions"
   			dd if=/dev/urandom of=swapfile.crypt bs=1024k count=$MEM
		fi
		# Reuse prior loop if had already been created
		loop=`losetup | grep swapfile.crypt | cut -d' ' -f1 | head -1`
		if [ -z $loop ]; then   # otherwise add new loop
		    loop=$(losetup -f)
		    losetup ${loop} swapfile.crypt
		fi
		cryptsetup open --type plain --key-file /dev/urandom ${loop} swapfile
		mkswap /dev/mapper/swapfile
		swapon -f /dev/mapper/swapfile
	else
		echo "Error locating device /dev/${SF} and mounting to /mnt/${SF}"
		echo "Either not specified as a kernel boot parameter i.e. swappartition=sda1"
		echo "or this script run without a parameter i.e. $0 sda1"
		echo "or in mc, highlight the script and run the mc command line"
		echo "sh %f sda1 (or whatever alternative location to sda1)"
		sleep 4
	fi
else
   clear
   echo "no swappartition=sda1 (or similar) swap partition kernel boot parameter"
   echo "or no sda1 (or similar) command line parameter for swap"
   echo 
   echo "If you're running this script from within mc then highlight it and run"
   echo "if from the mc command line using something like ... sh %f sda1"
   sleep 5
fi
If you store data outside of the OS region then there's little need to use 'save'. And when you do want to reconfigure things just boot, reconfigure and save - so the changes are applied and stored from a known 'clean' version. Otherwise just boot, use, shutdown without saving.

I don't save bookmarks in the browser as the above way means a clean browser after each reboot. Instead I have a text file in data space that I load up in a tilda terminal window that is available via F1 (that toggles show/hide of tilda) and where tilda supports clickable links so text of http://murga-linux.com/puppy in that text file for instance will underline when the mouse is hovered over it, and activate chromium to that link if clicked. A little annoying to go to a regular web site and have it pop up with all the first time visitor popups, but after a while you just accept that and click the OK to accept the terms/conditions.

It's way better to be booting a known clean system each and every time than it is to try and keep nasty stuff out 24/7 and where even just a single slip at some point in the past might have seen the system turn from clean to dirty. Yet as the above scripts outline, its relatively simple to set up a reasonably secure setup/system (data and online security are separate matters).
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#683 Post by rufwoof »

As per the other day, I guess if rdsh1.plug, main sfs and changes sfs can all be stored/run-from a ntfs folder, then so also might a encrypted swap file. If you're also booting from usb then a Windows HDD might remain pretty much as-is (best to just create a folder such as c:\voidlinux and store the sfs's, .plug and swapfile in that directory). Other than that, just need to get the Windows box to boot from usb before hdd so whenever the usb is attached at bootup (or selected) it boots into voidlinux instead of Windows. Whilst the intrusion detection mechanism ensures that even if the Windows box is infected, then the linux boot will detect anything 'unusual'.

When I build a voidlinux setup I record copies of the build scripts within that (including the firstrib plug (build script)) i.e. on usb, so worse come to worse, 20 minutes of rebuild time or whatever using those scripts could have the system back up and running again as before.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#684 Post by rufwoof »

md5sum is a lag factor when the file being checked is large. So on my two core laptop I did some timing trials of using straight md5sum compared to splitting it into two halves (two md5sum's) running concurrently and each assigned to its own core (nice ... timeset). A crude test with some overlap of the two halves. Overall the comparison of straight md5sum was much the same as the split based execution timing :(

Code: Select all

#!/bin/sh

# Might overlap slightly, but I'm looking at creating two md5sum's instead of
# just one as potentially quicker for applying to large files

SZ=`wc -c $1 | awk '{print $1}'`
echo size is $SZ
HALF=`expr $SZ / 2`
echo half is $HALF

nice -n -20 taskset 1 dd if=$1 bs=$HALF count=1 | md5sum &
nice -n -20 taskset 2 dd if=$1 bs=$HALF skip=1 | md5sum &

wait # wait for both background tasks to complete
Busybox in initramfs does have nice, but I don't know whether it actually supports taskset (binding to a particular cpu core) anyway, so even had it been quicker in user space not so sure it would have worked in initramfs.

As a reminder taskset is a bitwise operator 2 = 10 binary = core 1, 1 = 01 binary = core 0 (3 = 11 binary = both core 0 and 1 ..etc.). Nice -n -20 is allocate the highest priority possible (negative 20) ...down to least favourable for the process of 19 (positive 19).
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#685 Post by wiak »

I've had trouble with 'nice' command in the pass not actually changing process priority, but so long ago I can't remember the details. Something to do with Linux scheduler not using it. I'll have to google and see if I can remember why it didn't work the way its manage suggested. Or maybe I'm dreaming that issue since distant past attempts anyway.

I meant to look into that USB random partition boot matter. I've never had that on my machine but had meant to use a flag file for safety and had thought of the code but then forgot to implement.

wiak

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#686 Post by wiak »

rufwoof wrote:This is the full version of my /etc/rc.shutdown code.
I put link to your post from the User Contributed HowTo's thread (I put it under the Utilities section, thanks):

http://www.murga-linux.com/puppy/viewto ... 29#1029029

I've promised myself to try some save changes on exit mechanisms soon as I have some time back to see what will work well for me. I probably won't have any complex system (e.g. I don't use encrypted swap or filesystems) but will experiment and try out some parts of your variations too rufwoof.

wiak

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#687 Post by rockedge »

I am wondering how the kernel upgrades are used in WeeDog.
with xbps-install -Su the updates roll along but I am never using the newer upgraded version it seems.

Do I have to re-run the build_weedog_initramfs05_s205.sh script and build a new vmlinuz ?

I've run

Code: Select all

vkpurge  list
vkpurge rm all
and have installed xtools and looked at

Code: Select all

xcheckrestart list
_

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#688 Post by wiak »

rockedge wrote:I am wondering how the kernel upgrades are used in WeeDog.
_
Good question. The thought had crossed my mind too.
Yes, you certainly need to get the appropriate modules into 01firstrib_rootfs.sfs and also to rebuild the initramfs05.gz so these modules are also available in there (unless using 00firstrib_firmware_modules.sfs arrangement, in which case you'd want to rebuild that, albeit only with modules inside, not firmware - the sfs name was used because previoulsy used for Bionic zdrv firmware_modules, but with Void it only needs to contain modules if using it). However, you won't lose any of your previous work via the above procedures since that remains untouched.

EDIT: At the moment what I'd do is (untested right now):

EDIT2: Currently working on a script and testing it (current code printed towards end of this post FYI but, as I say, not guaranteed yet since just under test)

EDIT3: yes, this is tricky... There are issues with vkpurge - doesn't really work as expected in chroot, and though you 'could' upgrade kernel (new one ending up, for example, in upper_changes), the initramfs would also have to be upgraded in sync with that (since needs new kernel modules); I'm continuing to look into this less-than-trivial issue. Below is not a complete answer at all I think.

After CHECKING you really want to proceed (especially since about to delete an existing initramfs05.gz)... remove any existing:

firstrib_rootfs_for_initramfs_sNNN
uncompressed firstrib_rootfs
initramfs05.gz
vmlinuzNNNNN

Though I'd suggest making backups first, either manually, or using code such as:

Code: Select all

[ -d squashfs-root ] && rm -rf squashfs-root 
[ -d firstrib_rootfs_for_initramfs_sNNN ] && mv firstrib_rootfs_for_initramfs_sNNN BKUPfirstrib_rootfs_for_initramfs_sNNN
[ -d firstrib_rootfs ] && mv firstrib_rootfs BKUPfirstrib_rootfs
[ -f 01firstrib_rootfs.sfs ] && mv 01firstrib_rootfs.sfs BKUP01firstrib_rootfs.sfs
[ -f initramfs05.gz ] && mv initramfs05.gz BKUPinitramfs05.gz
I suggest putting BKUP at front of name as above, rather than at end, so has effect also of NNlayer disabled. Also alphabetically groups all the backups together in file list.

Then:

1. unsquash the 01filesystem.sfs (or its backup) and make sure the resultant squashfs-root folder is renamed to firstrib_rootfs. That is:

Code: Select all

unsquashfs BKUP01firstrib_rootfs.sfs
mv squashfs-root firstrib_rootfs 
EDIT NOTE WELL: Don't do WRONG 2 and WRONG 3 immediately below... you instead need to remove the current kernel using vkpurge, inside a chroot, as I detail later (just done a quick try)
WRONG: 2. rm -rf firstrib_rootfs/usr/lib/modules # DON'T DO THIS
WRONG: 3. rm -rf firstrib_rootfs/boot/vmlinuz* # DON'T DO THIS

CORRECT (I think... but still testing...):

2.

Code: Select all

./mount_chrootver007.sh
to chroot into firstrib_rootfs, and from there:

3. Now you could check current kernel(s) installed using, say (-l is small ell). However, since going to use vkpurge in next step, this check is probably unnecessary in practice:

Code: Select all

xbps-query -l | grep linux
EDIT3: Better is:

Code: Select all

xbps-query --regex -s '^linux[0-9.]+-[0-9._]+'
You need to examine the resulting list for the kernel name (you can try the meta-package/template name but maybe need main kernel version name; I tested but forget).

and then (untested but I think it is correct - thanks rockedge):

Code: Select all

vkpurge rm all
(not sure if you also need to remove previous firmware at this stage - I expect you don't need to).

4. followed by installing, for example, the current latest kernel with:

Code: Select all

xbps-install -Suy linux
which should install the new kernel and its related modules
Follow by:
5.

Code: Select all

exit
, and then

Code: Select all

./umount_chrootver007.sh
to clean up the chroot mounts, and lastly:
6.

Code: Select all

./build_weedog_initramfs05_XXX.sh void [options]
to rebuild the initramfs05.gz with the new modules inside it, which also outputs the new kernel for booting via grub menu.lst frugal arrangement, and a new 01firstrib_rootfs.sfs as usual

I guess you could write a simple script to automate the above (using for example a cat heredocument, similar to that used in build firstrib rootfs, to automate the xbps-install -Suy linux part done inside the chroot). Easy enough to do above manually though, and sometimes its best not to over-automate in case something odd happens...

I'm currently trying the following script - I'll post later if successful

EDIT: I think the following needs more work... There are issues with vkpurge I briefly mention in following posts. vkpurge is a shell script in /usr/bin that was designed to be used on running Void system (so not from say a chroot running on other Linux host).

Code: Select all

#!/bin/sh
# rebuild_weedog_void_frugal_001 with current kernel/modules
# Revision 0.0.1 Date: 10 Oct 2019
# Copyright wiak 10 Oct 2019+; Licence MIT (aka X11 license)

#### variables-used-in-script:
# script commandline arguments $1 $2 $3 $4
# "$1" is distro (e.g. void); "$2" is optional mksquashfs compression (or "default");
# "$3" is optional "huge" initramfs (or "default); $4 is optional busybox url to use

kernel="$1"
case "$1" in
   '-v'|'--version') echo "Rebuild WeeDog Void Frugal Revision 0.0.1";exit;;
   '-h'|'--help'|'-?') printf '
Usage:
./rebuild_weedog_void_frugal_001.sh void [OPTIONS]

-v --version       display version information and exit
-h --help -?       display this help and exit

First argument void is required to undertake rebuild
Optional second argument is mksquashfs compression (or "default")
Optional third argument is "huge" (or "default") initramfs
"huge" includes 01firstrib_rootfs.sfs inside initramfs/boot/initramfsNN
Optional fourth argument is busybox_url (in case, e.g. arm64 required)
fourth argument can optionally also be "default"

EXAMPLES:
./rebuild_weedog_void_frugal_NNN.sh void
./rebuild_weedog_void_frugal_NNN.sh void "-comp lz4 -Xhc"
./rebuild_weedog_void_frugal_NNN.sh void default huge

For more details read the attached README and/or
visit: https://github.com/firstrib/firstrib
';exit;;
   '') echo 'First argument must be distro name: void';exit;;
esac

[ -d squashfs-root ] && rm -rf squashfs-root
[ -d firstrib_rootfs_for_initramfs_sNNN ] && mv firstrib_rootfs_for_initramfs_sNNN BKUPfirstrib_rootfs_for_initramfs_sNNN
[ -d firstrib_rootfs ] && mv firstrib_rootfs BKUPfirstrib_rootfs
[ -f 01firstrib_rootfs.sfs ] && mv 01firstrib_rootfs.sfs BKUP01firstrib_rootfs.sfs
[ -f initramfs05.gz ] && mv initramfs05.gz BKUPinitramfs05.gz
echo 'Unsquashing BKUP01firstrib_rootfs.sfs ... Please wait ...'
unsquashfs BKUP01firstrib_rootfs.sfs
mv squashfs-root firstrib_rootfs

# bind mount host virtual filesystes required for chroot into firstrib_rootfs to work on host system
# as in mount_chrootXXX.sh
mkdir -p firstrib_rootfs/run/udev
mount --bind /proc firstrib_rootfs/proc && mount --bind /sys firstrib_rootfs/sys && mount --bind /dev firstrib_rootfs/dev && mount -t devpts devpts firstrib_rootfs/dev/pts && mount --bind /tmp firstrib_rootfs/tmp && mount --bind /run/udev firstrib_rootfs/run/udev && cp /etc/resolv.conf firstrib_rootfs/etc/resolv.conf

# The following commands gets done inside firstrib_rootfs (via here_document pipe to chroot):
cat << INSIDE_CHROOT | chroot firstrib_rootfs sh
# Remove all old kernel/modules
vkpurge rm all
echo 'Installing current kernel and modules ... Please wait ...'
xbps-install -Suy linux
exit
INSIDE_CHROOT

# Clean up the bind mounts that were used for the chroot
# as in umount_chrootXXX.sh:
umount -l firstrib_rootfs/proc && umount -l firstrib_rootfs/sys && umount -l firstrib_rootfs/dev/pts && umount -l firstrib_rootfs/dev && umount -l firstrib_rootfs/tmp && umount -l firstrib_rootfs/run/udev

printf "\nAt this stage you can manually modify firstrib_rootfs if you wish
Once you are ready to complete the rebuild press Enter to continue
"
read go

# Rebuild initramfs05.gz, 01firstrib_rootfs.sfs and output new vmlinuzXXX
echo "Executing $0 $@ ... Please wait ..."
exec ./build_weedog_initramfs05_s204.sh "$@"
exit
-----------------------------------------------

As far as my current efforts are concerned, Arch pacman reminds me of xbps a bit but pacman static isn't independent (it requires other code bits and pieces to be made available for it) like xbps static so I'm probably going to firstrib include it using archbootstrap or whatever it is called.

wiak
Last edited by wiak on Sat 12 Oct 2019, 12:30, edited 9 times in total.

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#689 Post by wiak »

@rockedge: You are correct, I should be using vkpurge. When I wrote my immediately above post I hadn't read:

https://docs.voidlinux.org/config/kernel.html

EDIT: I'm now editing my immediately above post with script I'm currently testing. I'll update later (and post script as file) once that script code is ready for use.
EDIT2: However, in my current tests, 'vkpurge list' isn't finding the old kernels I've installed (using xbps-install) for the test, so 'vkpurge rm all' isn't removing them. I'm looking into solution.

Also can check what already installed locally with:

Code: Select all

xbps-query --regex -s '^linux[0-9.]+-[0-9._]+'
I'll see if I can modify my above script accordingly to update kernel automatically.

Also, according to Void Linux wiki, to get a list of all packages installed without version numbers, you can use:

An example command line to get a list of all installed package names without their version is:

https://wiki.voidlinux.org/Frequently_A ... ackages.3F

Code: Select all

xbps-query -l | awk '{ print $2 }' | xargs -n1 xbps-uhelper getpkgname | fmt
wiak

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#690 Post by wiak »

Apparently, 'vkpurge rm all' safely remove all old kernel/modules that were installed via updates (but leaves the most recent kernel installed). However, it does not remove explicity installed kernels; I guess you need to use xbps-remove linux-NNNetc for that.

I can't say that the Void documentation on vkpurge is particularly clear on all of this.

wiak

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#691 Post by wiak »

Came across an interesting vkpurge-related link:

vkpurge is a shell script. Below is a modified version.

https://sysdfree.wordpress.com/2019/05/11/263/

NOTE that I haven't tested any of this (hardly looked at it yet); just saving it for later perusal. Also, I'm not suggesting this modified version is useful - just keeping it for interest at this stage.

EDIT: remove copy of modified script since not sure it is useful. Better to simply check script /usr/bin/vkpurge which contains the line:

Code: Select all

installed=$(xbps-query -o "/boot/vmlinu[xz]-*
" 2>/dev/null | awk '{print $2}')
That line is used to determine kernels that were explicity installed (via xbps-install) that vkpurge script then ignores during list or rm command.

I'll use this info to modify my kernel update script later.
EDIT2: Actually vkpurge probably best as it is. We don't want to remove explicity installed kernel (since that could include the current one) - I'm still thinking/experimenting with this.

wiak

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#692 Post by rockedge »

Success!

I was able to upgrade the WeeDog kernel via xbps-install -Suy and rebuild the vmlinuz-XX.XX, 01firstrib_rootfs.sfs and initramfs05.gz that booted cleanly and runs with a /70upper_changes, / 60upper_changes and /upper_changes. Unknown is the effects of updating with the changes occurring in the /upper_changes that occurred before mounting the uncompressed firstrib_rootfs
using the mount_chrootver007.sh script and running the command xbps-install -Suy. So far no conflicts have jumped out that I notice.

the firstrib_rootfs directory / filesystem must be present.
I used another WeeDog-s205 as the Host system for the upgrade process. Also tested using Puppy Linux Bionic 32 & 64 as the host.


The process was surprisingly easy!
(the paths may differ in your case)

Code: Select all

cd /mnt/sdb1/firstrib05-s103u-E
mv 01firstrib_rootfs.sfs 01firstrib_rootfs.sfs.bak
mv initramfs05.gz  initramfs05.gz.bak
./mount_chrootver007.sh
in the terminal inside the chroot'ed firstrib._rootfs :

Code: Select all

xbps-install -Suy
vkpurge list
vkpurge rm all
exit
Exit from firstrib_rootfs
and run:

Code: Select all

./umount_chrootver007.sh
now in the terminal run

Code: Select all

./build_weedog_initramfs05_s205.sh void
Open the Grub4Dos menu.lst and I use the entire vmlinuz name so the change went from vmlinuz-5.2.14_1 to vmlinuz-5.2.21_1
This could be simplified bly changing the name to vmlinuz after each upgrade. Also the usbwait= can be left off which is auto-detected in the lastest version of build_weedog_initramfs05

Code: Select all

title firstrib05-s103u-E (Void Linux)
  uuid 1ef03b67-ebf1-447a-808c-eaa32169e89a
  kernel /firstrib05-s103u-E/vmlinuz-5.2.21_1  net.ifnames=0 usbwait=6 bootfrom=/mnt/sdb1/firstrib05-s103u-E
  initrd /firstrib05-s103u-E/initramfs05.gz
after a successful reboot remove the .bak files

Code: Select all

rm /mnt/sdb1/firstrib05-s103u-E/initramfs05.gz.bak
rm /mnt/sdb1/firstrib05-s103u-E/01firstrib_rootfs.sfs.bak
In my case I then remove :

Code: Select all

rm  /mnt/sdb1/firstrib05-s103u-E/vmlinuz-5.2.14_1
this process updated WeeDog's initramfs05.gz and vmlinuz


_

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#693 Post by wiak »

rockedge wrote:Also tested using Puppy Linux Bionic 32 & 64 as the host.
...

Code: Select all

cd /mnt/sdb1/firstrib05-s103u-E
mv 01firstrib_rootfs.sfs 01firstrib_rootfs.sfs.bak
mv initramfs05.gz  initramfs05.gz.bak
./mount_chrootver007.sh
in the terminal inside the chroot'ed firstrib._rootfs :

Code: Select all

xbps-install -Suy
vkpurge list
vkpurge rm all
exit
Thanks rockedge. I think using a second WeeDog (that has the same running kernel) is a great idea. However, I'm surprised vkpurge lines worked correctly when different running kernel (e.g. when you used Bionic). I'll have to give it a try.

I'm out today and tomorrow but will get back to taking a further look at this (and vkpurge script) again thereafter.

wiak

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#694 Post by rockedge »

In theory I think if one boots a WeeDog into RAM, the update via mounting the firstrib_rootfs then umount and running the build_initramfs script it might work to update the running system without crashing the system.

for best results so far I did the update process from a host system, after I booted into the WeeDog and ran xbps-install -Suy in there, I rebooted into a host system and ran the mount script and did the xbps-install -Su and ran the vkpurge commands and exited with umount script, after that ran build_initramfs script in the host OS

One case I needed to delete the /upper_changes to get firefox to work again but all the other tests did not break firefox.

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#695 Post by wiak »

rockedge wrote:In theory I think if one boots a WeeDog into RAM, the update via mounting the firstrib_rootfs
Yes, for many reasons doing the update of kernel/modules via a WeeDog booted into RAM would be good. Will need to reboot to get new kernel running, but WeeDog boots fast so not much downtime there. I'm also still looking into this update kernel via simple script possibilty - I'm particularly looking at the vkpurge code since a modified version of that could also be used to remove old kernel/modules that were explicity installed (rather than just being the result of upgrades, which standard vkpurge takes care of).

To illustrate the more general issue (that isn't an issue if generally only upgrading kernels. I have a firstrib_rootfs that has four kernel/modules installed. Two of these were installed manually and the other two are upgrades to these (as far as I remember...). When I run vkpurge list command only the original two get seen as ready for removal. However, both of the upgrades then remain... That, I believe (but still have to check), is because the following vkpurge code marks them as upgrades of what were explicity installed kernels and thus 'vkpurge rm all' is coded to not delete any explicitly installed (rather than upgraded) kernels:

From inside chroot of ./mount_chrootver007.sh:

Code: Select all

xbps-query -o "/boot/vmlinu[xz]-*" 2>/dev/null | awk '{print $2}'
reports:

Code: Select all

/boot/vmlinuz-4.19.79_1
/boot/vmlinuz-5.2.21_1
whereas:

Code: Select all

vkpurge list
reports:

Code: Select all

4.19.69_1
5.2.20_1
What I want to arrange to happen for this more general case is that all of the above except for vmlinuz-5.2.21_1 get removed (which is not the case with current vkpurge rm all; so I'm planning to make a 'modified/special' weedog_vkpurge script to only leave most recent kernel/modules.

EDIT: Changed my mind. Better, is to allow more than one kernel to be installed in firstrib_rootfs and get build_weedog_initramfs to either (most simply) just use the latest of these, or give build script user choice.

wiak

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

build_weedog_initramfs05_s207.sh uploaded

#696 Post by wiak »

build_weedog_initramfs05_s207.sh uploaded to usual downloads location:

http://www.murga-linux.com/puppy/viewto ... 24#1035524

Changes:

Per official Void, firstrib_rootfs is now allowed to contain multiple versions of kernels/modules. The build_weedog_initramfs script will now give you a pause (via shell 'read' command) to allow you to (optionally) manually remove some kernels if you wish (via, for example, mount_chrootXXX.sh and then vkpurge or xbps-remove, and umount_chrootXXX.sh), and once you continue the script presents you with a number list menu of kernels that remain in firstrib_rootfs/boot. You then enter the number of the kernel you want to use or if you just press Enter at that stage the most recent kernel (according to alphanumeric order) in the list will be used during the initramfs build.

Also fixed the usbwait loop so correctly now has timeout of 30 seconds.

Has only had a little testing, so I am not as yet removing version s205. Using odd numbers (s207 in this case) to simply indicate not tested much as yet...

wiak
Last edited by wiak on Mon 14 Oct 2019, 09:11, edited 3 times in total.

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#697 Post by rockedge »

or give build script user choice
maybe add another parameter to the command line that will specify the available kernel to use and if not used, defaults to the latest version.

I had a system built that was using kernel 5.2.14 that I upgraded to kernel 5.2.21 by doing a system update then mounting the firstrib_rootfs and running the system update in there to hopefully insure all the dependencies and lib's are the same versions so /upper_changes is not layering in older versions and causing system breakage when initramfs05.gz and vmlinuz the 01firstrib_rootfs.sfsare rebuilt after the update.

Does it make sense to be able to merge /xxupper_changes into the firstrib_rootfs file system before squashing it or before running the build script?

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#698 Post by wiak »

rockedge wrote:
or give build script user choice
maybe add another parameter to the command line that will specify the available kernel to use and if not used, defaults to the latest version
EDIT: See my post below.
rockedge wrote:Does it make sense to be able to merge /xxupper_changes into the firstrib_rootfs file system before squashing it or before running the build script?
Anything to do with merging is, I feel, best left as a separate utility (a user may anyway not always want to merge...). There are different ways of merging and different likely user needs and overall flexibility therefore, such merging should IMO be a separate extra-script/process. The core build scripts cover core needs, but no more than that. If however any separate utility needed something already provided for in the core scripts of course should look into adding that. I doubt that's the case here though - I think merging can be arranged by a separate utility (perhaps sometimes one that can be executed from inside a running WeeDog). I'm still looking into a more definite separate weedog_vkpurge utility for cleaning out kernels, but again that will be an optional utility. But by all means if you have some specific code lines that add to overall WeeDog core flexibility/functionality which couldn't be conveniently simply used in separate utility we should consider that.

One definite good thing is that WeeDog, one way or another, is pretty much future-proof so could well be still around 10 or 15 or more years from now (Puppy itself been around for 15 years now, though been alot of dev work needed over time to keep that presence). It remains relatively easy to modify WeeDog (and I hope to keep it that way) and as long as upstream Void, Debian, Ubuntu, or Devuan (and later Arch) are still alive, so is WeeDog... and since build initramfs doesn't really care what the upstream distro is, could also be used with others.

wiak
Last edited by wiak on Mon 14 Oct 2019, 09:10, edited 5 times in total.

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#699 Post by wiak »

So, though I didn't add a new commandline parameter, if your firstrib_rootfs does contain more than one kernel (as in firstrib_rootfs/boot/vmlinuz*) then build_weedog_initramfs05_s207.sh will give you a pause (via read command) to allow you to (optionally) remove some if you wish (via, for example, mount_chrootXXX.sh and then vkpurge or xbps-remove, and umount_chrootXXX.sh), and then present you with a number list menu of kernels that remain in firstrib_rootfs/boot. You then enter the number of the kernel you want to use or if you just press Enter at that stage the most recent kernel (according to alphanumeric order) in the list will be used during the initramfs build.

So what I wrote earlier about possibility of adding N in front of vmlinuz name is no longer required. I'll amend my posts above accordingly.

Note that, the script is designed to work with busybox sh (not just bash) so bash select command couldn't be used for the simple menu - instead I'm using cat -n to number the list and sed to determine the selection. The menu is thus basic and simple but seems to work, but all needs better testing than I've had time to give it...

wiak

User avatar
rockedge
Posts: 1864
Joined: Wed 11 Apr 2012, 13:32
Location: Connecticut, United States
Contact:

#700 Post by rockedge »

the tests worked well. Built both a 32 and 64 bit versions using my firstrib-00-XX.plug file

I booted into the new WeeDog and thought why not cd into the firstrib_rootfs directory, open a terminal and run the mount_chrootver007.sh script. So using a chroot'ed file system of itself
in which I ran xbps-install -Su in both host and the jailed firstrib_rootfs simultaneously

this is working really well and possibly has some uses in this configuration. Since it is a host system running a container of itself

I am now testing building zoneminder with the script in the chroot mounted firstrib_rootfs and then rebuild initramfs05.gz The end result will be a ready to run simple desktop with a LHMP and zoneminder installed....I hope. Then maybe create a bootable ISO of it.

Although just WeeDog running a jailed version of itself but different is pretty fun to play with


__

Post Reply