Page 11 of 20

Posted: Sat 19 Jan 2013, 14:36
by goingnuts
No doubt busybox is needed. We could get through without it using ams-utils or the "multical call binary-technique" (mcb) of original apps...but BB delivers most of what is needed out of the box. Would be nice if they delivered some overview of which version is the most stable and content of apps between versions.
Also a "long term maintenance" version of BB could be nice. Upgrading BB (like upgrading other software) has the potential of breaking your existing scripts - so every script needs to be verified that it does what it is meant to do. On the other hand the new versions might offer apps that eliminate use of org apps and by then potentially reduce complexity and size.

Posted: Sat 19 Jan 2013, 19:06
by technosaurus
Ibidem wrote:
greengeek wrote:Does puppy HAVE to have busybox? What would happen to Puppy if it used a full command set?
Can you spell B-L-O-A-T?

To elaborate, I can build a ~700k static busybox that provides everything needed to boot (I have run this as an OS). This equates to:
coreutils:12 MB
bash: 800k
dhcpcd (127 k) or dhclient 643 k
net-tools: 927k
module-init-tools: 340k
netcat: 192 k
sysklogd: 200 k
iputils-ping: 131 k
netbase: 98 k
sed: 52 k
awk: 322 k
util-linux: 2 MB
e2fs-progs: 2 MB
wget: 2 MB
bzip2: 156 k
...
ie, at least 20 MB.
Also you get slower startup, more RAM used, etc.
I wrote a small c init that will boot into X with jwm that compiles to a few kb. It wouldn't be that much more to add more of the boot process in parallel if anyone is interested. Tinycore has a patch to allow the initrd to be swappable (the initramfs I was using isn't) as well as a couple of other kernel patches that will speed up booting. Basically all you need for the desktop is xvesa/xfbdev, jwm, rxvt and a shell, but I never made them into a threadsafe mcb, so they can just as well be separate static builds.

over 90% of the boot process that happens before x can be in parallel.

Posted: Sat 19 Jan 2013, 20:25
by amigo
"Does puppy HAVE to have busybox?" Yes, Puppy probably does. No one else should have to suffer a complete system loaded with cut-down versions of everything. Yeah, there's a big size difference, but I like using the same toys that the BIG boys use... LOL

Posted: Sat 19 Jan 2013, 20:33
by greengeek
technosaurus wrote:I wrote a small c init that will boot into X with jwm that compiles to a few kb. It wouldn't be that much more to add more of the boot process in parallel if anyone is interested. Tinycore has a patch to allow the initrd to be swappable (the initramfs I was using isn't) as well as a couple of other kernel patches that will speed up booting. Basically all you need for the desktop is xvesa/xfbdev, jwm, rxvt and a shell, but I never made them into a threadsafe mcb, so they can just as well be separate static builds.
This sounds rather interesting. I fear it will take me a long time to get to grips with all the things I don't understand about the boot process and componentry, but the more I look at the simpler stuff, the more I am learning.

Posted: Sat 19 Jan 2013, 21:26
by technosaurus
to start X all that needs to happen is to mount /dev
to get a terminal emulator working, you also need to mount /dev/pts
and some (non-crucial) apps will need /sys and /proc mounted
this is one line of C each

While X is starting up you can load additional kernel modules, mount file systems, do checks, set settings etc... from kernel boot parameters (boot parameters should be the way most settings are stored so that you don't have to mount a file system to get them, but no-one else does it that way)

Once X and the wm are up, a terminal or gui setup wizard can be used for the rest (and jwm has a convenient startup tag... also restart and shutdown, but that is another topic)

Posted: Sun 20 Jan 2013, 05:01
by greengeek
technosaurus wrote:to start X all that needs to happen is to mount /dev
to get a terminal emulator working, you also need to mount /dev/pts
and some (non-crucial) apps will need /sys and /proc mounted
... you can load additional kernel modules, mount file systems, do checks, set settings etc... from kernel boot parameters
Once X and the wm are up, a terminal or gui setup wizard can be used for the rest (and jwm has a convenient startup tag...
I used the phrase "simpler stuff" but what I really meant was "more minimalistic stuff". Each time I read your posts I realise that the minimalistic stuff is way harder than the bigger stuff (like booting a fullblown puppy from CD...that I can do!) :-)

Understanding all this grassroots stuff about booting with the minimum required code is like getting to the moon on a bicycle. Enticing yet very difficult for the untrained :-)

Thank God for google!

Posted: Sun 20 Jan 2013, 06:57
by technosaurus
The only reason it is complex is because we made it that way over time. For example puppy's init has tons of bloat dealing with finding the correct partitions, sfs and save file when it could just be added as a parameter to the kernel command line (same goes for kb, resolution, drivers to load, language and many others). Perhaps early bootloaders lacked this functionality and this is a holdover that exists because things still work? I don't know, but I literally spent days turning on and off kernel options to boil it down to the salts and then did the same with init. Once I realized how little was actually needed, it just made sense to write it in c since it was basically:

Mount /dev and /dev/pts
Fork and exec X
Set some basic env variables normally found in /etc/profile and DISPLAY
Fork and exec the wm
(Logic to reboot/shutdown/restartx)

Posted: Sun 20 Jan 2013, 14:42
by Ibidem
If you use Xorg or xfbdev, you'll need to load modules first.
And an autoconfigured X requires udev or hal. But then, most people use udev for /dev and loading modules.

Posted: Sun 20 Jan 2013, 18:07
by technosaurus
Ibidem wrote:If you use Xorg or xfbdev, you'll need to load modules first.
And an autoconfigured X requires udev or hal. But then, most people use udev for /dev and loading modules.
there _is_ a builltin option, IMHO everything needed by init _should_ be built-in unless there is some conflict (but in that case I would just have a separate version for the broken device)

Speaking of loading modules, there is a recent kernel patch to load modules by filename (previously they had to be memmapped) that would make my flat module method much easier in c (by flat I mean all modules in 1 dir). Even with standard tools it was faster this way because it didn't need to recurse into subdirs. I know this doesn't sound like much, but loading a single module previously required several lines of code, much of which was unnecessarily complex.

Speaking of udev, it has grown into a behemoth, and the things it does _can_ be simple. I'll probably try to patch toybox's mdev instead.

Posted: Wed 23 Jan 2013, 17:41
by PANZERKOPF
technosaurus wrote: Speaking of udev, it has grown into a behemoth, and the things it does _can_ be simple.
Agree.
technosaurus wrote: I'll probably try to patch toybox's mdev instead.
Busybox already has mdev, it can be used as simple udev replacement.
I made some scripts and mdev.conf file (Idea was borrowed from Alpine linux).
One problem was discovered there: It does not create device node for parallel port (LPT), althought all needed drivers was successfully loaded (parport, parport_pc).

Posted: Wed 23 Jan 2013, 19:20
by goingnuts
PANZERKOPF: Never got mdev working...tried you scripts (deactivated udevd in init), run rc.mdev afterlogin. Had to add groups disk, uucp, cdrom and video. Then only minor warnings running mdev -s or rc.mdev.
But no drivers loaded afterwards...
When to launch rc.mdev? Do you need additional scripts/setup?

Btw. diethotplug is an other close to udevd replacement.

Posted: Wed 23 Jan 2013, 22:22
by PANZERKOPF
goingnuts wrote:PANZERKOPF: Never got mdev working...tried you scripts (deactivated udevd in init), run rc.mdev afterlogin. Had to add groups disk, uucp, cdrom and video. Then only minor warnings running mdev -s or rc.mdev.
But no drivers loaded afterwards...
When to launch rc.mdev? Do you need additional scripts/setup?
Weird....
Usually mdev must be launched after mounting sysfs and proc.
Maybe Your busybox has different configuration?
Above code is a part of my system initscript. Attached tarball contains this script and
busybox configuration file.

Posted: Thu 24 Jan 2013, 00:54
by Ibidem
Looking at the mdev scripts, I see a couple things...
1. mdev.conf is missing this section (from examples/mdev_fat.conf in the busybox source):

Code: Select all

# Syntax:
# [-]devicename_regex user:group mode [>|=path] [@|$|*cmd args...]
#
# =: move, >: move and create a symlink
# @|$|*: run $cmd on delete, @cmd on create, *cmd on both

# support module loading on hotplug
$MODALIAS=.*	root:root 660 @modprobe "$MODALIAS"
This is what handles hotplug module loading, which is important.

2. Most of the mknod stuff is unneeded with a proper mdev.conf.
I know /dev/{console,full,null,zero,urandom,random} are automatically created, even with mdev.conf absent.

3. sysctl -w kernel.hotplug=/sbin/mdev allows use wthout mounting /proc

4. Alpine uses/used something like:

Code: Select all

find /sys -name modalias |xargs sort -u |xargs modprobe -a -b
to load modules for devices that aren't hotplugged. This should really speed things up.
And to coldplug USB devices, you may need this:

Code: Select all

	# mdev -s will not create /dev/usb[1-9] devices with recent kernels
	# so we trigger hotplug events for usb for now
	for i in $(find /sys/devices -name 'usb[0-9]*'); do
		[ -e $i/uevent ] && echo add > $i/uevent
	done

Alpine's stuff is at:
http://git.alpinelinux.org/cgit/aports/ ... nitscripts

Posted: Thu 24 Jan 2013, 06:31
by greengeek
PANZERKOPF wrote: Attached tarball contains this script and
busybox configuration file.
The rc.shutdown is very much simpler than some of the puppy shutdown scripts I have been looking at recently. Where would you use this? Is it taken from another distro or would it work in a cutdown puppy?

Posted: Thu 24 Jan 2013, 13:57
by PANZERKOPF
Ibidem wrote:Looking at the mdev scripts, I see a couple things...
1. mdev.conf is missing this section (from examples/mdev_fat.conf in the busybox source):

Code: Select all

# Syntax:
# [-]devicename_regex user:group mode [>|=path] [@|$|*cmd args...]
#
# =: move, >: move and create a symlink
# @|$|*: run $cmd on delete, @cmd on create, *cmd on both

# support module loading on hotplug
$MODALIAS=.*	root:root 660 @modprobe "$MODALIAS"
This is what handles hotplug module loading, which is important.
We can use above trick as well as making script for particular event (see /lib/mdev/usbdev for example).
Ibidem wrote: 4. Alpine uses/used something like:

Code: Select all

find /sys -name modalias |xargs sort -u |xargs modprobe -a -b
to load modules for devices that aren't hotplugged. This should really speed things up.
And to coldplug USB devices, you may need this:

Code: Select all

	# mdev -s will not create /dev/usb[1-9] devices with recent kernels
	# so we trigger hotplug events for usb for now
	for i in $(find /sys/devices -name 'usb[0-9]*'); do
		[ -e $i/uevent ] && echo add > $i/uevent
	done

I learned Alpine scripts before making my one. That script unfortunately cannot detect
all devices plugged in my test boxes (like Broadcom wireless card).
Some devices hasnt modalias file in /sys directory but has uevent with modalias string inside.
Also, my version based on "pure shell", no grep, find, xargs etc. That is not important but
should be faster (IMHO) :)

Posted: Thu 24 Jan 2013, 14:27
by PANZERKOPF
greengeek wrote:The rc.shutdown is very much simpler than some of the puppy shutdown scripts I have been looking at recently. Where would you use this? Is it taken from another distro or would it work in a cutdown puppy?
I use this in my own "pocket Linux", most things are written from scratch but some ideas
and functions were borrowed from other projects.

Posted: Fri 25 Jan 2013, 06:39
by Ibidem
PANZERKOPF wrote:
Ibidem wrote: 4. Alpine uses/used something like:

Code: Select all

find /sys -name modalias |xargs sort -u |xargs modprobe -a -b
to load modules for devices that aren't hotplugged. This should really speed things up.
And to coldplug USB devices, you may need this:

Code: Select all

	# mdev -s will not create /dev/usb[1-9] devices with recent kernels
	# so we trigger hotplug events for usb for now
	for i in $(find /sys/devices -name 'usb[0-9]*'); do
		[ -e $i/uevent ] && echo add > $i/uevent
	done

I learned Alpine scripts before making my one. That script unfortunately cannot detect
all devices plugged in my test boxes (like Broadcom wireless card).
Some devices hasnt modalias file in /sys directory but has uevent with modalias string inside.
Also, my version based on "pure shell", no grep, find, xargs etc. That is not important but
should be faster (IMHO) :)
A loop to load modules will probably be slower than loading all at once: you may save the time of invoking find/grep/xargs once, but you pay by reading each file serially and invoking modprobe once per module.
Plus, a for loop has some inherent overhead.

I timed the different methods with modprobe converted to an echo:
Anyway, here's an attempt to handle that case; there is some overhead to parsing the uevent files.
Brute force with find @ 0.39 seconds:

Code: Select all

find /sys/devices -name uevent | xargs grep -h MODALIAS 2>/dev/null |cut -d = -f 2 
Your way @0.14 seconds (goes down by half without the second /sys/bus/*):

Code: Select all

#!/bin/sh

#is there really a reason to try twice?
for ONEBUS in /sys/bus/* /sys/bus/*; do
    for ONEDEVICE in ${ONEBUS}/devices/*; do
	[ -r ${ONEDEVICE}/uevent ] || continue
	while read ONEUEVENTSTR; do
	    case $ONEUEVENTSTR in
		MODALIAS=*)
		    echo ${ONEUEVENTSTR#MODALIAS=} 2>/dev/null
		;;
		*)
		    continue
		;;
	    esac
	done < ${ONEDEVICE}/uevent
    done
done
And a mix @0.035 seconds:

Code: Select all

grep -h MODALIAS /sys/bus/*/devices/*/uevent 2>/dev/null |cut -d = -f 2
The "official" method is more like

Code: Select all

find /sys -name uevent -exec echo add > {} \;
but, that would likely have even worse performance (for every entry, a context switch to kernel mode and exec mdev in user mode, then exec modprobe if needed)

Each testcase outputs a list of module aliases.

Posted: Sun 27 Jan 2013, 07:16
by Ibidem
goingnuts wrote:PANZERKOPF: Never got mdev working...tried you scripts (deactivated udevd in init), run rc.mdev afterlogin. Had to add groups disk, uucp, cdrom and video. Then only minor warnings running mdev -s or rc.mdev.
But no drivers loaded afterwards...
When to launch rc.mdev? Do you need additional scripts/setup?

Btw. diethotplug is an other close to udevd replacement.
FYI: diethotplug is from 2002; you might want hotplug2.
Then there's mdev and ndev, and Busybox hotplug, and eudev, and probably a few other systems...
But udev/eudev provide one thing the other solutions don't: they can enumerate devices, which is what Xorg wants them for.

Posted: Mon 28 Jan 2013, 16:38
by PANZERKOPF
Ibidem wrote: But udev/eudev provide one thing the other solutions don't: they can enumerate devices, which is what Xorg wants them for.
Yes, latest Xorg servers need udev but as I know, It can be configured/compiled without udev.

Posted: Mon 28 Jan 2013, 19:47
by goingnuts
PANZERKOPF & Ibidem: Thanks for all your input! I guess I stick with udev-124 as its quite small and it works for me. Cant get BB mdev working, diethotplug is fast but lacks some features, hotplug2 seems to favor glibc-dynamic linking and eudev needs Autoconf version 2.68 or higher and I do not want to upgrade now...

And the udevd seems to work on kernel from P216, P412, P431 & P525 without recompile.

Revisited the static build of tcl/tk 8.5 to get attached 3 games running (bubbles.tcl CrystalsBattle.tcl gemgame.tcl) where the first & last will be known from Puppy3...quite addictive...