Updating the kernel - typical problems

[tempestuous]

I thought I would make a separate post about this, so all developers here might see it.
Each time a new Puppy version is released with an updated kernel, we typically gain some brand new drivers (kernel modules). This provides a few problems:

1. New wifi drivers won't be recognised in the Network Wizard as being WPA encryption-capable, until the developers spot the new or updated driver, and manually update the Network Wizard configuration scripts.

2. New drivers, especially wifi drivers, often require separate firmware, and this firmware will be absent until the developers identify the missing firmware and add it.

3. The addition of this firmware also requires an update to Puppy's configuration scripts.

We often don't learn about these problems until someone on the forum reports some item of non-functioning hardware, and we go through the often awkward process of trying to coach new users into providing the necessary diagnostic information. The resistance we encounter to new users running a manual command in the terminal is one of the great frustrations of being involved on this forum, and IMO the greatest roadblock to being able to provide accurate help.

I have some suggestions about fixing these 3 problems:

[tempestuous]

1. The Network Wizard
In /usr/sbin/wag-profiles.sh
lines 311 to 343 need a major overhaul. This section does three things:
i) it identifies whether the the wifi device is the "conventional" type which is configured with "iwconfig" and "wpa_supplicant", or is the unconventional prism2_usb driver which requies special handling via the "wlanctl-ng" utility.
ii) it identifies the wifi driver's "interface type" for how it should be handled by wpa_supplicant ...
but this is completely outdated now, and I think that wpa_supplicant now interacts with every wifi driver (except the prism2_usb driver previously mentioned) via the same "wext" interface-type.

iii) if a wifi driver is not listed in this section (lines 320-329) it means that the driver is not WPA-compatible.

So this part of the Network Wizard now just needs to identify whether the wifi driver in use is "prism2_usb" - and handle it accordingly, otherwise in all other situations the setup will be conventional, and the wpa_supplicant interface-type will always be "wext" ...
... except where the wifi driver is non-WPA compatible - and I think the logic of the Wizard needs to be reversed in this respect - it's far easier to list all the non-WPA compatible drivers instead of listing all the compatible ones, as is the case presently.

[tempestuous]

2. Firmware collection
I think that each time a new kernel is added, the latest firmware collection needs to be added at the same time.
The latest firmware collection is available as a git download from -
git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

There's just one little problem - not all of the firmware required by Puppy's collection of drivers is available there!
For example, the Intel ipw2100 and ipw2200 firmware was obtained from Sourceforge. We don't need to re-acquire this firmware each time, but we probably need a list of firmware currently in Puppy's filesystem which is not available from kernel.org.

[tempestuous]

3. Firmware handling
Historically Puppy has had some complicated methods of locating firmware into the correct directories, in the interests of keeping the filesystem as trim as possible.
Back in the Puppy 2.x days firmware and drivers were located totally externally to the main compressed filesystem, in a separate file called the "zdrv". At first boot Puppy would extract whatever it needed from the zdrv, leaving all unneeded drivers and firmware outside the filesystem. This process was shown to be unreliable ...
so later it was decided to include all drivers and firmware within the filesystem, but keep the firmware compressed in a separate directory, and only uncompress/relocate the firmware that was needed.

Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>

[Karl Godt]

First, tempestuous, myself I have not a real idea what this firmware is all about. I simply seemed not to have any need for additional hardware firmware on my relative few machines .

What I am using daily from all-firmware is pgprs-connect, which is not firmware hardware specific .

I really have the same idea to deprecate /lib/modules/all-firmware for at least the "packages" that contain only scripts like for /etc/init.d/ .


I am running since a longer time a rc.services like that :

Code: Select all

LSMODS=`lsmod | cut -f 1 -d ' ' | grep -v -w -E 'snd|processor|ac'`
for i in $LSMODS ; do
echo "$i"
find /etc/init.d/ -maxdepth 1 \( -type f -o -type l \) -iname "*$i*" -exec echo chmoding +x {} \; -exec chmod +x {} \;
done
LSMODS2=`echo "$LSMODS" | tr '\n' '|' | sed 's#|$##'`
echo LSMODS2="$LSMODS2"
for i in /etc/init.d/FIRMWARE/* ; do
#if [ -z "`echo "$LSMODS" | grep -i "$(basename "$i")"`" ] ; then

if [ -z "`echo "$i" | grep -i -E "$LSMODS2"`" ] ; then
echo "chmod -x $i"
chmod -x "$i"
else
echo "chmod +x $i"
chmod +x "$i"
fi
done
for_func(){
for service_script in /etc/init.d/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script

for service_script in /etc/init.d/DEFAULT/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script

for service_script in /etc/init.d/FIRMWARE/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script
}

find -L /etc/init.d/ \
\( -type f -o -type l \) \
\( -perm /777 -o -perm /775 -o -perm /771 -o -perm /755 -o -perm /751 -o -perm /711 \) \
-exec echo {} \; -exec {} start \;

###END###

This code tries to determine, which drivers are loaded and chmod 'es the firmware scripts according if the driver found loaded .
The all-firmware scripts are in /etc/init.d/FIRMWARE/ directory and the common scripts in /etc/init.d/ directory like *alsa* or cups . alsa had to be renamed alsa_snd , because there is no alsa.ko, but snd.ko .


*


Puppy includes few special outside drivers like sysproof.ko, ndiswrapper and several unknown to me extra/ modem drivers . I never had the inner "drive" to do all that compiling of these outside drivers, if I compile a kernel . I personally don't need them, and could never find out what the purpose of the sysproof driver would be .. It's GUI was always just not comprehensive to me .


*


I see, that you do a lot for others in regards to drivers .
Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic. ... 939#223939 915resolution-0.5.3-patched20110617.tar.gz ,
because it doesn't download for me any more. Some of my attachment have same "An Error Occurred" since they possibly are larger than 256.00 KB , but yours is just 8.61 KB .


*

The resistance we encounter to new users running a manual command in the terminal is one of the great frustrations of being involved on this forum, and IMO the greatest roadblock to being able to provide accurate help.

http://en.wikipedia.org/wiki/Beerware :

Should the user of the product meet the author and consider the software useful, he is encouraged to buy the author a beer "in return" (or, in some variations, to drink a beer in the author's honor).

[anikin]

tempestuous,
I'm glad you started this thread, and would like to bring the following to your attention:
http://lists.debian.org/debian-user/201 ... 00126.html
and this
http://packages.debian.org/wheezy/intel-microcode
I've tried hard and failed to make it run on all the latest puppies (Raring, Slacko, Wheezy). Your thoughts will be greatly appreciated.

Thank you in advance.

[mavrothal]

1. The Network Wizard
In /usr/sbin/wag-profiles.sh
lines 311 to 343 need a major overhaul.

<snip>

So this part of the Network Wizard now just needs to identify whether the wifi driver in use is "prism2_usb" - and handle it accordingly, otherwise in all other situations the setup will be conventional, and the wpa_supplicant interface-type will always be "wext" ...
... except where the wifi driver is non-WPA compatible

You mean something like this?

Code: Select all

    INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
    INTMODULE=${INTMODULE##*/}
	case "$INTMODULE" in 
	prism2_*) USE_WLAN_NG="yes" ;;
	# orther_no_wext_card) add as needed ;;
	*) CARD_WPA_DRV="wext" ;; 
	esac

[mavrothal]

3.

<snip>
Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>

So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

[jamesbond]

3.

<snip>
Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>

So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

+1 Yes. Pros:
1. Faster
2. Easier to manage
If size is concern, then pack them into squashfs. This is what Fatdog does.

[tempestuous]

Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic. ... 939#223939
915resolution-0.5.3-patched20110617.tar.gz

OK, done. I found a more recent version of my patched 915resolution, dated May 21st 2012.
Bear in mind that from about Sandy Bridge onwards, 915resolution is no longer compatible.

http://packages.debian.org/wheezy/intel-microcode
I've tried hard and failed to make it run on all the latest puppies

The firmware that I have been discussing is the type that's required to work with a related kernel module. Without such firmware, the kernel module will fail to work altogether.
The Intel processor microcode, on the other hand, is quite different. It appears to be more of a "hotfix" than an essential driver component. And since your computer can obviously boot and run without it, this firmware certainly cannot be considered "essential", so I wouldn't lose sleep over it.
The reason you cannot get it working with Puppy is simple; here is the relevant section of Puppy's kernel configuration -

Code: Select all

# CONFIG_MICROCODE is not set

If you REALLY want to get it working, the first thing you must do is lobby the Puppy developers to change the kernel configuration. This will yield a new driver called "microcode", which is used in conjunction with the microcode_ctl utility to load the firmware at each boot up, as launched from a script in /etc/rc.d

You mean something like this?

Code: Select all

    INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
    INTMODULE=${INTMODULE##*/}
	case "$INTMODULE" in 
	prism2_*) USE_WLAN_NG="yes" ;;
	# orther_no_wext_card) add as needed ;;
	*) CARD_WPA_DRV="wext" ;; 
	esac

I like the simplicity of your code, and you're on the right track, but no, the whole logic needs to be adjusted -
the CARD_WPA_DRV variable is completely redundant, and we can dispense with this variable altogether.

There are now 3 possible wifi driver scenarios:
i) prism2_usb - the wlan-ctl setup regime is used.
ii) all other wifi drivers, which are WPA-compatible - conventional (iwconfig) setup regime, both WEP and WPA encryption offered.
iii) all other wifi drivers, which are NOT WPA-compatible - conventional (iwconfig) setup regime, only WEP encryption offered.

In scenario ii) the wpa_supplicant interface-type will ALWAYS be "wext", so the wpa_supplicant command can be hard-coded with this parameter - there's no need to vary it.

So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.

Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.

If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.

[01micko]

If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.

All the firmware is inside the main sfs anyway.

[Karl Godt]

Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic. ... 939#223939
915resolution-0.5.3-patched20110617.tar.gz

OK, done. I found a more recent version of my patched 915resolution, dated May 21st 2012.
Bear in mind that from about Sandy Bridge onwards, 915resolution is no longer compatible.

Code: Select all

# CONFIG_MICROCODE is not set

If you REALLY want to get it working, the first thing you must do is lobby the Puppy developers to change the kernel configuration. This will yield a new driver called "microcode", which is used in conjunction with the microcode_ctl utility to load the firmware at each boot up, as launched from a script in /etc/rc.d

So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.

Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.

If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.

Thank you very much !

*

DOTconfig_2.6.37.4-KRG-iatom-1:CONFIG_MICROCODE=m
Now I know what this might be used for . Thanks again for the very good explanation .


*

I don't think that the firmware_module_func
#101120 new firmware_tarball_func, as /sbin/pup_event_backend_modprobe rewritten. that is in functions4puppy4 and in *_backend_modprobe cannot be totally replaced with a 99% solution at once .
This will take several puppy releases I think . Step by Step .
Both functions are essentially the same with a USLEEP difference .
The whole serialize code to prevent multiple loadings of modules is as heavy as firing modprobe all the time IMHO .

pgprs* can be installed into /usr/bin/ by default and the pgprs-shell script dropped then .
[edit : wrote pgprs-setup, meant pgprs-shell ]
*

Was firmware ever a component of standard puppy_initrd.gz ?

[mavrothal]

the whole logic needs to be adjusted -
the CARD_WPA_DRV variable is completely redundant, and we can dispense with this variable altogether.

There are now 3 possible wifi driver scenarios:
i) prism2_usb - the wlan-ctl setup regime is used.
ii) all other wifi drivers, which are WPA-compatible - conventional (iwconfig) setup regime, both WEP and WPA encryption offered.
iii) all other wifi drivers, which are NOT WPA-compatible - conventional (iwconfig) setup regime, only WEP encryption offered.

In scenario ii) the wpa_supplicant interface-type will ALWAYS be "wext", so the wpa_supplicant command can be hard-coded with this parameter - there's no need to vary it.

So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.

Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.

If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.

Thank you for the clarifications. It makes a lot of sense to be adopted by "puppy-next".
Would you happen to have a draft of the the code for the proposed changes?ie are you (will be) working on it assuming puppy goes with all firmware in /lib/firmware (SFS'ed or not)?

[tempestuous]

I don't think that the firmware_module_func
#101120 new firmware_tarball_func, as /sbin/pup_event_backend_modprobe rewritten. that is in functions4puppy4 and in *_backend_modprobe cannot be totally replaced with a 99% solution at once .
This will take several puppy releases I think . Step by Step .

I disagree. The complexity of the current situation has confused us all, and we seem reluctant to change it!
But the current regime is IMO way more complicated than it needs to be, and it's really not difficult to remove.

In /sbin/pup_event_backend_firmware, and in /etc/rc.d/functions4puppy4
just remove the entire "firmware_tarball_func" section.

/etc/modules/firmware.dep.<kernelversion> can be completely removed.

/sbin/pup_event_backend_firmware must REMAIN. This is the core firmware loading mechanism - it's triggered by a udev rule:
/etc/udev/rules.d/50-udev-puppy-basic.rules

pgprs* can be installed into /usr/bin/ by default and the pgprs-shell script dropped then.

Well, partly. Sure, once /lib/modules/all-firmware/pgprs has been copied into the filesystem, you should then delete from /usr/bin/pgprs-shell the section relating to "firmware" ...
but you might want to keep the "fixmenus" section.

Was firmware ever a component of standard puppy_initrd.gz ?

Not as far as I know. The initrd consists mainly of the static kernel. There are only a few kernel modules included, and none of these require external firmware.

Would you happen to have a draft of the the code for the proposed changes?

The changes relating to firmware handling are very simple - I just explained them in this post.

Regarding the firmware collection, itself, I would be prepared to help build a list of all firmware that is NOT obtained from kernel.org.

Regarding the Network Wizard, I'm not a programmer, so I can only advise on the logic and correct processes - that's exactly how I helped with the current Network Wizard, in collaboration with forum member Dougal, who wrote the code.
Unfortunately Dougal is no longer active on the forum. Someone else needs to step in and offer to update the Network Wizard code.

[mavrothal]

Regarding the firmware collection, itself, I would be prepared to help build a list of all firmware that is NOT obtained from kernel.org.

That would be very helpful.
Please do so when you have the chance.
Thx

[mavrothal]

it's far easier to list all the non-WPA compatible drivers instead of listing all the compatible ones, as is the case presently.

Would you happen to have such a list or pointers?
Thx

[mavrothal]

You mean something like this?

Code: Select all

    INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
    INTMODULE=${INTMODULE##*/}
	case "$INTMODULE" in 
	prism2_*) USE_WLAN_NG="yes" ;;
	# orther_no_wext_card) add as needed ;;
	*) CARD_WPA_DRV="wext" ;; 
	esac

I like the simplicity of your code, and you're on the right track, but no, the whole logic needs to be adjusted -
the CARD_WPA_DRV variable is completely redundant, and we can dispense with this variable altogether.

We could leave in CARD_WPA_DRV to keep changes to a minimum.
What about this

Code: Select all

# In showProfilesWindow() function line 309-443

INTERFACE="$1"
INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
INTMODULE=${INTMODULE##*/}
case "$INTMODULE" in 
#Drivers with no WPA support
driver1|driver2|driver3|driver4|...)
		if [ -f "$Extra_WPA_Modules_File" ] &&\
	        CARD_WPA_DRV=$(grep -m1 "^$INTMODULE:" $Extra_WPA_Modules_File) ; then
			CARD_WPA_DRV=${CARD_WPA_DRV#*:}
		else
			CARD_WPA_DRV=""
         NO_CARD_WPA_DRV="Yes" 
			giveNoWPADialog
		fi
	;;
prism2_*) USE_WLAN_NG="yes" 
	;;
*) CARD_WPA_DRV="wext"
   ;;
esac

# In setupNewProfile () function lines 956-989

	if [ "$NO_CARD_WPA_DRV" ] ; then 
		ENABLE_WPA_BUTTON='false'
		ENABLE_WPA2_BUTTON='false'
	fi

[mikeb]

Thanks for the confirmation.

I changed puppy 4.12 to a slax kernel and was unsure but I added/moved firmware to the usual /lib/firmware unpacked as per convention and it seemed happy. I thought I was missing something but apparently not. It did seem a little odd to compress items in a squashfs archive.

So if there is a point its to confirm that doing this is no problem.

not 100% of why the firmware is loaded via the backend script since other distros do not seem to need this (udev doing it directly?) Kernel messages seem to suggest the kernel locates and loads.

I also prefer the standard kernel blacklist file only leaving overrides ( I actually tried hacking the kernel modules for the latter...ie removed unwanted device ids.. not exactly normal but actually works.)

mike

[Karl Godt]

Just looked into /lib/modules/all-firmware/pgprs :

All files can be moved into the / filesystem :
/etc/ppp/peers/gprs* ( 3 files ) ,
/usr/bin/pgprs* ( 2 files )
and
/usr/share/applications/pgprs*.desktop ( 2 files )
as if they were already installed .

I thought that pgprs* would be linked to some modem /etc/init.d/Driver_start_stop_script like /etc/init.d/Option , which mainly creates the /dev/modem link , but that is not the case.

This would be a woof level issue . I am not sure how woof handles .desktop files to create a jwmrc menu file with the entries found in /usr/share/applications/ automatically before squashing together the puppy_name_version.sfs .

[mikeslr]

Hi All,

We often don't learn about these problems until someone on the forum reports some item of non-functioning hardware, and we go through the often awkward process of trying to coach new users into providing the necessary diagnostic information. The resistance we encounter to new users running a manual command in the terminal is one of the great frustrations of being involved on this forum, and IMO the greatest roadblock to being able to provide accurate help.

:

Sorry I can't help with the technical problems, but the foregoing could perhaps be resolved by including a GUI app on the networking submenu. It might only need a yad or gtk-dialog script. Noobs only have to click it, respond as necessary to the dialog, and post the resulting log.

mikesLr