Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 27 Nov 2014, 13:38
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Updating the kernel - typical problems
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 3 [31 Posts]   Goto page: 1, 2, 3 Next
Author Message
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Mon 02 Dec 2013, 23:47    Post subject:  Updating the kernel - typical problems  

I thought I would make a separate post about this, so all developers here might see it.
Each time a new Puppy version is released with an updated kernel, we typically gain some brand new drivers (kernel modules). This provides a few problems:

1. New wifi drivers won't be recognised in the Network Wizard as being WPA encryption-capable, until the developers spot the new or updated driver, and manually update the Network Wizard configuration scripts.

2. New drivers, especially wifi drivers, often require separate firmware, and this firmware will be absent until the developers identify the missing firmware and add it.

3. The addition of this firmware also requires an update to Puppy's configuration scripts.

We often don't learn about these problems until someone on the forum reports some item of non-functioning hardware, and we go through the often awkward process of trying to coach new users into providing the necessary diagnostic information. The resistance we encounter to new users running a manual command in the terminal is one of the great frustrations of being involved on this forum, and IMO the greatest roadblock to being able to provide accurate help.

I have some suggestions about fixing these 3 problems:
Back to top
View user's profile Send private message 
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Mon 02 Dec 2013, 23:49    Post subject:  

1. The Network Wizard
In /usr/sbin/wag-profiles.sh
lines 311 to 343 need a major overhaul. This section does three things:
i) it identifies whether the the wifi device is the "conventional" type which is configured with "iwconfig" and "wpa_supplicant", or is the unconventional prism2_usb driver which requies special handling via the "wlanctl-ng" utility.
ii) it identifies the wifi driver's "interface type" for how it should be handled by wpa_supplicant ...
but this is completely outdated now, and I think that wpa_supplicant now interacts with every wifi driver (except the prism2_usb driver previously mentioned) via the same "wext" interface-type.

iii) if a wifi driver is not listed in this section (lines 320-329) it means that the driver is not WPA-compatible.

So this part of the Network Wizard now just needs to identify whether the wifi driver in use is "prism2_usb" - and handle it accordingly, otherwise in all other situations the setup will be conventional, and the wpa_supplicant interface-type will always be "wext" ...
... except where the wifi driver is non-WPA compatible - and I think the logic of the Wizard needs to be reversed in this respect - it's far easier to list all the non-WPA compatible drivers instead of listing all the compatible ones, as is the case presently.
Back to top
View user's profile Send private message 
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Mon 02 Dec 2013, 23:52    Post subject:  

2. Firmware collection
I think that each time a new kernel is added, the latest firmware collection needs to be added at the same time.
The latest firmware collection is available as a git download from -
git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

There's just one little problem - not all of the firmware required by Puppy's collection of drivers is available there!
For example, the Intel ipw2100 and ipw2200 firmware was obtained from Sourceforge. We don't need to re-acquire this firmware each time, but we probably need a list of firmware currently in Puppy's filesystem which is not available from kernel.org.
Back to top
View user's profile Send private message 
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Mon 02 Dec 2013, 23:53    Post subject:  

3. Firmware handling
Historically Puppy has had some complicated methods of locating firmware into the correct directories, in the interests of keeping the filesystem as trim as possible.
Back in the Puppy 2.x days firmware and drivers were located totally externally to the main compressed filesystem, in a separate file called the "zdrv". At first boot Puppy would extract whatever it needed from the zdrv, leaving all unneeded drivers and firmware outside the filesystem. This process was shown to be unreliable ...
so later it was decided to include all drivers and firmware within the filesystem, but keep the firmware compressed in a separate directory, and only uncompress/relocate the firmware that was needed.

Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 3972
Location: Kiel,Germany

PostPosted: Tue 03 Dec 2013, 05:48    Post subject:  

First, tempestuous, myself I have not a real idea what this firmware is all about. I simply seemed not to have any need for additional hardware firmware on my relative few machines .

What I am using daily from all-firmware is pgprs-connect, which is not firmware hardware specific .

I really have the same idea to deprecate /lib/modules/all-firmware for at least the "packages" that contain only scripts like for /etc/init.d/ .


I am running since a longer time a rc.services like that :
Code:
LSMODS=`lsmod | cut -f 1 -d ' ' | grep -v -w -E 'snd|processor|ac'`
for i in $LSMODS ; do
echo "$i"
find /etc/init.d/ -maxdepth 1 \( -type f -o -type l \) -iname "*$i*" -exec echo chmoding +x {} \; -exec chmod +x {} \;
done
LSMODS2=`echo "$LSMODS" | tr '\n' '|' | sed 's#|$##'`
echo LSMODS2="$LSMODS2"
for i in /etc/init.d/FIRMWARE/* ; do
#if [ -z "`echo "$LSMODS" | grep -i "$(basename "$i")"`" ] ; then

if [ -z "`echo "$i" | grep -i -E "$LSMODS2"`" ] ; then
echo "chmod -x $i"
chmod -x "$i"
else
echo "chmod +x $i"
chmod +x "$i"
fi
done
for_func(){
for service_script in /etc/init.d/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script

for service_script in /etc/init.d/DEFAULT/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script

for service_script in /etc/init.d/FIRMWARE/*
do
 [ -x $service_script ] && echo "$service_script" && $service_script start
done
unset service_script
}

find -L /etc/init.d/ \
\( -type f -o -type l \) \
\( -perm /777 -o -perm /775 -o -perm /771 -o -perm /755 -o -perm /751 -o -perm /711 \) \
-exec echo {} \; -exec {} start \;

###END###

This code tries to determine, which drivers are loaded and chmod 'es the firmware scripts according if the driver found loaded .
The all-firmware scripts are in /etc/init.d/FIRMWARE/ directory and the common scripts in /etc/init.d/ directory like *alsa* or cups . alsa had to be renamed alsa_snd , because there is no alsa.ko, but snd.ko .


*


Puppy includes few special outside drivers like sysproof.ko, ndiswrapper and several unknown to me extra/ modem drivers . I never had the inner "drive" to do all that compiling of these outside drivers, if I compile a kernel . I personally don't need them, and could never find out what the purpose of the sysproof driver would be .. It's GUI was always just not comprehensive to me .


*


I see, that you do a lot for others in regards to drivers .
Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic.php?p=223939#223939 915resolution-0.5.3-patched20110617.tar.gz ,
because it doesn't download for me any more. Some of my attachment have same "An Error Occurred" since they possibly are larger than 256.00 KB , but yours is just 8.61 KB .


*


Quote:
The resistance we encounter to new users running a manual command in the terminal is one of the great frustrations of being involved on this forum, and IMO the greatest roadblock to being able to provide accurate help.

http://en.wikipedia.org/wiki/Beerware :
Quote:
Should the user of the product meet the author and consider the software useful, he is encouraged to buy the author a beer "in return" (or, in some variations, to drink a beer in the author's honor).

Wink
Back to top
View user's profile Send private message Visit poster's website 
anikin

Joined: 10 May 2012
Posts: 529

PostPosted: Tue 03 Dec 2013, 06:34    Post subject:  

tempestuous,
I'm glad you started this thread, and would like to bring the following to your attention:
http://lists.debian.org/debian-user/2013/09/msg00126.html
and this
http://packages.debian.org/wheezy/intel-microcode
I've tried hard and failed to make it run on all the latest puppies (Raring, Slacko, Wheezy). Your thoughts will be greatly appreciated.

Thank you in advance.
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 1808

PostPosted: Tue 03 Dec 2013, 13:26    Post subject:  

tempestuous wrote:
1. The Network Wizard
In /usr/sbin/wag-profiles.sh
lines 311 to 343 need a major overhaul.

<snip>

So this part of the Network Wizard now just needs to identify whether the wifi driver in use is "prism2_usb" - and handle it accordingly, otherwise in all other situations the setup will be conventional, and the wpa_supplicant interface-type will always be "wext" ...
... except where the wifi driver is non-WPA compatible


You mean something like this?
Code:
    INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
    INTMODULE=${INTMODULE##*/}
   case "$INTMODULE" in
   prism2_*) USE_WLAN_NG="yes" ;;
   # orther_no_wext_card) add as needed ;;
   *) CARD_WPA_DRV="wext" ;;
   esac

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 1808

PostPosted: Tue 03 Dec 2013, 13:35    Post subject:  

tempestuous wrote:
3.

<snip>
Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>


So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2230
Location: The Blue Marble

PostPosted: Tue 03 Dec 2013, 14:11    Post subject:  

mavrothal wrote:
tempestuous wrote:
3.

<snip>
Most recently, the firmware is no longer held in compressed form ... but it's still held in an unconventional location (/lib/modules/all-firmware) and there's still a (kludgey) process whereby any relevant firmware is relocated to the correct directory as the associated driver loads for the first time.
But since the firmware is no longer compressed within Puppy's filesystem, this relocation process is now completely redundant - the firmware might as well be located in the correct directories in the first instance, and the relocation function can be disabled - it's in /sbin/pup_event_backend_modprobe
and it references /etc/modules/firmware.dep.<kernelversion>


So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?


+1 Yes. Pros:
1. Faster
2. Easier to manage
If size is concern, then pack them into squashfs. This is what Fatdog does.

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Wed 04 Dec 2013, 21:19    Post subject:  

Karl Godt wrote:
Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic.php?p=223939#223939
915resolution-0.5.3-patched20110617.tar.gz

OK, done. I found a more recent version of my patched 915resolution, dated May 21st 2012.
Bear in mind that from about Sandy Bridge onwards, 915resolution is no longer compatible.


anikin wrote:
http://packages.debian.org/wheezy/intel-microcode
I've tried hard and failed to make it run on all the latest puppies

The firmware that I have been discussing is the type that's required to work with a related kernel module. Without such firmware, the kernel module will fail to work altogether.
The Intel processor microcode, on the other hand, is quite different. It appears to be more of a "hotfix" than an essential driver component. And since your computer can obviously boot and run without it, this firmware certainly cannot be considered "essential", so I wouldn't lose sleep over it.
The reason you cannot get it working with Puppy is simple; here is the relevant section of Puppy's kernel configuration -
Code:
# CONFIG_MICROCODE is not set

If you REALLY want to get it working, the first thing you must do is lobby the Puppy developers to change the kernel configuration. This will yield a new driver called "microcode", which is used in conjunction with the microcode_ctl utility to load the firmware at each boot up, as launched from a script in /etc/rc.d


mavrothal wrote:
You mean something like this?
Code:
    INTMODULE=$(readlink /sys/class/net/$INTERFACE/device/driver)
    INTMODULE=${INTMODULE##*/}
   case "$INTMODULE" in
   prism2_*) USE_WLAN_NG="yes" ;;
   # orther_no_wext_card) add as needed ;;
   *) CARD_WPA_DRV="wext" ;;
   esac

I like the simplicity of your code, and you're on the right track, but no, the whole logic needs to be adjusted -
the CARD_WPA_DRV variable is completely redundant, and we can dispense with this variable altogether.

There are now 3 possible wifi driver scenarios:
i) prism2_usb - the wlan-ctl setup regime is used.
ii) all other wifi drivers, which are WPA-compatible - conventional (iwconfig) setup regime, both WEP and WPA encryption offered.
iii) all other wifi drivers, which are NOT WPA-compatible - conventional (iwconfig) setup regime, only WEP encryption offered.

In scenario ii) the wpa_supplicant interface-type will ALWAYS be "wext", so the wpa_supplicant command can be hard-coded with this parameter - there's no need to vary it.

mavrothal wrote:
So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.

mavrothal wrote:
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.


jamesbond wrote:
If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.
Back to top
View user's profile Send private message 
01micko


Joined: 11 Oct 2008
Posts: 7841
Location: qld

PostPosted: Wed 04 Dec 2013, 21:35    Post subject:  

tempestuous wrote:
jamesbond wrote:
If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.

All the firmware is inside the main sfs anyway.

_________________
Woof Mailing List | keep the faith Cool |
Back to top
View user's profile Send private message Visit poster's website 
Karl Godt


Joined: 20 Jun 2010
Posts: 3972
Location: Kiel,Germany

PostPosted: Wed 04 Dec 2013, 23:00    Post subject:  

tempestuous wrote:
Karl Godt wrote:
Myself would like to ask you to check http://murga-linux.com/puppy/viewtopic.php?p=223939#223939
915resolution-0.5.3-patched20110617.tar.gz

OK, done. I found a more recent version of my patched 915resolution, dated May 21st 2012.
Bear in mind that from about Sandy Bridge onwards, 915resolution is no longer compatible.


Code:
# CONFIG_MICROCODE is not set

If you REALLY want to get it working, the first thing you must do is lobby the Puppy developers to change the kernel configuration. This will yield a new driver called "microcode", which is used in conjunction with the microcode_ctl utility to load the firmware at each boot up, as launched from a script in /etc/rc.d


mavrothal wrote:
So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.
mavrothal wrote:
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.


jamesbond wrote:
If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.


Thank you very much !

*

DOTconfig_2.6.37.4-KRG-iatom-1:CONFIG_MICROCODE=m
Now I know what this might be used for . Thanks again for the very good explanation .


*

I don't think that the firmware_module_func
#101120 new firmware_tarball_func, as /sbin/pup_event_backend_modprobe rewritten. that is in functions4puppy4 and in *_backend_modprobe cannot be totally replaced with a 99% solution at once .
This will take several puppy releases I think . Step by Step .
Both functions are essentially the same with a USLEEP difference .
The whole serialize code to prevent multiple loadings of modules is as heavy as firing modprobe all the time IMHO .

pgprs* can be installed into /usr/bin/ by default and the pgprs-shell script dropped then .
[edit : wrote pgprs-setup, meant pgprs-shell ]
*

Was firmware ever a component of standard puppy_initrd.gz ?
Back to top
View user's profile Send private message Visit poster's website 
mavrothal


Joined: 24 Aug 2009
Posts: 1808

PostPosted: Thu 05 Dec 2013, 00:26    Post subject:  

tempestuous wrote:
the whole logic needs to be adjusted -
the CARD_WPA_DRV variable is completely redundant, and we can dispense with this variable altogether.

There are now 3 possible wifi driver scenarios:
i) prism2_usb - the wlan-ctl setup regime is used.
ii) all other wifi drivers, which are WPA-compatible - conventional (iwconfig) setup regime, both WEP and WPA encryption offered.
iii) all other wifi drivers, which are NOT WPA-compatible - conventional (iwconfig) setup regime, only WEP encryption offered.

In scenario ii) the wpa_supplicant interface-type will ALWAYS be "wext", so the wpa_supplicant command can be hard-coded with this parameter - there's no need to vary it.

mavrothal wrote:
So you suggest to remove the "firmware_tarball_func" of the script and /etc/modules/firmware.dep.<kernelversion>?

Yes.

mavrothal wrote:
Is the pinstall.${FIRMPKG}.sh script included in some firmware needed?

No. We simply analyse those scripts according to the Puppy architecture in use at the time, and add any feature (device node, configuration script, etc) specified by those scripts.


jamesbond wrote:
If size is concern, then pack them into squashfs. This is what Fatdog does.

Not a bad idea.


Thank you for the clarifications. It makes a lot of sense to be adopted by "puppy-next".
Would you happen to have a draft of the the code for the proposed changes?ie are you (will be) working on it assuming puppy goes with all firmware in /lib/firmware (SFS'ed or not)?

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
tempestuous

Joined: 10 Jun 2005
Posts: 5285
Location: Australia

PostPosted: Thu 05 Dec 2013, 04:37    Post subject:  

Karl Godt wrote:
I don't think that the firmware_module_func
#101120 new firmware_tarball_func, as /sbin/pup_event_backend_modprobe rewritten. that is in functions4puppy4 and in *_backend_modprobe cannot be totally replaced with a 99% solution at once .
This will take several puppy releases I think . Step by Step .

I disagree. The complexity of the current situation has confused us all, and we seem reluctant to change it!
But the current regime is IMO way more complicated than it needs to be, and it's really not difficult to remove.

In /sbin/pup_event_backend_firmware, and in /etc/rc.d/functions4puppy4
just remove the entire "firmware_tarball_func" section.

/etc/modules/firmware.dep.<kernelversion> can be completely removed.

/sbin/pup_event_backend_firmware must REMAIN. This is the core firmware loading mechanism - it's triggered by a udev rule:
/etc/udev/rules.d/50-udev-puppy-basic.rules


Karl Godt wrote:
pgprs* can be installed into /usr/bin/ by default and the pgprs-shell script dropped then.

Well, partly. Sure, once /lib/modules/all-firmware/pgprs has been copied into the filesystem, you should then delete from /usr/bin/pgprs-shell the section relating to "firmware" ...
but you might want to keep the "fixmenus" section.

Karl Godt wrote:
Was firmware ever a component of standard puppy_initrd.gz ?

Not as far as I know. The initrd consists mainly of the static kernel. There are only a few kernel modules included, and none of these require external firmware.


mavrothal wrote:
Would you happen to have a draft of the the code for the proposed changes?

The changes relating to firmware handling are very simple - I just explained them in this post.

Regarding the firmware collection, itself, I would be prepared to help build a list of all firmware that is NOT obtained from kernel.org.

Regarding the Network Wizard, I'm not a programmer, so I can only advise on the logic and correct processes - that's exactly how I helped with the current Network Wizard, in collaboration with forum member Dougal, who wrote the code.
Unfortunately Dougal is no longer active on the forum. Someone else needs to step in and offer to update the Network Wizard code.
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 1808

PostPosted: Thu 05 Dec 2013, 07:34    Post subject:  

tempestuous wrote:

Regarding the firmware collection, itself, I would be prepared to help build a list of all firmware that is NOT obtained from kernel.org.

That would be very helpful.
Please do so when you have the chance.
Thx

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 3 [31 Posts]   Goto page: 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1445s ][ Queries: 11 (0.0037s) ][ GZIP on ]