Kernel 2.6.27 and squashfs 3.4 patch problem [solved]

[MU]

yes, so we had to copy /lib/modules/2.6.27/.config to
/etc/modules/DOTconfig-K2.6.27

For those who did not download the source sfs, here is the config:
http://www.murga-linux.com/puppy/viewto ... 215#239215

Now I must find from which files the firmware.dep is created.
Maybe I have to write a small script, that generates it.

Mark

[big_bass]

jamesbond

I found:
/etc/modules/

It includes a file called
firmware.dep.2.6.25.16

I wonder how this file is created?
I think it had to be renamed to
firmware.dep.2.6.27

It also should be updated to the firmware that comes with the new kernel, but for a first test, we could ignore that.

I wonder, if your firmware now gets extracted correctly?

Who knows, how this file is created?
And what is /etc/modules/DOTconfig-K2.6.25.16 needed for?
Is it there for documentation, or is it scanned by some scripts?
Mark

Hey Mark

in the unleashed createpuppy script there is a reference to /lib/modules
not etc/modules


this all gets built in unleashed

********************************cut and pasted relevant lines of createpuppy script************
--------------------------------not the complete script ----------------------------------------


cp -a kernels/$USEKERNEL/firmware.dep.$USEKERNEL
echo "" >> modules-complete/lib/modules/firmware.dep.$USEKERNEL





echo '#These are the directories and files in each firmware pkg...' >> modules-complete/lib/modules/firmware.dep.$USEKERNEL






echo "Adding $ONEFIRM filelist to modules-complete/lib/modules/firmware.dep.$USEKERNEL..."
cp -af kernels/$USEKERNEL/all-firmware/$ONEFIRM/* modules-complete/
#need a list of all files in the package...
echo -n "${ONEFIRM}|" >> modules-complete/lib/modules/firmware.dep.$USEKERNEL
cd kernels/$USEKERNEL/all-firmware/$ONEFIRM
find ./ -mount | sed -e 's/^\.//g' | grep -v '^/$' | tr ' ' '_' | tr '\n' ' ' >> $PUPPYDIR/modules-complete/lib/modules/firmware.dep.$USEKERNEL
sync
echo >> $PUPPYDIR/modules-complete/lib/modules/firmware.dep.$USEKERNEL
cd $PUPPYDIR
done
sync
******************************************************************************************************

hope that helps btw that was a code snipit from 3.01 unleashed I dont know if in version
4.00 that got changed

big_bass

[MU]

ok thanks, that might help...

I encountered other issues in this context.
1.) I could not find out, where the firmware comes from.
I could not find it in the Kernelsource, that has much less firmware files.
So I think the firmware files are something, that Barry got somewhere else.

For example iwl3945.ko IS part of the Kernelsource 2.6.27.
But /lib/firmware/iwlwifi-3945-1.ucode is NOT.
That file is only available in Puppy 4.1.
I also could not find it in the 3rd party modules at:
http://puptrix.org/sources/kernel-2.6.25.16/

So I then wanted to use the /lib/modules/all-firmware/ from Puppy 4.1.
But I encountered the same problem as jamesbond.

When I type:
modprobe iwl3945
then the
/lib/modules/all-firmware/iwlwifi.tar.gz
is NOT extracted.

I then thought it might have to do with a new kerneloption
CONFIG_FIRMWARE_IN_KERNEL = y
http://cateee.net/lkddb/web-lkddb/FIRMW ... ERNEL.html

So I deactivated it, and recompiled the Kernel.
But without success.

2.) Then I ran Puppy 4.1 original again (2.6.25.16).
Here I had the same problem:
When I type:
modprobe iwl3945
then the
/lib/modules/all-firmware/iwlwifi.tar.gz
is NOT extracted.

Same is with
modprobe dvb-usb
That problem was reported by linuxcbon here:
http://www.murga-linux.com/puppy/viewtopic.php?p=237657

So it seems, there is something wrong with the extraction of the firmware modules.

I then thought, that I might extract them all.
But that would work just in Puppy 3, that has a modified modprobe script.
It executes installation scripts, that are included in the firmware packages.
Like pinstall.slmodem.sh.
In Puppy 4, this mechanism is no longer used, instead, it is executed by the new (for me nonworking) event based extraction.

Fortunately, only some modems seem to have these scripts:
pinstall.dgcmodem.sh
pinstall.ltmodem.sh
pinstall.mwavem-1.0.2.sh
pinstall.pgprs.sh
pinstall.slmodem.sh

So I think I will rebuild the iso with the extracted firmware, and accept, that these modems will not work, unless the scripts are executed by hand.

This seems the best compromize, until a solution/bugfix is found.
The disadvantage is, that some firmware packages put scripts in /etc/init.d/ , what might slow down startup.
But I will test it, Muppy Live uses this solution, too.

Mark

[MU]

I updated the Iso.
The Kernel now uses the lzma patch provided by Amigo.

I included the firmware from the original Puppy.
It is extracted, except these modem firmware packages:
dgcmodem.tar.gz
ltmodem.tar.gz
mwavem-1.0.2.tar.gz
pgprs.tar.gz
slmodem.tar.gz
Those have these pinstallscripts described above, so simply extracting them would make no sense.
Instead, the extraction script should be enhanced to work on every computer.

So Puppys firmware also should work, if on your computer the automatic extraction does not work.
The files in /etc/modules were updated/renamed.

I also upgraded the networkwizard to oct-11th.
And added the Glint xorg driver.

And I modified init again.
I removed the 4 seconds delay (and also the 3 seconds for classmate), and instead added a check, if usb-storage had finished to scan for drives.
Barry already had such code there, but for me, it did not work. I think, that is also, why he had added the 3 seconds for the classmate.
So I replaced it with own one, I hope this works for everybody.

Mark

[aragon]

hi mark,

i will test this evening.

one more question: did you compile the kernel with the "tickless idle"?

cheers
aragon

[jamesbond]

Thanks Mark,

I'm downloading and once finished, I'll test it.
However, your point about this is strange:

2.) Then I ran Puppy 4.1 original again (2.6.25.16).
Here I had the same problem:
When I type:
modprobe iwl3945
then the
/lib/modules/all-firmware/iwlwifi.tar.gz
is NOT extracted.

The laptop I'm using to test has this iwl3945 device. With Puppy 4.1 original, after boot up is finished, iwlwifi-3945-1.ucode is already extracted into /lib/firmware. I'm not sure of the mechanism - a few searches in Barry's blog such as this:
http://puppylinux.com/blog/?viewDetailed=00406
and this:
http://puppylinux.com/blog/?viewDetailed=00108
and this:
http://puppylinux.com/blog/?viewDetailed=00121
and this:
http://puppylinux.com/technical/module-loading.htm

Seems to be that the module extraction is done prior to modprobe, therefore, the example you show, by doing modprobe iwl3945 doesn't work because you don't have the actual iwl3945 hardware? (which is the one that triggers the uevent and therefore extract the module)?

Anyway - as long as it works - I don't mind

cheers!

[jamesbond]

I tested. The wireless iwl3945 works now.

(well, actually, it doesn't work from the network-wizard - I noted that you have thrown in the oct11 version --- I had to manually hack wpa_supplicant.conf, but that's another problem which doesn't belong in this thread).

Thanks !

[Sit Heel Speak]

Hi Mark,

This third version incorrectly shows my rig as having 4 processors in "System-->Hardinfo-->Summary" and "-->Processor" as I reported here.

Your second version correctly shows 2 processors.

***EDITED November 20: 4 processors _should_ be shown. Not sure why the 2nd was showing only 2.***

(rest deleted by poster)

[MU]

Sit Heel Speak

this is very astonishing.
I just made a change in this firmware option, and added lzma.

The change then required, to answer some more questions concerning firmware.
So I think, the issue will be here. I could not imagine, that the lzma patch would cause a slowdown.
On the other hand, it just saves around 350 kb.

I think the best will be, to take version 2 again.
Then add my initrd patch, and the compressed firmware.
Barry explained why the firmware was not extracted, it would require a reboot:
http://www.puppylinux.com/blog/?viewDetailed=00421

I just bought a DVBT Stick, so that I can check on my own, if the extraction works.
If I have success, I'll use this scenario for a new iso tomorrow.

Mark

[BarryK]

jamesbond

I found:
/etc/modules/

It includes a file called
firmware.dep.2.6.25.16

I wonder how this file is created?
I think it had to be renamed to
firmware.dep.2.6.27

It also should be updated to the firmware that comes with the new kernel, but for a first test, we could ignore that.

I wonder, if your firmware now gets extracted correctly?

Who knows, how this file is created?
And what is /etc/modules/DOTconfig-K2.6.25.16 needed for?
Is it there for documentation, or is it scanned by some scripts?
Mark

Hey Mark

in the unleashed createpuppy script there is a reference to /lib/modules
not etc/modules


this all gets built in unleashed

********************************cut and pasted relevant lines of createpuppy script************
--------------------------------not the complete script ----------------------------------------


cp -a kernels/$USEKERNEL/firmware.dep.$USEKERNEL
echo "" >> modules-complete/lib/modules/firmware.dep.$USEKERNEL





echo '#These are the directories and files in each firmware pkg...' >> modules-complete/lib/modules/firmware.dep.$USEKERNEL






echo "Adding $ONEFIRM filelist to modules-complete/lib/modules/firmware.dep.$USEKERNEL..."
cp -af kernels/$USEKERNEL/all-firmware/$ONEFIRM/* modules-complete/
#need a list of all files in the package...
echo -n "${ONEFIRM}|" >> modules-complete/lib/modules/firmware.dep.$USEKERNEL
cd kernels/$USEKERNEL/all-firmware/$ONEFIRM
find ./ -mount | sed -e 's/^\.//g' | grep -v '^/$' | tr ' ' '_' | tr '\n' ' ' >> $PUPPYDIR/modules-complete/lib/modules/firmware.dep.$USEKERNEL
sync
echo >> $PUPPYDIR/modules-complete/lib/modules/firmware.dep.$USEKERNEL
cd $PUPPYDIR
done
sync
******************************************************************************************************

hope that helps btw that was a code snipit from 3.01 unleashed I dont know if in version
4.00 that got changed

big_bass

it's /etc/modules/firmware.dep.<kernel version>.

You should be looking in the latest Unleashed, that is, for 4.1.

The firmware.dep files are manually created and are in Unleashed, in puppy-unleashed/kernels/<kernel version>/firmware.dep.<kernel version>

For 2.6.27, you can just copy all of puppy-unleashed/kernels/2.6.25.16/all-firmware, and copy firmware.dep.2.6.25.16 (and rename to firmware.dep.2.6.27) to puppy-unleashed/kernels/2.6.27/.

[MU]

ok, uploaded the 4th iso.
ftp://ks301128.kimsufi.com/testing/puppy4.1-k2.6.27/

it is based on version 2, the kernel did not change.
So no lzma.

- I added the firmware from Puppy 4.1, most is compressed.
I could confirm with a DVBT stick, that the extraction works, after modifying /etc/modules/firmware.dep.2.6.27 according to Barrys Blog:
http://www.puppylinux.com/blog/?viewDetailed=00421

- added networkwizard oct-16th.

- used the modified initrd.gz from version 3.

- I went through all the 3rd party modules.
Many did not compile, due to changes in the API.
So for those, new patches must be searched in the web.
But I think, those are used quite seldom, like the exotic modem drivers.

These did compile:
/lib/modules/2.6.27/extra/atl2.ko
/lib/modules/2.6.27/extra/ungrab-winmodem.ko
/lib/modules/2.6.27/slmodem/slamr.ko
/lib/modules/2.6.27/slmodem/slusb.ko
/lib/modules/2.6.27/kernel/drivers/net/r8101.ko

I also patched ndiswrapper and used the current madwifi snapshot.
Then these compiled, too.

The 2.6.27sourcecompiled_410.sfs includes all drivers sources (precompiled) in /usr/src/patches
I also added comments in:
modules-compile.txt
modules-compile-extra.txt
modules-compile-MU.txt

So you see, which ones did not compile (I pasted error-messages).
You could google for the error messages, to find patches for them.
That is, what I did for madwifi and ndiswrapper.

Please let me know, if this iso fixes the issues from the older versions.
If yes, I will start on monday, to upgrade Muppy with the same kernel.

Mark

[big_bass]

I updated this post to reflect new changes in the script #v4.02



#v4.02 firmware in /lib/all-firmware now tarballs. now /lib/modules/all-firmware.
#v4.02 /lib/modules/firmware.dep.<kern>,DOTconfig-K<kern> moved to /etc/modules.
#v406 now can select different kernels in same version. (IDE, SCSI, LZMA, etc.)
#v406 kernels/<sub dir> name can now be named <kern version>-<text>



#-------------------------ok the new relative code ---------------------
****this is cut and paste of the newly modified code for #ver4.02
and the changes pertaining to #v4.02 /lib/modules/firmware.dep.<kern>,DOTconfig-K<kern> moved to /etc/modules.
#-----------------------------------------------------------------------

#v2.12 now have complete set of modules...
echo "Creating modules-complete/..."
rm -rf modules-complete
cp -a kernels/$DIRKERNEL/all-modules modules-complete #v406
#mkdir -p modules-complete/etc/modules
#cp -a kernels/$USEREKNEL/firmware.dep.$USEKERNEL modules-complete/etc/modules/

---------------------------------------------------------------------


#v4.00 v406
cp -a kernels/$DIRKERNEL/DOTconfig-K${USEKERNEL} rootfs-complete/etc/modules/
cp -a kernels/$DIRKERNEL/firmware.dep.${USEKERNEL} rootfs-complete/etc/modules/
--------------------------------------------------------------------




#cp -a modules-complete/lib/modules/firmware.dep.$USEKERNEL boot/initrd-tree/etc/modules/
for ONECHOSENMOD in `cat /tmp/zdrvcutdown.list`
do
CHOSENPATH="`dirname $ONECHOSENMOD`"
mkdir -p boot/initrd-tree/lib/modules/$USEKERNEL/$CHOSENPATH
cp -af modules-complete/lib/modules/$USEKERNEL/$ONECHOSENMOD boot/initrd-tree/lib/modules/$USEKERNEL/$CHOSENPATH/
done
fi
sync
find boot/initrd-tree/lib/modules -type f -name *.ko |
while read ONEKO
do
gzip $ONEKO
sync
done

-------------------------------------------------------------------------------------------------------

P.S Thanks BarryK for this tip also
For 2.6.27, you can just copy all of puppy-unleashed/kernels/2.6.25.16/all-firmware, and copy firmware.dep.2.6.25.16 (and rename to firmware.dep.2.6.27) to puppy-unleashed/kernels/2.6.27/.

big_bass

[jamesbond]

Hi Mark,

Tested. I agree with SHS, your 4th disc (which uses the same kernel as the 2nd) is noticeably faster than 3rd disc.

iwl3945 Wireless flawless works also (this time with network-wizard).

cheers!

[Sit Heel Speak]

The 2nd disc is still the champion (but I do not use wireless).

The 4th disc still shows 4x Xeon processors in Hardinfo, just like the 3rd.

Zlib score (average of several runs)
is 16477 in the 4th (doesn't vary much between runs) and...
16201 in the 3rd (but with occasional, about 1-time-out-of-8, runs over 18000)...and...
in the high 17000's for the 2nd (varies a lot between runs, I saw as low as 16780 and as high as 18660).

The "missing hyperthread" problem, did show up just now, in the 3rd disc.

Seamonkey feels most crisp in the 2nd disc. In fact, it and ttuuxxx's firepup-.02 are very nearly as delightful-to-use in your 2nd disc, as Seamonkey is when running under my personalized Puppy 2.17-1, which uses a Con Kolivas-scheduler-patched 2.6.21.5 SMP kernel compiled in Gentoo.

I have downloaded the 4.1 Unleashed and kernel source packages and will follow your steps this weekend. Maybe I can spot some oddity.

[MU]

Sit Heel Speak
I'm puzzled,as the kernel is the same, it should at least show the same results in the amount of processor cores...

Can you please compare the size of vmlinuz in 2nd and 4th?
Maybe we talk of a different second release, and I used one, that already had changes?
Did you download the source of the second?
If yes, could you attach /usr/src/2.6.27/.config ?
Mark

[MU]

please also tell me the md5sum of the second iso, that works fastest and shows the corrects amount of processor cores.
md5sum Puppy-410-2.6.27.iso
Mark

[Sit Heel Speak]

On-2nd-disc vmlinuz size: 2284K
On-3rd-disc vmlinuz size: 1928K
On-4th-disc vmlinuz size: 2284K

md5sum's:

2nd: 1afd280c17563842e447db391bb4c11b Puppy-410-2.6.27.iso
3rd: 3f19c2e2403b606e24c3f94d19ae7b44 Puppy-410-2.6.27.iso
4th: fc44eb3cdae809c197ab64ba0ddf0027 Puppy-410-2.6.27.iso

All 3 match the md5.txt.

I downloaded the Unleashed + kernel source last night, and some overnight while I slept, so if there was a change in .config or the kernel source+patches then I do not have it.

And now Hardinfo is incorrectly showing 4x Xeon's

...***EDITED 4x CPU's is correct. So says everyone over on the Gentoo forum. Each hyperthread is seen as a separate CPU.***

[MU]

thanks,
md5sum and kernelsize indicate, that I used the "correct" second iso to create the 4th.
So the kernel is identical.
You could try to use vmlinuz from 2nd in the 4th, if you run them frugal, to verify it.

The one thing that changed, is initrd.gz (the delay).
So you could try, too to use the 2nd in the 4th.

The other thing that changed, is that there are some more additional modules and firmware from the 3rd-party drivers (modem, network).
Because of this, also /lib/modules/2.6.27/modules.dep is huger in version 4.
You could try to use the one from version 2 to see, if it makes any difference. But I don't think so.
And delete (move somwhere) the two files in /etc/modules.
So Puppys eventsystem ignores them.

So I have no idea at moment, why now 4 cores are shown

Mark

[Sit Heel Speak]

@Mark: you may wish to wait another day before building your new Muppy. I think I have built a better 4.1 SMP kernel than yours. I have --finally,...it took all day-- figured out amigo's lzma patch--at least, one way of making it work correctly...

It is late here now, and I must hit the sack. I have tomorrow (Monday) free and am strongly tempted to write a How-To.

[MUguest]

ok, I will wait for it, great.

I must build some other packages like openoffice with languagepacks, so will do this today.

Mark