Remove unneeded modules and firmware - Zdrv_Cutter

Core libraries and systems
Post Reply
Message
Author
User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

Remove unneeded modules and firmware - Zdrv_Cutter

#1 Post by technosaurus »

Thanks to amigo for the assist on this one...

This should remove about 20MB compressed of unneeded modules from your zdrv.... or during a remaster you can delete the contents of /lib/modules from your pupXXXsfs working directory and place the zdrv file in the iso directory ... thats about 1/5th the size of the iso

This is something I thought might be useful for the hardware specific distros to (pupeee, olpc etc...) and for low resource computers that still want to do a frugal install... or just the minimalist at heart

0.1 - just cutdown modules - still add all firmware(10-15MB)
0.2 - cutdown both modules and firmware(~5MB)
0.3 - removes initrd modules (~1MB)
TODO
#0.4 cover /lib/firmware (~2MB)
#0.5 cover xorg drivers (~4MB)
#0.6 patch woof to allow xorg drivers in zdrv
#0.7 patch remaster script to integrate this as option
#0.8 autobuild using /etc/modules/firmware.dep{,(uname -r)}
#0.9 bugfixes and GUI improvements
#1.0 Final release

#1.X main sfs support: stripping binaries (strip), removing manpages locales, DEV files and broken links; optimize images (optipng), symlink duplicate files (rdfind), compress large binaries (upx)

Development thread here:
http://www.murga-linux.com/puppy/viewtopic.php?t=51450
Attachments
zdrv_cutter-0.3.pet
also removes initrd modules since they get built automatically
adds some dialogs and chance to edit/check the build
(2.13 KiB) Downloaded 1953 times
zdrv_cutter-0.2.pet
cuts both modules and firmware
(2.02 KiB) Downloaded 1669 times
zdrvcutr.gz
original script - just for reference
(825 Bytes) Downloaded 1762 times
Last edited by technosaurus on Thu 21 Jan 2010, 14:25, edited 5 times in total.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#2 Post by technosaurus »

firmware is in tar.gz thus the loaded modules don't necessarily reference them, so the will need to be copied manually for now

example:
iwlwifi contains iwlwifi-3945... but the module name is iwl3945

iwlcore should get /lib/modules/all-firmware/iwlwifi.tar.gz

I added a couple lines that copy them all until we can figure out the cases (still cuts out 15MB)-

Does someone have a list of what module gets what firmware? rerwin? tempestuous?

Code: Select all

#!/bin/sh
. /etc/DISTRO_SPECS #gets $DISTRO* variables
KERNELVER=$(uname -r) # get the version of your kernel
PREFIX1CHAR="`echo -n "$DISTRO_FILE_PREFIX" | cut -c 1`"
KERNEL3CHARS="`echo -n "$KERNELVER" | tr -d '.' | tr -d '\-' | tr -d '[a-z]' | rev | cut -c 1,2,3 | rev`"
ZDIR="z${PREFIX1CHAR}${DISTRO_VERSION}${KERNEL3CHARS}"
# set this to the name of a temp dir:
TMPDIR=/dev/shm/$ZDIR #faster to use /dev/shm
# create the module directory in the TMP dir:
mkdir -p $TMPDIR/all-firmware
lsmod |tail +2 |cut -f1 -d' ' > $TMPDIR/mods #just a list of modules
sed -i "s/_/?/g" $TMPDIR/mods #replaces the _ with ? so that - and . will not give false evaluation
cd /
for module in $(cat $TMPDIR/mods) ; do
   # uncomment the first and comment the second one to just see the list
   # find lib/modules/$KVERS -type f -name "$module*"
   find lib/modules/$KVERS -type f -name "$module*" -exec cp --parents {} $TMPDIR/ \;
done
cp /lib/modules/all-firmware/* $TMPDIR/lib/modules/all-firmware #needed until we make case for each tarball
mv /lib/modules /lib/modules~ # can't force a link over a directory
ln -s -f $TMPDIR/lib/modules /lib #use link so busybox depmod works
depmod #-b $TMPDIR/lib/modules/$KERNELVER -e -F /path/to/kernel/System.map
rm $TMPDIR/mods #cleanup
dir2sfs $TMPDIR
mv $TMPDIR"_"$DISTRO_VERSION.sfs ~/$ZDIR.sfs #move to $HOME
mv $TMPDIR"_"$DISTRO_VERSION.sfs-md5.txt ~/$ZDIR.sfs-md5.txt #move to $HOME
rm -rf $TMPDIR
rm -rf /lib/modules
mv /lib/modules~ /lib/modules
Xdialog --left --buttons-style text --title "Zdrv Cutter" --ok-label "Show file" --cancel-label "Exit" --yesno "Cutdown z"${PREFIX1CHAR}${DISTRO_VERSION}${KERNEL3CHARS}".sfs is now in ~" 0 0
[ $? -eq 0 ] && rox ~ &
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#3 Post by technosaurus »

beta 0.2 is now released (see initial post)

added firmware cutdown section using info from /etc/modules/firmware.dep - removes an additional 5MB compressed

a typical zdrv is now ~4.5 MB instead of 25 MB
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#4 Post by technosaurus »

beta 0.3 is now released

removes /lib/modules/$(uname -r)/initrd since it gets built at startup
added some dialogs giving opportunity to check before and after it is built
fixed some typos
Attachments
zdrv_cutter-0.3.pet
(2.13 KiB) Downloaded 1646 times
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#5 Post by technosaurus »

you can do something similar with xorg drivers ~4MB

#this code tells you your xorg driver
grep -w card0driver /etc/X11/xorg.conf |cut -d "\"" -f 2

#files are located in /usr/X11R7/lib/xorg/modules/drivers/
#however some end in _drv.so and some just in .so
#but the ones without _drv are insignificantly small

There is no reason that these cannot go in the zdrv either AFAIK
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#6 Post by greengeek »

Does a smaller zdrv bring the benefit of a smaller RAM footprint and speed gains?

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#7 Post by technosaurus »

That depends on where the modules are originally located:
if they are in the pup*.sfs, you would need to remaster and it would reduce union/mount/decompression overhead and more with pfix=ram.
if they are already in a separate, larger zdrv.sfs, then it reduces the union/mount/decompression overhead.

Side note: after learning how modules are loaded (the entire module is memmapped first and then gets passed to init_module), I would guess that its possible to load the modules and then completely umount the zdrv.sfs
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
moeppyfan
Posts: 47
Joined: Thu 11 Feb 2016, 18:59

#8 Post by moeppyfan »

Hello technosaurus,

I tried your zdrv_cutter-0.3.pet on huge-kernel-3.14.55 (Slacko 6) and it shrinks my zdrv from 30MB to 2.8MB! Nice. The new 2.8MB zdrv boots to desktop - and trackpad, keyboard, video, sound all work on my Dell Latitude D410.

To get it to recognize my wireless, I had to recreate /lib/firmware dir and re-add these files:

Code: Select all

/lib/firmware/ipw2200-bss.fw
/lib/firmware/ipw2200-ibss.fw
/lib/firmware/ipw2200-sniffer.fw
And to get "depmod" to run without error, I had to re-add these two files:

Code: Select all

/lib/modules/3.14.55/modules.builtin
/lib/modules/3.14.55/modules.order
I'm not sure if copying that old module list is 100% accurate though, since stuff has been cut?
Everything seems to work now, so I'll keep using your cut version, and let you know if I spot anything else. :)

P.S. The user backi started a thread about your script here, which is how I found out about it.

edit: Tried also on Dell Latitude E4300. The process was similar, except I had to re-add /lib/firmware/b43 folder for wireless.

Pelo

I will try it with iwlwifi.

#9 Post by Pelo »

Very usefull topic, 2016 most of us surf the web with at laptop at home. Wireless just needed for your firmware sure makes the Puppy Lighter.
I will try it with iwlwifi.
zdrv_vivid_6.5.sfs: 30 M
staging: 3735 K seems a place to study.. but all that is in english, and i am not a programer.

jabu2
Posts: 46
Joined: Tue 08 Apr 2008, 03:19
Location: Australia

#10 Post by jabu2 »

Likewise not a programmer,
but TechnoS has once more come up with what looks like something really useful -if I'm understanding his concepts.

TechnoS identified it as "for low resource computers that still want to do a frugal install... or just the minimalist at heart ".

I think its also what is needed for a custom job, to use on a laptop/portable taken travelling - running on CD or USB.
Minimalist, but secure, and low risk of loss (if running on an oldie but goodie), uncluttered but doing the basics needed: =email, internet (top browser), file managing for whatever storage/dumping space the oldie has, etc, and easy wifi and internet connection?

Is that correct? Or are there better options for such a travelling OS?

Thanks TechnoSaurus for your efforts/initiative,
and to anyone who can give us an end-product using Google chrome and simple connectivity choices built-in.

ATVB (all the very best)
jabu2

stemsee

#11 Post by stemsee »

When will 1.0 be available?

Sailor Enceladus
Posts: 1543
Joined: Mon 22 Feb 2016, 19:43

#12 Post by Sailor Enceladus »

stemsee wrote:When will 1.0 be available?
I would love a version 0.3b that patches modules.builtin and modules.order to match the new zdrv, rather than deletes them.

Pelo

being tested with my X-Slim Slacko

#13 Post by Pelo »

being tested with my X-Slim Slacko backi information;
however i must keep iwlwifi, which is one of the biggest so no many MBs to delete

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#14 Post by technosaurus »

I don't really use this method now since the kernel's introduction of allmodconfig and allyesconfig.
make allmodconfig rebuilds the kernel without support for unused modules.
make allyesconfig will then build those used modules into the kernel
(for the full effect you need to boot into the former before building the latter)

The down side is that you need the devx and kernel source (I normally do), so I can see how this may still be useful.

most of the modules.* files are either completely unnecessary or automatically rebuilt by other things like depmod and will end up just getting duplicated in the savefile, so putting it in the sfs would be wasteful and redundant.

I may be able to do something about keeping the appropriate firmware though.

I have to stress that this method is just a hack. If you are doing a commercial product, the above kernel build method is more appropriate (probably followed by iterations of make menuconfig for anticipated upgrades)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#15 Post by rufwoof »

technosaurus wrote:I don't really use this method now since the kernel's introduction of allmodconfig and allyesconfig.
make allmodconfig rebuilds the kernel without support for unused modules.
make allyesconfig will then build those used modules into the kernel
(for the full effect you need to boot into the former before building the latter)

The down side is that you need the devx and kernel source (I normally do), so I can see how this may still be useful.
Fumbling my way through building a kernel, using fatdog 8.2 I downloaded the devx and kernel sources sfs and (booted frugally from a usb) unsquashed the kernel source sfs on my hdd and sfs loaded the devx.

Ran 'make' and it took hours. I even ctrl-c broke out of that as it had passed my zzzz time. The next day (or rather later that same day) I and ran make localyesconfig and then ran make again. It seemingly picked up from where it had left off. A hour or so later and it reported it had created the bzImage.

With the bzImage built I renamed that vmlinuz-mine. Made a folder /boot and copied that into there, along with the System.map. With my frugal booted normally, usb left attached, wireless net connected, I then ran make modules install. Don't think that was necessary as it just created another vmlinuz the same size as the first one.

Opened the default fatdog initrd and removed fd64.sfs (main sfs) and kernel-modules.sfs out of that. Reclosed it and gzip'd that initrd (that I named initrd-mine.gz).

Around 16MB combined vmlinuz and initrd.gz ... that boots fine. Within init (/sbin/system-init in fatdogs case) I exit to shell prior to the fd64.sfs loading (aufs layering) being set up, so this is just a tui/cli based boot. But it boots with wireless net connected OK and reads the usb ok. Which is all I want for this particular boot i.e. I use that busybox/initrd based boot to ssh into hashbang - from where I have tmux (multiple windows), irc, email, w3m web browser, mail lists, BBS access ...etc. (more than enough to keep me occupied when otherwise bored). As expected from such a small size, it boots very quickly even from usb. No pseudo terminals, no X, so very secure. And as I ssh into hashbang to run things from there my ISP/state only sees that encrypted link, no dns or other traffic. And remote sites only see the hashbang servers IP, not my IP. The likes of reddit (ssh redditbox.us) and yahoo finance using w3m are lightning fast, whereas when accessed using chrome they're more often relatively slow/laggy. Booted (from usb) and then the usb disconnected (running in ram), it uses around 76MB of ram. I have a swap partition on HDD and within init I set that up as a encrypted swap, so even though running in ram (4GB) it has that swap space (16GB partition) on top of that. To test that out I dd'd a 10GB file and that was created fine and the system continued to function as normal.

But as the attached image indicates (blurry text due to scaling down image - a clearer version is available here), you do need to recompile things again if new hardware is going to be used/attached. With the kernel built once however subsequent compiles run through much much quicker as it only recompiles the things that have been changed.
Attachments
localyesconfig400.png
(72.77 KiB) Downloaded 439 times
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

Post Reply