Saluki, Puppy Remastered

[Iguleder]

Now I'm doing another experiment, since the Slackware experiment failed. Dependency resolution was bad I guess.

Now I wrote a tool similar to Woof and Pdebthing that can download DEB packages (of Ubuntu, Debian, ...) with their dependencies, but this time it's written in Perl and all its components are independent, so I can use this as the base for a package manager. It's similar to the relationship between Woof and PPM.

I wrote it from scratch with more of the secret UNIX philosophy sauce on top

At the moment I run it with these packages and the Ubuntu Maverick repos:

ubuntu-minimal
xorg
desktop-base
menu-xdg
extra-xdg-menus
kdelibs-kde3
kdebase-kde3

While Pdebthing detected around 90 dependencies for the last 4 packages, this beast detects almost 300. It's 100% percise but much heavier. Lots of recursion work

Just a working X ... is that too much to ask for?

EDIT: X won't start and I noticed things like dbus and hal are missing, so now I'm doing a second run with those:

ubuntu-minimal
ubuntu-standard
xorg
desktop-base
lubuntu-desktop

As you see it's pretty much the entire Lubuntu, I want to start with all packages and then start the trimming. I just want a working X. Puppy's X works but it's broken, missing icons and stuff because the Puppy init scripts create a desktop icons "cache" and run the "event manager" ... I don't have those in the Calf Linux init.

I also noticed many packages have post-install scripts so I must reverse-engineer this too, I need to write some script that installs a DEB package to a directory ... then I need to run it for all packages in one huge directory and make a SFS from this mess

[technosaurus]

Here is a status update on my gcc-4.5.2(snapshot) + binutils-2.20.51(snapshot) + glibc-2.12.1 build tree:
Please let me know if any of these choices have straye from our goals.

gcc built without cloog-ppl due to autoconf conflicts (it is in a transition from cloog-ppl + ppl to just cloog + ppl) there are still some size regressions compared to some older versions, but I think it is worth testing to see if the other improvements are worth it. (libelf,mpc,gmp & mpfr are compiled/linked static)

I used the binutils snapshot to pick up commits to work with gcc

I chose glibc-2.12.1 because Versions of the GNU C library up to and including 2.11.1 included an incorrect implementation of the cproj function. and patched stat.h to allow compilation with -Os

Updated libz to 1.2.5 (there are patches in the mail list that could improve performance by up to 50%, however discussion has died off and I couldn't find the latest combined patch - so it is not applied)

xml2 works for gtk2 with --with-minimal --with-push to come pretty close to the size of expat; however, other various programs do require different additional options that end up making it at least 4x larger, so plan on having gtk based on expat.

freetype required some manual configuration in header files (no configure option available) to remove bloat (3 different types of debugging) Note that the lzw patents have expired so lzw is enabled by default via its own internal implementation (this is the case for many applications and it would be a good project to begin porting these to use liblzw or possibly adding lzw support to libz instead for libz-2.0)

fontconfig - patched the makefile for libfontconfig object files to be compiled with -fpic (I don't know why this is not the default. AFAIK all shared libraries should have position independent code) and set the default font path to /usr/share/font
Tonight I will have a go at the graphic libs: png + apng patch, gif & jpeg

For all of the above the following were used:
CFLAGS=" -Os -finline-small-functions -findirect-inlining -march=i586 -mtune=i686 "
./configure options: --prefix=/usr --sysconfdir=/etc --localstatedir=/var --build=i486-t2-linux-gnu (for backwards compat) --enable-static --disable-shared
replaced -fPIC with -fpic for all shared libs (it is smaller and faster)
disabled debugging
left NLS enabled if it is by default (nls mainly affects the output of binary programs that are distributed with the libs and are mostly obfuscated from the user by various gui wrappers with better localization so maybe nls could/should be eliminated in these to reduce wrapper complexity??? non-english speakers please advise)
I read some comparisons on X11 vs. xcb - based on those xcb _should_ be the right choice (This may require building a separate static Xvesa)

Perhaps we should think about having Xvesa, jwm and rxvt as static uclibc builds so that no matter what happens we can always get a functional desktop? for instance if libc,z,xcb,X11... gets broken

[goingnuts]

Any thoughts on compressed binaries with upx and compressed libraries with zlibc?

A small test showed the following:

Binary----------------------Size of main sfs-----------Size of initrd
jwm static 1000K:--------------2940K--------------------2105K---
jwm static upxed 596K:-------2844K--------------------2219K---

everything else the same.
So seems that LZMA has better compression and that upxed in initrd increase size whereas you get a smaller size using upxed bins in squashfs-files...

Squashfs LZMA might be a choice for main sfs-file?

[Iguleder]

2.6.36 already has LZO, .37 or .38 should have LZMA.

And regarding Xvesa, it's a dead project with a problematic license, that's why xorg was started. I think it's a bad idea to use it ... but I think it could be interesting if we could experiment with two X packages, one static with just fb/vesa and a full server.

[technosaurus]

everything else the same.
So seems that LZMA has better compression and that upxed in initrd increase size whereas you get a smaller size using upxed bins in squashfs-files...

Squashfs LZMA might be a choice for main sfs-file?

what if instead only the files in the init are compressed (if the init was just cpio with no gzip or lzma compression it would load faster) it would be hard to time the difference though

Slax has patches for squashfs-lzma already (for a while now)

2.6.36 already has LZO, .37 or .38 should have LZMA.

And regarding Xvesa, it's a dead project with a problematic license, that's why xorg was started. I think it's a bad idea to use it ... but I think it could be interesting if we could experiment with two X packages, one static with just fb/vesa and a full server.

lzo _just_ came off patent and lzma has been looked at several times already thanks to slax (they are pretty much opposites fastest+decent compression vs decent speed+best compression)
There is also a 2.6.31-rc8 patch for xz that could be adapted ... xz is similar to lzma but it will allow faster decompression than lzma if the "-e" parameter is pass during compression (it looks like xz-5.0.0 finally came out and they don't have the patch readily accessible right now)
[/quote]

[Iguleder]

squashfs-lzma is against Squashfs 3.x, which means it needs 2.6.27 or below. And regarding a raw cpio initramfs - it takes more time to read a larger file. Consider the btrfs/ext4 benchmarks we see everywhere ... I think it's better to compress it with LZO or something to reduce size without too much impact on boot times and use some internal compression.

[jemimah]

I vote for lzma or xz for the main sfs if we can get it to work. I don't think the performance difference for the on-the-fly decompression would be very noticeable - but that's fairly easy to test.

if the initrd is kept relatively small, I don't think it matters much how you compress it. I compile all the kernel modules needed for booting straight into the kernel and then compress the kernel with lzo. That makes it real easy to swap kernels. There's probably good reasons not to do it that way for general purpose hardware though.

[technosaurus]

if the initrd is kept relatively small, I don't think it matters much how you compress it. I compile all the kernel modules needed for booting straight into the kernel and then compress the kernel with lzo. That makes it real easy to swap kernels. There's probably good reasons not to do it that way for general purpose hardware though.

fuse,ntfs-3g & other out of tree modules? however I wonder if we should just place the extra kernel modules inside the kernel's builtin internal initramfs (would require an additional kernel build) ... to be honest it would be nice for development purposes to have everything but the init script built into it ... maybe a couple of others??

@Iguleder - have you tried Alpine linux - it is a complete uclibc system

for some real cutting edge ideas check out xpud http://penkia.blogspot.com/2010/11/xpud ... thing.html
Its not uclibc based, but it has some really innovative approaches

also there are at least 2 xcb based window managers awesomewm and mcwm

I did a quick test of mcwm and it works fine and only use 1.5Mb of resources even compiled against shared glibc and libxcb (though I prefer different keybindings) ... it could make an excellent fallback or even be the preferred wm for low resource machines - I may test it with static uclibc since xcb seems to be laid out much better than X11... it appears the plan is to eventually eliminate libX11 anyways

[goingnuts]

Guess it all ends up in a compromise between size of the different elements, speed and ram-usage.
For me the best compromise is:
Kernel holding drivers/modules needed to boot on most hardware (>year 2000)
All other modules in zdrv (and the possibility to remove unneeded with zdrvctr after install)
initrd holding only what is needed to get to the switch-root point/a repair console (consider asm-tools for speed and size...?)
Main system in lzma-squash compressed file containing only bare-bone applications (could be pure CLI)

After that a package system (could be theme packages like asm-based, ncurses-based, gtk1-based, small-size-based, fastest speed-based, lowest resource-use-based etc) to let the user decide/choose the further customization of Saluki.

On top of that an on demand merger of personal-save-file and main-sfs - consolidating the personal additions and refresh of save-file...

All this can be done even with re-build of P412 (the best of them all and in which I am stuck...) - think its just a question of making a decision on which compromise to choose...

[technosaurus]

gtk2 will not build with xcb as it stands!! gtk-xcb was forked over 4 years ago and no further work is being done on it in gtk-main - there isn't even a branch and as far as I can tell nobody is working on it at all.
This seems very odd to me, as directfb continues to be supported (but could easily be replaced by xcb)

In case anyone is interested in taking it over, here is the old gtk-xcb port
http://gtk-xcb.svn.sourceforge.net/viewvc/gtk-xcb/gtk+/
To update it with gtk2-main, mostly only the gdk/X11 directory needs to be updated (I forced configure to let it try to build and almost all other objects built fin). A lot can come from the old port, but now all of gtk's dependencies will build with xcb, so the rest can use xcb, pango, cairo and now even gdk-pixbuf since it has been split off.

In looking through this stuff I found a different *randr that may fix issue with refresh rates that people have been having with xrandr:
http://cgit.freedesktop.org/xcb/demo/tree/xcbrandr.c

[Lobster]

Sorry to intrude.
I find am barely able to comprehend the latest conversations.

Can anyone clarify?

Is a base being worked on? In other words are we beyond theory and moving towards a practice? Who is creating the base?

That might not even be the right question.
Just trying to find out what's happening?

Thanks guys.

Puppy Linux
Elite Geekery available

[ttuuxxx]

I personally would like to see bigbass take lead development of puppy 6, I have too many other projects to do such a thing and he has proven himself as a dedicated puppian, The only restraints that I would like to see is that glibC and gtk matches a current main stream release like Debian, Ubuntu, Slackware. I don't care which one it is. But without the extra Gnome bloat would be nice as seen in series 5.
I'm not trying to start something, I just want to see puppy stay at its objectives from series 1-4. Keep a small distro and a small memory/cpu footprint. I will help with packages etc, as long as I have a basic compiling environment.
ttuuxxx

[technosaurus]

@Lobster

• In order to slim down Puppy, compared to recent versions, I have built and rebuilt the base system several times to figure out the best combinations for the leanest build. Here are the results:
  
  gcc - needs a build later than 4.5.1 (recent snapshots have significantly reduced many significant code size regressions) - I would _rather_ wait for the upcoming release than use a snapshot version (otherwise 4.3.5 is the best current "release" version - maybe 4.4.5 fixed the worst regressions but I haven't checked it yet)
  
  kernel - build with small (-Os) enabled and Denys Vlasenko's patches to allow garbage collecting function and data sections (and add some extra options for speed: -finline-small-functions, -findirect-inlining, ...)
  
  glibc - build with small patch to enable -Os (and use some extra options for speed) - or use eglibc
  
  libX11 - build --with-xcb to enable future development toward smaller, faster xcb based apps (see mcwm for a current example)
  
  libpng - patch with apng (needed by mozilla)
  
  Pango - use Patched version of 1.24.5 to avoid the large libstdc++ overhead introduce in Pango-1.25
  
  latest versions of gtk and its dependencies unless noted above
  
  cups - using 1.3.X (due to libstdc++, but some other considerations too) and compile a small static cupsd - see next item
  
  daemonized apps - separate as static binaries to reduce resource usage (syslogd, klogd, getty, init, ... run top to see others) also use separate smaller shell (static dash) if speed is irrelevant
  
  script cleanup - doing this inside of bashbox: it significantly increases script speed where previously several different scripts were called since the extra files don't need to be found and opened and read and executed (they are now just function calls)

@tttuuxxx

• you _can't_ easily use any of these distros now _and_ avoid the bloat (except _maybe_ slackware), because they all end up using some extra requirements in the dependent libraries and don't compile everything with --as-needed .... even if they did, you would still need to implement so many work-arounds (wrapper scripts with LDPRELOAD missingstuff.o etc...) that you may as well build your own. There is a reason 5.X continues to grow.

[jemimah]

Wow, it sounds like you're really close. I can't wait to see how fast this thing runs.

[Lobster]

In order to slim down Puppy, compared to recent versions, I have built and rebuilt the base system several times to figure out the best combinations for the leanest build.

I gets it. Thanks for letting us know. Look forward to any start base you come up with

[big_bass]

Hey Jeff (ttuuxxx)

I sent you a PM thanks for the kind words friend


Joe

[ttuuxxx]

@tttuuxxx

• you _can't_ easily use any of these distros now _and_ avoid the bloat (except _maybe_ slackware), because they all end up using some extra requirements in the dependent libraries and don't compile everything with --as-needed .... even if they did, you would still need to implement so many work-arounds (wrapper scripts with LDPRELOAD missingstuff.o etc...) that you may as well build your own. There is a reason 5.X continues to grow.

yes but would it be so much to ask to use the same version of glibC and gtk/glib/pango etc as a major release like slackware,debian,ubuntu and to each their on puppy 5. ssl doesn't work and it no longer loads in memory due to the shear size of it.
http://www.murga-linux.com/puppy/viewto ... 878#469878
I have stated from day one it was too large buy hey what do I know.
ttuuxxx

[big_bass]

would it be so much to ask to use the same version of glibC

some basic simple things are defined as a standard which define the distro
and any fair chance of binary compatibility

I personally would never compromise building fresh without binary compatibility
of a major distro .

most people already know what I prefer but I will say it anyway
glibc and tool chain from slackware current for starters

compile the packages you need to keep it cut down but
a good supply (binary compatible) of packages available (since packages could be removed
if you need the full package you could uninstall the light package and install the full package)

and keeping things easy


an end goal first must be defined

a logical standardized way to build and install packages




I have a way to pre test all of that by the way I built TXZ
since removing or installing packages can be done and kernels could be replaced too
if all the packages were rebuilt with a new glibc and kernel
its really a new distro



you just cant unscramble scrambled eggs
after they are cooked

Joe

[ttuuxxx]

would it be so much to ask to use the same version of glibC

some basic simple things are defined as a standard which define the distro
and any fair chance of binary compatibility

I personally would never compromise building fresh without binary compatibility
of a major distro .

most people already know what I prefer but I will say it anyway
glibc and tool chain from slackware current for starters

compile the packages you need to keep it cut down but
a good supply (binary compatible) of packages available (since packages could be removed
if you need the full package you could uninstall the light package and install the full package)

and keeping things easy


an end goal first must be defined

a logical standardized way to build and install packages




I have a way to pre test all of that by the way I built TXZ
since removing or installing packages can be done and kernels could be replaced too
if all the packages were rebuilt with a new glibc and kernel
its really a new distro



you just cant unscramble scrambled eggs
after they are cooked

Joe

Yesssssssss finally 10000000% of what I said, but better explained I agree totally.
We do it our way but with extra deps etc it could be possible to cross the tracks and use other distro's software.
Look at 4 series heck they had kb3 packaged from ubuntu because it had the same glibc, sure it was massive the extra libs needed, but they had fun making it happen. But if we restrict glibC and gtk. Then it takes a bit of the fun away.
ttuuxxx

[jemimah]

My understanding is that as long as glibc is not too old, most binaries tend to work if their dependencies are met. I don't think package compatibility should be a major focus of Saluki. I think we should choose our own packages, based on what will be smallest, and fastest.