Alternative way to build Ubuntu / Debian Puppy [RETIRED]

[jamesbond]

@Jamesbond

I use kernel 3.15.1Emsee 64bit as posted in repacked fatdog631. iso on your fatdog 631 thread! I boot a frugal install on bare metal!

Ah ok. Can you help to do two things?
a) upload the kernel only + kernel-modules.sfs for that kernel (my ADSL is rather slow for the 500MB ISO)
b) try using with the supplied 64bit kernel (which Mick builds) instead

This way we can try to figure out where the problem is.

[stemsee]

@jamesbond

kernel and modules
https://drive.google.com/file/d/0B4GhZV ... sp=sharing

[Iguleder]

I'm doing an experiment right now - I've built a static initramfs with stuff like init and a kernel without modules (everything is built-in - it's 13 MB). I want to boot this thing with a main SFS containing debootstrap's output, while Debian runs in a container

EDIT: boots just fine, but trivial stuff like poweroff fail hard, because it doesn't have Debian's init. Now it's time to work around these rough edges and get a working JWM/ROX-Filer desktop

[stemsee]

Interesting stuff going on!

I built saucy 64bit booted to desktops fine. Hardware not detected fully, notably trackpad (mine i8042 needs kernel arg i8042.noloop) and ethernet/wlan.

I built two versions 1. From Woof-ce-testing :- Slower but much more useable end product. Hardware detected, ext.usb mouse ok, cannot connect to network, iwlist and ifplugd problems. ifplugd-old is flattened is it needed?
2. Woof-next :- very fast, incomplete menus and poor hardware detection, trying again with hal added!

[smokey01]

it's all gone a bit quiet in here just as it was getting interesting.

I built an iso from jamesbonds first link in his first post. I can get it to a desktop but I can't get it to recognise my wifi, without I can't use slapt-get to populate the distro.

This system is very fast and quite easy to use. It is rather large at 183M considering there are not many apps included.

Cheers

[jamesbond]

smokey, I'm still around

Yes, I've got the same comment about the speed as well. CatDude said his regular Woof builds is in the order of 6 hours; with the new build system he got an ISO after 10 minutes.

The build system is more or less finished (other than some more trimming can still be done, more "parent" distro can be added, etc); but the core Puppy infrastructure needs re-work too. I've only done the minimalist changes required to get it to boot to graphical desktop, but a lot more still needs to be done, which I hope the community will pick-up.

Now that we can get a complete build in about 10 mins or less, it should make it easier and faster to improve these scripts as the edit-build-test cycle is much shorter.

cheers!

[smokey01]

James,

Are there any differences between the two links below?

https://github.com/jamesbond3142/woof-C ... next-james

https://github.com/puppylinux-woof-CE/w ... /woof-next

Ta

[jamesbond]

James,

Are there any differences between the two links below?

https://github.com/jamesbond3142/woof-C ... next-james

https://github.com/puppylinux-woof-CE/w ... /woof-next

Ta

Changes goes to woof-next-james first. Then I will issue pull request and after a review it will go to woof-next. So generally speaking woof-next-james is a bit more bleeding edge than woof-next.

[saintless]

Hi, jamesbond.

I had some time to test these days using https://github.com/jamesbond3142/woof-C ... next-james to build 32 bit debian sid puppy.sfs. Booted with initrd and kernel-modules from Slacko6.
I don't know if this helps you but some fixes I added to make dpkg/apt-get work. This one is critical (otherwise locales will be removed because of missing dependency). Change in this section in /var/lib/dpkg/status from:

Code: Select all

Package: libc6
Status: hold ok installed
Priority: required
Section: libs
Maintainer: unspecified
Architecture: i386
Version: 2.19-10
Depends: libgcc1
Description: libc6 installed by deb-build.sh

to:

Code: Select all

Package: libc6
Status: hold ok installed
Priority: required
Section: libs
Maintainer: unspecified
Architecture: i386
Version: 2.19-10
Replaces: libc6-i386
Provides: glibc-2.19-1
Depends: libgcc1
Description: libc6 installed by deb-build.sh

also added in /etc/group:

Code: Select all

utempter:x:103:
staff:x:50:

For example installing xterm will fail if utempter is not added. I guess installing more packages will show other missing groups also.
gettext package was missing and installing it shows conflicting dependencies:

Code: Select all

Unpacking xz-utils (5.1.1alpha+20120614-2) ...
dpkg: error processing archive /var/cache/apt/archives/xz-utils_5.1.1alpha+20120614-2_i386.deb (--unpack):
 trying to overwrite '/usr/bin/xz', which is also in package bblinks 1.0
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Maybe gettext is better to be taken from other place instead debian repository.
/var/lib/dpkg/available file is empty. I don't know what is the reason yet. It should contain all packages installed/uninstalled on the system with dpkg/apt-get.
I will try to play some more in the next days if I can find something more.

Toni

[saintless]

Hi, jamesbond.

There is something wrong with the process of installing packages in chroot-sid-i386 directory. Just to make clear I use real debian system with full dpkg/apt-get installed packages (not puppy with dpkg).
One of the problems is gettext which seems to be installed but actually is not installed.
The package gettext_0.19.2-2_i386.deb does not contain /usr/bin/gettext and gettext.sh (needed for most puppy utilities to work). The real gettext binary is part of gettext-base_0.19.2-2_i386.deb which is dependency package of gettext_0.19.2-2_i386.deb

Now here is the problem - reading /DEBIAN/Control file in gettext_0.19.2-2_i386.deb the dependency is there (+ many others). Here is the full content:

Code: Select all

Package: gettext
Version: 0.19.2-2
Architecture: i386
Maintainer: Santiago Vila <sanvila@debian.org>
Installed-Size: 6340
Depends: libc6 (>= 2.17), libcroco3 (>= 0.6.2), libglib2.0-0 (>= 2.12.0), libgomp1 (>= 4.9), libncurses5 (>= 5.5-5~), libtinfo5, libunistring0, libxml2 (>= 2.7.4), gettext-base, dpkg (>= 1.15.4) | install-info
Recommends: curl | wget | lynx-cur, autopoint, libasprintf-dev, libgettextpo-dev
Suggests: gettext-doc
Breaks: autopoint (<= 0.17-11)
Section: devel
Priority: optional
Multi-Arch: foreign
Homepage: http://www.gnu.org/software/gettext/
Description: GNU Internationalization utilities
 Interesting for authors or maintainers of other packages or programs
 which they want to see internationalized.

But this is what we get inside puppy.sfs /var/lib/dpkg/status for gettext (only dpkg as dependency):

Code: Select all

Package: gettext
Status: install ok installed
Priority: optional
Section: devel
Maintainer: unspecified
Architecture: i386
Version: 0.19.2-2
Depends: dpkg
Description: gettext installed by deb-build.sh

Same problem appears for most packages in /var/lib/dpkg/status which means dpkg database is not correct from the start.

The second problem that does not have solution yet (or only partly solved for now) is /var/lib/dpkg/available is empty and not updated at all after installing/removing packages with apt-get. Even if I populate it manually the content does not change after installing/removing packages. But proper populated and updated /var/lib/dpkg/available file is the only way to fix broken /var/lib/dpkg/status file.

Is such information useful for you or it is better to keep it for my self and try to solve the problems manually in puppy.sfs?

Toni

[smokey01]

Toni what sort of internet connection are you working with, eth or wifi?

[saintless]

Hi, Smokey, I use wired connection (eth).
It also does not work for me without some manually settings. Running the included SNS breaks already working internet. There is missing text editor also (or I still can't find it).Creating savefile/folder on shutdown also does not work and I use already created savefile/folder before first boot. But this are small problems compared to the incorrect populated dpkg database.

I'm not sure what you can do about wifi but if you have eth connection here is how to get internet working:

1. Run Puppy with working internet connection and copy /etc/resolv.conf and /usr/bin/geany and /usr/local/sbin/dhcpcd in /mnt/home (or other location outside Puppy save file). The locations might be different . Check the exact path with:

Code: Select all

which dhcpcd
/usr/sbin/dhcpcd

2. Boot puppy.sfs (or puppy.iso) made with woof-next scripts.

3. Replace /etc/resolv.conf with the one from /mnt/home and copy dhcpcd in /usr/sbin

4. Run in terminal dhcpcd and with some luck you should have internet connection now.

Backup plan with manually settings:
Here are the commands how to setup manually IP, netmask and Default gateway from terminal. Just replace them with yours:

Code: Select all

ifconfig eth0 192.168.100.2 netmask 255.255.255.0
route add default gw 192.168.100.1 eth0

Add manually dns servers in /etc/resolv.conf using geany. Mine for example are:

Code: Select all

nameserver 192.168.100.1
nameserver 198.41.0.4

Save /etc/resolv.conf and the internet should be working.

BTW shutdown, reboot works for me this way - first click on reboot and then logout and the system reboots. First shutdown and then logout will poweroff the computer.

Toni

[smokey01]

I'm using wifi wlan0 and none of the modules appear to be loading. Even after loading what I think are the correct modules ifconfig can't find the device.

BTW geany is included and works if you run it from a terminal. The desktop file points to /usr/local/bin/defaulttexteditor which then points to beaver that's why the desktop icon doesn't work.
Geany is in /usr/bin.

[saintless]

Geany is in /usr/bin.

Seems strange but I do not have /usr/bin/geany in puppy.sfs.
I will rebuild it again in the next days to confirm. Maybe I should use woof-next instead woof-next-james.

Toni

[jamesbond]

saintless, thank you. By all means, please post your findings and bug fixes here. The idea of this is to make puppy building easier, so by sharing you're improving the overall experience for all, as opposed to if you just tinker with the finished product (puppy.sfs).

I need to say that the project has slowed down a bit, though. I'll still try to find the time to get your changes into the system and push it down to woof-next sooner or later. If you find me not responsive enough, by all means fork the code, make the necessary changes and issue me a pull request.

About that extra lines: "Replaces: libc6-i386" and "Provides: glibc-2.19-1" - is that available for all packages? Is that available in the Debian package list, or is this only available from inside the deb itself?

About this: "trying to overwrite '/usr/bin/xz', which is also in package bblinks 1.0" - perhaps its better to install xz-utils (or whatever debian package that has xz in it) rather than having bblinks creates busybox symlink for xz.

As for the content of "/var/lib/dpkg/status", IIRC this is mostly manufactured information if you don't have real dpkg (apt-get is not used during build, only dpkg). It has been a while since I played with this, so I need to check whether it is mostly manufactured or comes from dpkg. If it is manufactured it won't be complete - I only took the *minimum* necessary to get dpkg working.

In anyway, you can enable the use of real dpkg using the directive in the pkglist.

@Ibidem - thanks for the info. I have no idea whether Puppy can be fitted into Alpine Linux - it seems that a lot of the Puppy base tools are missing. But it is still an interesting idea in its own right to build a Puppy-like system using Alpine packages. The package format doesn't seem to difficult, and the fact that it has statically compiled pkgtool helps a lot.

[saintless]

Hi, jamesbond.
Respond when you have time. I jast needed to make sure this type of fixes are useful since I do not change the code. Changing woof-next code is new area for me and It will take me more time to learn it than fixing the problems in puppy.sfs I know nothing about puppy init, shutdown and save scripts and I'm not much interested learning them. My interest is learning more about how flexible debian can become.

I'll still try to find the time to get your changes into the system and push it down to woof-next sooner or later.

Do not rush to change anything yet. I still cant get apt-get to work correct and have to test more fixes first.

About that extra lines: "Replaces: libc6-i386" and "Provides: glibc-2.19-1" - is that available for all packages? Is that available in the Debian package list, or is this only available from inside the deb itself?

The information is even more complete here: http://ftp.au.debian.org/debian/dists/s ... ackages.xz but inside the deb package Control file it is almost the same. I'm sure the information added in /var/lib/dpkg/status is not from packages.xz or from deb package Control file. It should come from somewhere else.

About this: "trying to overwrite '/usr/bin/xz', which is also in package bblinks 1.0" - perhaps its better to install xz-utils (or whatever debian package that has xz in it) rather than having bblinks creates busybox symlink for xz.

Yes, I think installing the debian files is better than symlinks to busybox.

As for the content of "/var/lib/dpkg/status", IIRC this is mostly manufactured information if you don't have real dpkg (apt-get is not used during build, only dpkg). It has been a while since I played with this, so I need to check whether it is mostly manufactured or comes from dpkg. If it is manufactured it won't be complete...

I will try to find out this also.

Toni

[saintless]

OK, I think the main dpkg database problem is inside /woof-CE-woof-next-james/builders/debian-build.sh (in update_pkg_status function):

Code: Select all

update_pkg_status() {
	{
echo \
"Package: $1
Status: install ok installed
Priority: $2
Section:  $3
Maintainer: unspecified
Architecture: $ARCH
Version: $4"
[ "${5%,}" ] && echo "Depends: ${5%,}" 
echo "Description: $1 installed by deb-build.sh
"
	} >> "$CHROOT_DIR/$ADMIN_DIR/status"

}

If this is the function that updates /var/lib/dpkg/status file then it can't produce proper dpkg database as it is now. I don't think this is the proper way of generating status file. I need to read more about chroot before making suggestions how to change it.
We need the debian way of pupulating /var/lib/dpkg/status and available. Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

Toni

[smokey01]

Geany is in /usr/bin.

Seems strange but I do not have /usr/bin/geany in puppy.sfs.
I will rebuild it again in the next days to confirm. Maybe I should use woof-next instead woof-next-james.

Toni

Toni I was building a slacko version so it may not be avaiable in the debian build.

[saintless]

Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

The building script from Emil's thread uses only apt-get. Unfortunately puppy can't use same install method and woof-ce aims to use puppy as base distro for building so the best option is lost for us.

Jamesbond,
Reading the functions in deb-build.sh am I right to assume the script actually downloads the deb packages, extracts them and update_pkg_status function generates /var/lib/dpkg/status file? If this is correct we have a big problem. This means none of the packages is configured proper and since most of the dependencies are missing in /var/lib/dpkg/status it can't be configured proper with apt-get later.

It is not very clear for me yet how the build script generates the files in /var/lib/dpkg/info directory, but the information there is also not correct. For example adduser package in puppy.sfs has only /var/lib/dpkg/info/adduser.list but installed with apt-get/dpkg it generates also adduser.conffiles, adduser.config, adduser.md5sums, adduser.postinst, adduser.postrm. The situation is the same with all packages in puppy.sfs. This means the package configure process can't be executed after first boot because all configuration files are missing in /var/lib/dpkg/info
This also could be the reason apt-get/dpkg does not populate /var/lib/dpkg/available list - because apt-get is not configured proper.

Is it possible to use in chroot from puppy real install with:

Code: Select all

dpkg -i /path/to/*.deb

At least this should generate proper status file and info directory content and the configuration process will continue proper with apt-get after first boot.

If it is not possible, I sugest testing minimal build without Synaptic, Jwm, Xorg. Only what is needed for console boot and working network, dpkg and apt-get and option to use save file. If it works better we can provide script to download/install the rest after first boot (if it is acceptable).
This is what I see as safest way to restore and keep apt-get functional as much as possible. Even if we keep the same dpkg extracting process it will be much easier to make proper status file and info directory with less packages.

And one more important thing - /etc/group must be populated with more groups used in debian. Otherwise many post install scripts from later installed packages will fail. I will do this and post example /etc/group file.

Any tips where to search to remove most of the packages and test minimal noX build? And where I can download the puppy-base, bblinks and other non-debian files included in the build as deb or pet packages?

Thank you, Smokey, I guess it is different in slacko build.

Toni

[jamesbond]

Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

The building script from Emil's thread uses only apt-get. Unfortunately puppy can't use same install method and woof-ce aims to use puppy as base distro for building so the best option is lost for us.

I haven't looked at the details, but Emil's script seems to be starting off an existing Debian setup (the "live cd" setup), while woof-next starts from nothing. In a way, deb-build.sh is more like debootstrap, and yes, I did consult debootstrap source to get ideas of how it works.

And yes, woof-next is aimed for using puppy as the "host" distro (where you do the build). I am assuming that people who wants to build puppy are puppy users themselves, those who know the virtues of puppy - thus naturally using puppy as the host. That being said, deb-build.sh *should* work on any reasonable distro, not only puppies. Anything other than should be considered as a bug and should be fixed.

Reading the functions in deb-build.sh am I right to assume the script actually downloads the deb packages, extracts them and update_pkg_status function generates /var/lib/dpkg/status file?

Correct.

If this is correct we have a big problem. This means none of the packages is configured proper and since most of the dependencies are missing in /var/lib/dpkg/status it can't be configured proper with apt-get later.

Hmm, while not all fields inside dpkg/status, I made sure that those that do, are there. And I do take care of dependencies, listing them in dpkg/status too. Perhaps not as complete as a real dpkg, but close enough.

It is not very clear for me yet how the build script generates the files in /var/lib/dpkg/info directory, but the information there is also not correct. For example adduser package in puppy.sfs has only /var/lib/dpkg/info/adduser.list but installed with apt-get/dpkg it generates also adduser.conffiles, adduser.config, adduser.md5sums, adduser.postinst, adduser.postrm.

Indeed, deb-build.sh only generates *.list files, and nothing else.

The situation is the same with all packages in puppy.sfs. This means the package configure process can't be executed after first boot because all configuration files are missing in /var/lib/dpkg/info

Indeed. My reasoning here is that most base packages do not need to have their postinst scripts.

This also could be the reason apt-get/dpkg does not populate /var/lib/dpkg/available list - because apt-get is not configured proper.

Not apt-get, but "dpkg" is not used to install the package.

Is it possible to use in chroot from puppy real install with:

Code: Select all

dpkg -i /path/to/*.deb

At least this should generate proper status file and info directory content and the configuration process will continue proper with apt-get after first boot.

Yes, but you can only do this after certain setup has been done on the rootfs (ie, installing glibc and a few other core libs/utilities). %dpkgchroot directive is what you want. Some package do need this, see for example when I installed Xorg - this needs pre-install and whatnot and thus needs the complete dpkg treatment. Simple extraction works, but it will cause problem with apt-get later, so you do have a point. I just don't use this for most packages because installing via real dpkg (chroot or not) is slow.

If it is not possible, I sugest testing minimal build without Synaptic, Jwm, Xorg. Only what is needed for console boot and working network, dpkg and apt-get and option to use save file. If it works better we can provide script to download/install the rest after first boot (if it is acceptable).

You don't need to do this, even if you think that this is the way. You can build the minimum rootfs with console tools, then chroot into it, and run whatever that you need to update the status files etc.

That being said, I'm not very fond of the idea that the system runs "pre-install" steps during first boot unless it is absolutely necessary. If it can be done at build-time, why do it later? Example: I'm running update-mime-database, gtk-query-immodules etc at build time, not at first-boot.

This is what I see as safest way to restore and keep apt-get functional as much as possible. Even if we keep the same dpkg extracting process it will be much easier to make proper status file and info directory with less packages.

Well, in my build, "apt-cache pkgnames" shows me all the packages from the repo (at the time of build). "apt-get install icewm" indeed would install icewm with its 11 library dependencies.

And one more important thing - /etc/group must be populated with more groups used in debian. Otherwise many post install scripts from later installed packages will fail. I will do this and post example /etc/group file.

Agreed, thanks.

Any tips where to search to remove most of the packages and test minimal noX build?

I don't have it anymore. You may want to look at my github commit history - the earliest version of woof-next builds a console-only debian/ubuntu puppy. Alternatively, you can look at basesfs and starts removing unncessary packages ...

And where I can download the puppy-base, bblinks and other non-debian files included in the build as deb or pet packages?

They aren't in the build, puppy-base is rootfs-skeleton, and bblinks is dynamically created from "busybox --list-full". Their status files are then manufactured, giving the appearance that they come from .deb packages; but in reality the don't.

cheers!