Alternative way to build Ubuntu / Debian Puppy [RETIRED]

[smokey01]

Toni what sort of internet connection are you working with, eth or wifi?

[saintless]

Hi, Smokey, I use wired connection (eth).
It also does not work for me without some manually settings. Running the included SNS breaks already working internet. There is missing text editor also (or I still can't find it).Creating savefile/folder on shutdown also does not work and I use already created savefile/folder before first boot. But this are small problems compared to the incorrect populated dpkg database.

I'm not sure what you can do about wifi but if you have eth connection here is how to get internet working:

1. Run Puppy with working internet connection and copy /etc/resolv.conf and /usr/bin/geany and /usr/local/sbin/dhcpcd in /mnt/home (or other location outside Puppy save file). The locations might be different . Check the exact path with:

Code: Select all

which dhcpcd
/usr/sbin/dhcpcd

2. Boot puppy.sfs (or puppy.iso) made with woof-next scripts.

3. Replace /etc/resolv.conf with the one from /mnt/home and copy dhcpcd in /usr/sbin

4. Run in terminal dhcpcd and with some luck you should have internet connection now.

Backup plan with manually settings:
Here are the commands how to setup manually IP, netmask and Default gateway from terminal. Just replace them with yours:

Code: Select all

ifconfig eth0 192.168.100.2 netmask 255.255.255.0
route add default gw 192.168.100.1 eth0

Add manually dns servers in /etc/resolv.conf using geany. Mine for example are:

Code: Select all

nameserver 192.168.100.1
nameserver 198.41.0.4

Save /etc/resolv.conf and the internet should be working.

BTW shutdown, reboot works for me this way - first click on reboot and then logout and the system reboots. First shutdown and then logout will poweroff the computer.

Toni

[smokey01]

I'm using wifi wlan0 and none of the modules appear to be loading. Even after loading what I think are the correct modules ifconfig can't find the device.

BTW geany is included and works if you run it from a terminal. The desktop file points to /usr/local/bin/defaulttexteditor which then points to beaver that's why the desktop icon doesn't work.
Geany is in /usr/bin.

[saintless]

Geany is in /usr/bin.

Seems strange but I do not have /usr/bin/geany in puppy.sfs.
I will rebuild it again in the next days to confirm. Maybe I should use woof-next instead woof-next-james.

Toni

[jamesbond]

saintless, thank you. By all means, please post your findings and bug fixes here. The idea of this is to make puppy building easier, so by sharing you're improving the overall experience for all, as opposed to if you just tinker with the finished product (puppy.sfs).

I need to say that the project has slowed down a bit, though. I'll still try to find the time to get your changes into the system and push it down to woof-next sooner or later. If you find me not responsive enough, by all means fork the code, make the necessary changes and issue me a pull request.

About that extra lines: "Replaces: libc6-i386" and "Provides: glibc-2.19-1" - is that available for all packages? Is that available in the Debian package list, or is this only available from inside the deb itself?

About this: "trying to overwrite '/usr/bin/xz', which is also in package bblinks 1.0" - perhaps its better to install xz-utils (or whatever debian package that has xz in it) rather than having bblinks creates busybox symlink for xz.

As for the content of "/var/lib/dpkg/status", IIRC this is mostly manufactured information if you don't have real dpkg (apt-get is not used during build, only dpkg). It has been a while since I played with this, so I need to check whether it is mostly manufactured or comes from dpkg. If it is manufactured it won't be complete - I only took the *minimum* necessary to get dpkg working.

In anyway, you can enable the use of real dpkg using the directive in the pkglist.

@Ibidem - thanks for the info. I have no idea whether Puppy can be fitted into Alpine Linux - it seems that a lot of the Puppy base tools are missing. But it is still an interesting idea in its own right to build a Puppy-like system using Alpine packages. The package format doesn't seem to difficult, and the fact that it has statically compiled pkgtool helps a lot.

[saintless]

Hi, jamesbond.
Respond when you have time. I jast needed to make sure this type of fixes are useful since I do not change the code. Changing woof-next code is new area for me and It will take me more time to learn it than fixing the problems in puppy.sfs I know nothing about puppy init, shutdown and save scripts and I'm not much interested learning them. My interest is learning more about how flexible debian can become.

I'll still try to find the time to get your changes into the system and push it down to woof-next sooner or later.

Do not rush to change anything yet. I still cant get apt-get to work correct and have to test more fixes first.

About that extra lines: "Replaces: libc6-i386" and "Provides: glibc-2.19-1" - is that available for all packages? Is that available in the Debian package list, or is this only available from inside the deb itself?

The information is even more complete here: http://ftp.au.debian.org/debian/dists/s ... ackages.xz but inside the deb package Control file it is almost the same. I'm sure the information added in /var/lib/dpkg/status is not from packages.xz or from deb package Control file. It should come from somewhere else.

About this: "trying to overwrite '/usr/bin/xz', which is also in package bblinks 1.0" - perhaps its better to install xz-utils (or whatever debian package that has xz in it) rather than having bblinks creates busybox symlink for xz.

Yes, I think installing the debian files is better than symlinks to busybox.

As for the content of "/var/lib/dpkg/status", IIRC this is mostly manufactured information if you don't have real dpkg (apt-get is not used during build, only dpkg). It has been a while since I played with this, so I need to check whether it is mostly manufactured or comes from dpkg. If it is manufactured it won't be complete...

I will try to find out this also.

Toni

[saintless]

OK, I think the main dpkg database problem is inside /woof-CE-woof-next-james/builders/debian-build.sh (in update_pkg_status function):

Code: Select all

update_pkg_status() {
	{
echo \
"Package: $1
Status: install ok installed
Priority: $2
Section:  $3
Maintainer: unspecified
Architecture: $ARCH
Version: $4"
[ "${5%,}" ] && echo "Depends: ${5%,}" 
echo "Description: $1 installed by deb-build.sh
"
	} >> "$CHROOT_DIR/$ADMIN_DIR/status"

}

If this is the function that updates /var/lib/dpkg/status file then it can't produce proper dpkg database as it is now. I don't think this is the proper way of generating status file. I need to read more about chroot before making suggestions how to change it.
We need the debian way of pupulating /var/lib/dpkg/status and available. Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

Toni

[smokey01]

Geany is in /usr/bin.

Seems strange but I do not have /usr/bin/geany in puppy.sfs.
I will rebuild it again in the next days to confirm. Maybe I should use woof-next instead woof-next-james.

Toni

Toni I was building a slacko version so it may not be avaiable in the debian build.

[saintless]

Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

The building script from Emil's thread uses only apt-get. Unfortunately puppy can't use same install method and woof-ce aims to use puppy as base distro for building so the best option is lost for us.

Jamesbond,
Reading the functions in deb-build.sh am I right to assume the script actually downloads the deb packages, extracts them and update_pkg_status function generates /var/lib/dpkg/status file? If this is correct we have a big problem. This means none of the packages is configured proper and since most of the dependencies are missing in /var/lib/dpkg/status it can't be configured proper with apt-get later.

It is not very clear for me yet how the build script generates the files in /var/lib/dpkg/info directory, but the information there is also not correct. For example adduser package in puppy.sfs has only /var/lib/dpkg/info/adduser.list but installed with apt-get/dpkg it generates also adduser.conffiles, adduser.config, adduser.md5sums, adduser.postinst, adduser.postrm. The situation is the same with all packages in puppy.sfs. This means the package configure process can't be executed after first boot because all configuration files are missing in /var/lib/dpkg/info
This also could be the reason apt-get/dpkg does not populate /var/lib/dpkg/available list - because apt-get is not configured proper.

Is it possible to use in chroot from puppy real install with:

Code: Select all

dpkg -i /path/to/*.deb

At least this should generate proper status file and info directory content and the configuration process will continue proper with apt-get after first boot.

If it is not possible, I sugest testing minimal build without Synaptic, Jwm, Xorg. Only what is needed for console boot and working network, dpkg and apt-get and option to use save file. If it works better we can provide script to download/install the rest after first boot (if it is acceptable).
This is what I see as safest way to restore and keep apt-get functional as much as possible. Even if we keep the same dpkg extracting process it will be much easier to make proper status file and info directory with less packages.

And one more important thing - /etc/group must be populated with more groups used in debian. Otherwise many post install scripts from later installed packages will fail. I will do this and post example /etc/group file.

Any tips where to search to remove most of the packages and test minimal noX build? And where I can download the puppy-base, bblinks and other non-debian files included in the build as deb or pet packages?

Thank you, Smokey, I guess it is different in slacko build.

Toni

[jamesbond]

Maybe something from here will help to fix the problem for debian build:
http://www.murga-linux.com/puppy/viewtopic.php?t=91832

The building script from Emil's thread uses only apt-get. Unfortunately puppy can't use same install method and woof-ce aims to use puppy as base distro for building so the best option is lost for us.

I haven't looked at the details, but Emil's script seems to be starting off an existing Debian setup (the "live cd" setup), while woof-next starts from nothing. In a way, deb-build.sh is more like debootstrap, and yes, I did consult debootstrap source to get ideas of how it works.

And yes, woof-next is aimed for using puppy as the "host" distro (where you do the build). I am assuming that people who wants to build puppy are puppy users themselves, those who know the virtues of puppy - thus naturally using puppy as the host. That being said, deb-build.sh *should* work on any reasonable distro, not only puppies. Anything other than should be considered as a bug and should be fixed.

Reading the functions in deb-build.sh am I right to assume the script actually downloads the deb packages, extracts them and update_pkg_status function generates /var/lib/dpkg/status file?

Correct.

If this is correct we have a big problem. This means none of the packages is configured proper and since most of the dependencies are missing in /var/lib/dpkg/status it can't be configured proper with apt-get later.

Hmm, while not all fields inside dpkg/status, I made sure that those that do, are there. And I do take care of dependencies, listing them in dpkg/status too. Perhaps not as complete as a real dpkg, but close enough.

It is not very clear for me yet how the build script generates the files in /var/lib/dpkg/info directory, but the information there is also not correct. For example adduser package in puppy.sfs has only /var/lib/dpkg/info/adduser.list but installed with apt-get/dpkg it generates also adduser.conffiles, adduser.config, adduser.md5sums, adduser.postinst, adduser.postrm.

Indeed, deb-build.sh only generates *.list files, and nothing else.

The situation is the same with all packages in puppy.sfs. This means the package configure process can't be executed after first boot because all configuration files are missing in /var/lib/dpkg/info

Indeed. My reasoning here is that most base packages do not need to have their postinst scripts.

This also could be the reason apt-get/dpkg does not populate /var/lib/dpkg/available list - because apt-get is not configured proper.

Not apt-get, but "dpkg" is not used to install the package.

Is it possible to use in chroot from puppy real install with:

Code: Select all

dpkg -i /path/to/*.deb

At least this should generate proper status file and info directory content and the configuration process will continue proper with apt-get after first boot.

Yes, but you can only do this after certain setup has been done on the rootfs (ie, installing glibc and a few other core libs/utilities). %dpkgchroot directive is what you want. Some package do need this, see for example when I installed Xorg - this needs pre-install and whatnot and thus needs the complete dpkg treatment. Simple extraction works, but it will cause problem with apt-get later, so you do have a point. I just don't use this for most packages because installing via real dpkg (chroot or not) is slow.

If it is not possible, I sugest testing minimal build without Synaptic, Jwm, Xorg. Only what is needed for console boot and working network, dpkg and apt-get and option to use save file. If it works better we can provide script to download/install the rest after first boot (if it is acceptable).

You don't need to do this, even if you think that this is the way. You can build the minimum rootfs with console tools, then chroot into it, and run whatever that you need to update the status files etc.

That being said, I'm not very fond of the idea that the system runs "pre-install" steps during first boot unless it is absolutely necessary. If it can be done at build-time, why do it later? Example: I'm running update-mime-database, gtk-query-immodules etc at build time, not at first-boot.

This is what I see as safest way to restore and keep apt-get functional as much as possible. Even if we keep the same dpkg extracting process it will be much easier to make proper status file and info directory with less packages.

Well, in my build, "apt-cache pkgnames" shows me all the packages from the repo (at the time of build). "apt-get install icewm" indeed would install icewm with its 11 library dependencies.

And one more important thing - /etc/group must be populated with more groups used in debian. Otherwise many post install scripts from later installed packages will fail. I will do this and post example /etc/group file.

Agreed, thanks.

Any tips where to search to remove most of the packages and test minimal noX build?

I don't have it anymore. You may want to look at my github commit history - the earliest version of woof-next builds a console-only debian/ubuntu puppy. Alternatively, you can look at basesfs and starts removing unncessary packages ...

And where I can download the puppy-base, bblinks and other non-debian files included in the build as deb or pet packages?

They aren't in the build, puppy-base is rootfs-skeleton, and bblinks is dynamically created from "busybox --list-full". Their status files are then manufactured, giving the appearance that they come from .deb packages; but in reality the don't.

cheers!

[anikin]

The building script from Emil's thread uses only apt-get. Unfortunately puppy can't use same install method and woof-ce aims to use puppy as base distro for building so the best option is lost for us.

I haven't looked at the details, but Emil's script seems to be starting off an existing Debian setup (the "live cd" setup), while woof-next starts from nothing. In a way, deb-build.sh is more like debootstrap, and yes, I did consult debootstrap source to get ideas of how it works.

You won't see much looking at the details. What matters is the bigger picture behind Emil's project. The engine, that drives it is the Debian Live-Build tool.
>> live-build is a set of scripts to build Debian Live system images. The idea behind live-build is a tool suite that uses a configuration directory to completely automate and customize all aspects of building a Live image <<
Emil's project is just a configuration directory (within live-build), that he has setup to build his variant of Debian Wheezy. You can reconfigure it any way you wish. Build a sid/jessy/trusty/utopic/precise variant plus many more. Customize anything you want - make it minimal, chubby, bloated, etc. And most important, Live-Build starts off from nothing too. Or to be more exact, from debootstrap and ends up in chroot - nothing more, nothing less. It can be installed to and run off of any distro. It does not start off an existing Debian setup. Regarding Emil's building script, that uses only apt-get - when you're following the process in the terminal, you will see, dpkg is doing its job too. Once you get a taste of live-build, you will never want to use or invent anything else. It makes Woof and friends look a clumsy and useless contraption.

[jamesbond]

It makes Woof and friends look a clumsy and useless contraption.

Then why are you posting here? This is a Woof-thread, if you don't like it because it is useless and clumsy, please post else where.

[anikin]

Why are you asking?
I'm not a Puppy basher, but that's how I feel comparing the two build systems.
And the point of my post was to clarify, that Emil's project starts from scratch, not off an existing live CD.

edit: jamesbond,
"clumsy and useless" *absolutely* do not apply to your or anyone's work - let me set it straight. The words should be read in a very narrow context of the above comparison - the two build systems.

[smokey01]

Anikin, will emil's system build other than debian systems. Currently woof-next can manage 4 different distros. Have you built using both systems.
I'm only asking to try and ascertain if one system is really better than the other and you are comparing apples with apples.
Thanks

[anikin]

Smokey,

I didn't even compare woof-next to anything, let's exclude it.
By woof I meant the original one and its current fork. Again, Emil's project is based on an official Debian/Ubuntu build system, which, I feel deserves the attention of Puppy devs. That was the only reason I posted here.

[smokey01]

I must admit I'm not too keen on the original woof as I find it difficult to drive and it can take seven hours to build a distro.
Woof-next on the other hand is easy to drive and only takes about 10 - 15 minutes to build a distro that will boot to a desktop.
When I get a chance I will have a look at emil's work.
Cheers

[saintless]

Hi, Jamesbond.

And yes, woof-next is aimed for using puppy as the "host" distro (where you do the build). I am assuming that people who wants to build puppy are puppy users themselves, those who know the virtues of puppy - thus naturally using puppy as the host. That being said, deb-build.sh *should* work on any reasonable distro, not only puppies. Anything other than should be considered as a bug and should be fixed.

I understand. I do not use puppy as host distro and do not know much about puppy structure and scripts but hope my testing will help you to improve debian sid build process. Still think dpkg database is not correctly builded from start and this will create problems for the user later. I will post some fixes soon.

Toni

[saintless]

Fixes for Woof-Next-James-Debian-Sid-Build:

1. Added in /usr/bin files from puppy:
dhcpcd
geany

Added deb packages:
gettext-base
libasprintf0c2
libcap2
startpar

Added /usr/local/bin/rox script to start file (home) desktop icon.

2. Remove /var/lib/dpkg/info/bblinks.list and "Package: bblinks" section from /var/lib/dpkg/status
Puppy BusyBox links are still in the system but will be replaced with debian files when it is needed without apt-get errors.

3. Added in /etc/group:

Code: Select all

sys:x:3:
adm:x:4:
mail:x:8:
news:x:9:
man:x:12:
proxy:x:13:
fax:x:21:
voice:x:22:
sudo:x:27:
www-data:x:33:
backup:x:34:
operator:x:37:
list:x:38:
irc:x:39:
src:x:40:
gnats:x:41:
shadow:x:42:
utmp:x:43:
sasl:x:45:
staff:x:50:
games:x:60:
nogroup:x:65534:
libuuid:x:101:
crontab:x:102:
scanner:x:104:saned
colord:x:106:
ssl-cert:x:108:
Debian-exim:x:109:
mlocate:x:110:
bluetooth:x:114:
saned:x:116:
utempter:x:103:
fuse:x:111:
ntp:x:112:

4. added in /etc/gshadow:

Code: Select all

kmem:*::
dialout:*::
fax:*::
voice:*::
cdrom:*::
floppy:*::
tape:*::
sudo:*::
audio:*::
www-data:*::
backup:*::
operator:*::
list:*::
irc:*::
src:*::
gnats:*::
shadow:*::
utmp:*::
video:*::
sasl:*::
plugdev:*::
staff:*::
games:*::
libuuid:!::
crontab:!::
scanner:!::
messagebus:!::
colord:!::
lpadmin:!::
ssl-cert:!::
Debian-exim:!::
mlocate:!::
netdev:!::
bluetooth:!::
saned:!::
utempter:!::
fuse:!::
ntp:!::

5. Changed section in /var/lib/dpkg/status from:

Code: Select all

Package: libc6
Status: hold ok installed
Priority: required
Section: libs
Maintainer: unspecified
Architecture: i386
Version: 2.19-10
Depends: libgcc1
Description: libc6 installed by deb-build.sh

to:

Code: Select all

Package: libc6
Status: hold ok installed
Priority: required
Section: libs
Maintainer: unspecified
Architecture: i386
Version: 2.19-10
Replaces: libc6-i386
Provides: glibc-2.19-1
Depends: libgcc1
Description: libc6 installed by deb-build.sh

If I find something more I will post it again.

Toni

[emil]

When I get a chance I will have a look at emil's work.

Currently the project is available on github
https://github.com/emilwidmann/Debian-light-builder

I removed it from the boxen.math server, so the download links in my old posts don't work at the moment. Eventually I can reupload it.

I want to add that I just created a tiny part of these scripts. Most work is done by the debian live-builder. Thiery Monteil wrote a creative module system to configure and run those scripts. I just wrote a special module which tries to create a simple, puppy like system.

I would like that my project is not compared to the work of others (like jamesbonds scripts) in a judgemential (better/worse) way. I think all those different projects share a similar goal and come from a special need to improve on some deep founded weakness of puppy, while at the same time keeping its strengths and simplicity - generally all traits why we love the Puppy Linux system and care for it.

Saintless and the team of DebianDog* proved, that a puppy like debian system is possible. There was lot of insight and experience gained. Possibly I return to my build scripts, and anbody who wants to join to inject some good portions of puppy linux is very welcome.

with respect and kind regards
emil

* aka: Saintless and the DebianDogs (TM)

[jamesbond]

This project is now retired.

The purpose of this project is to bring fun back to puppy building (along with exploring different ideas); during the time that woof-CE development seemed stagnant.

Woof-CE has just seen a resurgence of activity; and a lot of things have been improved. Maintaining a separate branch of Woof-CE (and trying to sync between them) by myself is not something that I can do concurrently with my other projects (hobby and paid).

Therefore I have decided to retire this project. People should focus on the official Woof-CE (the "testing" branch) and contributes there.

I have just done one final update (updating the repo-url) so that it should still work; but remember you are on your own. Please feel free to experiment with it, fork it, improve it etc.

I may revert my decision and re-activate this if I get enough feedback and contributions - but I don't hold my breath. Until then, this is it.

So long and thanks for all the fish.

====

PS: The build system which is the core of Woof-CE NG (which is mine) will live along elsewhere in another project called "Fatdog-Like".

"Fatdog-Like" is a derivative-based Fatdog. One can build Ubuntu-based Fatdog (=Fatdog-Like Ubuntu), Fatdog-Like Slackware, etc. It's still in the early stage and is not public yet, but I can already boot 32-bit barebones Fatdog-Like Tahr under BIOS and UEFI to desktop with jwm

Fatdog-Like is a project that I have been pondering for a long time. As soon as I started this, I was thinking about it - but I didn't do it because I don't want people (and myself) to get distracted.

Now that this Woof-CE NG is officialy retired, I can pursue this project with leisure.