Woof-based Puppy builders wanted

[starhawk]

I wish I could help. I've got a couple ideas for some interesting (to me) Pups... but I barely know enough to troubleshoot things that I accidentally break. Knowledge of how things work is one of the big things, if not the biggest, that separates user from dev -- and by that measure, I am most definitely a user at this time...

I guess what I'm asking is, how and where can I learn the highly technical details of how Puppy functions...? I can read bash script (with a little help from Google) and I've plenty of time...

[jamesbond]

I guess what I'm asking is, how and where can I learn the highly technical details of how Puppy functions...? I can read bash script (with a little help from Google) and I've plenty of time...

Really? Ok, here we go.

---

Step 1: Read this: https://github.com/iguleder/roar-ng-ii/ ... ramfs/init. This is roar-ng-ii's /init script (Roar NG II was the minimalist distro Iguleder worked before he switched his attention to LazyUx and packdude - I wanted to show you the /init script from Lazyux but I can't find any).

It is the first program executed by a waking-up kernel (everything before that is "boot loader" and "kernel initialisation" which is from the realm beyond puppy).

This script, at 287 lines, is a lot shorter and simpler than Puppy's init script but it does more or less the same thing, if you can understand this then it would help you to better grasp puppy's /init script. It is also beautifully written and is heavily commented to help with your understanding.

---

Step 2: The last line of /init will execute /sbin/init, which is a busybox binary. /sbin/init will do actions according to what you tell it to do in /etc/inittab, so go and read https://github.com/iguleder/roar-ng-ii/ ... tc/inittab. Notice the first few lines.

---

Step 3: Ahh, so /etc/inittab instructs /sbin/init to execute /etc/rc.d/rc.sysinit as the first thing to do! May as well read what what rc.sysinit does ... https://github.com/iguleder/roar-ng-ii/ ... rc.sysinit.

---

Step 4: Finished already? Don't get the clue on what's next? Well as soon as rc.sysinit finishes, it returns back to /sbin/init, which will find stuff to do. Hint: look at those tty1, tty2 lines ... hmm, all of them seems to be running getty (="get tty", which runs login, which display the user/name prompt); but Puppy never ask me for username and password. Perhaps we should look at Puppy's *real* inittab https://github.com/puppylinux-woof-CE/w ... tc/inittab ...

Eh, what's this? Puppy's inittab for tty1 says "mingetty --autologin". That's not the same as roar-ng-ii's. Mingetty is a replacement for getty, and "--autologin" means what exactly what it is: "login automatically". So what happens after one logins? the "login" program will launch the "login shell".

"login shell" is just a normal "shell" (ie, /bin/sh) that will do certain things because the shell executed at "login time". There are other deeper meanings, but let's skimp on them at the moment.

A standard, posix "login shell" will first execute ("sources", to be exact) anything in /etc/profile, so lets look at that ... https://github.com/iguleder/roar-ng-ii/ ... tc/profile.


---

Step 5: When done, the "login shell" will also run $HOME/.profile (in this case, user is "root" so its $HOME is "/root", so lets look at roar-ng-ii's root folder ... https://github.com/iguleder/roar-ng-ii/ ... ootfs/root.

Eh! Roar-ng-ii's root folder is empty, there is no .profile !!! So what happens next?

What happens next, is, the shell becomes interactive (=ie accepts input, waiting for your command) and display the familiar "#" prompt that you love and see from terminal. You've dropped at that Linux console. This is something that you only see if you choose to "Exit X" from jwm, usually accompanied by scary messages about how to run xorgwizard if you can't get the graphical desktop started again - but be assured, there is nothing scary here.

Hang one a minute, you say. Puppy does *NOT* stop at Linux console, not unless *I* tell it to stop by giving pfix=nox as the boot parameter. You can't be right! Well, my dear, you are looking at roar-ng-ii's /root/.profile here, and indeed if you boot roar-ng-ii you will be dropped at the linux console. Perhaps then we should look at *Puppy*'s /root/.profile, here: https://github.com/puppylinux-woof-CE/w ... t/.profile, should we?

Aha !!! It is *definitely* not empty. And what it is near the end of the file? ... it runs "xwin" ... That's it, it starts the graphical desktop !

---

Step 6: What's next? There is still so much story to tell, but before that, we need to step back and re-trace our path. In most of the steps above, we have been examining roar-ng-ii's files. The order is identical, the commands that get executed is identical, the rough purpose of the scripts are identical - I hope by this stage you have a clear understanding of what every script does.

When you are ready, you should then feel brave enough to look at the real thing:
Puppy's init https://github.com/puppylinux-woof-CE/w ... tree0/init - from Step 1
Puppy's inittab https://github.com/iguleder/roar-ng-ii/ ... tc/inittab from step 2.
Puppy's rc.sysinit https://github.com/puppylinux-woof-CE/w ... rc.sysinit from step 3
Puppy's /etc/profile https://github.com/puppylinux-woof-CE/w ... tc/profile from step 4
And you you have already reviewed Puppy's /root/.profile from step 5.

And when you feel you have understood them, you will see scripts from above will call other scripts - make sure you go and read them and understand them too. Only when you've done that, you will be prepared to continue the journey to the wonders of Graphical Desktop and Puppy event system.

---

To be continued (when I'm motivated enough and have enough free time).

EDIT: typos.

[jamesbond]

Just to be clear.. I am not gone.

Not sure if this is a good thing or a bad thing
Looks like you scared everyone away

[starhawk]

...that is a lot of reading.

Let me grab some lunch and I'll get started... can't concentrate on an empty stomach!

[stemsee]

@JamesBond

That's just what I was looking for ... but didn't know how to ask?

[starhawk]

@jamesbond -- started reading that first script, and my brain melted a little.

I don't think today's a day I should be trying to parse that stuff. I've bookmarked it for later...

[jamesbond]

Mavrothal sparked the interested by posting this thread. Let me add to it: how about building puppy from tahr/utopic, *directly from ubuntu repo*, no conversion to PET required?

I have a proof-of-concept under-300-lines script that builds puppy.sfs *directly* from Ubuntu, tested with trusty and utopic (32-bit).
It uses genuine Woof-CE puppy scripts and genuine Woof-CE initrd (kernel is Fatdog's but that's because I'm lazy). So far it builds command-line only (bash, dash, tar, grep, gawk), but it's easy to extend. Boots under qemu.

Note:
1. This is proof-of-concept and *unfinished work*.
2. Only builds puppy.sfs. The initrd I used for testing is manually prepared (easy, only 3 steps - copy DISTRO_SPEC, edit DISTRO_SPEC, comment out one stupid line in /init, then usual cpio)
3. Tested with trusty and utopic, it should be easy to extend this to Debian too (any versions).
4. The only requirement is you need to have recent-enough full version of "dpkg" (and dpkg-deb) installed (easy, if you're under ubuntu-based puppy just "apt-get dpkg"; for other puppies just compile from source)

5. Script *only builds* from Debian-style packages. Don't ask for building from slackware packages, RPM packages, etc - question like that > /dev/null.
6. You still need Woof-CE as all the puppy core lives there.
7. Uses native Debian package manager. Does not use PET.

Any interest? Comments?

[stemsee]

Built entirely from .debs no pets?
Your 64 bit kernel?
So resulting sfs has full xorg/xinput etc?

[neerajkolte]

*directly from ubuntu repo*, no conversion to PET required?
Uses native Debian package manager. Does not use PET.

Seems very interesting.
For me having and maintaining pet packages repos seemed like having a parallel universe.
If it uses native debian package manager and take care of dependencies that would really be great.

Then again dosen't Debiandog do exactly that.
I haven't used it but it says so.

[jamesbond]

Built entirely from .debs no pets?

Yes.

Your 64 bit kernel?

As a stop-gap. We should use proper 32-bit kernel instead.

So resulting sfs has full xorg/xinput etc?

I have not included Xorg in the build yet, but when I do, then plan is to bring use Ubuntu's native Xorg. Thus, if Ubuntu's xorg support xinput, then this will do, too.

If it uses native debian package manager and take care of dependencies that would really be great.

Yes, it does. It uses native debian package manager (dpkg) and it does take care of dependencies. Dpkg is not the one that usually sees though, people usually use apt-get or synaptics (both are one-step higher than dpkg and uses dpkg internally). I plan to get there too.

Then again dosen't Debiandog do exactly that.
I haven't used it but it says so.

Hahaha, yeah. Many roads to Rome, eh? Debiandog takes an existing Debian LiveCD and converts it to puppy. The result is fully Debian to the core, yet feels and works like puppy. Saintless has done a good work on that.
Me, I take the Ubuntu (or Debian) packages and build them up. The result is Puppy to the core, yet will work and is fully compatible with Ubuntu/Debian (I hope )

[mavrothal]

I have a proof-of-concept under-300-lines script that builds puppy.sfs *directly* from Ubuntu, tested with trusty and utopic (32-bit).
It uses genuine Woof-CE puppy scripts and genuine Woof-CE initrd

<snip>

6. You still need Woof-CE as all the puppy core lives there.
7. Uses native Debian package manager. Does not use PET.

Any interest? Comments?

Yeh I'm interested
Puppy does use a number of pets though not in woof and characteristic to puppy (jwmconfig for example).
However this could be fixed bu adding them flat into the recently added woof-CE/woof-code/rootfs-packages directory
Does you 300-liner considers rootfs-packages and if it does does it also runs any pinstall included there?

This can be a really interesting development as rootfs-skeleton and rootfs-packages can hold what really makes puppy a Puppy and then a package manager (apt-get, yum, pacman, slaptget, packdude) can offer full distro compatibility

[saintless]

Hi, Jamesbond.

I take the Ubuntu (or Debian) packages and build them up. The result is Puppy to the core, yet will work and is fully compatible with Ubuntu/Debian (I hope )

Unfortunately fully compatible is impossible in my opinion if you keep Puppy kernel, but the result will be much more Debian/Ubuntu repository compatible system than Dpup or Lucid.
Still this is very interesting project and it worth separate development. Is it possible to upload some ready for testing resulted iso?

What I see as a problem from quick thoughts:

1. Different system structure - for example rc.local is located different in Debian and Ubuntu and .xinitrc is .xsession. I know such small things seems not important but some packages in Debian or Ubuntu repository may and most probably will depend on them.

2. Forget about packages related to kernel downloaded from Ubuntu or Debian - linux-headers for example and maybe VirtualBox and similar programs.

3. I know it is not very welcome replacement here, but still... Forget about using systemd and programs depending on systemd from Debian or Ubuntu repository. With Debian/Ubuntu kernel and initrd.img you can boot with systemd by adding this to the boot code init=/bin/systemd With Puppy kernel and initrd.gz this option is lost forever.

4. Some programs from Debian/Ubuntu repository will not work without changes for root account.

5. Dpkg reads the information from /var/lib/dpkg/info , status, available. It is very easy to break dpkg database using different package managers as option to install pet or other distro packages. It should be dpkg (+apt-get or Synaptic) only or you are looking for troubles.

Toni

[mavrothal]

As far as I know all package managers allow for non-updatable packages.
Kernel can be a problem when it comes to drivers, but ubuntu at least is using pretty recent kernels and as long as the puppy kernel is the same version and has devtmpfs=y should not have issues, even if the user decides to go with the you-do-as-I-say systemd

[jamesbond]

Puppy does use a number of pets though not in woof and characteristic to puppy (jwmconfig for example).

Yup, I'm aware of that, though I don't know what exactly are needed.

However this could be fixed bu adding them flat into the recently added woof-CE/woof-code/rootfs-packages directory

Either that, or a collection of tarballs we can simply expand upon building Or if you want to be proper, make them into DEBs

Does you 300-liner considers rootfs-packages and if it does does it also runs any pinstall included there?

Yes.

This can be a really interesting development as rootfs-skeleton and rootfs-packages can hold what really makes puppy a Puppy and then a package manager (apt-get, yum, pacman, slaptget, packdude) can offer full distro compatibility

Yes, that's the plan.

Unfortunately fully compatible is impossible in my opinion if you keep Puppy kernel, but the result will be much more Debian/Ubuntu repository compatible system than Dpup or Lucid.

Thanks Toni, it's good observation. All of your observations are valid. You are right, the word "fully" must be qualified.

Still this is very interesting project and it worth separate development.

I'm posting this here in the hope of attracting current and future Woof developers (hopefully enticing the guys who want to do Woof but currently thinks that Woof is "too hard"). If there is enough interest I will start a new thread in Cutting Edge section

Is it possible to upload some ready for testing resulted iso?

The current script produces puppy.sfs, not iso, yet. But it is just a quantum leap from puppy.sfs to iso.

1. Different system structure - for example rc.local is located different in Debian and Ubuntu and .xinitrc is .xsession. I know such small things seems not important but some packages in Debian or Ubuntu repository may and most probably will depend on them.

I haven't checked them. I thought most distros have .xinitrc and .xsession in $HOME. Puppy does, too. In any case, going forward, we can align Puppy's structure to be more "standards"-compliant, so to speak, for compatibility reasons. That being the case, there is a limit to how far we can mold Puppy before it becomes not-Puppy so your concern is valid.

2. Forget about packages related to kernel downloaded from Ubuntu or Debian - linux-headers for example and maybe VirtualBox and similar programs.

Yes, in this case, we need to provide our own DEB repository. Aligning Puppy to work with standard kernel can be done but it's so far off I wouldn't even consider it at the time being.

3. I know it is not very welcome replacement here, but still... Forget about using systemd and programs depending on systemd from Debian or Ubuntu repository. With Debian/Ubuntu kernel and initrd.img you can boot with systemd by adding this to the boot code init=/bin/systemd With Puppy kernel and initrd.gz this option is lost forever.

Correct, although, going forward, anybody's who likes systemd can propose modification to Puppy core so that it can work with systemd.

4. Some programs from Debian/Ubuntu repWhat I see as a problem from quick thoughts:ository will not work without changes for root account.

Correct. I can't remember, I think dejan or whoever was planning to enable multi-user to Puppy and has committed some code too.

5. Dpkg reads the information from /var/lib/dpkg/info , status, available. It is very easy to break dpkg database using different package managers as option to install pet or other distro packages. It should be dpkg (+apt-get or Synaptic) only or you are looking for troubles.

No other package manager, it's dpkg (and its front-ends like apt-get/synaptic onwards). If you want to install pet, use a pet2deb converter and then install it with dpkg.

I guess going forward, we need to have names for these (right now, all these independent components are all called "Woof-CE" - which is super confusing.
a) The name of the repository that holds puppy-builder code, and puppy core code ("Woof repository").
b) Puppy builder code ("Woof build system").
c) Puppy core code (rootfs-skeleton, initrd-skeleton).

Please decide on what you want to call these three, but please don't call all of them them "Woof-CE". The script I'm talking about touches on point b, the puppy builder code (with minimal modification to puppy core code). Toni's concern are mainly for Puppy core (point c).

[saintless]

As far as I know all package managers allow for non-updatable packages.

If you install not dpkg registered package it should be very carefully configured.
Make sure not to overwrite existing executables and libs.
Very important because you will change them with different version that will probably not work for the package that installed the overwritten executable or lib.
And after uninstalling the same non-updatable package you will remove the executable or libs that were installed and overwritten already. The result is missing files and dpkg can't solve the problem because this files are actually manually removed and the related program does not work anymore, even dpkg says "All is fine here".

Kernel can be a problem when it comes to drivers, but ubuntu at least is using pretty recent kernels and as long as the puppy kernel is the same version and has devtmpfs=y should not have issues, even if the user decides to go with the you-do-as-I-say systemd

No. It will not work. Tested with official Debian kernel and porteus-initrd.img Even in this case systemd boot code does not work. No way to make it work with puppy kernel + puppy inird without changes.

Toni

[saintless]

...I thought most distros have .xinitrc and .xsession in $HOME. Puppy does, too. In any case, going forward, we can align Puppy's structure to be more "standards"-compliant, so to speak, for compatibility reasons. That being the case, there is a limit to how far we can mold Puppy before it becomes not-Puppy

Yes, .xinitrc is in $HOME as .xsession. Just the name is different for puppy and debian. /etc/rc.local in debian is /etc/rc.d/rc.local in puppy. Simple symlinks could fix this differences without structure changes I think but never tested.

Aligning Puppy to work with standard kernel can be done but it's so far off I wouldn't even consider it at the time being.

It's too good to be true... I tried to make something similar but failed bad. Too complicated for me.

...I think dejan or whoever was planning to enable multi-user to Puppy and has committed some code too.

Yes, and it works quite nice as multiuser system. I think even spot can be used for such applications by running spot in terminal and then the program name. Needs testing again.

If you want to install pet, use a pet2deb converter and then install it with dpkg.

See my last post. It has to be very well configured deb to prevent troubles with dpkg. I doubt auto-convert script can cover all packages safe. It must add conflict: option to prevent overwriting other files.

Toni

[jamesbond]

As far as I know all package managers allow for non-updatable packages.

If you install not dpkg registered package it should be very carefully configured.

True. *All* packages, down to the very first one (libgcc1) is registered with dpkg, including puppy-base files (rootfs-skeleton) and dynamically created busybox symlinks.

IIRC the non-updatable package list is in apt-get, not in dpkg --- dpkg allows you to do *anything*. But my knowledge in dpkg is limited, saintless should know better.

No. It will not work. Tested with official Debian kernel and porteus-initrd.img Even in this case systemd boot code does not work. No way to make it work with puppy kernel + puppy inird without changes.

Every initrd has its own assumptions on how the kernel is compiled. With proper configuration, one can reduces this assumption - down to a standard kernel. But it's difficult - thus my earlier note that this will be far off, even if it is ever considered

See my last post. It has to be very well configured deb to prevent troubles with dpkg. I doubt auto-convert script can cover all packages safe. It must add conflict: option to prevent overwriting other files.

I didn't consider this case. Anyone who installs foreign packages run the risk of borking his/her system, as much as running a self-compiled package (just as today, anyone who installs debian package on wary-puppy (T2-based puppy) run the risk of borking his/her system. There is some amount of sanity checks we can do during pet2deb conversion process but I doubt we can (or want to) cover all bases.

cheers!

[jamesbond]

More teaser to pique interest:

Code: Select all

# ubuntu puppy pkglist
#
# generic commands: %exit %include %makesfs
# special package commands: %get_pkgs_by_priority %puppy %bblinks
# installer commands: %bootstrap %dpkg %depend %nodepend
# startup default: bootstrap, depend
#
# extra param for commands, params can be quoted
# %include         include-file
# %makesfs         output.sfs [squashfs-param]
# %get_pkg_by_prio priority ["inclusion-egrep"] ["exclusion-egrep"]
# %bblinks         [nousr]
#

%get_pkgs_by_priority "required" ".*lib.*|^tzdata|^bash|^dash|^lsb-base|^ncurses.*|bsdutils|kmod" "^klibc|.*plymouth.*"
%dpkg # switch to dpkg installer
dpkg  # install dpkg
grep 
gawk
sed
tar
mawk
gzip
cpio
mingetty

# final
busybox-static
%puppy
%bblinks # fallback for missing utilities
%makesfs puppy.sfs -comp gzip -Xcompression-level 1

This will install about 79 packages, enough to boot to shell. Works on tahr and utopic (just need to say VERSION=trusty ./deb-build.sh or VERSION=utopic ./deb-build.sh - the pkglist remains the same). By now deb-build.sh is 293 lines.

Come on, how come it's only me, mav, saintless and stemsee? Others?

[mavrothal]

However this could be fixed bu adding them flat into the recently added woof-CE/woof-code/rootfs-packages directory

Either that, or a collection of tarballs we can simply expand upon building Or if you want to be proper, make them into DEBs

Yes but this would mean another repo to maintain for each binary_compat.
Since they are mostly scripts or icons, having them flat in woof-CE also facilitates maintenance.

I guess going forward, we need to have names for these (right now, all these independent components are all called "Woof-CE" - which is super confusing.
a) The name of the repository that holds puppy-builder code, and puppy core code ("Woof repository").
b) Puppy builder code ("Woof build system").
c) Puppy core code (rootfs-skeleton, initrd-skeleton).

Please decide on what you want to call these three, but please don't call all of them them "Woof-CE". The script I'm talking about touches on point b, the puppy builder code (with minimal modification to puppy core code). Toni's concern are mainly for Puppy core (point c).

There is a major headache in this. Puppy is not standard compliant, specially when it comes to menus and file location that's why it has all these package-templates in woog-code to rework standard packages.

I think the way to go is get a woof3 branch going where, we keep woof-code/{boot,rootfs-skeleton,rootfs-packages} and maybe kernel-skeleton
Also keep kernel-kit.
Then make things in rootfs-skeleton standard-compliant and rewrite the build scripts/tweaks.
It sound much easier than it actually is

[stemsee]

LXC-create container. I haven't tried it yet, but I imagine it creates a container with it's own special code included, which presumably create a 'root-fs tree. lxc-create -n ubuntu (option) .... lxc then downloads and installs ubuntu in this container. This container might accept furnishing with woof-CE root-fs and then continue with the normally ubuntu/deb/fed/arch/ insltallation. Make some adjustments and links, then squashfs the container and boot with any kernel and initrd or load into ram or chroot (lxc = chroot on steroids.)