Alternative way to build Ubuntu / Debian Puppy [RETIRED]

[Iguleder]

Makes sense.

So ... now it's time for us to set up an Apt repository?

There's an alternative - merging Puppy-specific packages into rootfs-skeleton.

[stemsee]

For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :

What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.

[Keef]

Doesn't "localyesconfig" select the modules just for the current running system?

[Iguleder]

For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :

What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.

They're huge - a kernel package is ~50 MB with the modules. This means, the distro RAM usage starts at the uncompressed size of the kernel package.

Having all the modules in the main SFS doesn't have any downsides. A kernel with all the modules jamesbond specified should be 2-3 MB.

[stemsee]

yep, make sure everything is on! ... then after run make menuconfig and modify accordingly! Might save some time!

[gcmartin]

Questions:

• In 2014 is GTK3 all that bad for this new build tool?
• Would life be simpler for the march into the future if we embraced it now?

Everyone knows I am not a developer, merely a testor-user.

Curious

[mavrothal]

Just generated the woof-next branch in woof-CE.
Basically moved all woof building scripts to woof-code/woof2-scripts_and_files and changed merge2out not to copy over all the distro-package-...
It does not have jamesbond's changes so it can not build anything right now, but does have jamesbond as a woof-CE member

[01micko]

Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.

[mavrothal]

Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.

VFAT for sure.
f2fs if possible too.
What about booting from NTFS volumes?
Although is sacrilegious to run linux in a Mac ( ) there is a bunch of 32bit intel core duo (not core2duo) Macs that are stuck with OSX 10.6 and not supported anymore. So go for it.
I guess the other question is SCSI/IDE/SATA etc.

Another option would be to have the entire kernel in the initrd so you do not really need to wary about what is where. XOpup does that but it has really few modules.
Should not really affect boot times since this file will be loaded from somewhere and will likely decrease the kernel memory since a bunch of unused modules will not be in vmlinuz.

[Iguleder]

All we need is:
- Drivers for USB mass storage devices
- All hard drive drivers (they're tiny)
- SCSI CDROM drivers
- Drivers for all file systems we want to be able to boot from - ext2, ext3, ext4, FAT16/32, NTFS (?), XFS, reiserfs (for unlucky users who installed an old distro and kept the file system when they moved on to Puppy) and btrfs ... isn't f2fs too young?
- Squashfs and Aufs

This is the classic Puppy recipe - it should produce a 2-3 MB kernel.

I think sticking to a LTS kernel is the way to go, because pretty often, Aufs gets messed up with the latest and greatest (segfaults, in the kernel ). I've been using 3.10.x since it received its LTS status and it's simply awesome.

EDIT: 01micko - here's my kernel building script for 3.10. It's very Puppy-like, except the fact it's for Linux-libre and doesn't have Aufs. It took some time and additional gray hair to get UEFI to work (i.e, it boots but you get a black screen because it doesn't have the UEFI framebuffer module ), so it's a good reference configuration.

EDIT 2: I have a static toybox and many static daemons I wrote on my own (i.e syslogd and klogd). Functionality-wise, they should be compatible with the BusyBox ones. Maybe we could use them in the initrd instead of BusyBox. They're much lighter and smaller.

[01micko]

Thanks Iguleder - using your DOTconfig with 3.12.21 with aufs added. Just raw, as in I haven't checked if new modules are available. Compiling now.

I'll add James' firmware to a kernel-modules.sfs and see if the thing boots - more soon.

EDIT: BTW 3.12 is LTS too and what slacko64 was using before the 3.13 experimental release.

EDIT: failed to boot, initramfs problem, anyway rebuilding using fatdog-630-rc2 config ..which is what i had on hand ..should "just work"

EDIT: FD config booted but no keyboard at console for me. I honestly don't know if it worked in fatdog itself at console on this machine so I'll have to check. (LATER: checked .. yes it did work in FD-630-rc2 k3.12.6)

NEXT Morning
EDIT: depmod call in /etc/rc.d/rc.sysinit fails so that's why kernel modules didn't load. I ran depmod on the kernel modules before squashing and now it works . I should be able to mod kernel kit appropriately to produce a fatdog style kernel.

Code: Select all

Linux puppypc29976 3.12.21 #1 SMP Sun Jun 8 07:25:04 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/DISTRO_SPECS
DISTRO_NAME='ubuntu Puppy'
DISTRO_VERSION=7.0
DISTRO_BINARY_COMPAT='ubuntu'
DISTRO_FILE_PREFIX='ubuntu'
DISTRO_COMPAT_VERSION='ubuntu'
DISTRO_XORG_AUTO='yes'
DISTRO_TARGETARCH='x86'
DISTRO_DB_SUBNAME='ubuntu'
DISTRO_PUPPYSFS=puppy.sfs
DISTRO_ZDRVSFS=kernel-modules.sfs

Ignore the $DISTRO_TARGETARCH.. uname certainly wants us to know that this is x86_64, 3 times

[01micko]

I just did a fresh install of slacko-5.7 to test out kernel-kit modifications on 32 bit and the Ubuntu install using tahr bins. I have just pulled james' latest so it will have the mods he put in (so far I haven't updated since initial pull).

If all goes well I'll post kernel kit here for testing and if that goes well I'll push it to woof-CE/testing. Kernel is compiling now. (again 3.12.21). Kernel kit will have both configs. BTW, it's 4G not PAE so I can test on my old centrino lap top. That can be easy changed in the config for those who like PAE, and beauty of the FatDog kernel method is simplicity in switching kernels. Just replace vmlinuz and kernel-modules.sfs and reboot.

Mavrothal.. it might actually be time soon for a new git repo.. woof-NG

[stemsee]

This new kernel, initrd and kernel-modules.sfs combination, has to be the most significant breakthrough in Puppy Development for ages. The time I have wasted fiddling around with DISTRO-SPECS in main.sfs and initrd and swapping modules around and separating modules for initrd, and/or compressing/decompressing *.ko.gz mods, and all the other stuff was utterly unnecessary! And this method will, hopefully, prove it!

Kudos to you. Anything I can do let me know!

Edit: Any chance of ensuring all puppys build with the ability to 'xinput create-master second' , only Lighthouse64 has that ability and after research and experimentation I can shed no light no the issue. It is the one thing which gives superiority to Debian based systems in my opinion.

[01micko]

Ok, 32 bit kernel is a success and posting from it now, just added dhcpcd to the default, hacked build-iso.sh to grab the kernel from my personal repo and added firefox and we are away.

Code: Select all

# uname -a
Linux puppypc27288 3.12.21 #1 SMP Sun Jun 8 15:24:18 EST 2014 i686 athlon i686 GNU/Linux

More later.. dinner time!

[01micko]

Time for some code

Diff to build iso (grabs kernel)

Code: Select all

--- build-iso.sh.orig	2014-06-03 20:54:06.660580810 +1000
+++ build-iso.sh	2014-06-08 18:02:47.560002531 +1000
@@ -10,6 +10,7 @@ ISO_ROOT=${ISO_ROOT:-$OUTPUT_DIR/iso-roo
 PUPPY_SFS=${PUPPY_SFS:-puppy.sfs}
 KERNEL_VERSION=${KERNEL_VERSION:-3.12.9}
 PARENT_DISTRO=${PARENT_DISTRO:-ubuntu} # or debian
+KERNEL_URL=${KERNEL_URL:-http://distro.ibiblio.org/fatdog/kernels/700}
 
 WOOF_ISO_ROOT=${WOOF_ISO_ROOT:-boot}
 WOOF_INITRD=${WOOF_INITRD:-boot/initrd-tree0}
@@ -29,7 +30,7 @@ install_boot_files() {
 install_kernel() {
 	for p in vmlinuz kernel-modules.sfs; do
 		! [ -e $ISO_ROOT/$p ] &&
-		wget -P $ISO_ROOT -c http://distro.ibiblio.org/fatdog/kernels/700/$p-$KERNEL_VERSION
+		wget -P $ISO_ROOT -c $KERNEL_URL/$p-$KERNEL_VERSION
 		mv $ISO_ROOT/$p-$KERNEL_VERSION $ISO_ROOT/$p
 	done
 }
@@ -58,7 +59,7 @@ make_iso() {
 	-volid "Puppy-Linux" \
 	-iso-level 4 -D -R  \
 	-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table $ISO_ROOT/
-	isohybrid -o 64 "$OUTPUT_DIR/$OUTPUT_ISO"
+	isohybrid "$OUTPUT_DIR/$OUTPUT_ISO" #-o 64
 }
 
 ### main
# 

NB: The third hunk of the diff will not be necessary for FatDog (isohybrid part)

Apply that and run with KERNEL_URL as an arg (will work as I have uploaded that kernel)

Code: Select all

KERNEL_VERSION=3.12.21-slacko32FD4G KERNEL_URL=http://01micko.com/packages/ ./build-iso.sh

I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.

[stemsee]

I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.

Question on kernel-kit-ng

''# DO NOT set it to 0 (zero) >> cooked machine
JOBS=-j6''

Having -J0 results in cooked cpu, dual core cpu has 4 threads (-j4) kernel-kitt is set default to ''JOBS=-j6'' will that not cook a machine also unless it has 3 or more cores?

Also, ../dist/sources/patches directory missing/not created only a patches file, hence no patches to apply ... copied manually. Maybe that is how it is supposed to be ... what do I know??

Kernel compiled using new kit = 3.14.5-32FD-pae and sfs i don't have the new initrd which I imagine needs an amended init pointing to the modules.sfs so I haven't tested it. Also the DOTconfig has some significant changes, for example deselected RTC, and included additional fs, i486. Also, added firmware. The kernel gets dumped in dist/packages/ outside of the linux_kernel-*/boot directory.

kernel
https://drive.google.com/file/d/0B4GhZV ... edit?usp=1
modules.sfs
https://drive.google.com/file/d/0B4GhZV ... edit?usp=1

[mavrothal]

I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.

Looks good.
added dm_crypt in vmlinuz just in case
Aufs mm/fremap.c patching failed in 3.12
Also in line 396 the sfs sufix is in the middle
Another little issue is that will not empty or stop and warn idf the output directories are not empty so if you build 2 karnels (different conf or $FD) may end up in a mix up.

[01micko]

@stemsee

''# DO NOT set it to 0 (zero) >> cooked machine
JOBS=-j6''

Having -J0 results in cooked cpu, dual core cpu has 4 threads (-j4) kernel-kitt is set default to ''JOBS=-j6'' will that not cook a machine also unless it has 3 or more cores?

Well, I don't think jobs are the same as threads.. I should have commented it however I have been using -j6 forever on my athlon X2. Cooked machine is a possibility but more likely is the thermal cut out protection.

Also, ../dist/sources/patches directory missing/not created only a patches file, hence no patches to apply ... copied manually. Maybe that is how it is supposed to be ... what do I know??

There is intentionally only 1 patch in patches. Others are generated in the script and archived in dist/ for future reference.

Kernel compiled using new kit = 3.14.5-32FD-pae and sfs i don't have the new initrd which I imagine needs an amended init pointing to the modules.sfs so I haven't tested it. Also the DOTconfig has some significant changes, for example deselected RTC, and included additional fs, i486. Also, added firmware. The kernel gets dumped in dist/packages/ outside of the linux_kernel-*/boot directory.

There are 2 ways you can test a kernel resulting from the FD build.

• 1. build a distro using jamesbond's scripts
  2. build a 64 bit kernel and replace the FatDog64 kernel

I hope these answers help

-

@mavrothal

Aufs mm/fremap.c patching failed in 3.12

Yes I am aware. This needs to be reported to Junjiro and that's why I changed from "exit 1" to a pause and inspect routine.

Also in line 396 the sfs sufix is in the middle

Intentional to be compatible with jamesbond's build-iso.sh. As stated up the page, the diff needs to be applied so you can use the KERNEL_URL as a param. The suffix gets stripped off in the script; vmlinuz gets the same treatment.

Another little issue is that will not empty or stop and warn idf the output directories are not empty so if you build 2 karnels (different conf or $FD) may end up in a mix up.

Yes, I suppose I should add a "clean" param to the script and a note in README. I don't want to automate that, so much hand holding is ok but it's the kernel we are talking about here and if a user doesn't know what they are doing then they shouldn't be using the scripts. It doesn't check for space either

Thanks for reports.

[stemsee]

I built a debian with jamesbond's scripts 'build-deb'. However I realise that i misunderstood kernel-kit-ng... just have to recompile kernel and then insert for ./3builddistro

So to clarify. The kernel does not need to be in the ../dist/linux_kernel-*/boot directory nor does that directory need to dir2tgz-ed and tgz2pet -ed and added to local-repositories for inclusion in the build ?

[01micko]

James' script doesn't touch 3builddistro. I believe he only used the woof-CE framework so that the generic puppy scripts (including the initrd) could be imported to the build. After deb-build.sh run build-iso.sh. This pulls in the kernel from the fatdog ibiblio repo. My diff just adds an option to pull in a kernel from elsewhere and I uploaded mine to my repo to test.

To use your kernel, just comment the install_kernel line (near the bottom), add a read command directly under it, which effectively pauses the script so that you can insert your kernel (into iso/iso-root/) , continue the script and an iso will be built with your kernel. You could be more elegant than my hacky method if you want to spend the time to add the code pulling in a kernel from a local directory.