alternative puppy build system

[bigpup]

1

[wanderer]

hi s243a

i will read your stuff
and try to follow it
and report back

i will not use devuan base
because it is too big

i will use tinycore as the base
because it has been minimized

wanderer

[wanderer]

hi s243a

i have been thinking
this is our opportunity to improve the system

why perpetuate mistakes
we dont owe any allegiance to poor design

the boot system of puppy is a monstrosity
why load one operating system
just to have it load another similar operating system
but have it still hang around
in some sort of bizarre tree structure

the only unique thing in the puppy initrd file
is the init file
and that is just a series of commands

i will use the tinycore boot system
which is superior

also another monstrosity
is the propensity of puppy to mix everything together
so that components cannot be undone or even identified easily

i will also use the tinycore method of modularizing components
but will turn it into the symlinked app directory system

the other part
is the layered file system
and the symlinked file system
which can coexist just fine together

boot process and filesystems
will be set up by the combined boot chain tc-configure and init files

so this will be the organization of this project

1. puppy and tinycore merged base

2. puppy layer - pets -> sfs

3. devuan layer - debs -> sfs apt-get ?

this should actually be more manageable
than trying to work around puppy's design flaws
and make devuan a small and simple system

so the woof-next system
will just be designed to put these things together
from their repositories
based on a template

basefs and devx

which i think should be combined into one file

anyway since much of this is already done
we can start with tinycore 9
and make it into a puppy tinycore devuan hybrid

what do you think s243a

wanderer

[musher0]

1

Speak for yourself!

[zagreb999]

suggestion

boot option as fluppy 013-
it is very fast!

based on debian with lz4 support
and remaster tool

synaptic pm

as fred's debiandog

regards

[s243a]

hi s243a

i have been thinking
this is our opportunity to improve the system

why perpetuate mistakes
we dont owe any allegiance to poor design

the boot system of puppy is a monstrosity
why load one operating system
just to have it load another similar operating system
but have it still hang around
in some sort of bizarre tree structure

the only unique thing in the puppy initrd file
is the init file
and that is just a series of commands

i will use the tinycore boot system
which is superior

You haven't sold me on this option. If said system can't support standard puppy layered file system (and init) then I suspect there will be less interest in it.

However, if it can also support other types of init (e.g. tinycore and fatdog) then I think there will be a larger interest. I think it can support more. Jamesbond has just built a fatdog devaun hybird using this system. See post:
http://murga-linux.com/puppy/viewtopic. ... 51#1028251

Fatdog like tinycore uses an initrd that is a full system. Therefore if we can build a fatdog like system with woof-next maybe we can also build a tinycore like system with woof-next.

also another monstrosity
is the propensity of puppy to mix everything together
so that components cannot be undone or even identified easily

i will also use the tinycore method of modularizing components
but will turn it into the symlinked app directory system

The tinycore core.gz suffers from the same problem as the rootfs-skeleton of woofCE in that everything is mixed together.

the other part
is the layered file system
and the symlinked file system
which can coexist just fine together

boot process and filesystems
will be set up by the combined boot chain tc-configure and init files

so this will be the organization of this project

1. puppy and tinycore merged base

2. puppy layer - pets -> sfs

3. devuan layer - debs -> sfs apt-get ?

I see two approaches here. If we use devuan's package manager we will have to seperate the layers out after the fact using something like I created in the thread, "Stripping Down a Puppy". Alternatively we can use sc0ttman's package manager (i.e. PKG):
http://murga-linux.com/puppy/viewtopic.php?t=112927

under the premise that we will need much fewer dependencies this way.

In the prior approach "Stripping after the fact" we'll have to repatch the merged base because the devuan package manager might over-write both the the symlinks to busybox and scripts that stand in as a minimal version of some linux utility.

this should actually be more manageable
than trying to work around puppy's design flaws
and make devuan a small and simple system

This might not be the best thread for "marketing tinycore". I think we should instead focus on solutions here.

so the woof-next system
will just be designed to put these things together
from their repositories
based on a template

basefs and devx

which i think should be combined into one file

anyway since much of this is already done
we can start with tinycore 9
and make it into a puppy tinycore devuan hybrid

what do you think s243a

wanderer

[s243a]

I don't think it was mentioned yet so I'll post it.

Here is the command that I think one uses to pull the woof-next branch from github:

Code: Select all

git clone https://github.com/puppylinux-woof-CE/woof-CE.git --branch woof-next

https://stackoverflow.com/questions/323 ... ub-project

Edit: The above command seemed to work. Now I'll outline some steps I'll try (will update this post as I go)
1. copy /woof-distro/x86/devuan and rename it tiny_devuan
2. replace /woof-distro/x86/tiny_devuan/basesfs with the file at:
https://pastebin.com/chn1TMSB (edits in progress for this file)
3. Download sc0ttman's package manager:
https://gitlab.com/sc0ttj/Pkg/-/archive ... ter.tar.gz
4. Extract scottman's package manager and copy the contents to:
woof-code/rootfs-packages/PKG
**Maybe we should give this folder a more descriptive name than PKG?

related to this I added "%addpkg PKG" after adding the tinycore_base_sfs, we'll also need to patch in some puppy stuff here. For now on lines 36 & 37 we have:

Code: Select all

%addpkg tinycore_base_sfs
%addpkg PKG

5. In woof-code/rootfs-packages/PKG rename installer.sh to pinstall.sh
6. more steps to come... (post being edited.

[s243a]

wanderer, did you want to use the core.gz from the tinycore 9 iso as the base or do you have a merged one that might be better with stuff from puppies roots-skeleton?

Edit 1 I have an idea here. We can take core.gz from the tinycore iso and patch it with some essential puppyfiles that mistfire identified in her build kit.

[jamesbond]

Here is the command that I think one uses to pull the woof-next branch from github:

Code: Select all

git clone https://github.com/puppylinux-woof-CE/woof-CE.git --branch woof-next

Or you can download the latest copy from here: https://github.com/puppylinux-woof-CE/w ... ext.tar.gz (I gave this a few posts back, but I thought I'd just re-posted it for people who are allergic to git).

[s243a]

Here is the command that I think one uses to pull the woof-next branch from github:

Code: Select all

git clone https://github.com/puppylinux-woof-CE/woof-CE.git --branch woof-next

Or you can download the latest copy from here: https://github.com/puppylinux-woof-CE/w ... ext.tar.gz (I gave this a few posts back, but I thought I'd just re-posted it for people who are allergic to git).

You did but I thought there might be some advantages to using the "git" command if one wants to make changes and publish the code back to github. Perhaps there aren't. Reposting it is good though, in case for some reason someone doesn't like using the "git" command.

[s243a]

I don't think it was mentioned yet so I'll post it.
....

Edit: The above command seemed to work. Now I'll outline some steps I'll try (will update this post as I go)
1. copy /woof-distro/x86/devuan and rename it tiny_devuan
2. replace /woof-distro/x86/tiny_devuan/basesfs with the file at:
https://pastebin.com/chn1TMSB (edits in progress for this file)
3. Download sc0ttman's package manager:
https://gitlab.com/sc0ttj/Pkg/-/archive ... ter.tar.gz
4. Extract scottman's package manager and copy the contents to:
woof-code/rootfs-packages/PKG
**Maybe we should give this folder a more descriptive name than PKG?

related to this I added "%addpkg PKG" after adding the tinycore_base_sfs, we'll also need to patch in some puppy stuff here. For now on lines 36 & 37 we have:

Code: Select all

%addpkg tinycore_base_sfs
%addpkg PKG

5. In woof-code/rootfs-packages/PKG rename installer.sh to pinstall.sh
6. more steps to come... (post being edited.

Regarding patching tinycore's core.gz, I'll use the files in tazpup64-core-files-190515.tar.gz (older version, related to thread) as a starting point.

This is a modified version of the folder tazpup-core-files from mistife's TazPup buildkit. Anyway within, tazpup64-core-files-190515.tar.gz there are some folders that we will make into packaes to be installed after the tinycore core.gz to patch some puppy stuff into it.

The first folder that we will make into a package is:
/tazpup-core-files/puppy
**Note later on I'll pull these files from WoofCE but for now I'll take it from my buildkit.
**I'll call this package "puppycore_noarch"

The other folder that we will need made into a package is:
/tazpup-core-files/build/i386

This is architecture specific stuff such as mount-full. This could be in theory extracted from the related package for a specific distribution but if they were made into static files then we don't have to worry about this. For tinycore 9 the related files from dpup strech should work. I'll replace the files from those in dpup strech and call it puppycore_i386_strech.

[jamesbond]

wanderer asked of what is the package list for CLI-based system. Obivously, the answer depends on what CLI tools you need, but here is an example of pkglist that will give you Devuan Ascii (or Debian Stretch) minimal CLI with apt-get capability.

Code: Select all

# minimal CLI-only pkglist for basesfs with apt-get support

# essential packages
%pkgs_by_priority "required" ".*lib.*|^tzdata|^bash|^dash|^lsb-base|^ncurses.*|^bsdutils|^kmod|^mount|^insserv|^mount|^sysvinit-utils|^procps|^makedev" "^klibc|.*plymouth.*|mountall"
%depend

coreutils
grep 
gawk
mawk
sed
tar
gzip
cpio
dialog
gettext
unzip
bzip
xz-utils

diffutils
util-linux 
e2fsprogs 
findutils 

debianutils
multiarch-support
sensible-utils
libdb5.3

apt

module-init-tools
eudev
netbase
mime-support

# enable proper support for package signing
%dpkgchroot
%reinstall devuan-keyring apt
%bootstrap

# final steps
# remove extremely toxic packages, then setup the dummy
%remove initscripts ifupdown sysv-rc upstart mountall login
%dummy  initscripts ifupdown sysv-rc upstart mountall login

# these useless packages got pulled by apt-get -f install, so prevent it from getting installed
%rm /sbin/init # don't use systemd-init, use busybox one instead
%remove plymouth libplymouth2 plymouth-theme-ubuntu-text 
%dummy  plymouth libplymouth2 plymouth-theme-ubuntu-text 
%remove busybox-initramfs initramfs-tools-bin klibc-utils initramfs-tools
%dummy  busybox-initramfs initramfs-tools-bin klibc-utils initramfs-tools
%dummy  adduser base-files

# install busybox and its symlinks, fallback for missing utilities
busybox-static
%bblinks

# install puppy-base - MUST BE LAST - unless overriding puppy-base
%dpkg_configure --force-all -a
%addbase
%addpkg debian-setup # specific debian setup, overriding puppy-base
%lock puppy-base puppy-base-arch libc6 # example: never update puppy base and libc6

# cutdown the size
%cutdown all # maximum cutdown

# make the sfs (optional)
%makesfs iso/iso-root/puppy.sfs -comp gzip # -Xcompression-level 1

Copy and paste the code above and use it to replace the content of "basesfs".

The basesfs above is for Devuan Ascii, if you want to use Debian Stretch as the base, replace "devuan-keyring" with "debian-keyring", and replace "eudev" with "udev".

It only supports wired network as I don't install wireless-tools, wpasupplicant, etc. I used xenial-4.4.95 kernel for testing.

Here is how to test it in qemu (which you can do just by running ./runqemu in the workdir): Boot it, and after you answer the usual puppy startup questions, you will be dropped in the linux console.
Then type the following commands, one line at at time.

Code: Select all

ifconfig eth0 up
udhcpc -qi eth0
echo nameserver 8.8.8.8 > /etc/resolv.conf
apt-get update
apt-get install mc
mc

The ISO size for this is about 148MB for Devuan but notice that 81 MB of that is the kernel + modules + firmware. The size of the puppy.sfs itself is only 66MB (and this is using gzip compression, if you use xz compression it will be smaller). It is 180MB if you use Debian Stretch as the base.

Can this still be minimised further? Yes. Instead of using automatic dependency pulling, you can specify the exact package you want to install, one by one. But for me, it's not worth my time doing that. (EDIT: the pkglist above will pull about 118 packages).

----

More edit: to make devx, you just need to put this into "devx" pkglist:

Code: Select all

### Development environment
%import devx-holder # import headers and libs output by "%cutdown dev"
%depend
build-essential
%makesfs iso/devx.sfs -comp gzip

And then run "./build-sfs.sh devx"

In fact, you can use the same methods to build any other SFS. Just list the package list in a file somewhere (say, "mysfs"), and then run "./build-sfs.sh mysfs". Note the last command %makesfs to actually make the SFS, and where the output should go.

If you need symlink, then the builders understand the %symlink command.

---

EDIT: This is probably my final contribution to this thread regarding woof-next. I think I've given enough examples. I will still entertain questions from time to time but I know that s243a would be able to answer most of them too, so I probably won't show up unless there nobody can answer it.

And, I certainly hope that this thread will never get deleted as I have put a lot of effort to write all the stuff I wrote here.

Good luck everyone.

[wanderer]

hi s243a and everyone

the idea of this project
is to be able to combine stuff
from every distro

it is made up of 6 components

1. the bootloader syslinux

2. the kernel

3. the initrd core.gz - console only

4. the cde folder basic x and x-apps xterm gtkedit emelfm dillo

additional apps

5. puppy.sfs file(s) made with devuan and puppy parts

6. other sfs files made from other distros-sources

since the files will be sfs files
they can be mounted either by symlinks or a layered file system
in the same iso


the first step is to combine
the puppy initrd
and the tinycore core.gz

and the 2 files tc-config and puppy init
which are equivalent

puppy init is very complex
and the only thing we will really need from it
is the layered file system code
so we will use tc-config
and add puppy stuff to it
i have already added a bootcode to tc-config
as an exercise

to make things manageable
we will start with a working system
that is already organized this way - tinycore 9

also in the future we may want to look at dcore
since it can build combined app sfs files from debian
and steal its code


woof-next will need to

read the template basefs and devx

get the bootloader

get the kernel

get the core.gz

get the cde folder

get the debs and pets and make the the puppy-devuan.sfs file(s)

get the tcz or pets or debs and make the other distro-souces sfs file(s)

make the iso


it already does a lot of this
and a lot of stuff is already done

our first iso
will be a modified core

our second iso
will be a modified core with cde folder

our third iso
will be a modified core cde folder and puppy.sfs

and then on into infinity

what say you all

wanderer

[wanderer]

addendum

i am in the process of making these isos as we speak

wanderer

[s243a]

I don't think it was mentioned yet so I'll post it.
....

Edit: The above command seemed to work. Now I'll outline some steps I'll try (will update this post as I go)
1. copy /woof-distro/x86/devuan and rename it tiny_devuan
2. replace /woof-distro/x86/tiny_devuan/basesfs with the file at:
https://pastebin.com/chn1TMSB (edits in progress for this file)
3. Download sc0ttman's package manager:
https://gitlab.com/sc0ttj/Pkg/-/archive ... ter.tar.gz
4. Extract scottman's package manager and copy the contents to:
woof-code/rootfs-packages/PKG
**Maybe we should give this folder a more descriptive name than PKG?

related to this I added "%addpkg PKG" after adding the tinycore_base_sfs, we'll also need to patch in some puppy stuff here. For now on lines 36 & 37 we have:

Code: Select all

%addpkg tinycore_base_sfs
%addpkg PKG

5. In woof-code/rootfs-packages/PKG rename installer.sh to pinstall.sh
6. more steps to come... (post being edited.

Regarding patching tinycore's core.gz, I'll use the files in tazpup64-core-files-190515.tar.gz (older version, related to thread) as a starting point.

This is a modified version of the folder tazpup-core-files from mistife's TazPup buildkit. Anyway within, tazpup64-core-files-190515.tar.gz there are some folders that we will make into packaes to be installed after the tinycore core.gz to patch some puppy stuff into it.

The first folder that we will make into a package is:
/tazpup-core-files/puppy
**Note later on I'll pull these files from WoofCE but for now I'll take it from my buildkit.
**I'll call this package "puppycore_noarch"

The other folder that we will need made into a package is:
/tazpup-core-files/build/i386

This is architecture specific stuff such as mount-full. This could be in theory extracted from the related package for a specific distribution but if they were made into static files then we don't have to worry about this. For tinycore 9 the related files from dpup strech should work. I'll replace the files from those in dpup strech and call it puppycore_i386_strech.

I put this stuff on github (it was a bit of a headache to change the remote origin). Also I forked all of woof-CE when I only wanted to fork only woof-next. The intention of my for was only for extermination and not to replaces jamesbond's branch. Anyway, My update files are:
/woof-next/woof-distro/x86/tiny_devuan/ascii/basesfs
/woof-CE/tree/woof-next/woof-code/rootfs-packages/puppycore_i386_strech
/woof-CE/tree/woof-next/woof-code/rootfs-packages/puppycore_noarch
/woof-code/rootfs-packages/tinycore_base_gz
/woof-code/rootfs-packages/PKG

I haven't tested this yet. I'll test it now. Woof-next isn't my main focus. I'm just trying to give wander some ideas on how one might use Woof-next to utilize tinycore concepts.

Also note that currently I have procesps inside puppycore_i386_strech. I plan to make this it's own package but first I want to figure out some kind of logic to see if the native version (to the compatible distribution) is installed before installing this package.

[s243a]

I put this stuff on github (it was a bit of a headache to change the remote origin).

Let me see if I can recall the steps (based on my bash history).

Code: Select all

git remote set-url origin git@github.com:s243a/woof-CE.git #You need to do this if you pulled the code before forking
git config --global user.email "sombody@somewhere.com"
git config --global user.name s243a
git add .
git commit -m "s243a's third commit"
git pull #Need to do this only if git changed since your last pull. This will be the case if you fork via the web interface but never pulled. 
git push -u origin woof-next

Some background info:
https://help.github.com/en/articles/cha ... emotes-url
https://stackoverflow.com/questions/925 ... -on-github
https://stackoverflow.com/questions/192 ... 7_19216491
https://help.github.com/en/articles/fork-a-repo

Not mentioned above was how to create the github keys. This is done as follows:
1. install openssh-client then

Code: Select all

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
eval "$(ssh-agent)" #Starts the SSH Agent
ssh-add ~/.ssh/id_rsa #Add the key to the ssh agent

https://www.pearltrees.com/s243a/ssh-github/id16286594
**Note that you must copy id_rsa.pup to github. There are several ways to do this.
https://www.pearltrees.com/s243a/add-ne ... id16286671

[s243a]

I don't think it was mentioned yet so I'll post it.
....

Edit: The above command seemed to work. Now I'll outline some steps I'll try (will update this post as I go)
1. copy /woof-distro/x86/devuan and rename it tiny_devuan
2. replace /woof-distro/x86/tiny_devuan/basesfs with the file at:
https://pastebin.com/chn1TMSB (edits in progress for this file)
3. Download sc0ttman's package manager:
https://gitlab.com/sc0ttj/Pkg/-/archive ... ter.tar.gz
4. Extract scottman's package manager and copy the contents to:
woof-code/rootfs-packages/PKG
**Maybe we should give this folder a more descriptive name than PKG?

related to this I added "%addpkg PKG" after adding the tinycore_base_sfs, we'll also need to patch in some puppy stuff here. For now on lines 36 & 37 we have:

Code: Select all

%addpkg tinycore_base_sfs
%addpkg PKG

5. In woof-code/rootfs-packages/PKG rename installer.sh to pinstall.sh
6. more steps to come... (post being edited.

Regarding patching tinycore's core.gz, I'll use the files in tazpup64-core-files-190515.tar.gz (older version, related to thread) as a starting point.

This is a modified version of the folder tazpup-core-files from mistife's TazPup buildkit. Anyway within, tazpup64-core-files-190515.tar.gz there are some folders that we will make into packaes to be installed after the tinycore core.gz to patch some puppy stuff into it.

The first folder that we will make into a package is:
/tazpup-core-files/puppy
**Note later on I'll pull these files from WoofCE but for now I'll take it from my buildkit.
**I'll call this package "puppycore_noarch"

The other folder that we will need made into a package is:
/tazpup-core-files/build/i386

This is architecture specific stuff such as mount-full. This could be in theory extracted from the related package for a specific distribution but if they were made into static files then we don't have to worry about this. For tinycore 9 the related files from dpup strech should work. I'll replace the files from those in dpup strech and call it puppycore_i386_strech.

I put this stuff on github (it was a bit of a headache to change the remote origin). Also I forked all of woof-CE when I only wanted to fork only woof-next. The intention of my for was only for extermination and not to replaces jamesbond's branch. Anyway, My update files are:
/woof-next/woof-distro/x86/tiny_devuan/ascii/basesfs
/woof-CE/tree/woof-next/woof-code/rootfs-packages/puppycore_i386_strech
/woof-CE/tree/woof-next/woof-code/rootfs-packages/puppycore_noarch
/woof-code/rootfs-packages/tinycore_base_gz
/woof-code/rootfs-packages/PKG

I haven't tested this yet. I'll test it now. Woof-next isn't my main focus. I'm just trying to give wander some ideas on how one might use Woof-next to utilize tinycore concepts.

Also note that currently I have procesps inside puppycore_i386_strech. I plan to make this it's own package but first I want to figure out some kind of logic to see if the native version (to the compatible distribution) is installed before installing this package.

I'm not sure I'm doing this right. Running setup.sh didn't produce any working directory for me and for some reason all the kernals to chose from in the list were 64 bit versions.

Anyway, so for now I ignore the lack of working directory and try to create a builder called:
/woof-CE/builders/tiny_devaun-build.sh

Code: Select all

#!/bin/sh
export PKGLIST=${PKGLIST:-pkglist}
export ARCH=${ARCH:-i386} # or amd64
export VERSION=${VERSION:-ascii}
export DISTRO_PREFIX=${DISTRO_PREFIX:-puppy}
export DISTRO_VERSION=${DISTRO_VERSION:-700} # informative only

export DEFAULT_REPOS=${REPO_URLS:-http://deb.devuan.org/merged/dists/|$VERSION|main:universe|Packages.xz}
#KEEP_DUPLICATES=1 # keep multiple versions of package in pkgdb
#WITH_APT_DB= # default is don't include apt-db

# dirs
export REPO_DIR=${REPO_DIR:-repo-$VERSION-$ARCH}
export CHROOT_DIR=${CHROOT_DIR:-chroot-$VERSION-$ARCH}
export DEVX_DIR=${DEVX_DIR:-devx-holder}
export NLS_DIR=${NLS_DIR:-nls-holder}
export BASE_CODE_PATH=${ROOTFS_BASE:-rootfs-skeleton}
# BASE_ARCH_PATH= # inherit - arch-specific base files, can be empty
export EXTRAPKG_PATH=${EXTRAPKG_PATH:-rootfs-packages}
deb-build.sh

I haven't tried running it yet, I'm going to take a break from this now. I'm not sure which of these above parmaters are used and what other ones. I might need to change. I changed.

Code: Select all

 
DEFAULT_REPOS=${REPO_URLS:-http://deb.devuan.org/merged/dists/|$VERSION|main:universe|Packages.xz}
and making the extention xz in Packages.xz. 

I also changed

Code: Select all

export VERSION=${VERSION:-ascii}

[musher0]

@jamesbond:

Thanks for the devx recipe. And thanks for inventing this build system.

Best wishes of success in your new ventures.

[wanderer]

hi musher0


you might like this new system
it has a bunch of window managers besides jwm


wanderer

[wanderer]

hi jamesbond

thanks again for making this great system
and all the help

i hope you will check in now and then
to see how things are going

i promise i will never ask for another thread to be deleted
i have learned my lesson

wanderer