Minimum Operating Systems to host package manager in chroot

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

Minimum Operating Systems to host package manager in chroot

#1 Post by s243a »

When I was experimenting with a 64bit version of TazPup, I found it helpful to be able to use slacko64's package manager to download packages, in the event that tazpkg didn't have available in it's repo's something that I wanted to install (e.g. vlc). However, extracting the basesfs of Slacko64 obviously gave me many more files then I would need to run puppies package manager.

I've often wanted a slimer base system for puppylinux, where only the most core files were in the base sfs and the other less core packages were moved to the adrv (or other sfs files). Having the package manager as the target application provides a design goal for this objective. This isn't necessarily the best design goal for a minimal system, rather it is the goal which suited my application.

My initial idea was that PuPnGo might make a good base system and that I could add a package manager to it from either tazpkg or x-slacko-slim. The reason that I thought that x-slacko slim might be a good choice is that mistfire has adapated petget in it to be able to download and install packages from more repo's then a typical puppy and this includes void and tinycore packages/repos.

PuPnGo is based on puppylinux 412, so the most compatible repos would be the puppylinux 412 repos or if I was looking for a comparable SliTaz repo I would choose version 4 of SliTaz (based on comparing bash versions). Obviously comparing glib versions would be more applicable but I haven't done this yet.

The reason that I looked at bash was that I thought an older version of bash might save significant space and I thought that PuPnGo's bash might be smaller since it may have been compiled against ulibc.

It turns out though that the bash in PuPnGo is not smaller. I presume that this is because it is statically compiled. However, at least in the case of slitaz the dependency libtinfo is only 128k. Moreover, in either case bash is only around 1MB and while packages of this size can add up, bash is fairly core in my opinion (although not required for tazpkg).

Anyway core PupNGo is around 8MB but that doesn't even include busybox! Perhaps PuPnGo pulls busybox from initrd. I'm not sure. The complete tazpkg with all it's dependencies is only 13MB.

The dpendencies of tazpkg are:

Code: Select all

  busybox (824K)
    slitaz-base-files (568K)
      gettext-base (116K)
        glibc-base (4M)
    ncurses-common (120K)
I'm not sure what the functionality differences are between the base PuPnGo and a minimum tazpkg but for my purposes PuPnGo isn't offering any clear advantages except for perhaps more static libs, which might be advantages if one is messing around with core dependencies.

Anyway, I'll look at how much things grow once I try to get a whole petget to work but for now here are some base systems for tazpkg:

tazpkg_all_deps 13MB (includes busybox, slitaz-base-files, glibc-base, ncurses-common)
tazpkg_slim 6.8MB (includes slitaz-base-files)
tazpkg_slimist_4 6MB (stripped down slitaz-base-files, using slitaz version 4 as the repo)
tazpkg_slimist_cooking 6MB (stripped down slitaz-base-files, using slitaz cooking as the repo. Cooking refers to rolling release version of Slitaz)
rootfs-tazpkg+bash 15.1MB (tazpkg_all_deps + bash + some preliminary files for petget)

The idea of tazpkg_slimist_4 was it might be a compatible starting point to combine with PuPnGo since as noted above the SliTaz 4 repo might have similar version of libraries as PupnGo. ALL SliTaZ files in the above systems are from either the cooking branch or the next32 branch. My intent was to use the next32 branch but by default the packages are configured to use the cooking branch so I'm not completely sure which versions I got.

These base systems are built using tazpkg since tazpkg allows you to install packages at an alternative root. The one exception being the last item on the above list (i.e. rootfs-tazpkg+bash) which was built by taking tazpkg_all_deps and then adding both bash-4.4.5 and libtinfo-5.9.tazpkg (a dependency for bash), in the SliTaz repo. These last two packages were added by some code that I'm working on:
install-min-pkgs.sh
mk-min-taz-functions.sh

The code is designed so that you can install packages from either a directory or a file. In the script, I use uextract to extract the files. I convert pet specs to tazpkg receipts and vis versa if one of them is available otherwise I use dir2pet to generate the pet specs. The downside here is that there is no option in dir2pet to not pack the archive. I'll have to do some mods to fix this.

Another downside is that uextract duplicates the extraction folder if it already exists but this is probably a bug in my code as I probably want to delete the old one first if it exists before running uextract.

Anyway, the above code (i.e. install-min-pkgs.sh) adds the information needed for both petget and tazpkg to know what is installed. The idea is that one can provide this type of info for one or multiple package managers:

Code: Select all

  mkdir -p "$dest_root/root/.packages/builtin_files"
  mkdir -p "$dest_root/var/lib/tazpkg/installed/$pkg_short_name"  
  for spec_type in $(eval 'echo "${'$spec_types_arry_name'[@]}"'); do
      case "$spec_type" in
      pet|.pet) 
        pet_specs_fm_dir "$source_dir" "$dest_root"/
        cp "$source_dir/pet.specs" \
           "$dest_root/root/.packages/builtin_files/$pkg_short_name"
        ;;
      tazpkg)
        tazpkg_specs_fm_dir "$source_dir"
        
        cp "$source_dir/$pkg_name" "$dest_root/var/lib/tazpkg/installed/$pkg_short_name/files.list"
        ;;
      esac
  done
Currently though, I haven't written a sync mechanism for adding this type of information to a new package manager once it is added to the system, so that the new package manager is up to date. Anyway, this code is very preliminary and should be run in a sandbox...although I haven't.

The sandbox helps to protect against coding errors where a source or destination root is not assigned properly, or if a symlink targets the system rootfs (remember that non relative symbolic links will point to the root system if one is not running in a chroot environment).

The purpose of the above systems is to have a small system that one can play with in a chroot environment to download packages and examine dependencies. All code is very preliminary so I recommend people wait before it is more mature before playing with it.
Last edited by s243a on Wed 03 Apr 2019, 18:10, edited 2 times in total.

wanderer
Posts: 1098
Joined: Sat 20 Oct 2007, 23:17

#2 Post by wanderer »

hi s243a

I am very interested in what you are doing
I hope you will continue
if you need a tester or just a user opinion I am available

I once got a puppy ramdrive core down to 2 mgs
and it would boot and symlink a usr directory to the ramdrive
everything big was in the usr directory
which was uncompressed on the hard drive so you could mess with it

I did the same thing with moving everything to the save file
in a normal puppy

but I didn't have the expertise to add many applications
just dillo gtkedit and emelfm

I think what you are doing
could be very useful to the puppy community
both as a base and a distro
and could spawn a whole puppy genre

wanderer

oui

Re: Minimum Operating Systems to host package manager in chroot

#3 Post by oui »

s243a wrote:The idea of tazpkg_slimist_4 was it might be a compatible starting point to combine with PuPnGo since as noted above the SliTaz 4 repo might have similar version of libraries as PupnGo. ALL SliTaZ files in the above systems are from either the cooking branch or the next32 branch. My intent was to use the next32 branch but by default the packages are configured to use the cooking branch so I'm not completely sure which versions I got.
As far as I know (i was so! I don't know if it continues to stay so today)

a- all packages from the releases are offered as binaries on http://mirrors.slitaz.org, being downloadable as "download all the web site" or web site part, and

b- SliTaz was/is relatively easy to use a local repository as pseudo website directly on the own computer using /var/www , in SliTaz always donnected per default!

I suppose it is important to proceed so as there is no security that changes did not happen from one minute to the next making the chances from an homogeneous work as null (it would be the same thing in NuTyx: You willl be tried to try to compile as, it is astonish, the changes at the git always happens as YOU are try a new compilation :oops: ...). It would be necessary to introduce a version's selector in the scripts building in assembly language and to store a part of the more recent old versions...

.
Last edited by oui on Wed 03 Apr 2019, 20:46, edited 2 times in total.

oui

Re: Minimum Operating Systems to host package manager in chroot

#4 Post by oui »

s243a wrote:When I was experimenting with a 64bit version of TazPup, I found it helpful to be able to use slacko64's package manager to download packages, in the event that tazpkg didn't have available in it's repo's something that I wanted to install (e.g. vlc).
I am not software designer at all.

And I never open the motor hood or look under the kirt!

But as far I know, SliTaz64 has to be a 64 bit kernel running full with 32 bit stuff like the usual SliTaz distribution (declaration of Aleksej, the actual main developer else if he names himself with modesty "hobbyist coder", on the SliTaz forum).

Barry Kauler did also announce us such a design at his time!

If I am right, this would mean you develop an not existing SliTaz full 64 (*1 outside of the SliTaz community!

Would that be right?

(*1 with VLC over that! As I hate (*2 the VLC monopolist influence, the horror pure for me! I hope it will be easy to desintall VLC and use our old versions instead of VLC without damage at the system?

(*2 why I hate VLC? VLC is one of the software requiring a confirmation from user to submit to the VLC conditions. As I never use some save file / dir, and start each day sometimes more that one time each day a PUPPY FIRST START, as it was in Puppy in the long first years until release (about, I am not sure any more if it did happen at that release) 1.7 this detail is a great inconvenient and shows that VLC is absolutely no free but spy software and remember it each new start!

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#5 Post by musher0 »

Thanks, s243a.

Interesting.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

Re: Minimum Operating Systems to host package manager in chroot

#6 Post by s243a »

oui wrote:
s243a wrote:When I was experimenting with a 64bit version of TazPup, I found it helpful to be able to use slacko64's package manager to download packages, in the event that tazpkg didn't have available in it's repo's something that I wanted to install (e.g. vlc).
I am not software designer at all.

And I never open the motor hood or look under the kirt!

But as far I know, SliTaz64 has to be a 64 bit kernel running full with 32 bit stuff like the usual SliTaz distribution (declaration of Aleksej, the actual main developer else if he names himself with modesty "hobbyist coder", on the SliTaz forum).

Barry Kauler did also announce us such a design at his time!

If I am right, this would mean you develop an not existing SliTaz full 64 (*1 outside of the SliTaz community!

Would that be right?

(*1 with VLC over that! As I hate (*2 the VLC monopolist influence, the horror pure for me! I hope it will be easy to desintall VLC and use our old versions instead of VLC without damage at the system?

(*2 why I hate VLC? VLC is one of the software requiring a confirmation from user to submit to the VLC conditions. As I never use some save file / dir, and start each day sometimes more that one time each day a PUPPY FIRST START, as it was in Puppy in the long first years until release (about, I am not sure any more if it did happen at that release) 1.7 this detail is a great inconvenient and shows that VLC is absolutely no free but spy software and remember it each new start!
Please see my response, in the thread, " TazPup64 (pre-alpha7) - Verified Sound Works. ", note that; I haven't tried booting directly from the ISO yet. Instead I booted from the files in the ISO.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#7 Post by rufwoof »

mpv is my first port of call. vlc only if I have to.

mpv plays sounds, videos and can preview images, in a folder full of .png's for instance mpv *.png ... and it steps through them one by one.
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

wanderer
Posts: 1098
Joined: Sat 20 Oct 2007, 23:17

#8 Post by wanderer »

hi s243a

this is just a question from a non guru


what i understand you are doing is

1. you will have a core

2. and a package manager that will download and resolve applications
from various sources and load them onto the core


so you will only need to maintain the core and the package manager
to have an infinitely variable minimal build system


also would it be simpler to use apps symlinked to an initrd
rather than a layered file system


wanderer

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#9 Post by s243a »

wanderer wrote:hi s243a


also would it be simpler to use apps symlinked to an initrd
rather than a layered file system


wanderer
I'm open to different approaches but I need to learn more about the symlinked approach. Also making a fully running system is sort of a second step. The primary core systems that I have in mind are focused on a chroot environment and might possibly (but not necessarily) be re-organized when being transformed into a fully bootable system.

wanderer
Posts: 1098
Joined: Sat 20 Oct 2007, 23:17

#10 Post by wanderer »

s243a

what you are doing here is truly groundbreaking

it will revolutionize the build system

and allow people to create amazing distros

thanks for doing it

wanderer

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#11 Post by musher0 »

Hi s243a and all.

@s243a: if considering the symlinked approach, perhaps you will find the
GNU command < stow > of interest.

As starting points:
https://linuxconfig.org/how-to-use-gnu- ... d-dotfiles

https://www.ecosia.org/search?q=GNU+sto ... opensearch

IHTH
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#12 Post by s243a »

musher0 wrote:Hi s243a and all.

@s243a: if considering the symlinked approach, perhaps you will find the
GNU command < stow > of interest.

As starting points:
https://linuxconfig.org/how-to-use-gnu- ... d-dotfiles

https://www.ecosia.org/search?q=GNU+sto ... opensearch

IHTH


Thankyou for the links and I'll consider all of this stuff but as I noted above; building a functional ISO is a secondary goal. The chroot system is the primary goal. For a functional ISO I will probably do a traditional puppy first and this is mostly because I already have scripts for this based on mistfire's build-kit for TazPup.

That said in a chroot environment, I probably don't want all the overhead of a layered file-system and may first try sym-links (and/or binds) in the chroot environment before trying a layered file system approach. Another option for the chroot approach is a hybird approach where one works directly with one of the layers but sym-links to a fully layered system. In this approach perhaps the PATH and LD_LIBRARY_PATH can be modified to search the fully layered system if the file isn't found directly on a given layer.

(As I note above?), I already have scripts to dissect a rootfs into separate units stored in separate directories (see thread, "stripping down a puppy") so I can aim to make the build system flexible enough that one can choose either a symlink or a layered file system approach.

Regarding the "stripping down thread"' this thread takes the opposite approach but despite this the strip down approach used in the other thread was still educational in that it lets me organize the packages of a working puppy into functional units and furthermore as I note above, the dissect_rootfs() function that I developed in the "strip down approach" can be used in the bottom-up approach for organizing packages into various units.

Code: Select all

dissect_rootfs(){
  #arr_names=${1:-ALL_ArrNames}
  arry_names_name=${1:-ALL_ArrNames}
  for arr_name in $(eval 'echo "${'$arry_names_name'[@]}"'); do
    eval "arr=( "\${"$arr_name"[@]}" )" 
    eval "arr_action=\$"$arr_name"_action"
    arr_action=${arr_action:-"$action"}
    #Move is faster but perhaps copy is safer
    arr_action=${arr_action:-cp} 
    set -x
    eval 'target="$'$arr_name'_target"'
    target=${target:-"$curdir/$arr_name"}
    set +x
    if [ "${s_rootfs:0:1}" = "/" ]; then
      s_rootfs_prefixed="$s_rootfs"    
    else
      s_rootfs_prefixed="$curdir/$s_rootfs"     
    fi  
    for app in "${arr[@]}"; do
      echo "arr_action=$arr_action"
      case "$arr_action" in
      cp|mv|retry)  
          echo "app=$app" 
          copy_built_in "$app"        
          ;;
      esac
      if [ -f "/tmp/trim_puppy/$app" ] && [ $chroot_remove_builtin -eq 1 ]; then
        file_list="$curdir/$s_rootfs/root/.packages/builtin_files/$app"
        rm "$file_list"
        cp "/tmp/trim_puppy/$app" "$file_list"
      fi
      #chroot "$s_rootfs_prefixed" remove_builtin "$app"
    done 
  done
}

Post Reply