Pkg - CLI package manager

[sc0ttman]

I noticed an error that is causing double slashes.

the variable whoami isn't defined until line 100

Code: Select all

whoami=$(whoami)

/usr/sbin/pkg#L100

but this variable is used to define the tmp directory on line 40 before this variable is set:

Code: Select all

TMPDIR=/tmp/pkg/${whoami}  # set the tmp dir

/usr/sbin/pkg#L40

Yes this is a genuine bug, probably from when I moved some code around..

It causes stuff to be written into /tmp/pkg/ instead of /tmp/pkg/root/ ... or something like that .. not fatal, but will fix it anyway..

[sc0ttman]

I eventually did get an error (prior to fixing distro specs). The error was:

Code: Select all

	Failed to download db file: 
	 http://packages.devuan.org/merged/dists//main/binary-/Packages.xz
	...exited from 0setup script.

so pkg could sand this log file for the word "Failed"

Will look into doing something like that if 0setup fails, thanks.

[sc0ttman]

As a side note, one can see the builtin packages with the command:

Code: Select all

HIDE_BUILTINS=false pkg --list-installed | grep ^hicolor

^ all packages will be shown (builtins, devx files and also user installed packages).

Pkg should probably separate devx packages out, so they can be hidden/shown with:

Code: Select all

HIDE_DEVX_PACKAGES=true/false pkg --list-installed

Is that useful? Pkg counts devx packages as installed by default, so they don't get re-downloaded & intstalled...

**BTW although I was missing the petspec record for libc6

Where? You mean ~/.packages/**.files?

in my opinion having the file list should be enough to prevent the package from being installed...but this is a minor point.

Happy for this to be the default Pkg behaviour, if it isn't already .. might look into on a standard Puppy.

[s243a]

Of course, I first and foremost intend to get Pkg working in official Puppies ...
Your devuan acsii version is an 'edge case', for now...

Not sure if the following is an "edge case issue" or a woof-CE issue

If you are not running a standard Puppy (which you are not), then for Pkg it's 100%
an "edge case" - because Pkg is designed (mainly) for standard Puppy Linux releases.

I have no idea if your custom OS (half Puppy, half something else) has everything setup
correctly... I don't even know if it was built with Woof... Hence, your OS is the "edge-case"
as far as Pkg development is concerned.

I started a thread on it:
tiny_puduan_ascii-PreAlpha11 (made via a woof-next fork)


It's not built via woof but it uses puppy init scripts and most of the rootfs-skeleton.

Code: Select all

Anyway, I tried to work out if it's a Pkg issue....

thankyou

I know that the package manager...

Which one?? Pkg? PPM? PPM 3.0?

..writes to the actual save layer but in my case
the save layer is:

Code: Select all

/initrd/mnt/dev_save

.....

Edit: After further examination this might be an issue with mistfire's package manager.

I search for pup_ro1 returns nothing that seems relevant in the gitlab branch for pkg.

Then I think you should probably post it at the PPM 3.0 thread, or even better work out if it's
actually a Woof-CE issue before posting at either... I guess..?

I did notice though that pup_r01 is used in sfs_loadr.

Code: Select all

   6) SAVE_LAYER='/pup_rw'; PUP_HOME='/pup_rw'; PUPSAVE='sda1,ext2,/';;
   12) SAVE_LAYER='/pup_rw'; PUP_HOME='/mnt/dev_save';;
   13) SAVE_LAYER='/pup_ro1'; PUP_HOME='/mnt/dev_save';;
   77) SAVE_LAYER='/pup_ro1'; PUP_HOME=''; PUPSAVE='sr0,iso9660,/2011-01-27-20-26';;

/usr/sbin/sfs_loadr#L1697

....

I think that PUP_HOME should either be /mnt/home or /initrd/mnt/dev_save/

**note that /mnt/home is a symlink pointing to /initrd/mnt/dev_save

Yep.. That is what I have on my system (Stretch RC something..) and most Pups I'd imagine..
But it depends on the install method/type... Sorry but I can't reproduce those issues..

...More generally ...

I think maybe you're posting "too soon", if you don't mind me saying so...

The first long series of posts you posted to the Pkg thread turned out to be your /etc/DISTRO_SPECS
not being setup correctly, which wouldn't have happened in a standard Puppy...

In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):

Code: Select all

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup 

The path to the DB file is:

Code: Select all

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz    
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   

the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

Note that at the moment some of this is outside the scope of "pkg" because the repo update is done by files which are currently part of puppies package manager.

Code: Select all

And sorry, but I also don't want this thread mixed up with mistfires PPM 3.0 (which literally runs 
[i]against[/i] to the goals of this project).. Pkg is not designed to support PPM 3.0 (unless it becomes 
Puppys official/main PPM).. even then, Pkg aims to become independent of any PPM..

...

I think you should probably test Pkg on a standard Puppy to compare to your OS before posting any 
issues here in order to make sure the issues are with Pkg and not your custom OS.. 

Currently pkg still depends on some files from the puppy package manager. These files could be from the current puppy package manager, mistfires version or the latest woof-CE code. Since pkg aims to be independent of the ppm, may I suggest having a variable that points to the directory where the needed ppm files are. One could make this independent of puppies ppm by simply pointing to a different directory than the default puppy ppm.
**although it might also want to use a different directory for temporary files.

Anyway, regarding testing on a standard puppy; it seems to be the case that there are a lot of woof-CE changes related to puppies package manager. Some of these are related to improvements by mistfire. If pkg aims to be compatible with the lastest woof-CE code then the code is too unstable at this point. If pkg aims to use legacy puppy package manager code then it should include these files as part of the project and put them in a separate directory than the default puppy package manager.

Anyway, the reason that mistfire's code interested me was that in her x-slacko slim release the ppm appeared to be able to downloaded packages from many different repos.
**In theory pkg should also be able to do this based on the soures file.

For this to work, I think that the dependencies on DISTRO_SPECS should be removed, in mistfire's ppm 3.0 thread, it appears to me that there are still some dependencies on DISTRO_SPECS related to updating the package data base files. Therefore, I'm not sure if x-slacko slim properly updates databases which aren't part of the compatible distro.

[sc0ttman]

In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):

Code: Select all

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup 

The path to the DB file is:

Code: Select all

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz    
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   

the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

Ah OK, that may be another area where we can stop Pkg relying on Puppy config files... Will have a look into it at some point..

In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

It already does do some of its own arch checking, so should be do-able..

As for Pkg/Woof-CE compatibility - yes there is a way to go yet...

And the only trouble I see with using Pkg alongside PPM 3.0 is that I've no idea what changes were made - except I do know some of the ~/.package/ files have been re-organised ... Obvioulsy if you can make Pkg work alongside PPM 3.0 that is great, but I won't be bug-fixing Pkg to make it happen ... Only to align it with whatever PPM is in Woof-CE ..

[s243a]

I'm not completly sure if the following is a bug or not but I noticed that you do the "direct save path" slightly different than puppy:

Code: Select all

# set $DIRECTSAVEPATH (where we want to install pkgs)
if [ $PUPMODE -eq 3 -o $PUPMODE -eq 7 -o $PUPMODE -eq 13 ];then
  DIRECTSAVEPATH="/initrd${SAVE_LAYER}" #SAVE_LAYER is in /etc/rc.d/PUPSTATE.
elif [ "$PUPMODE" = "2" ]; then
  DIRECTSAVEPATH=""
fi

https://gitlab.com/sc0ttj/Pkg/blob/mast ... n/pkg#L147

Code: Select all

	   if [ ! "$DIRLINK" ];then
	    if [ -h "/initrd${SAVE_LAYER}${ONESPEC}" ];then #120107
	     DIRLINK="`readlink -m "/initrd${SAVE_LAYER}${ONESPEC}" | sed -e "s%/initrd${SAVE_LAYER}%%"`" #SAVE_LAYER: see /etc/rc.d/PUPSTATE. 120107
	     xDIRLINK="`readlink "/initrd${SAVE_LAYER}${ONESPEC}"`" #120107
	    fi
	   fi
	   if [ "$DIRLINK" ];then
	    if [ -d "$DIRLINK"  ];then
	     if [ "$DIRLINK" != "${ONESPEC}" ];then #precaution.
	      mkdir -p "${DIRECTSAVEPATH}${DIRLINK}" #120107
	      cp -a -f --remove-destination ${DIRECTSAVEPATH}"${ONESPEC}"/* "${DIRECTSAVEPATH}${DIRLINK}/" #ha! fails if put double-quotes around entire expression.
	      rm -rf "${DIRECTSAVEPATH}${ONESPEC}"
	      if [ "$DIRECTSAVEPATH" = "" ];then
	       ln -s "$xDIRLINK" "${ONESPEC}"
	      else
	       DSOPATH="`dirname "${DIRECTSAVEPATH}${ONESPEC}"`"
	       DSOBASE="`basename "${DIRECTSAVEPATH}${ONESPEC}"`"
	       rm -f "${DSOPATH}/.wh.${DSOBASE}" #allow underlying symlink to become visible on top.
	      fi
fi

/woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L443

The key difference here is that in puppy they are using the "readlink" function, so the puppy package manager gets the real path whereas you are using a symlink which points to the "direct save path" The consequence is that when you you extrat the package archive via the tar command, you are extracting the package archive to the symlink rather than to the actual directory.

Code: Select all

tar --absolute-names $tarops "./${PKGNAME}.tar.$TAREXT" ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog 

/master/usr/sbin/pkg#L4563

Now long ago (perhaps prior to when woof-CE was on github there was the following fix:

Code: Select all

#120102 install may have overwritten a symlink-to-dir.
#120107 rerwin: need quotes around some paths in case of space chars. remove '--unlink-first' from tar (was introduced 120102, don't think necessary).

woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L26

I'm guessing that the --unlink-first option was used in conjuction with the "-h" option:

-h, --dereference
follow symlinks; archive and dump the files they point to

http://murga-linux.com/puppy/viewtopic. ... 49#1030949

However, I assume that neither of these options are necessary now that the puppy package manager is using the realpath of pup_ro1

Anyway, this error only manifested itself (see post) when I broke either "busybox mount" or mount.aufs (the latter used in mount-FULL) and therefore, I don't know yet whether or not this is an actual bug in "pkg".

Edit:

The following section of the "tar manpage" suggests that /initrd/pup_ro1 will be overwritten if the following option isn't supplied (testing required to verify):

--keep-directory-symlink
Don't replace existing symlinks to directories when
extracting.

http://man7.org/linux/man-pages/man1/tar.1.html

[s243a]

In the meantime I took a stab at this (commit e4b000bda57f13a6eba5a2f3fb2bd43e856189ce)

I probably made some changes which weren't necessary.

You should probably define the right package dir at the top of the /usr/sbin/pkg script as a variable,
then use that variable later throughout the script..

Or even better, define it in the main config RC script: ~/.pkg/pkgrc

Then it can be set to various different places - easier to run Pkg in chroot, ssh, local folder for unit tests , etc..

Okay, I took your suggestion (see commit: ea299b8ff3054173df6b5725bc8b7eef658762c6). I put the following

Code: Select all

[ -z "$PKGS_DIR" ] && PKGS_DIR="$(realpath "${HOME}/.packages")"

/woof-code/rootfs-packages/PKG/usr/sbin/pkg#L60

after "~/.pkg/pkgrc" is sourced. Therefore this folder can be defined in ~/.pkg/pkgrc.

I left in the previous realpath statements I added. I don't think they hurt anything but are probably redundant.

**Note that this last change was done by a simple string replace and I haven't done any testing yet.

[s243a]

I'm not sure if the following is being done by ppm v3.0 or sc0tmann's pkg but some process seems to be moving all my files out of the /usr/lib/i386-linux-gnu/ folder....

...This time the whole i386-linux-gnu folder seems to be replaced with a symlink. I know this is standard puppy to do this but maybe there is a reason that Debian/Devaun creates the i386-linux-gnu folder (e.g. to separate architectures on multi architecture systems).

Maybe pkg and PPM..? Don't quote me on that..

But this code in Pkg expects a symlink:

Your are correct. This is related to the "muti-arch hack" of the puppy package manager. I oppened an issue on github to discuss whether said muti-arch hack is a good or bad thing:

https://github.com/puppylinux-woof-CE/w ... ssues/1475

http://murga-linux.com/puppy/viewtopic. ... 59#1030959

I found the code that sc0tmann is talking about here:

Code: Select all

					# woofce: NO_MULTIARCH_SYMLINK=1 (DISTRO_SPECS)
					if [ -z "$NO_MULTIARCH_SYMLINK" ] ; then
            if [ -f $HOME/.packages/${PKGNAME}.files ];then
              pkg_has_archdir="$(grep -m1 "$DISTRO_ARCHDIR" "$HOME/.packages/${PKGNAME}.files")"
            fi

						# Workaround to avoid overwriting the $DISTRO_ARCHDIR symlink.
						if [ "$DISTRO_ARCHDIR" != "" -a -f $HOME/.packages/${PKGNAME}.files -a "$pkg_has_archdir" != "" ]; then
						  mkdir -p /tmp/$PKGNAME
						  rm -rf /tmp/$PKGNAME/*
						  dpkg-deb -x ${CURDIR}/${PKGNAME}.deb /tmp/$PKGNAME/ 2> $TMPDIR/$SELF-cp-errlog
						  for f in $(find /tmp/$PKGNAME \( -type f -o -type l \))
						  do
							  xpath=$(echo "$f" |  cut  -f 4-30 -d "/" | sed "s/$DISTRO_ARCHDIR\///")
							  mkdir -p ${DIRECTSAVEPATH}/$(dirname "$xpath")
							  cp -a "$f" ${DIRECTSAVEPATH}/$(dirname "$xpath")/
						  done
						  rm -rf /tmp/$PKGNAME &>/dev/null
						else
						  dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
						fi
 					else
						dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
 					fi

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L4614

BTW: according to wdlkmpx, the "multi-arch hack" has been removed:
https://github.com/puppylinux-woof-CE/w ... -505873406

[s243a]

The following doesn't look correct to me:

Code: Select all

dpkg-deb -e "${CURDIR}/${PKGNAME}/${PKGNAME}.deb" /DEBIAN #130112 extracts deb control files to dir /DEBIAN. may have a post-install script, see below.

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L4644

I think it should be:

Code: Select all

dpkg-deb -e "${CURDIR}/${PKGNAME}.deb" /DEBIAN #130112 extracts deb control files to dir /DEBIAN. may have a post-install script, see below.

It also looks like it is code that was copied from woof-CE so there may have been (or be) a similar bug in woof-CE.

[sc0ttman]

Something good:

I (stupidly) only just realised there is an easy way to get updated packages.

And it will get around the horrible merging of repos that the PPM insists on
doing during repo updates...

How?

You can manually add the "backports" or "unstable" (etc) repos to Puppy
as separate repos, by doing the following:

Code: Select all

pkg add-repo http://deb.debian.org/debian stretch-backports main contrib

^ this makes it possible to add extra repos (updates, backports or testing) to your
Ubuntu/Debian based Puppies very easily..

After adding the backports repo to my Puppy Stretch 7.5 RC2, I now have
a user-added repo (called stretch-backports) that has lots of updated versions of the
packages in my main repos - something the default Puppy repo update (`0setup`)
fails to do (for some reason!)...


Something bad:

There is a serious bug in the `add-repo` command:
Sometimes the newly added repo overwrites another repo!

Details:

If you add a 3rd-party Debian repo that has streams matching your default
system repos, then the new repo will replace the system one whose stream
names it matches, wiping out the main repo in the process.

Example which breaks the repos:

Code: Select all

pkg add-repo https://deb.torproject.org/torproject.org stretch main

^this doesn't work cos the last 2 fields (the repo "streams") are "stretch" and "main",
so the "stretch-main" repo gets overritten!!

Workaround for now:

Install tor from the Ubuntu PPA repos instead:

Code: Select all

pkg add-repo http://deb.torproject.org/torproject.org/ bionic main

[josejp2424]

I love this program in my case it works very well. Thank you sc0ttman.
and I don't report an error, because I didn't find them.

[josejp2424]

http://murga-linux.com/puppy/viewtopic. ... 79#1033179

pkg seems to be working well now

Have you added any fixes to Pkg to make it work?

What was the problem, if I may ask?

.

hello sc0ttman.
Regarding this question.
in the previous version woof-ce. the x86_64-linux-gnu folders.
They had link to / lib. and pkg when installing packages modified the folder. I stopped having a / lib link.
And corrupted the system.

in the new version of woof-ce.
those folders no longer have a direct link to / lib.


http://murga-linux.com/puppy/viewtopic.php?t=116615

[s243a]

I think I know why pkg seems to be installing [1] some items which are already installed.

Pkg removes dependencies from the list of packages to install with the following line:

Code: Select all

 comm -23 ${TMPDIR}all_deps_0 ${TMPDIR}installed_pkgs | sort -u | grep -v ^$ > ${TMPDIR}all_deps_1

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L6033

The listof files ${TMPDIR}installed_pkgs is produced by the following function:

Code: Select all

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

  if [ "${HIDE_INSTALLED}" = true -a "$FORCE" = false ];then
    # add user installed pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/user-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ "${HIDE_BUILTINS}" = true ];then
    # add builtins to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/woof-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/devx-installed-packages ];then
    # add devx pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/devx-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/layers-installed-packages ];then
    # add layers list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/layers-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L5947

I think in the first to if blocks the logic is backwards but I'm not sure why we have the iff blocks at all. Why not just do the following instead:

Code: Select all

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

    cut -f2 -d'|' "$USER_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$WOOF_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$DEVX_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$LAYER_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}

For now I'll just comment out code to effectively do this in case I misunderstood something.

Notes
--------------------
I say "seems to be installing" because some of the packages that were installed as dependencies look like built-ins to me. I haven't verified this yet though.

[s243a]

I think I know why pkg seems to be installing [1] some items which are already installed.
...
Notes
--------------------
I say "seems to be installing" because some of the packages that were installed as dependencies look like built-ins to me. I haven't verified this yet though.

I looked into this further. I tested this on tiny_puduan_ascii-PreAlpha11.4.iso (see post). I have the builtin file libstdc++6-6.3.0-18+deb9u1. The following package was installed (despite my above mods) libstdc++-5.0.6 (as part of installing dependencies for firefox-esr). Maybe this is because it is a different version number? I'll look into it more.

[s243a]

I
Pkg removes dependencies from the list of packages to install with the following line:

Code: Select all

 comm -23 ${TMPDIR}all_deps_0 ${TMPDIR}installed_pkgs | sort -u | grep -v ^$ > ${TMPDIR}all_deps_1

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L6033

The listof files ${TMPDIR}installed_pkgs is produced by the following function:

Code: Select all

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

  if [ "${HIDE_INSTALLED}" = true -a "$FORCE" = false ];then
    # add user installed pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/user-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ "${HIDE_BUILTINS}" = true ];then
    # add builtins to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/woof-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/devx-installed-packages ];then
    # add devx pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/devx-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/layers-installed-packages ];then
    # add layers list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/layers-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L5947

Trying to trace what is going on here might be slightly difficult due to concurancy but maybe I'll make good progress in understaning this by writing this post.

I noticed the currency via line L6311 of the following code:

Code: Select all

5984 get_deps(){
...
5990 list_all_installed_pkgs
...
6311  # wait until list_deps() is finished (if it's running at all...)
6312  while [ -f /tmp/pkg/list_deps_busy ];do
6313    echo -n '.'
6314    sleep 0.75
6315  done

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L6311

My debug output execuded via

Code: Select all

bash -x /usr/sbin/pkg 2>$1 | tee logfile

produces output like:

Code: Select all

+ list_all_installed_pkgs
+ echo -n ''
+ cut -f2 '-d|' /var/packages/user-installed-packages
++ get_pkg_ext firefox-esr_60.6.3esr-1
++ '[' '!' firefox-esr_60.6.3esr-1 ']'
++ local pkg_installed=
++ local pkg_filename=
++ local pkg_ext=
++ local pkg_name=
++ case "$1" in
+ cut -f2 '-d|' /var/packages/woof-installed-packages
+++ grep -m1 '^firefox-esr_60.6.3esr-1'
+ cut -f2 '-d|' /var/packages/devx-only-installed-packages
+++ list_installed_pkgs firefox-esr_60.6.3esr-1
+++ local user_pkgs_list=
+++ local builtins_list=
+++ local devx_pkgs_list=
+++ user_pkgs_list=/var/packages/user-installed-packages
+++ '[' true '!=' true -a -f /var/packages/devx-only-installed-packages ']'
+++ '[' true '!=' true ']'
+++ '[' '!' firefox-esr_60.6.3esr-1 ']'
cut: /var/packages/devx-only-installed-packages: No such file or directory
+ cut -f2 '-d|' '/var/packages/layers-installed packages'
+++ grep '^firefox-esr_60.6.3esr-1'
+++ cut -f1 '-d|' /var/packages/user-installed-packages

What seems to be happening based on this output is that get_deps() is getting called multiple times in the background before the function list_all_installed_pkgs() finishes. "+" is the first instance "++" is the second instance and "+++" is the third instance of get_deps. The flag:

Code: Select all

/tmp/pkg/list_deps_busy

seems to be used to prevent multiple instances of get_deps from calling list_all_installed_pkgs()


Note the output is from my modified version of pkg but the code that I'm citing is from sc0tmann's version. I'm speculating a bit here about concurrency because I haven't yet identified the code that launches the background processes.

[s243a]

Here is the first relevant instance of concurancy that I found related to the previous post:

Code: Select all

list_deps "$pkg_name" > ${TMPDIR}/${pkg_name}_dep_list &

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L5458

I've been picking out what looks to be one instance of the function list_deps. Because of concurancy my debugging output is mixed up with various subprocesses but in most cases I can follow where there is a single "+" to try to follow the logic:
https://pastebin.com/PTXYqEx9 (incomplete output...will be updated)

I'm having more troubles understanding this "HIDE_BUILTINS" variable as it isn't obvious to me by the name what affect it should have in all cases. I think there should almost be seperate enviornmental variables for different functions. For instance, we could have another variable called "REINSTALL_BUILTINS" which will reinstall builts for all dependencies and if we only want to reinstall the builting of a single package, maybe a sinilar name for the enrironmental variable but without the trailing "S". To me the "HIDE_BUILTINS" enviornmental variable should only apply to when we are listing variables.

Anyway, I'm going to postepone this renaming idea but I want to make the following change. I want to change:

Code: Select all

            if [ "$HIDE_BUILTINS" = true ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L6077
to

Code: Select all

            if [ "$HIDE_BUILTINS" = true ] || [ -z "$HIDE_BUILTINS" ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi

because by default we shouldn't install a package if it is a builtin. I think I'll follow the same logic in the function
list_all_installed_pkgs(), which I mentioned in a previous post.

Regarding the concurrancy (see prevoius post and the start of this current post)

I'm conerned that various instances of the function list_deps() will overwrite (or even delete) $TMPDIR/installed_pkgs prior to it being used to filterout already installed packages (see post).

My suggest is that the function list_all_installed_pkgs(). should only be installed once and then each time a package is installed it should be appended to the list and than sorted.

A simple way to do this is check for the existance of $TMPDIR/installed_pkgs and only create said file if it doesn't exist. I think I'll try this.

Edit: The actual changes that I made related to my comments in this post are discussed at: http://murga-linux.com/puppy/viewtopic. ... 70#1035370

[sc0ttman]

I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution... I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point.. Keep up the good work!

[s243a]

I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution...

I think I can get some speed improvements by better use of concurrency.

I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point..

The silver lining here is that it will give me some time to flesh out some ideas. I hope that I can do it fast enough.

Keep up the good work!

Thanks for the encouragement

[s243a]

One thing that I'm working on is improved use of aliases. My understanding is that pkg currently only uses aliases for listing available packages

Code: Select all

list_pkg_names(){                
...
      echo $ALIAS_LIST | while read ALIAS
      do
...
ALIAS_RES="`LANG=C cut -f1 -d'|' ${HOME}/.packages/${REPOFILE} 2>/dev/null | grep "^$ALIAS"`" #

https://gitlab.com/sc0ttj/Pkg/blob/mast ... /pkg#L2078

and not with the --get option. I would like to make it so people can use aliases with the --get option but first some simpler fixes.

I don't believe that in the way pkg uses aliases that it checks properly for a terminating character "either a comma ',' or a line ending. We can check for this by using alternatives which are available in grep if we use extended regular expressions (i.e. the -E switch). The pattern to match a comma or a line termination character is:

Code: Select all

'([,]|$)'

I'm writing a list_all_aliases() function. It is probably more complex than it needs to be and is only partially fleshed out. The file in the above link stripped out a lot of things from pkg to make debugging easier (i.e. it's a testing file so the trace output will be smaller).

I programmed it so that the alias list can be either a variable or a file. If no argument is suppled for the alias list it first checks for the existance of the file "${TMPDIR}/pkg_aliases" and if it doesn't exist it uses the variable, "$PKG_NAME_ALIASES". With my test code the file exists but the function is written in such a way not to create a tmp file if it doesn't exist.

Here is the code which produces the alias list for a given dependency.

Code: Select all

  if [ -z "$ALIAS_LIST" ]; then
    if [ -f ${TMPDIR}/pkg_aliases ]; then
      ALIAS_LIST="${TMPDIR}/pkg_aliases"
    else
      ALIAS_LIST="$PKG_NAME_ALIASES"
    fi
  fi
  if [ -e "$ALIAS_LIST" ]; then #ALIAS_LIST is a file
       ALIAS_LIST=( `grep -E "$subdep"'([,]|$)' "$ALIAS_LIST"  2>/dev/null | tr ',' ' '` )
  else #[[ "$ALIAS_LIST" = *,* ]]; then #ALIAS_LIST is string
    ALIAS_LIST=( $(echo "$ALIAS_LIST" | tr ' ' '\n' | grep -E "$subdep"'([,]|$)'  2>/dev/null | tr ',' '\n') )
  fi

https://pastebin.com/F8UL4ius

bellow that there is some checking code (e.g. does the package exist in the package database), but we can add more checks if we want like the wget spider function. However, if we do we first have to ensure the package databases are up to date.

I eventually plan to have the list_all_aliases() function be able to also download packages (provide a suitable option is supplied to the function). This way we can start downloading as soon as we find a suitable alias. I plan some kind of process management (based on named pipes) so that the user can limit the number of process if they desire. If the process limit is reached then the remaining work to do will be stored in a file.

[s243a]

What is the intent of "sources-user". It doesn't seem to be documented in the wiki.