Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 15 Jul 2019, 20:48
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Pkg - CLI package manager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 9 of 9 [128 Posts]   Goto page: Previous 1, 2, 3, ..., 7, 8, 9
Author Message
sc0ttman


Joined: 16 Sep 2009
Posts: 2691
Location: UK

PostPosted: Sun 23 Jun 2019, 12:36    Post subject:  

s243a wrote:
I noticed an error that is causing double slashes.

the variable whoami isn't defined until line 100
Code:

whoami=$(whoami)

/usr/sbin/pkg#L100

but this variable is used to define the tmp directory on line 40 before this variable is set:
Code:

TMPDIR=/tmp/pkg/${whoami}  # set the tmp dir

/usr/sbin/pkg#L40

Yes this is a genuine bug, probably from when I moved some code around..

It causes stuff to be written into /tmp/pkg/ instead of /tmp/pkg/root/ ... or something like that .. not fatal, but will fix it anyway..

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2691
Location: UK

PostPosted: Sun 23 Jun 2019, 12:38    Post subject:  

s243a wrote:
I eventually did get an error (prior to fixing distro specs). The error was:

Code:

   Failed to download db file:
    http://packages.devuan.org/merged/dists//main/binary-/Packages.xz
   ...exited from 0setup script.

so pkg could sand this log file for the word "Failed"

Will look into doing something like that if 0setup fails, thanks.

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2691
Location: UK

PostPosted: Sun 23 Jun 2019, 12:45    Post subject:  

s243a wrote:

As a side note, one can see the builtin packages with the command:
Code:

HIDE_BUILTINS=false pkg --list-installed | grep ^hicolor

^ all packages will be shown (builtins, devx files and also user installed packages).

Pkg should probably separate devx packages out, so they can be hidden/shown with:

Code:
HIDE_DEVX_PACKAGES=true/false pkg --list-installed

Is that useful? Pkg counts devx packages as installed by default, so they don't get re-downloaded & intstalled...

Quote:
**BTW although I was missing the petspec record for libc6
Where? You mean ~/.packages/**.files?
Quote:
in my opinion having the file list should be enough to prevent the package from being installed...but this is a minor point.

Happy for this to be the default Pkg behaviour, if it isn't already .. might look into on a standard Puppy.

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1941

PostPosted: Sun 23 Jun 2019, 18:44    Post subject:  

sc0ttman wrote:
s243a wrote:
sc0ttman wrote:

Of course, I first and foremost intend to get Pkg working in official Puppies ...
Your devuan acsii version is an 'edge case', for now...

Not sure if the following is an "edge case issue" or a woof-CE issue

If you are not running a standard Puppy (which you are not), then for Pkg it's 100%
an "edge case"
- because Pkg is designed (mainly) for standard Puppy Linux releases.

I have no idea if your custom OS (half Puppy, half something else) has everything setup
correctly... I don't even know if it was built with Woof... Hence, your OS is the "edge-case"
as far as Pkg development is concerned.


I started a thread on it:
tiny_puduan_ascii-PreAlpha11 (made via a woof-next fork)


It's not built via woof but it uses puppy init scripts and most of the rootfs-skeleton.

Code:

Anyway, I tried to work out if it's a Pkg issue....

thankyou Smile

Quote:

Quote:
I know that the package manager...
Which one?? Pkg? PPM? PPM 3.0?
Quote:
..writes to the actual save layer but in my case
the save layer is:

Code:

/initrd/mnt/dev_save


.....

Edit: After further examination this might be an issue with mistfire's package manager.

I search for pup_ro1 returns nothing that seems relevant in the gitlab branch for pkg.

Then I think you should probably post it at the PPM 3.0 thread, or even better work out if it's
actually a Woof-CE issue before posting at either... I guess..?

Quote:

I did notice though that pup_r01 is used in sfs_loadr.

Code:

   6) SAVE_LAYER='/pup_rw'; PUP_HOME='/pup_rw'; PUPSAVE='sda1,ext2,/';;
   12) SAVE_LAYER='/pup_rw'; PUP_HOME='/mnt/dev_save';;
   13) SAVE_LAYER='/pup_ro1'; PUP_HOME='/mnt/dev_save';;
   77) SAVE_LAYER='/pup_ro1'; PUP_HOME=''; PUPSAVE='sr0,iso9660,/2011-01-27-20-26';;

/usr/sbin/sfs_loadr#L1697

....

I think that PUP_HOME should either be /mnt/home or /initrd/mnt/dev_save/

**note that /mnt/home is a symlink pointing to /initrd/mnt/dev_save

Yep.. That is what I have on my system (Stretch RC something..) and most Pups I'd imagine..
But it depends on the install method/type... Sorry but I can't reproduce those issues..

...More generally ...

I think maybe you're posting "too soon", if you don't mind me saying so...

The first long series of posts you posted to the Pkg thread turned out to be your /etc/DISTRO_SPECS
not being setup correctly, which wouldn't have happened in a standard Puppy...


In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):
Code:

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup


The path to the DB file is:
Code:

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz   
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   


the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

Note that at the moment some of this is outside the scope of "pkg" because the repo update is done by files which are currently part of puppies package manager.

Code:
And sorry, but I also don't want this thread mixed up with mistfires PPM 3.0 (which literally runs
[i]against[/i] to the goals of this project).. Pkg is not designed to support PPM 3.0 (unless it becomes
Puppys official/main PPM).. even then, Pkg aims to become independent of any PPM..

...

I think you should probably test Pkg on a standard Puppy to compare to your OS before posting any
issues here in order to make sure the issues are with Pkg and not your custom OS..


Currently pkg still depends on some files from the puppy package manager. These files could be from the current puppy package manager, mistfires version or the latest woof-CE code. Since pkg aims to be independent of the ppm, may I suggest having a variable that points to the directory where the needed ppm files are. One could make this independent of puppies ppm by simply pointing to a different directory than the default puppy ppm.
**although it might also want to use a different directory for temporary files.

Anyway, regarding testing on a standard puppy; it seems to be the case that there are a lot of woof-CE changes related to puppies package manager. Some of these are related to improvements by mistfire. If pkg aims to be compatible with the lastest woof-CE code then the code is too unstable at this point. If pkg aims to use legacy puppy package manager code then it should include these files as part of the project and put them in a separate directory than the default puppy package manager.

Anyway, the reason that mistfire's code interested me was that in her x-slacko slim release the ppm appeared to be able to downloaded packages from many different repos.
**In theory pkg should also be able to do this based on the soures file.

For this to work, I think that the dependencies on DISTRO_SPECS should be removed, in mistfire's ppm 3.0 thread, it appears to me that there are still some dependencies on DISTRO_SPECS related to updating the package data base files. Therefore, I'm not sure if x-slacko slim properly updates databases which aren't part of the compatible distro.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2691
Location: UK

PostPosted: Mon 24 Jun 2019, 14:17    Post subject:  

s243a wrote:
In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):
Code:

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup


The path to the DB file is:
Code:

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz   
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   


the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

Ah OK, that may be another area where we can stop Pkg relying on Puppy config files... Will have a look into it at some point..

Quote:
In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

It already does do some of its own arch checking, so should be do-able..

As for Pkg/Woof-CE compatibility - yes there is a way to go yet...

And the only trouble I see with using Pkg alongside PPM 3.0 is that I've no idea what changes were made - except I do know some of the ~/.package/ files have been re-organised ... Obvioulsy if you can make Pkg work alongside PPM 3.0 that is great, but I won't be bug-fixing Pkg to make it happen ... Only to align it with whatever PPM is in Woof-CE ..

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1941

PostPosted: Tue 25 Jun 2019, 23:32    Post subject:  

I'm not completly sure if the following is a bug or not but I noticed that you do the "direct save path" slightly different than puppy:

Code:

# set $DIRECTSAVEPATH (where we want to install pkgs)
if [ $PUPMODE -eq 3 -o $PUPMODE -eq 7 -o $PUPMODE -eq 13 ];then
  DIRECTSAVEPATH="/initrd${SAVE_LAYER}" #SAVE_LAYER is in /etc/rc.d/PUPSTATE.
elif [ "$PUPMODE" = "2" ]; then
  DIRECTSAVEPATH=""
fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L147

Code:

      if [ ! "$DIRLINK" ];then
       if [ -h "/initrd${SAVE_LAYER}${ONESPEC}" ];then #120107
        DIRLINK="`readlink -m "/initrd${SAVE_LAYER}${ONESPEC}" | sed -e "s%/initrd${SAVE_LAYER}%%"`" #SAVE_LAYER: see /etc/rc.d/PUPSTATE. 120107
        xDIRLINK="`readlink "/initrd${SAVE_LAYER}${ONESPEC}"`" #120107
       fi
      fi
      if [ "$DIRLINK" ];then
       if [ -d "$DIRLINK"  ];then
        if [ "$DIRLINK" != "${ONESPEC}" ];then #precaution.
         mkdir -p "${DIRECTSAVEPATH}${DIRLINK}" #120107
         cp -a -f --remove-destination ${DIRECTSAVEPATH}"${ONESPEC}"/* "${DIRECTSAVEPATH}${DIRLINK}/" #ha! fails if put double-quotes around entire expression.
         rm -rf "${DIRECTSAVEPATH}${ONESPEC}"
         if [ "$DIRECTSAVEPATH" = "" ];then
          ln -s "$xDIRLINK" "${ONESPEC}"
         else
          DSOPATH="`dirname "${DIRECTSAVEPATH}${ONESPEC}"`"
          DSOBASE="`basename "${DIRECTSAVEPATH}${ONESPEC}"`"
          rm -f "${DSOPATH}/.wh.${DSOBASE}" #allow underlying symlink to become visible on top.
         fi
fi

/woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L443

The key difference here is that in puppy they are using the "readlink" function, so the puppy package manager gets the real path whereas you are using a symlink which points to the "direct save path" The consequence is that when you you extrat the package archive via the tar command, you are extracting the package archive to the symlink rather than to the actual directory.

Code:

tar --absolute-names $tarops "./${PKGNAME}.tar.$TAREXT" ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog

/master/usr/sbin/pkg#L4563

Now long ago (perhaps prior to when woof-CE was on github there was the following fix:
Code:

#120102 install may have overwritten a symlink-to-dir.
#120107 rerwin: need quotes around some paths in case of space chars. remove '--unlink-first' from tar (was introduced 120102, don't think necessary).

woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L26

I'm guessing that the --unlink-first option was used in conjuction with the "-h" option:
Quote:

-h, --dereference
follow symlinks; archive and dump the files they point to

http://murga-linux.com/puppy/viewtopic.php?p=1030949#1030949

However, I assume that neither of these options are necessary now that the puppy package manager is using the realpath of pup_ro1

Anyway, this error only manifested itself (see post) when I broke either "busybox mount" or mount.aufs (the latter used in mount-FULL) and therefore, I don't know yet whether or not this is an actual bug in "pkg".

Edit:

The following section of the "tar manpage" suggests that /initrd/pup_ro1 will be overwritten if the following option isn't supplied (testing required to verify):

Quote:

--keep-directory-symlink
Don't replace existing symlinks to directories when
extracting.

http://man7.org/linux/man-pages/man1/tar.1.html

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 1941

PostPosted: Wed 26 Jun 2019, 12:33    Post subject:  

sc0ttman wrote:
s243a wrote:
In the meantime I took a stab at this (commit e4b000bda57f13a6eba5a2f3fb2bd43e856189ce)

I probably made some changes which weren't necessary.


You should probably define the right package dir at the top of the /usr/sbin/pkg script as a variable,
then use that variable later throughout the script..

Or even better, define it in the main config RC script: ~/.pkg/pkgrc

Then it can be set to various different places - easier to run Pkg in chroot, ssh, local folder for unit tests Wink , etc..


Okay, I took your suggestion (see commit: ea299b8ff3054173df6b5725bc8b7eef658762c6). I put the following

Code:

[ -z "$PKGS_DIR" ] && PKGS_DIR="$(realpath "${HOME}/.packages")"

/woof-code/rootfs-packages/PKG/usr/sbin/pkg#L60

after "~/.pkg/pkgrc" is sourced. Therefore this folder can be defined in ~/.pkg/pkgrc.

I left in the previous realpath statements I added. I don't think they hurt anything but are probably redundant.

**Note that this last change was done by a simple string replace and I haven't done any testing yet.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 1941

PostPosted: Wed 26 Jun 2019, 21:14    Post subject:  

s243a wrote:
sc0ttman wrote:
s243a wrote:
I'm not sure if the following is being done by ppm v3.0 or sc0tmann's pkg but some process seems to be moving all my files out of the /usr/lib/i386-linux-gnu/ folder....

...This time the whole i386-linux-gnu folder seems to be replaced with a symlink. I know this is standard puppy to do this but maybe there is a reason that Debian/Devaun creates the i386-linux-gnu folder (e.g. to separate architectures on multi architecture systems).

Maybe pkg and PPM..? Don't quote me on that..

But this code in Pkg expects a symlink:



Your are correct. This is related to the "muti-arch hack" of the puppy package manager. I oppened an issue on github to discuss whether said muti-arch hack is a good or bad thing:

https://github.com/puppylinux-woof-CE/woof-CE/issues/1475

http://murga-linux.com/puppy/viewtopic.php?p=1030959#1030959

I found the code that sc0tmann is talking about here:

Code:


               # woofce: NO_MULTIARCH_SYMLINK=1 (DISTRO_SPECS)
               if [ -z "$NO_MULTIARCH_SYMLINK" ] ; then
            if [ -f $HOME/.packages/${PKGNAME}.files ];then
              pkg_has_archdir="$(grep -m1 "$DISTRO_ARCHDIR" "$HOME/.packages/${PKGNAME}.files")"
            fi

                  # Workaround to avoid overwriting the $DISTRO_ARCHDIR symlink.
                  if [ "$DISTRO_ARCHDIR" != "" -a -f $HOME/.packages/${PKGNAME}.files -a "$pkg_has_archdir" != "" ]; then
                    mkdir -p /tmp/$PKGNAME
                    rm -rf /tmp/$PKGNAME/*
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb /tmp/$PKGNAME/ 2> $TMPDIR/$SELF-cp-errlog
                    for f in $(find /tmp/$PKGNAME \( -type f -o -type l \))
                    do
                       xpath=$(echo "$f" |  cut  -f 4-30 -d "/" | sed "s/$DISTRO_ARCHDIR\///")
                       mkdir -p ${DIRECTSAVEPATH}/$(dirname "$xpath")
                       cp -a "$f" ${DIRECTSAVEPATH}/$(dirname "$xpath")/
                    done
                    rm -rf /tmp/$PKGNAME &>/dev/null
                  else
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                  fi
                else
                  dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L4614

BTW: according to wdlkmpx, the "multi-arch hack" has been removed:
https://github.com/puppylinux-woof-CE/woof-CE/issues/1475#issuecomment-505873406

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 9 of 9 [128 Posts]   Goto page: Previous 1, 2, 3, ..., 7, 8, 9
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1039s ][ Queries: 12 (0.0378s) ][ GZIP on ]