Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 10 Jul 2020, 19:31
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Pkg - CLI package manager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 10 of 11 [152 Posts]   Goto page: Previous 1, 2, 3, ..., 8, 9, 10, 11 Next
Author Message
s243a

Joined: 02 Sep 2014
Posts: 2626

PostPosted: Sun 25 Aug 2019, 00:01    Post subject:  

Here is the first relevant instance of concurancy that I found related to the previous post:

Code:

list_deps "$pkg_name" > ${TMPDIR}/${pkg_name}_dep_list &

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L5458

I've been picking out what looks to be one instance of the function list_deps. Because of concurancy my debugging output is mixed up with various subprocesses but in most cases I can follow where there is a single "+" to try to follow the logic:
https://pastebin.com/PTXYqEx9 (incomplete output...will be updated)

I'm having more troubles understanding this "HIDE_BUILTINS" variable as it isn't obvious to me by the name what affect it should have in all cases. I think there should almost be seperate enviornmental variables for different functions. For instance, we could have another variable called "REINSTALL_BUILTINS" which will reinstall builts for all dependencies and if we only want to reinstall the builting of a single package, maybe a sinilar name for the enrironmental variable but without the trailing "S". To me the "HIDE_BUILTINS" enviornmental variable should only apply to when we are listing variables.

Anyway, I'm going to postepone this renaming idea but I want to make the following change. I want to change:

Code:

            if [ "$HIDE_BUILTINS" = true ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L6077
to

Code:

            if [ "$HIDE_BUILTINS" = true ] || [ -z "$HIDE_BUILTINS" ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi


because by default we shouldn't install a package if it is a builtin. I think I'll follow the same logic in the function
list_all_installed_pkgs(), which I mentioned in a previous post.

Regarding the concurrancy (see prevoius post and the start of this current post)

I'm conerned that various instances of the function list_deps() will overwrite (or even delete) $TMPDIR/installed_pkgs prior to it being used to filterout already installed packages (see post).

My suggest is that the function list_all_installed_pkgs(). should only be installed once and then each time a package is installed it should be appended to the list and than sorted.

A simple way to do this is check for the existance of $TMPDIR/installed_pkgs and only create said file if it doesn't exist. I think I'll try this. Smile

Edit: The actual changes that I made related to my comments in this post are discussed at: http://murga-linux.com/puppy/viewtopic.php?p=1035370#1035370

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2806
Location: UK

PostPosted: Thu 05 Sep 2019, 10:08    Post subject:  

I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution... I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point.. Keep up the good work!
_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2626

PostPosted: Sat 14 Sep 2019, 22:00    Post subject:  

sc0ttman wrote:
I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution...

I think I can get some speed improvements by better use of concurrency.
Quote:
I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point..

The silver lining here is that it will give me some time to flesh out some ideas. I hope that I can do it fast enough.
Quote:
Keep up the good work!

Thanks for the encouragement Smile

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2626

PostPosted: Sat 14 Sep 2019, 22:09    Post subject:  

One thing that I'm working on is improved use of aliases. My understanding is that pkg currently only uses aliases for listing available packages
Code:

list_pkg_names(){               
...
      echo $ALIAS_LIST | while read ALIAS
      do
...
ALIAS_RES="`LANG=C cut -f1 -d'|' ${HOME}/.packages/${REPOFILE} 2>/dev/null | grep "^$ALIAS"`" #

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L2078

and not with the --get option. I would like to make it so people can use aliases with the --get option but first some simpler fixes.

I don't believe that in the way pkg uses aliases that it checks properly for a terminating character "either a comma ',' or a line ending. We can check for this by using alternatives which are available in grep if we use extended regular expressions (i.e. the -E switch). The pattern to match a comma or a line termination character is:
Code:

'([,]|$)'


I'm writing a list_all_aliases() function. It is probably more complex than it needs to be and is only partially fleshed out. The file in the above link stripped out a lot of things from pkg to make debugging easier (i.e. it's a testing file so the trace output will be smaller).

I programmed it so that the alias list can be either a variable or a file. If no argument is suppled for the alias list it first checks for the existance of the file "${TMPDIR}/pkg_aliases" and if it doesn't exist it uses the variable, "$PKG_NAME_ALIASES". With my test code the file exists but the function is written in such a way not to create a tmp file if it doesn't exist.

Here is the code which produces the alias list for a given dependency.

Code:

  if [ -z "$ALIAS_LIST" ]; then
    if [ -f ${TMPDIR}/pkg_aliases ]; then
      ALIAS_LIST="${TMPDIR}/pkg_aliases"
    else
      ALIAS_LIST="$PKG_NAME_ALIASES"
    fi
  fi
  if [ -e "$ALIAS_LIST" ]; then #ALIAS_LIST is a file
       ALIAS_LIST=( `grep -E "$subdep"'([,]|$)' "$ALIAS_LIST"  2>/dev/null | tr ',' ' '` )
  else #[[ "$ALIAS_LIST" = *,* ]]; then #ALIAS_LIST is string
    ALIAS_LIST=( $(echo "$ALIAS_LIST" | tr ' ' '\n' | grep -E "$subdep"'([,]|$)'  2>/dev/null | tr ',' '\n') )
  fi

https://pastebin.com/F8UL4ius

bellow that there is some checking code (e.g. does the package exist in the package database), but we can add more checks if we want like the wget spider function. However, if we do we first have to ensure the package databases are up to date.

I eventually plan to have the list_all_aliases() function be able to also download packages (provide a suitable option is supplied to the function). This way we can start downloading as soon as we find a suitable alias. I plan some kind of process management (based on named pipes) so that the user can limit the number of process if they desire. If the process limit is reached then the remaining work to do will be stored in a file.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2626

PostPosted: Sun 17 Nov 2019, 04:47    Post subject:  

What is the intent of "sources-user". It doesn't seem to be documented in the wiki.
_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2806
Location: UK

PostPosted: Mon 18 Nov 2019, 14:05    Post subject:  

Quote:
What is the intent of "sources-user". It doesn't seem to be documented in the wiki.


User added repos are listed in there, if they are not PPA, Debian or Slackware repos.

(in other words, if they are 3rd party repos, created using Pkg, and added by the user as an additional or third party repo), then they get listed in "sources-user".

EDIT:

more info, might make sense:

Pkg tries to use the existing repo config files of other pkg managers (not only PPM) where possible.. So

- user-added Debian/Ubuntu/PPA repos go in /etc/sources.list (Pkg and apt-get read/write to this file)

- user-added Slackware repos go in /etc/slackpkg/mirrors (Pkg and slackpkg read/write to this file)

- other user-added repos will be Puppy or Pkg repos, so they go in ~/.pkg/sources-user (only Pkg uses this file)


...this means Pkg will "see" the repos added by other package managers...
And can then convert them to Puppy supported formats Smile

Just do `pkg repo-update` Cool

So users could copy over the config files of their user-added repo from Ubuntu/Debian/Slackware,
and Pkg would be able to "install" them, to be used like any other repo.

PPM can't do that Wink

(but then PPM doesn't let you add your own repos either, without hacking it)

_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search
Back to top
View user's profile Send private message 
rockedge


Joined: 11 Apr 2012
Posts: 1870
Location: Connecticut, United States

PostPosted: Sat 28 Dec 2019, 12:28    Post subject:  

Hello sc0ttman!

I have almost completed a script that will install Hiawatha, mariadb, PHP7 and ZoneMinder, using pkg for most of the heavy lifting. With a few more adjustments it should set up a zoneminder system from a stock Bionic64.

I have though a small question. The Ubuntu PPA for Hiawatha doesn't seem to be up to date so I grab the Debian build with wget and then use
Code:
wget https://files.tuxhelp.org/hiawatha/hiawatha_10.10_amd64.deb
pkg deb2pet hiawatha_10.10_amd64.deb
petget hiawatha_10.10_amd64.pet


but using petget requires user input and it seems I can not use any option to auto confirm and run petget fully automatic.

So how can I use wget to grab the .deb and use pkg to install it?

will this work?
Code:
cd /root/Downloads
wget https://files.tuxhelp.org/hiawatha/hiawatha_10.10_amd64.deb
pkg install hiawatha_10.10_amd64.deb


I'll give it a try!
Back to top
View user's profile Send private message Visit poster's website 
mavrothal


Joined: 24 Aug 2009
Posts: 3108

PostPosted: Sun 29 Dec 2019, 13:40    Post subject:  

Used the Pkg GUI in a fresh Dpupbuster64 RC2 (GPkgdialog 0.5 - pkg 1.9.23) and I think the gui may be still a bit rough.
Starting gives a blank window and you must click the checkmark box to get the repo. This takes 8-9 sec to process in an i7 machine (!) plenty of time to get you worried… (it really feels/is very slow).

Then clicking on an entry opens the options window and if you choose “Add to Queue” there is no way to process the queue without clicking on another or the same entry.
Not showing what is in the queue you may be adding the same entry multiple times and interestingly is trying to install it multiple times.

It would also appear that the dependencies installation is independent of the main package. In one case the intended package (audacious_3.10.1-dmo1) could not be found but all the dependencies were installed regardless. More important the failure appeared only in the beginning and was very easy to miss as it scrolls up in the install window.
Interesting the “installed packages” tab does not update after installation and you must restart GPkgdialog to populate it, so you have no way to know what happened.

Searching is also cumbersome. It defaults to only searching the current repo. Even if you change it in the settings to search all as soon as you define/change repo it automatically shifts back to searching only the current repo.

Another thing that is questionable is the “Build from source” tab that is empty with no obvious way to populate it from the GUI (and rather tricky from cli with its long syntax requirements)
In general there is no way to update repo or sources from the GUI.

I do not know if what is in dpupbuster is the current versions and I can appreciate that pkg was developed as a cli package manager and the GUI is an add-on, but is sitting on the desktop of dpupbuster (and new puppies I guess) and in its current form is not doing justice to the code underneath, I think.

PS.: sorry no time to delve into it Embarassed

_________________
== Here is how to solve your Linux problems fast ==
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2806
Location: UK

PostPosted: Sun 29 Dec 2019, 15:10    Post subject:  

mavrothal wrote:
Used the Pkg GUI in a fresh Dpupbuster64 RC2 (GPkgdialog 0.5 - pkg 1.9.23) and I think the gui may be still a bit rough.

Yes, the Gtkdialog GUI was a "rough sketch".. I never use it. It needs work.

mavrothal wrote:
Starting gives a blank window and you must click the checkmark box to get the repo.

This is wldmpks doing from a recent commit .. I don't know why he did that.

mavrothal wrote:
This takes 8-9 sec to process in an i7 machine (!) plenty of time to get you worried… (it really feels/is very slow).

On my old thinkpad (i3), loading the stretch-main repo (49,000 packages) NOT including DEV, NLS and DOC packages takes a similar time..

mavrothal wrote:
Then clicking on an entry opens the options window and if you choose “Add to Queue” there is no way to process the queue without clicking on another or the same entry.
Not showing what is in the queue you may be adding the same entry multiple times and interestingly is trying to install it multiple times.

Yeah it probably needs removing or fixing.

mavrothal wrote:
It would also appear that the dependencies installation is independent of the main package. In one case the intended package (audacious_3.10.1-dmo1) could not be found but all the dependencies were installed regardless.


GPkgdialog (the frontend itself) calls Pkg (the backend) in 2 steps ... If the first (getting the main package fails, we need to skip the 2nd step (getting the deps)..

mavrothal wrote:
More important the failure appeared only in the beginning and was very easy to miss as it scrolls up in the install window.
Interesting the “installed packages” tab does not update after installation and you must restart GPkgdialog to populate it, so you have no way to know what happened.

Yep.. Also, the installed window is filtered by the search, which is only visible in the main window - bad design, and something that also needs to be fixed.

mavrothal wrote:
Searching is also cumbersome. It defaults to only searching the current repo. Even if you change it in the settings to search all as soon as you define/change repo it automatically shifts back to searching only the current repo.


You need to change the settings in Pkg to search all repos... Search should prob be all repos by default, like `pkg names-all <pkgname>`.

mavrothal wrote:
Another thing that is questionable is the “Build from source” tab that is empty with no obvious way to populate it from the GUI (and rather tricky from cli with its long syntax requirements)
In general there is no way to update repo or sources from the GUI.

Yeah, if petbuild is chosen as BUILDTOOL, and its not setup, we should have a button in the GUI to kick start that process.

I presume you didn't try SboPackage or BuildPet as the build tools? It will be different in those cases... I can't remember what the GUI shows if Sbo package

mavrothal wrote:
I do not know if what is in dpupbuster is the current versions and I can appreciate that pkg was developed as a cli package manager and the GUI is an add-on, but is sitting on the desktop of dpupbuster (and new puppies I guess) and in its current form is not doing justice to the code underneath, I think.

PS.: sorry no time to delve into it Embarassed

No you're right.. The GUI is a toy ... A demo of what Pkg can do...

It was never intended to be "the" GUI replacement for PPM ... I actually spent more time on Pkgdialog (the ncurses/Xdialog frontend).. But even that is not "complete", still more of a demo..

I'm happy for someone to make a newer, better GUI for Pkg (like a nice web based one running off localhost, that looks like a nice proper "software center"...

_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search

Last edited by sc0ttman on Sun 29 Dec 2019, 15:18; edited 1 time in total
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2806
Location: UK

PostPosted: Sun 29 Dec 2019, 15:14    Post subject:  

rockedge wrote:
Hello sc0ttman!

Yo.

rockedge wrote:
I have almost completed a script that will install Hiawatha, mariadb, PHP7 and ZoneMinder, using Pkg for most of the heavy lifting. With a few more adjustments it should set up a zoneminder system from a stock Bionic64.

Nice Smile

rockedge wrote:
I have though a small question. The Ubuntu PPA for Hiawatha doesn't seem to be up to date so I grab the Debian build with wget and then use
Code:
wget https://files.tuxhelp.org/hiawatha/hiawatha_10.10_amd64.deb
pkg deb2pet hiawatha_10.10_amd64.deb
petget hiawatha_10.10_amd64.pet


but using petget requires user input and it seems I can not use any option to auto confirm and run petget fully automatic.

So how can I use wget to grab the .deb and use pkg to install it?

will this work?
Code:
cd /root/Downloads
wget https://files.tuxhelp.org/hiawatha/hiawatha_10.10_amd64.deb
pkg install hiawatha_10.10_amd64.deb


I'll give it a try!

Should do...

To install local packages using Pkg, you just do:

pkg install path/to/package.deb

or, if the file to install is already in the ~/pkg/ folder:

pkg install package.deb

_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search
Back to top
View user's profile Send private message 
rockedge


Joined: 11 Apr 2012
Posts: 1870
Location: Connecticut, United States

PostPosted: Sun 29 Dec 2019, 20:44    Post subject:  

Quote:
To install local packages using Pkg, you just do:

pkg install path/to/package.deb

or, if the file to install is already in the ~/pkg/ folder:

pkg install package.deb


Works! went very well. a few more test runs and I'll be posting the script.
Back to top
View user's profile Send private message Visit poster's website 
wiak

Joined: 11 Dec 2007
Posts: 2074
Location: not Bulgaria

PostPosted: Mon 30 Dec 2019, 01:33    Post subject:  

I always said that pkg (or similar commandline package manager) would be the future to Puppy development and that is certainly proving to be the case. GUI tools are all very well, but when it comes to dev utilities the GUI should always be the last thing added, and separate via an API, so different GUI can always be written and work with same underlying commandline util. When writing applications it is sometimes not so easy to separate the GUI from underlying functions but generally can be conveniently done to a greater or lesser extent and certainly best to always try.

wiak

_________________
WeeDogLinux forum: https://weedoglinux.rockedge.org/viewforum.php?f=4
Tiny Linux Blog: https://www.tinylinux.info/
Check Firmware: http://murga-linux.com/puppy/viewtopic.php?p=1022797
Back to top
View user's profile Send private message 
darry19662018

Joined: 31 Mar 2018
Posts: 731
Location: Rakaia

PostPosted: Mon 30 Dec 2019, 03:46    Post subject:  

Have to agree Wiak - found pkg to be an awesome package manager. To be honest the problems with ppm and check dependencies script have lead me to using Stretch Dog full time.
_________________
Puppy Linux Wiki: http://wikka.puppylinux.com/HomePage

https://freemedia.neocities.org/
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2806
Location: UK

PostPosted: Sun 12 Jan 2020, 12:05    Post subject: plugin support
Subject description: new feature coming soon
 

A new feature, coming soon:

Plugin support
Code here: https://gitlab.com/sc0ttj/Pkg/merge_requests/28

so far I have created a simple plugin system, with 2 plugins:

post_install/wings3d/00-wings3d_installdeps.sh

This plugin will install extra deps for wings3d, on Debian-based Pups only.



post_install/gimp/50_install_plugins.sh



This plugin will ask to install Gimp plugins, if --ask was given when installing Gimp.

_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search
Back to top
View user's profile Send private message 
josejp2424


Joined: 01 Aug 2010
Posts: 554

PostPosted: Sun 12 Jan 2020, 18:22    Post subject: Re: plugin support
Subject description: new feature coming soon
 

sc0ttman wrote:
A new feature, coming soon:

Plugin support
Code here: https://gitlab.com/sc0ttj/Pkg/merge_requests/28

so far I have created a simple plugin system, with 2 plugins:

post_install/wings3d/00-wings3d_installdeps.sh

This plugin will install extra deps for wings3d, on Debian-based Pups only.



post_install/gimp/50_install_plugins.sh



This plugin will ask to install Gimp plugins, if --ask was given when installing Gimp.

from when it will be available.
I want to run it on dpupbuster

Thanks sc0ttman for your great work.

_________________
Shiba Inu | Pupjibaro jessie | My Blog
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 10 of 11 [152 Posts]   Goto page: Previous 1, 2, 3, ..., 8, 9, 10, 11 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1116s ][ Queries: 12 (0.0299s) ][ GZIP on ]