Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 15 Nov 2019, 13:59
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Pkg - CLI package manager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 10 of 10 [139 Posts]   Goto page: Previous 1, 2, 3, ..., 8, 9, 10
Author Message
s243a

Joined: 02 Sep 2014
Posts: 2147

PostPosted: Sun 25 Aug 2019, 00:01    Post subject:  

Here is the first relevant instance of concurancy that I found related to the previous post:

Code:

list_deps "$pkg_name" > ${TMPDIR}/${pkg_name}_dep_list &

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L5458

I've been picking out what looks to be one instance of the function list_deps. Because of concurancy my debugging output is mixed up with various subprocesses but in most cases I can follow where there is a single "+" to try to follow the logic:
https://pastebin.com/PTXYqEx9 (incomplete output...will be updated)

I'm having more troubles understanding this "HIDE_BUILTINS" variable as it isn't obvious to me by the name what affect it should have in all cases. I think there should almost be seperate enviornmental variables for different functions. For instance, we could have another variable called "REINSTALL_BUILTINS" which will reinstall builts for all dependencies and if we only want to reinstall the builting of a single package, maybe a sinilar name for the enrironmental variable but without the trailing "S". To me the "HIDE_BUILTINS" enviornmental variable should only apply to when we are listing variables.

Anyway, I'm going to postepone this renaming idea but I want to make the following change. I want to change:

Code:

            if [ "$HIDE_BUILTINS" = true ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L6077
to

Code:

            if [ "$HIDE_BUILTINS" = true ] || [ -z "$HIDE_BUILTINS" ];then
              is_builtin=`is_builtin_pkg "$subdep"`
              [ "$is_builtin" = true ] && continue
              is_in_devx=`is_devx_pkg "$subdep"`
              [ "$is_in_devx" = true ] && continue
            fi


because by default we shouldn't install a package if it is a builtin. I think I'll follow the same logic in the function
list_all_installed_pkgs(), which I mentioned in a previous post.

Regarding the concurrancy (see prevoius post and the start of this current post)

I'm conerned that various instances of the function list_deps() will overwrite (or even delete) $TMPDIR/installed_pkgs prior to it being used to filterout already installed packages (see post).

My suggest is that the function list_all_installed_pkgs(). should only be installed once and then each time a package is installed it should be appended to the list and than sorted.

A simple way to do this is check for the existance of $TMPDIR/installed_pkgs and only create said file if it doesn't exist. I think I'll try this. Smile

Edit: The actual changes that I made related to my comments in this post are discussed at: http://murga-linux.com/puppy/viewtopic.php?p=1035370#1035370

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2741
Location: UK

PostPosted: Thu 05 Sep 2019, 10:08    Post subject:  

I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution... I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point.. Keep up the good work!
_________________
Pkg, mdsh, Woofy, Akita, VLC-GTK, Search
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2147

PostPosted: Sat 14 Sep 2019, 22:00    Post subject:  

sc0ttman wrote:
I'll happily look into any clean ups, simplifications, logic fixes, bug fixes and especially speed improvements that you can make to the dep resolution...

I think I can get some speed improvements by better use of concurrency.
Quote:
I'm keeping an eye on this thread... Yet to get back to Pkg for a while now, but will do at some point..

The silver lining here is that it will give me some time to flesh out some ideas. I hope that I can do it fast enough.
Quote:
Keep up the good work!

Thanks for the encouragement Smile

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2147

PostPosted: Sat 14 Sep 2019, 22:09    Post subject:  

One thing that I'm working on is improved use of aliases. My understanding is that pkg currently only uses aliases for listing available packages
Code:

list_pkg_names(){               
...
      echo $ALIAS_LIST | while read ALIAS
      do
...
ALIAS_RES="`LANG=C cut -f1 -d'|' ${HOME}/.packages/${REPOFILE} 2>/dev/null | grep "^$ALIAS"`" #

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L2078

and not with the --get option. I would like to make it so people can use aliases with the --get option but first some simpler fixes.

I don't believe that in the way pkg uses aliases that it checks properly for a terminating character "either a comma ',' or a line ending. We can check for this by using alternatives which are available in grep if we use extended regular expressions (i.e. the -E switch). The pattern to match a comma or a line termination character is:
Code:

'([,]|$)'


I'm writing a list_all_aliases() function. It is probably more complex than it needs to be and is only partially fleshed out. The file in the above link stripped out a lot of things from pkg to make debugging easier (i.e. it's a testing file so the trace output will be smaller).

I programmed it so that the alias list can be either a variable or a file. If no argument is suppled for the alias list it first checks for the existance of the file "${TMPDIR}/pkg_aliases" and if it doesn't exist it uses the variable, "$PKG_NAME_ALIASES". With my test code the file exists but the function is written in such a way not to create a tmp file if it doesn't exist.

Here is the code which produces the alias list for a given dependency.

Code:

  if [ -z "$ALIAS_LIST" ]; then
    if [ -f ${TMPDIR}/pkg_aliases ]; then
      ALIAS_LIST="${TMPDIR}/pkg_aliases"
    else
      ALIAS_LIST="$PKG_NAME_ALIASES"
    fi
  fi
  if [ -e "$ALIAS_LIST" ]; then #ALIAS_LIST is a file
       ALIAS_LIST=( `grep -E "$subdep"'([,]|$)' "$ALIAS_LIST"  2>/dev/null | tr ',' ' '` )
  else #[[ "$ALIAS_LIST" = *,* ]]; then #ALIAS_LIST is string
    ALIAS_LIST=( $(echo "$ALIAS_LIST" | tr ' ' '\n' | grep -E "$subdep"'([,]|$)'  2>/dev/null | tr ',' '\n') )
  fi

https://pastebin.com/F8UL4ius

bellow that there is some checking code (e.g. does the package exist in the package database), but we can add more checks if we want like the wget spider function. However, if we do we first have to ensure the package databases are up to date.

I eventually plan to have the list_all_aliases() function be able to also download packages (provide a suitable option is supplied to the function). This way we can start downloading as soon as we find a suitable alias. I plan some kind of process management (based on named pipes) so that the user can limit the number of process if they desire. If the process limit is reached then the remaining work to do will be stored in a file.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 10 of 10 [139 Posts]   Goto page: Previous 1, 2, 3, ..., 8, 9, 10
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1047s ][ Queries: 12 (0.0364s) ][ GZIP on ]