Package categories help wanted

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

Package categories help wanted

#1 Post by BarryK »

My blog post says it all:

http://bkhome.org/blog2/?viewDetailed=00029

If anyone is interested, post here so that we don't have two people doing the same thing -- well, I am assuming that more than one person will be interested in doing this! Or even one for that matter.

I was thinking that this is something that someone can do who wants to contribute to Puppy development, but is not into shell/bacon/c or whatever programming, or anything system-level.
[url]https://bkhome.org/news/[/url]

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#2 Post by technosaurus »

all of this _could_ be accomplished by downloading the package, extracting the .desktop file and parsing it ... if it doesn't have a desktop file it would be pretty safe to assume building block. I wrote a similar script for pet packages that scottman used in akita last year.

it was along the lines of:

Code: Select all

for listing in `tar -tf package |grep desktop`; do tar -xf package listing; parse_func listing; done
I don't have that kind of bandwidth though
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

#3 Post by L18L »

technosaurus wrote:I don't have that kind of bandwidth though
I have some bandwidth, just give me parse_func and a little help 8)

------
edit 20:40
generating list of packages that have .desktop

0ad is 1st in HASdesktopFILE and is 1st in PKGS_HOMEPAGES
.
.
bouncy is 175th in HASdesktopFILE and is 843rd in PKGS_HOMEPAGES
.
and so on (wait some hours)
packages which have .desktop file: less than 20%

--
edited (110 minutes later)
--2012-11-13 21:17:54-- http://packages.ubuntu.com/precise/all/zziplib/filelist
Auflösen des Hostnamen »packages.ubuntu.com (packages.ubuntu.com)«.... 91.189.94.203
Verbindungsaufbau zu packages.ubuntu.com (packages.ubuntu.com)|91.189.94.203|:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: nicht spezifiziert [text/html]
Saving to: `filelist'

[ <=> ] 2.697 --.-K/s in 0s

2012-11-13 21:17:54 (7,63 MB/s) - `filelist' saved [2697]

2080 packages have desktop files
Attachments
HASdesktopFILE.gz
list of packages with desktop files
remove .gz
(20.82 KiB) Downloaded 940 times
Last edited by L18L on Tue 13 Nov 2012, 21:23, edited 9 times in total.

aragon
Posts: 1698
Joined: Mon 15 Oct 2007, 12:18
Location: Germany

#4 Post by aragon »

This shouldn't sound harsh and maybe i don't understand the problem but to to me building such a db sounds like a graet waste of lifetime.

If a puppy is based on other distros packages it needs to reflect the distros menu-categories or it needs a script that converts the offending categories to puppy-categories on installation.

Aragon

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#5 Post by technosaurus »

I almost forgot that debian/ubuntu keep a list of files in each package, which could help avoid unnecessary downloads.

@L18L - most of the code for parsing the desktop files is already written as part of jwm_menu_create, just have to remove the jwm xml bits and unneeded lines, but I don't have a good feel of what barry is checking against regarding package names from distro to distro. In actuality it would be easy enough to create a whole database template that only requires filling in the full package name and size for each distro and appending any that are missing from the template

@aragon, once the database is created, it should be applicable to most distros

@barry, I'm not sure how much the different distros vary in their package names and how you account for the different naming such as abiword-x.x.txz vs abiword-bin-x.x.deb, etc... if they all somehow listed the executable, that would be the easiest since I could just use the Exec= and Categories= lines from the desktop files, ... otherwise it would require some manipulation of the package name (I am assuming this is coded somewhere already)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

#6 Post by L18L »

technosaurus wrote:... debian/ubuntu keep a list of files in each package, which could help avoid unnecessary downloads.

@L18L - most of the code for parsing the desktop files is already written as part of jwm_menu_create, just have to remove the jwm xml bits and unneeded lines, but I don't have a good feel of what barry is checking against regarding package names from distro to distro. In actuality it would be easy enough to create a whole database template that only requires filling in the full package name and size for each distro and appending any that are missing from the template
I have grepped that filelist, results in 2080 packages (only :D ), see above.

Now I do not know how to "look" inside a .deb (extract the .desktop file) without installing

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#7 Post by BarryK »

all of this _could_ be accomplished by downloading the package, extracting the .desktop file and parsing it ... if it doesn't have a desktop file it would be pretty safe to assume building block. I wrote a similar script for pet packages that scottman used in akita last year.
technosaurus,
This already happens on a per-package basis in /usr/local/petget/installpkg.sh when a package is installed. The script tries to make sure the .desktop file "Categories" field is appropriate for the Puppy menu structure.
Doesn't always get it right though, which is why a 'categories.dat' would be nice to check against -- perhaps that file could be initially created with a script, then each entry manually checked for correctness -- but then it might be better to do it all manually, which was my original thinking, and why I suggested a non-developer would like this.
2080 packages have desktop files
L18L,
That's great, then probably a huge number of these are very exotic and could be ignored -- not put into categories.dat, meaning that the automatic category determination performed in debdb2pupdb will be deemed good enough.
If a puppy is based on other distros packages it needs to reflect the distros menu-categories or it needs a script that converts the offending categories to puppy-categories on installation.
aragon,
Puppy's menu structure is unique. .desktop files are modified at installation, as mentioned above. However, the automatic mechanism doesn't always get it right.
Also, the PPM needs to know the categories of all packages prior to installation, as they are displayed in categories, and icon thumbnails are assigned on a per-sub-category basis.
The categories that the PPM displays must fit the Puppy menu structure.
The categories are in every package database entry -- which is done by the -to-pup-db converters such as 'debdb2pupdb'.
[url]https://bkhome.org/news/[/url]

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#8 Post by BarryK »

technosaurus wrote:@aragon, once the database is created, it should be applicable to most distros
Yes. Well, applicable to all distros supported by Woof.
technosaurus wrote:@barry, I'm not sure how much the different distros vary in their package names and how you account for the different naming such as abiword-x.x.txz vs abiword-bin-x.x.deb, etc... if they all somehow listed the executable, that would be the easiest since I could just use the Exec= and Categories= lines from the desktop files, ... otherwise it would require some manipulation of the package name (I am assuming this is coded somewhere already)
PKGS_HOMEPAGES has generic names for packages. Ubuntu and Debian do use some odd names, and also split original packages up into many smaller ones -- however I have got around this problem as the Ubuntu/Debian database does also contain generic names.

Ubuntu/Debian keep packages in their online repos in folders named by the generic name, that's how I am able to obtain the generic name and check against lists such as categories.dat and PKGS_HOMEPAGES.

Though, the folder names in the Ubuntu/Debian online repos do not always correspond to the original package name, so it is not a perfect method.

Most other distros keep the original generic naming of packages, Slackware for example, so no problem.

Mageia is one example that splits up packages a lot and renames them. I don't yet know if the generic names can be determined for Mageia, probably some way to do it.
[url]https://bkhome.org/news/[/url]

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#9 Post by technosaurus »

L18L wrote:Now I do not know how to "look" inside a .deb (extract the .desktop file) without installing
undeb <pkg>,... but I forgot to send my undeb patch in to busybox for xz and I seem to have lost it, so you will need the full (and recent) package since they seem to be using xz in at least some if not all deb packages now.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

#10 Post by seaside »

technosaurus wrote:
L18L wrote:Now I do not know how to "look" inside a .deb (extract the .desktop file) without installing
undeb <pkg>,... but I forgot to send my undeb patch in to busybox for xz and I seem to have lost it, so you will need the full (and recent) package since they seem to be using xz in at least some if not all deb packages now.
Could just the desktop files be extracted by something like this

Code: Select all

tar -xfJ PACKAGE.tar --wildcards --no-anchored '*.desktop' 
Cheers,
s

aragon
Posts: 1698
Joined: Mon 15 Oct 2007, 12:18
Location: Germany

#11 Post by aragon »

Ok, sorry ... Now i see the points where the db is needed.

One more point about the menu-structure: it would be nice if the menu-points help and submenu shutdown would be implemented as desktop-files and desktop-catetegory. This would ease internationalization and the need to have static parts in templates would be superflous.

Aragon

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#12 Post by technosaurus »

aragon wrote:Ok, sorry ... Now i see the points where the db is needed.

One more point about the menu-structure: it would be nice if the menu-points help and submenu shutdown would be implemented as desktop-files and desktop-catetegory. This would ease internationalization and the need to have static parts in templates would be superflous.

Aragon
It may be too big for puppy but it isn't _that_ much to copy all of the desktop files to a separate directory like /usr/share/applications.uninstalled
and replace Exec=app with Exec=installer package_name (where "installer" is a wrapper around petget and removes the .uninstalled/*.desktop file after install) such that the whole thing can be parsed into an install menu ... we could also go through them and check the Icon= field to see if the icon exists and if not, make a symlink for it based on the Categories= field (jwm_tools already has a similar installer menu to this)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

#13 Post by L18L »

technosaurus wrote:...it isn't _that_ much to copy all of the desktop files to a separate directory...
Yes, and thank you for undeb that does work

BTW, there are more packages in that list that do NOT have .desktop
example:
389-admin
389-admin-console
389-adminutil
389-console
389-ds-base
389-ds-console
389-dsgw
accountsservice
aewm++
afterstep


wait some minutes...hours
script is running....

edit
the list NOTfound continues:

amarok
amiwm
antiword
apparmor
apper
arbtt
ark
at-spi2-core
audex
awesome
bangarang
bash-completion
basket
biomaj-watcher
blackbox
blinken
bluedevil

anyhow, seems to be all exotic

--edit
here is a very bad (buntu :lol: ) example
gamazons.desktop wrote:[Desktop Entry]
Encoding=UTF-8
Name=Gamazons
Name[ca]=Gamazons
Name[cs]=Gamazons
Name[da]=Gamazons
Name[de]=Gamazons
Name[el]=Gamazons
Name[es]=Gamazons
Name[et]=Gamazons
Name[fi]=Gamazons
Name[ga]=Gamazons
Name[gl]=Gamazons
Name[ja]=
Name[ko]=
Name[lt]=Gamazons
Name[ms]=Gamazons
Name[nl]=Gamazons
Name[no]=Gamazons
Name[pl]=Gamazons
Name[pt]=Gamazons
Name[pt_BR]=Gamazons
Name[ru]=
Name[sk]=Gamazons
Name[sl]=Gamazons
Name[sv]=
Name[vi]=Gamazons
Name[wa]=Gamazons
Name[zh_TW]=Gamazons
Comment=Gamazons game
Exec=gamazons
Icon=gnome-gamazons.png
Terminal=false
Type=Application
Categories=GNOME;Application;Game;BoardGame;
X-Desktop-File-Install-Version=0.3
OnlyShowIn=GNOME;
I will have to remove all translations in all .desktop files

----
...done

99 minutes
remove of translations (2 minutes) saved 785408 Bytes
directory containing 1902 desktop files is size 1983k
in attached archive 174k
Attachments
applications.tar.gz
(173.82 KiB) Downloaded 907 times

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

Package categories help wanted

#14 Post by L18L »

filtered packages that
- are already in categories.dat
- do not have an entry CATEGORIES= in .desktop (they exist!)
and sorted by category

remaining 1765 packages

The first one:
Accessibility;Utility;Settings;X-GNOME-Settings-Panel;X-GNOME-PersonalSettings; myunity
In /etc/xdg/menu/hierarchy we find Accessibility
Desktop :settings X-DesktopCountry,Accessibility
Thus this package myunity belongs to
Category Desktop
Subcategory Settings

Now myunity has to be inserted into the line containing Desktop settings in file categories.dat:
CONST PKG_CAT_Desktop_settings$=" jwmconfig2 myunity pupx xclock xkbconfigmanager "
Any volunteers?
Or split the work?
-multimedia
-games
-...

--
edit
... and the last one is:
X-XFCE;Utility; xfprint-manager
:D
Attachments
Categories_package.sorted.gz
remove extension .gz
(55.88 KiB) Downloaded 831 times

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#15 Post by technosaurus »

@L18L does it look like we need to do something like:
echo Package=$PACKAGENAME >> package/.../*.desktop
so we have a basis for determining the package name, or are most fairly straight forward

I am stuck in windows mode until I finish my programming languages project, so I can only give guidance right now, but if you open up jwm_menu_create (from jwm_tools) all of the language is there to generate all subcategories, all that needs to be done is:
set the subcategories variable to true (i forget the exact var name)
change the directory path from /usr/share/applications to the appropriate dir. remove the unneeded jwm markup
change the output to just give the format that Barry needs, which is ???
(p.s. my sub-category names may slightly differ but that is easily fixed by humans pretty quickly)
I assume that output could be:
category|subcategory|packagea packageb

(note to self: may need tolower so we can use strstr instead of needing strcasestr which is slower)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

Package categories help wanted

#16 Post by L18L »

Let's start with Fun :lol:
s5.sed wrote:s/Game; /Fun_Sub /
s/Game;ActionGame; /Fun_adventure /
s/Game;Action; /Fun_adventure /
s/Game;BlocksGame; /Fun_Sub /
s/Game;Blocks; /Fun_Sub /
s/Game;AdventureGame; /Fun_adventure /
s/Game;Adventure; /Fun_adventure /
s/Game;ArcadeGame; /Fun_arcade /
s/Game;Arcade; /Fun_arcade /
s/Game;BoardGame; /Fun_boardgame /
s/Game;Board; /Fun_boardgame /
s/Game;CardGame; /Fun_cardgame /
s/Game;Card; /Fun_cardgame /
s/Game;KidsGame; /Fun_Sub /
s/Game;LogicGame; /Fun_puzzle /
s/Game;RolePlaying; /Fun_adventure /
s/Game;Simulation; /Fun_Sub /
s/Game;SportsGame; /Fun_Sub /
s/Game;StrategyGame; /Fun_puzzle /
s/Game;Strategy; /Fun_puzzle /

Code: Select all

sed -f s5.sed Categories_package.sorted | sort > xxx
mv xxx Categories_package.sorted
has given puppy category and subcategory to 330 Fun packages

The sad rest is:
Game;Emulator; bsnes
Game;Emulator; dosbox
Game;Emulator; kcemu
Game;Emulator; rlvm
Game;FileTools; snespurify
Game;GPE; gpe-julia
Game;Music;Education linthesia
dosbox a game is not a joke. Is it?
So what now? Silently forget that rest?

@technosaurus, I think that I have now found one way to continue.
Inserting package names into categories.dat will be be the very last step....

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#17 Post by BarryK »

L18L,
Ha ha, 'dosbox' a game :lol:

Well, yesterday I did a test Precise build, and the 'debdb2pupdb' utility created this package database entry for dosbox:

Code: Select all

dosbox_0.74-2|dosbox|0.74-2||Graphic|2468K|pool/universe/d/dosbox|dosbox_0.74-2_i386.deb|+libasound21.0.24.1,+libc6&ge2.7,+libgcc1&ge4.1.1,+libgl1-mesa-glx,+libpng12-0&ge1.2.13-4,+libsdl-net1.2,+libsdl-sound1.2&ge1.0.1,+libsdl1.2debian&ge1.2.10-1,+libstdc++6&ge4.6,+libx11-6,+zlib1g&ge1.1.4|x86 emulator with Tandy/Herc/CGA/EGA/VGA/SVGA graphics sound and DOS|ubuntu|precise||
...assigned it as "Graphic" category. It would have picked up on that keyword "graphics" in the description and assigned the category -- obviously it has done that before discovering "emulator".

That's one reason a more fleshed-out categories.dat will be very helpful. debdb2pupdb will check that first, failing that can then fall back to the somewhat-less-than-perfect method of checking description keywords.

Thanks for all the work you are doing on this, it is great!
[url]https://bkhome.org/news/[/url]

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#18 Post by BarryK »

L18L,
while putting pkgs into categories.dat, or preparing to do so, if you think that there is a bunch of packages that really should have another sub-category, we can easily add that.

For example, maybe many of the top-level categories should have a "utility" sub-category. For example:

Code: Select all

CONST PKG_CAT_Business_Sub$=" "
CONST PKG_CAT_Business_spreadsheet$=" gnumeric "
CONST PKG_CAT_Business_planner$=" planner "
CONST PKG_CAT_Business_finance$=" acct expensetracker grisbi homebank moneymanagerx "
CONST PKG_CAT_Business_calc$=" calcoo cgtkcalc galculator gmeasures ycalc xcalc "
There may be some utility-type business apps, for which we should have "PKG_CAT_Business_utility" variable.

Note, in the menu we don't have to have horizontal lines between all of the sub-categories. Some can be grouped. So, these variables are for organizational purposes, actual display in the menu is flexible.

In fact, in Precise there are no horizontal bars between "spreadsheet" and "finance".
[url]https://bkhome.org/news/[/url]

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

#19 Post by L18L »

BarryK wrote:... a more fleshed-out categories.dat will be very helpful. debdb2pupdb will check that first, failing that can then fall back to the somewhat-less-than-perfect method of checking description keywords...
This dosbox example has made it very clear what to do:

Assign exactly 1 (puppy) category/subcategory to each package.
Existing Categories (in .desktop file) may have more than 1 category, example:
dosbox.desktop wrote:[Desktop Entry]
Type=Application
Name=DOSBox Emulator
Comment=Run old DOS applications
Icon=dosbox
Exec=dosbox
Terminal=false
Categories=Game;Emulator;
What can be done automatically?
Everything that has just 1 (sub)category.

I have continued to use some sed snippets and came to about
750 which cannot be automatically processed.
But before I publish this I will repeat this process to make sure that it is reproducible.

Then we will see if
there is a bunch of packages that really should have another sub-category,

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#20 Post by sunburnt »

While you`re writing specialized scripts to parse the desktop files...

Why not generate an app. library dependency data base with ldd.?

I`ve said that statistics on library usage is a key item for O.S. design.
Puppy`s installed libraries are probably a good list of common ones.

But many apps. need more, so which libs. are only used 1 or 2 times?

Include the commonly used libs. in Puppy or make an add-on SFS file.
The rarely used odd libs. just include with the apps. It`s good design!

# Sorry for stepping on your thread Barry... The thought struck me. :wink:

Post Reply