Universal package database format

[technosaurus]

Surely you didn't think I don't know how to list files in a package...? The list of files contained in a package is the first, most basic bit of data we can have. Can you imagine a package manager, or anything that calls itself a package manager which would be *unable* to produce a package fiile-list or remove a package. Oh wait, there is a distro that is famous for this... Still, I have some theoretical examples that are way more absurd.

Half of my posts are more of a note to myself than anything else. I forget simple things like that all the time (I still have to use stackoverflow almost every time I do a complex regex or any non-stanard printf format) The rest of the post follows my broken train of thought that if we want to ensure each file is un-corrupted, that a tarball listing won't be enough, since it doesn't do checksums. I have been browsing the repositories of different distros and really like the idea of having the package info in a separate file from the package; tiny core, for instance, uses squashfs packages and has the package info in a separate file. That seems to be the simplest way. I like simple.

I really re-posted here just to include the text of an email I just got from the rox-users mailing list -some very interesting and pertinent topics they are going to explore:

Message: 1
Date: Sat, 4 Nov 2017 14:10:40 +0100
From: Liam Proven <lproven@gmail.com>
To: ROX Desktop Mailing List <rox-users@lists.sourceforge.net>
Subject: [rox-users] FosDem
Message-ID:
<CAMTenCH=dOEUh45U3GBMbTU=CO+PMjsZTtH-xT_qxucwrKcaCA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

Will any ROX people be at FOSdem? Prime material for a presentation, I think...

Liam Proven ? Profile: https://about.me/liamproven
Email: lproven@cix.co.uk ? Google Mail/Talk/Plus: lproven@gmail.com
Twitter/Facebook/Flickr: lproven ? Skype/LinkedIn/AIM/Yahoo: liamproven
UK: +44 7939-087884 ? ?R/WhatsApp/Telegram/Signal: +420 702 829 053

---------- Forwarded message ----------

Online at:
https://lists.fosdem.org/pipermail/fosd ... 02648.html
...

I wonder if a live video feed / recorded lightning talk would be a possibility, since there's no way I could make it to Brussels. Rox specifically could benefit from the Mime-type + package manager integration parts. I'd also like to integrate the suggests/recommends items into the start menu via a right click menu... I wonder if I should submit a rewritten version of my previously mentioned Rox start menu? I stopped working on it after realizing that the XDG spec for .directory files was too ambiguous to the point that there is very little cross distro compatibility and the icon naming spec and desktop spec need some additional work to sync them up, but I don't recall the exact details. Though it would probably be better to patch Rox to properly handle mime-apps.list according to the spec ..instead of (or in addition to ) a directory full of shell scripts or links for handlers, use:

Code: Select all

[Default Applications]
mimetype1=default1.desktop;default2.desktop;

[technosaurus]

That said, lots of info which can be had at package build-time would serve no sane purpose if passed through *.desktop files to the run-time side of things like a Desktop Environment.

I must have miscommunicated something. What I meant was to use much of the data from a package's .desktop file(s) in the package database entry if it has one. It doesn't even need to be in the same format, just a format that is easily usable by common programming languages used by package managers; thus it should be easy to parse in shell, awk, c/c++, lua, tcl and easily dumped to an sqlite db. Amazingly enough that is something puppy's format is really good at, though tab separators would be more conventional. Rather than having umpteen (possibly unused) additional fields for translations of each translatable field though, I would generate a separate database file for each language and use google translate (or similar) to fill in any gaps.

P.S. I think the desktop file should split Mimetype into multiple fields like read, write and edit (maybe more); so that right click menus could properly deduce whether an application can edit or only view or only convert to a specific Mimetype... which would also trigger an update to the mime apps spec.

On a similar note, dependencies are not specific enough either. Not just build vs run time, there is no way to tell if missing a dep will simply reduce functionality or render the program inoperable... which goes toward the ambiguity of suggests, recommends and other similar terms. There is also some difference between binary, library and data dependencies as well as module and plugin dependencies varying across programming languages. Finding those outside of the package build environment can become extremely complex - from parsing the output of strace to parsing an entire shell script to find out what binaries it calls that aren't shell functions or builtins (how many shell scripts do you think leave off sed or grep from dependencies even though they use it) This was (is) a big problem with user built pet packages, because they often only specify the dependencies that are not in the standard puppy (if any at all)

Edit: Note to self - Look into getting a cert for signing packages... and figure out how to sign them.

[amigo]

Yes, package info is best kept in a separate file. With care, this can be formatted as sourceable -just a bunch of variables which need no parsing, etc.

list of files included in a package should also be in separate files -and in their own subdir for easy, one-pass grepping to find out which package owns a certain file. This is used at build-time of new packages to determine dependencies (requirements).

These requirements are also best kept in their own file, own directory. This makes queries for reverse depends a single-pass grep operation and also means that requirements for the package or readily listed.

All file lists are simply that, without any comments or blank lines, so they cane be 'while, read, do-ed' -hehe. The list-of-all files should be included in the package -in case anyone gets the idea they could simply keep the package and then list it (the package) and use that for removepkg functions. The list-of-all files is the basic unit of package real 'management'.

Very important: dependency/requires list must be generated at build-time, on a native build system. Otherwise they can never be depended on to be accurate. Period. How specific and detailed the info is, is debatable. I mentioned that tpkg does away with any >= or <= definitions -as a way of avoiding that next level dependency hell. src2pkg uses a bash patch which can read scripts and determine which external programs it calls, then they are tracked by package, so a script which is part of a package will have its' deps checked and recorded. This is not perfect, but pretty killer, still.

Degrees of dependency: for most PM's, a dependency is something that must be available for the program to successfully start -or run effectively. Obviously, the first usually means some library that a program must have in order to start and run. The second part 'run effectively' means like the program 'man' which *mist have the executable program 'groff' in order to do anything useful. 'suggests' is used for things like binary plugins, or for runtime options, or like accessory scipts which aren't really part of the packages main functionality.

tpkg system is scalable in that one can choose to include, or not, manifest files. These are generated at package build-time using the long output of 'tar', but re-arranging the data fields while, at the same time, generateing hashes of each file in the package and adding that data field to the manifest entries. Again, the info can only be valid and believed, if generated at build-time and included in the package. This provides a way to check anytime, whether a file on the system has been modified. The scalability is that, the user can configure tpkg/tpm to discard the manifests, on installation. They are the largest files in the database.

Usually, lists of included files and dependencies do *not* fill the 2K file size-limit. However, the info from is the most of any needed data for queries and pkg management. It's worth of every bit of 'wasted' space to be able to easily and *very* quickly list files or cross-reference deps and reverse-deps.

"every time I do a complex regex or any non-stanard printf format" Well, I really dislike code which can't be easily read and understood and eventually, fixed or modified. I really hate regex and avoid it if possible. A tiny bit of egrep-regex and with sed. But even sed lets us use '-e' to separate different passes. All my packaging tools, end-to-end, use grep/fgrep often, sed rarely and awk never. I admire awk, but I don't think it should be required for handling or building packages.

build-time-dependencies are bit harder to get, beyond deducing that the build requires the devel-files of any libs it needs. Pretty sure most build systems use, like src2pkg, at least some manual input or referencing lists. src2pkg runs potentially *lots* of commands figuring out what the sources consist of and how to get them to configure and build. It does track some of its' success -being able to write the build script for you, when an auto-build succeeds. No routines yet, for writing out/amending build-depends from that.

T2 has a tool which parses output from configure scripts/cmake, etc. to determine header calls. That tool is the big 'gem' from T2 -along with their lua-bash plugin. I have the lua-bash working here. Could a real C guru, like yourself, pull that configure-parser out to stand alone? Probably a 15 minute job, no?

[technosaurus]

Crap... just found this