The State of Package Management

What features/apps/bugfixes needed in a future Puppy

Should Puppy's package format be changed?

Yes, without backwards compatibility.
11
28%
Yes, with backwards compatibility.
10
26%
No, but the PET format should be standardized/stricter.
8
21%
No, the PET format works fine.
10
26%
 
Total votes: 39

Message
Author
amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#41 Post by amigo »

"The only reason Puppy cant run other distros packages well is because of the extremely stripped down kernel" Not at all true. the kernl has nothing to do with software compatibility. Software-compatibility is all down to the libs, mate. Things don't always run right because there is library version-mismatch or differing configuration options at compile-time.
As you said, "There is no point in trying to retro fit another OSes packages on top of Puppy."

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

Regarding package management

#42 Post by 2byte »

Regarding package management

Why oh why hasn’t anyone considered using a real database to oversee the package installation and removal? Puppy comes with one already, SQLite, a self-contained, serverless, zero-configuration, transactional SQL database engine. It can be used from bash, C or almost any other programming language.

Consider this scenario.
Create a SQLite system database with two tables, named Packages and PackageFiles.

1. When the base puppy is created add a ‘system’ name to the Packages table with an ID of 1.
Record the path/file name of all libraries, links, and whatever other files the system is comprised of and put a record of each file in PackageFiles table of the database. Add a value of 1 to the count field in the table. All of these files will belong to the ‘system’ package. This database becomes a permanent part of the distro.

2. Installing a package. (not part of a system upgrade)
When a package is to be installed store the package name in the Packages table’. It will have a unique ID number greater than 1.
Parse each path/file name in the package and run a SELECT query for the exact path/file name on the PackageFiles table. If no record is returned install that file, add the file name to the PackageFiles table and add 1 to the count field of that record. Give this record the same ID as the package name. If the file already exists then increment the count field of the existing record but do not install the file.
Repeat until all new files are copied.

3. Uninstalling a package
Query the database for all files that belong to that package. For each file returned examine the count field. If the count is more than 1, decrement the count and leave the file installed. If the count is 1 and it is not a system file (ID=1), delete the file and delete the file record from the PackageFiles table . If the count is 1 and it is a system file then it is part of the distro and is to be left alone.
Repeat until all valid files are removed.
Delete the package’s record from the Packages table.

There it is in a nut shell. No system files ever get deleted or over written by a software package, plus safe install and removal of software using only existing tools in Puppy. All that’s missing is integration with a PPM.

Additional benefits:
A record of all packages can be retrieved at any time.
A record of all files actually installed for any software package can be retrieved at any time.
A record of the original system files can be retrieved at any time.
The number of applications using a particular library can be retrieved.
Foreign packages could be installed without fear.
The database will be small in KB.
The package format could be almost the same as a pet. A compressed base directory containing all of the package files, links and dirs. A pkg.list file could be required to speed up installation. Do away with pinstall scripts too.
Attachments
pkg-tables.jpg
(14.46 KiB) Downloaded 608 times


User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#43 Post by sunburnt »

amigo; I assume that you mean other OS`s compiled packages.?
So making SFS files from Slackware or other source packages is best.?
The trick then becomes satisfying all the dependencies if not provided.

I`ve always had good luck making Debian binary packages work in Puppy.
But recently that doesn`t seem to be the case, don`t know what changed.
Mainly multimedia apps. don`t work ( lots of libraries ), so maybe the app.?

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#44 Post by amigo »

Any sort of software installation -whether 'really installed' or dynamically loaded (sfs tcz, etc) can only be sanely managed by respecting the 'unit'. That means one_source=one_package (except for split sub-packages NLS, DOC, etc). Anything which violates this principle is gonna be a mess -guaranteed.
If you want a system which always works and brings no surprises, then all binary material should be compiled on, by and for that very system -not taken from *anywhere* else. The only excpetion to that should be for non-open-source stuff where the sources are not available -and maybe, with lots of care to supply *all* the dependencies (from your own stock), things like firefox -where you aren't always allowed to re-compile it and keep the name.

The methods used around here are simply a huge mess -there has never been a single release which was free from the problems caused by mixing/matching bins from all over the place. Even if you decide to minimize this by sticking to only bins from a single distro, as soon as you see that you need different options for one of the standard libs/progs from upstream, then you've already begub to break things. Your own toolchain, your own binaries for everything or you are fighting an up-hill battle.

The trick is to supply your own packages where each source supplies one program/lib or group of programs/libs in one package. External dependenices get the same treatment -logically, orderly *packaged*. Every single file, directory and link should be accounted for by at least one package (yes there are duplications which does not always mean conflicts)

Sometimes it's good to accept a bit of old-fashioned methodology from the old guys who have been doing this for decades. Do you really think your ideas are new and have never been tried and possibly discarded for good reasons?

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

Re: Regarding package management

#45 Post by jemimah »

2byte wrote:Regarding package management

Why oh why hasn’t anyone considered using a real database to oversee the package installation and removal? Puppy comes with one already, SQLite, a self-contained, serverless, zero-configuration, transactional SQL database engine. It can be used from bash, C or almost any other programming language.
Puppy already has a flat file database. Using SQL would only make it less transparent.
2byte wrote: If the file already exists then increment the count field of the existing record but do not install the file.
This would cripple the functionality.
2byte wrote:
Additional benefits:
A record of all packages can be retrieved at any time.
A record of all files actually installed for any software package can be retrieved at any time.
A record of the original system files can be retrieved at any time.
We already have this.
2byte wrote: Foreign packages could be installed without fear.
Not really. As long as you allow binaries or scripts, damage can be done easily.
2byte wrote: Do away with pinstall scripts too.
We need this functionality - not for everything, but when it's needed - nothing else will do.

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#46 Post by 2byte »

I knew before posting the idea would be rejected out of hand. Amigo is 100% right, although I don’t understand the point of that snide remark in the last sentence.

Jemimah
It might cripple the functionality of the installed software but it would prevent the base system from being crippled by a poorly packaged pet. I am talking about user installed software, not system upgrades. Those things you say we already have, yep. The idea doesn’t do away with those as you pointed out. As for foreign packages, I was referring to installing without fear of overwriting critical system files. Damage from running foreign software is unpreventable.

As long as the PPM blindly installs everything in a pet and blindly executes any pinstall script and then blindly removes every file that was in the pet people are going to continue to fubar their system.

If the Puppy package management isn’t broken, then don’t fix it.

No hard feelings :)


User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#47 Post by jemimah »

If you want to prevent overwriting system files, by far the easiest and most effective way is to mount the save file below the main sfs - as we discovered in a saluki-008 bug. :lol:

Perhaps it is better to have the user confirm system file overwrite.

It is important to understand that good package management (in the style of other distros) and read-only filesystems can be rather at odds with each other.

If you want apt-style package management, the only option is the one Sickgut concluded with - you must design the entire distro around it. You must discard Puppy entirely and start over. Anything less is doomed to failure.

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#48 Post by 2byte »

Well I'll just say this, then butt out. I have been using this sort of thing as my own package control system since Lucid was first released and it has performed very well.

Best regards to all


User avatar
darkcity
Posts: 2534
Joined: Sun 23 May 2010, 19:16
Location: near here
Contact:

#49 Post by darkcity »

Hi 2byte

It is possible you could upload your system to the forum so other could test it?

:D

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#50 Post by Lobster »

If Sqlite is too complex a flatfile database that creates a text file
is available as a base
http://puppylinux.info/topic/p-data-alp ... e-database
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#51 Post by sunburnt »

amigo; I myself don`t assume SFS and non-union Squash files are new.
However they don`t seem to be used much even though they have many
important improvements over the classic legacy .tar.gz type package.

Tiny Core Linux uses them very effectively, but a little oddly in my opinion.
But the manner of usage is consistent with the way the packages are setup.

To statically compile app. Squash packages by the Linux distro. it`s to be
used on would seem to be in line with what amigo is suggesting.
Static compiling solves nearly all shared library mismatches that might occur.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#52 Post by amigo »

The problem with statically-linking everything is that you wind up shipping duplicates of libs and making each app much larger than necessary -unless you use binary compression which means each app winds up needing more RAM and cycles than necessary...

Anyway, simply following the natural arrangement of things, developers usually write software in a compartmented way. Just because you are trying to get a certain 'capability' working, doesn't mean you should deliver it as a single-file or single-stop-solution.

From a devs standpoint, it's much easier to maintain, bugfix or add features using a modular approach -despite the use of git, etc. Those guys are still thinking along the lines of 'stable' and experimental code, with stable being for the public consumption and experimental for the devs.

The matter of staying compatible for a useful amount of time also comes into play. If your desired program depends on somebody elses lib, then it makes sense for you to release your code where it is working against a 'release' version of those libs.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#53 Post by sunburnt »

amigo; Exactly!!!
it makes sense for you to release your code where it is working against a 'release' version of those libs.
This makes for an endless cycle of being chained to someone else`s whims.
I`m sure you know what I`m talking about... Constant changes to your app`s.
dependencies forcing you to update your app. in a never ending cycle.
I`ve had to update apps. several times because of changes made to Puppy.

If apps. are statically built, then this won`t happen nearly as much.
If a new Puppy has a different lib., your app. will probably keep working.
Yes the apps. are larger, but not by a lot, and ram`s no longer hoarded.

I have run a statistical analysis of library size verses it`s common usage.
Big common libs. would be shared, smaller non-common ones static.
First a process of sorting out the installed shared libs. is needed.
So when building apps., if the libs. not already installed then it`s static.

# Best for Barry to build many shared libs. for all of the Puppy types.
# Then app. building is easy to do without concern for building the libs.

It might be a fair assumption that Puppy has a lot of the common libs.
Not many more would be added to Puppy as shared libs. ( maybe 2x ).
This could be added to the Devx SFS file ( It has 322MB of libs. now...).

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#54 Post by jpeps »

sunburnt wrote:
# Best for Barry to build many shared libs. for all of the Puppy types.
# Then app. building is easy to do without concern for building the libs.

It might be a fair assumption that Puppy has a lot of the common libs.
Not many more would be added to Puppy as shared libs. ( maybe 2x ).
This could be added to the Devx SFS file ( It has 322MB of libs. now...).
In the interest of keeping things small, having a large central repository of available libs to use as needed would seem preferable.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#55 Post by sunburnt »

jpeps; Agreed, adding to the the Devx file may not be the best choice.
Splitting libs. to a few SFS files by usage may be the best approach.
A separate media libs. SFS file, as media is probably the biggest group.

The idea being... Libs. built by Puppy devs. for the main Puppy releases.
Then most of the Puppy variants should have no problems with the libs.
These extra lib. SFS files would make .sfs / .pet package building a snap.

And they would increase the chance of random app. binary files working.
I think some times a Debian app. binary would work, but the Debian lib. fails.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#56 Post by jpeps »

sunburnt wrote:
And they would increase the chance of random app. binary files working.
I think some times a Debian app. binary would work, but the Debian lib. fails.
well...there's the issue of versions, symlinks, etc etc. Good build scripts can work for static apps like skype, for example, so there's no need to store anything. Quite often, a renamed link to a another lib is all that is needed. Then there's dependency hell...lets say for some gnome app. The question is...how much do you want to override intelligence with automation?

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#57 Post by scsijon »

If you want a new Package Manager it needs to be able to handle the current format else there will be problems. However there is no reason why it should not have two sections, one that handles the current format and another the new format, but the new format would be need to be flexable enough to be changed/extended, rather than replaced in the future else people will grumble and eventually go elsewhere with complaints like 'it's too hard to add apps'.

What will need doing though if it is to suceed, is either creating a repointing table for those many faithfull old apps that will remain as is, to deal with the extra and changes in structure, ir an orderly update/repackage with only the pet.specs (and maybe the packagename changed to the new format with a ref to show it's part of an old series).

I looked at OpenSuSE's packaging, as I was considering adding OpenSuSE to our available base sets and build a OpenSuSE based Puppy.

In their case they set their dependancies quite differently to most others. Dependancies can be both Packages or/and individual files, including versions.

They also use the xlm format {xlm format url missing}, which is quite an eye-opener on what can be included and what can be in it.

For example, for the opensuse package ConsoleKit-0.4.5-6.2.2.i586.rpm, (in xlm format) the pet.specs is:

Code: Select all

##----------------------------------------
=Pkg: ConsoleKit 0.4.5 6.2.2 i586
=Cks: SHA1 3d3c52dd2b7a1bdb8710d7771e37984f5c79d2bf
+Req:
/bin/sh
/bin/sh
/bin/sh
config(ConsoleKit) = 0.4.5-6.2.2
libc.so.6
libc.so.6(GLIBC_2.0)
libc.so.6(GLIBC_2.1)
libc.so.6(GLIBC_2.1.2)
libc.so.6(GLIBC_2.1.3)
libc.so.6(GLIBC_2.3.4)
libc.so.6(GLIBC_2.4)
libck-connector.so.0
libdbus-1.so.3
libdbus-glib-1.so.2
libglib-2.0.so.0
libgobject-2.0.so.0
libgthread-2.0.so.0
libpam.so.0
libpam.so.0(LIBPAM_1.0)
libpam.so.0(LIBPAM_EXTENSION_1.0)
libpolkit-gobject-1.so.0
libpthread.so.0
libpthread.so.0(GLIBC_2.0)
libz.so.1
login
pam-config >= 0.22
pwdutils
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsLzma) <= 4.4.6-1
-Req:
+Prv:
config(ConsoleKit) = 0.4.5-6.2.2
libck-connector.so.0
pam_ck_connector.so
ConsoleKit = 0.4.5-6.2.2
ConsoleKit(x86-32) = 0.4.5-6.2.2
/etc/ConsoleKit
/etc/ConsoleKit/run-seat.d
/etc/ConsoleKit/run-session.d
/etc/ConsoleKit/seats.d
/etc/ConsoleKit/seats.d/00-primary.seat
/etc/dbus-1/system.d/ConsoleKit.conf
/usr/bin/ck-history
/usr/bin/ck-launch-session
/usr/bin/ck-list-sessions
/usr/sbin/ck-log-system-restart
/usr/sbin/ck-log-system-start
/usr/sbin/ck-log-system-stop
/usr/sbin/console-kit-daemon
-Prv:
=Grp: System/Daemons
=Lic: GPLv2+
=Vnd: openSUSE
=Src: ConsoleKit 0.4.5 6.2.2 src
=Tim: 1319908259
=Loc: 1 ConsoleKit-0.4.5-6.2.2.i586.rpm
=Siz: 89661 323799
and this is for just one package!

= are statements about the app
+ and - Req: are the file requirements boundaries (files/packages between must be already there.
+ and - Prv: is privlidge access requirements
and their are others.

As much as I would love to use it, I don't think that most puppy builders would be willing to spend that much time working on the package file for a single app, it would however fix that problem we have with multiple bases and missing dependancies though!

I might add that opensue works on requiring any dependancies installed BEFORE the new app is installed, that is one thing we should consider somehow making mandatory, as we fall down at that step. Especially as a growing number of apps are now integrating and the installs are either amending dependancy config files or their own if a dependancy is there and available.

anyway that's my initial thoughts on this topic

regards to all

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#58 Post by 2byte »

There is a lot of discussion about the current ppm. What I would like to know;
Is there anyone willing to band together and create a brand new ppm?
If we did create one, would it be incorporated into Woof-X? What criteria would it have to meet to become 'official'?
Barry, are you willing to comment about this?
darkcity wrote: It is possible you could upload your system to the forum so other could test it?
darkcity
The 'system' I use isn't an automated utility that could be uploaded for testing. Another thing about it is it's an all or nothing sort of thing; you either use Puppy's package manager or this so called system, but you can't use both.


jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#59 Post by jpeps »

scsijon wrote: I looked at OpenSuSE's packaging, as I was considering adding OpenSuSE to our available base sets and build a OpenSuSE based Puppy.

In their case they set their dependancies quite differently to most others. Dependancies can be both Packages or/and individual files, including versions.
Does that mean each and every dependency must be uploaded to a puppy repository?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#60 Post by sunburnt »

Doesn`t anyone see the value of not using loose files in an O.S.?
Squash files can`t get viruses ( I believe...), and don`t corrupt easily.
SFS files require almost no package management, only config. files.
They stay compressed, load to ram, and can be swapped on-the-fly.
And they can be mounted from anywhere, hd, ram, usb, lan, and web.

One of Puppy`s big strengths is it`s main SFS file holding most of the O.S.
And the devx file ( "DEVeloper eXtra", I think...) is it`s second best setup.

I suggest a few more SFS file extentions like medx ( "MEDia eXtra" ).
Having more "shared" libs. available in SFS form is a real good idea.

Have enough extra SFS lib. files covering most of the common libs.
Odd libs. are always needed, so static compile them into the apps.
An app. package builder should check shared libs. and list needed libs.

Post Reply