The State of Package Management

What features/apps/bugfixes needed in a future Puppy

Should Puppy's package format be changed?

Yes, without backwards compatibility.
11
28%
Yes, with backwards compatibility.
10
26%
No, but the PET format should be standardized/stricter.
8
21%
No, the PET format works fine.
10
26%
 
Total votes: 39

Message
Author
2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#46 Post by 2byte »

I knew before posting the idea would be rejected out of hand. Amigo is 100% right, although I don’t understand the point of that snide remark in the last sentence.

Jemimah
It might cripple the functionality of the installed software but it would prevent the base system from being crippled by a poorly packaged pet. I am talking about user installed software, not system upgrades. Those things you say we already have, yep. The idea doesn’t do away with those as you pointed out. As for foreign packages, I was referring to installing without fear of overwriting critical system files. Damage from running foreign software is unpreventable.

As long as the PPM blindly installs everything in a pet and blindly executes any pinstall script and then blindly removes every file that was in the pet people are going to continue to fubar their system.

If the Puppy package management isn’t broken, then don’t fix it.

No hard feelings :)


User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#47 Post by jemimah »

If you want to prevent overwriting system files, by far the easiest and most effective way is to mount the save file below the main sfs - as we discovered in a saluki-008 bug. :lol:

Perhaps it is better to have the user confirm system file overwrite.

It is important to understand that good package management (in the style of other distros) and read-only filesystems can be rather at odds with each other.

If you want apt-style package management, the only option is the one Sickgut concluded with - you must design the entire distro around it. You must discard Puppy entirely and start over. Anything less is doomed to failure.

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#48 Post by 2byte »

Well I'll just say this, then butt out. I have been using this sort of thing as my own package control system since Lucid was first released and it has performed very well.

Best regards to all


User avatar
darkcity
Posts: 2534
Joined: Sun 23 May 2010, 19:16
Location: near here
Contact:

#49 Post by darkcity »

Hi 2byte

It is possible you could upload your system to the forum so other could test it?

:D

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#50 Post by Lobster »

If Sqlite is too complex a flatfile database that creates a text file
is available as a base
http://puppylinux.info/topic/p-data-alp ... e-database
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#51 Post by sunburnt »

amigo; I myself don`t assume SFS and non-union Squash files are new.
However they don`t seem to be used much even though they have many
important improvements over the classic legacy .tar.gz type package.

Tiny Core Linux uses them very effectively, but a little oddly in my opinion.
But the manner of usage is consistent with the way the packages are setup.

To statically compile app. Squash packages by the Linux distro. it`s to be
used on would seem to be in line with what amigo is suggesting.
Static compiling solves nearly all shared library mismatches that might occur.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#52 Post by amigo »

The problem with statically-linking everything is that you wind up shipping duplicates of libs and making each app much larger than necessary -unless you use binary compression which means each app winds up needing more RAM and cycles than necessary...

Anyway, simply following the natural arrangement of things, developers usually write software in a compartmented way. Just because you are trying to get a certain 'capability' working, doesn't mean you should deliver it as a single-file or single-stop-solution.

From a devs standpoint, it's much easier to maintain, bugfix or add features using a modular approach -despite the use of git, etc. Those guys are still thinking along the lines of 'stable' and experimental code, with stable being for the public consumption and experimental for the devs.

The matter of staying compatible for a useful amount of time also comes into play. If your desired program depends on somebody elses lib, then it makes sense for you to release your code where it is working against a 'release' version of those libs.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#53 Post by sunburnt »

amigo; Exactly!!!
it makes sense for you to release your code where it is working against a 'release' version of those libs.
This makes for an endless cycle of being chained to someone else`s whims.
I`m sure you know what I`m talking about... Constant changes to your app`s.
dependencies forcing you to update your app. in a never ending cycle.
I`ve had to update apps. several times because of changes made to Puppy.

If apps. are statically built, then this won`t happen nearly as much.
If a new Puppy has a different lib., your app. will probably keep working.
Yes the apps. are larger, but not by a lot, and ram`s no longer hoarded.

I have run a statistical analysis of library size verses it`s common usage.
Big common libs. would be shared, smaller non-common ones static.
First a process of sorting out the installed shared libs. is needed.
So when building apps., if the libs. not already installed then it`s static.

# Best for Barry to build many shared libs. for all of the Puppy types.
# Then app. building is easy to do without concern for building the libs.

It might be a fair assumption that Puppy has a lot of the common libs.
Not many more would be added to Puppy as shared libs. ( maybe 2x ).
This could be added to the Devx SFS file ( It has 322MB of libs. now...).

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#54 Post by jpeps »

sunburnt wrote:
# Best for Barry to build many shared libs. for all of the Puppy types.
# Then app. building is easy to do without concern for building the libs.

It might be a fair assumption that Puppy has a lot of the common libs.
Not many more would be added to Puppy as shared libs. ( maybe 2x ).
This could be added to the Devx SFS file ( It has 322MB of libs. now...).
In the interest of keeping things small, having a large central repository of available libs to use as needed would seem preferable.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#55 Post by sunburnt »

jpeps; Agreed, adding to the the Devx file may not be the best choice.
Splitting libs. to a few SFS files by usage may be the best approach.
A separate media libs. SFS file, as media is probably the biggest group.

The idea being... Libs. built by Puppy devs. for the main Puppy releases.
Then most of the Puppy variants should have no problems with the libs.
These extra lib. SFS files would make .sfs / .pet package building a snap.

And they would increase the chance of random app. binary files working.
I think some times a Debian app. binary would work, but the Debian lib. fails.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#56 Post by jpeps »

sunburnt wrote:
And they would increase the chance of random app. binary files working.
I think some times a Debian app. binary would work, but the Debian lib. fails.
well...there's the issue of versions, symlinks, etc etc. Good build scripts can work for static apps like skype, for example, so there's no need to store anything. Quite often, a renamed link to a another lib is all that is needed. Then there's dependency hell...lets say for some gnome app. The question is...how much do you want to override intelligence with automation?

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#57 Post by scsijon »

If you want a new Package Manager it needs to be able to handle the current format else there will be problems. However there is no reason why it should not have two sections, one that handles the current format and another the new format, but the new format would be need to be flexable enough to be changed/extended, rather than replaced in the future else people will grumble and eventually go elsewhere with complaints like 'it's too hard to add apps'.

What will need doing though if it is to suceed, is either creating a repointing table for those many faithfull old apps that will remain as is, to deal with the extra and changes in structure, ir an orderly update/repackage with only the pet.specs (and maybe the packagename changed to the new format with a ref to show it's part of an old series).

I looked at OpenSuSE's packaging, as I was considering adding OpenSuSE to our available base sets and build a OpenSuSE based Puppy.

In their case they set their dependancies quite differently to most others. Dependancies can be both Packages or/and individual files, including versions.

They also use the xlm format {xlm format url missing}, which is quite an eye-opener on what can be included and what can be in it.

For example, for the opensuse package ConsoleKit-0.4.5-6.2.2.i586.rpm, (in xlm format) the pet.specs is:

Code: Select all

##----------------------------------------
=Pkg: ConsoleKit 0.4.5 6.2.2 i586
=Cks: SHA1 3d3c52dd2b7a1bdb8710d7771e37984f5c79d2bf
+Req:
/bin/sh
/bin/sh
/bin/sh
config(ConsoleKit) = 0.4.5-6.2.2
libc.so.6
libc.so.6(GLIBC_2.0)
libc.so.6(GLIBC_2.1)
libc.so.6(GLIBC_2.1.2)
libc.so.6(GLIBC_2.1.3)
libc.so.6(GLIBC_2.3.4)
libc.so.6(GLIBC_2.4)
libck-connector.so.0
libdbus-1.so.3
libdbus-glib-1.so.2
libglib-2.0.so.0
libgobject-2.0.so.0
libgthread-2.0.so.0
libpam.so.0
libpam.so.0(LIBPAM_1.0)
libpam.so.0(LIBPAM_EXTENSION_1.0)
libpolkit-gobject-1.so.0
libpthread.so.0
libpthread.so.0(GLIBC_2.0)
libz.so.1
login
pam-config >= 0.22
pwdutils
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsLzma) <= 4.4.6-1
-Req:
+Prv:
config(ConsoleKit) = 0.4.5-6.2.2
libck-connector.so.0
pam_ck_connector.so
ConsoleKit = 0.4.5-6.2.2
ConsoleKit(x86-32) = 0.4.5-6.2.2
/etc/ConsoleKit
/etc/ConsoleKit/run-seat.d
/etc/ConsoleKit/run-session.d
/etc/ConsoleKit/seats.d
/etc/ConsoleKit/seats.d/00-primary.seat
/etc/dbus-1/system.d/ConsoleKit.conf
/usr/bin/ck-history
/usr/bin/ck-launch-session
/usr/bin/ck-list-sessions
/usr/sbin/ck-log-system-restart
/usr/sbin/ck-log-system-start
/usr/sbin/ck-log-system-stop
/usr/sbin/console-kit-daemon
-Prv:
=Grp: System/Daemons
=Lic: GPLv2+
=Vnd: openSUSE
=Src: ConsoleKit 0.4.5 6.2.2 src
=Tim: 1319908259
=Loc: 1 ConsoleKit-0.4.5-6.2.2.i586.rpm
=Siz: 89661 323799
and this is for just one package!

= are statements about the app
+ and - Req: are the file requirements boundaries (files/packages between must be already there.
+ and - Prv: is privlidge access requirements
and their are others.

As much as I would love to use it, I don't think that most puppy builders would be willing to spend that much time working on the package file for a single app, it would however fix that problem we have with multiple bases and missing dependancies though!

I might add that opensue works on requiring any dependancies installed BEFORE the new app is installed, that is one thing we should consider somehow making mandatory, as we fall down at that step. Especially as a growing number of apps are now integrating and the installs are either amending dependancy config files or their own if a dependancy is there and available.

anyway that's my initial thoughts on this topic

regards to all

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#58 Post by 2byte »

There is a lot of discussion about the current ppm. What I would like to know;
Is there anyone willing to band together and create a brand new ppm?
If we did create one, would it be incorporated into Woof-X? What criteria would it have to meet to become 'official'?
Barry, are you willing to comment about this?
darkcity wrote: It is possible you could upload your system to the forum so other could test it?
darkcity
The 'system' I use isn't an automated utility that could be uploaded for testing. Another thing about it is it's an all or nothing sort of thing; you either use Puppy's package manager or this so called system, but you can't use both.


jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#59 Post by jpeps »

scsijon wrote: I looked at OpenSuSE's packaging, as I was considering adding OpenSuSE to our available base sets and build a OpenSuSE based Puppy.

In their case they set their dependancies quite differently to most others. Dependancies can be both Packages or/and individual files, including versions.
Does that mean each and every dependency must be uploaded to a puppy repository?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#60 Post by sunburnt »

Doesn`t anyone see the value of not using loose files in an O.S.?
Squash files can`t get viruses ( I believe...), and don`t corrupt easily.
SFS files require almost no package management, only config. files.
They stay compressed, load to ram, and can be swapped on-the-fly.
And they can be mounted from anywhere, hd, ram, usb, lan, and web.

One of Puppy`s big strengths is it`s main SFS file holding most of the O.S.
And the devx file ( "DEVeloper eXtra", I think...) is it`s second best setup.

I suggest a few more SFS file extentions like medx ( "MEDia eXtra" ).
Having more "shared" libs. available in SFS form is a real good idea.

Have enough extra SFS lib. files covering most of the common libs.
Odd libs. are always needed, so static compile them into the apps.
An app. package builder should check shared libs. and list needed libs.

User avatar
Ray MK
Posts: 774
Joined: Tue 05 Feb 2008, 09:10
Location: UK

#61 Post by Ray MK »

Doesn`t anyone see the value of not using loose files in an O.S.?
Squash files can`t get viruses ( I believe...), and don`t corrupt easily.
SFS files require almost no package management, only config. files.
They stay compressed, load to ram, and can be swapped on-the-fly.
And they can be mounted from anywhere, hd, ram, usb, lan, and web.

One of Puppy`s big strengths is it`s main SFS file holding most of the O.S.
And the devx file ( "DEVeloper eXtra", I think...) is it`s second best setup.

Agreed - absolutely.
Especially now that we have so many good "on-the-fly" loaders/un-loaders.

Also good for those with ram challenged kit.

Best regards - Ray

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#62 Post by jpeps »

sunburnt wrote: SFS files require almost no package management, only config. files.
They stay compressed, load to ram, and can be swapped on-the-fly.
And they can be mounted from anywhere, hd, ram, usb, lan, and web.
I don't know about that. All the configs have to match the current environment perfectly, or it won't load. I recall TC's issues...every time there was some minor change (like a file using "-" vs "_"), it wouldn't boot. It's probably fine if every app is completely self contained.

What's to keep users from making and combining SFS apps now? I do. I recall having to remove whiteouts to get an SFS to load when there was a previous version in base (maybe something unique).

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#63 Post by noryb009 »

Status update on makepuppypkg: I've done quite a bit of work on it, and it's working perfectly for me. However, I can't currently commit it to github, but I'll hopefully have it up on the weekend.
but the new format would be need to be flexable enough to be changed/extended, rather than replaced in the future else people will grumble and eventually go elsewhere with complaints like 'it's too hard to add apps'.
Definitely. Having a pet.specs that is separated by "|" may seem like a good idea, but it becomes hard to add more fields without either breaking backward compatibility or adding the field to the end. Having one field per line would fix this, as outdated package managers would ignore unknown fields, and package managers could have default values for outdated packages.
For example, for the opensuse package ConsoleKit-0.4.5-6.2.2.i586.rpm, (in xlm format) the pet.specs is:
<snip>
and this is for just one package!

As much as I would love to use it, I don't think that most puppy builders would be willing to spend that much time working on the package file for a single app
That is the pet.specs, but package builders do not have to worry about them. The program 'rpmbuild' reads a much nicer looking file and converts it to that file.

My 'makepuppypkg' scripts reads a file like arch's PKGBUILD.
Does that mean each and every dependency must be uploaded to a puppy repository?
In other distributions, if a package in a repository has a dependency, it would either be in that repository or an upstream one (in arch linux, a "extra" package can use a dependency from "core", but not vice versa).
And the devx file ( "DEVeloper eXtra", I think...) is it`s second best setup.
I agree with you saying that SFS files allow save files to stay small, but this quote also shows that bad part about the current SFS files - you get everything, or nothing.
SFS files would be great if either:
- every package in puppy is in it's own SFS file or
- every package in puppy is in it's own PET (or a new format) and users can make their own SFS files from it.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#64 Post by amigo »

Every file, dir and link on a fresh install (or bootable live) system, should be part of a single accountable 'unit'. Really, an SFS is no different from a package in some ways.
You nearly said it here:
"every package in puppy is in it's own PET"

The thing is that your concept of a single-file delivery doesn't hold up in practice -unless you are composing them from (usually) smaller units. An SFS which guarantees the usability of all its' 'members', should be composed of one or more 'packages'. The SFS itself is not really a good unit to use for packages simply because it's unhandy and CPU-intensive to work with from a creation standpoint.

An SFS is a file-system image, the same as an *.iso file or e2fs, e3fs. That aspect of it has no bearing on the contents of it. YXou could also create and distribute 'mega-packages' which combine the files of one or more 'units' of files.

Because of the very nature of how software/libs are created, there will nearly always be something 'external' needed by your delivered packge/FS-image. That means that the method of making your thing available and useful has to take that into account. It *is* possible (usually) to create fully static stuff, but if you go all the way wit that, then everything needs *loads* of stuff which winds up being duplicated -maybe thousands of time. How many progs are you running which depend on glibc? Do you have any idea?

The whole concept of shared libs is meant to avoid that, so statically including libs is nearly never the ritght thing to do.

But, the whole of idea of including everything necessary in one unit is simply wrong -unless you really want to upgrade, re-compile and re-assemble everything every time just one thing changes. If you have a program which is gonna need libs (for example), which are produced from other sources, you need to have a way to make sure those libs are available when your program is run -you need dependency-resolution. But resolving depends doesn't mean just having 'that library' on your system. It needs to be 'that library' exactly as the one used when you compiled your program -some stray library with the same name -even version, may not work, because of compile-time options and any further libs that get linked to simply because they exist at compilation time. Dependency resolution means you have a way to track the existence and origin of every file on the syetem.

While possible to create a list of all files included in your 'unit', if that unit is really composed of lots of little things from all over then you have an accounting mess. Processing any given source code is going to produce a limited number of files to be installed. The list of those files (as composed at built-time) is a unit -a package, if you will. Of course there can be split packages, if needed. But mega-packages which combine bits from various sources completely mess up the scheme -as a rule. There are exceptions there also, where several small utilities from several sources might be combined into one package. But when you start combining things which are, or are likely to be needed by any other programs, then your units start overlapping and accounting becomes difficult at best.

I'm tired, but instead of deleting this I'll let it stand just in case... You are going about it the wrong way around by addressing symptoms instead of the problem. The problem is that you have no way of accurately accounting for the necessary attributes of each and every item on your system at a particular time. That should be broken down into intelligible units. No, the user doesn't have to know or handle any of that, but 'devs' who don't do that have an un-maintainable mess.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#65 Post by jpeps »

amigo wrote: You are going about it the wrong way around by addressing symptoms instead of the problem. The problem is that you have no way of accurately accounting for the necessary attributes of each and every item on your system at a particular time. That should be broken down into intelligible units. No, the user doesn't have to know or handle any of that, but 'devs' who don't do that have an un-maintainable mess.
Very true, and that will inevitably require proposed apps to be submitted to some central authority prior to posting....a horrible idea. Multiply the censored disappearing forum posting issue by 1000X. In the present system, there's no need to maintain a huge database replete with source code for every item. I have yet to experience a boot failure in puppy related some mismatch with an app dependency.

Post Reply