The State of Package Management

What features/apps/bugfixes needed in a future Puppy
Post Reply

Should Puppy's package format be changed?

Yes, without backwards compatibility.
11
28%
Yes, with backwards compatibility.
10
26%
No, but the PET format should be standardized/stricter.
8
21%
No, the PET format works fine.
10
26%
 
Total votes: 39

Message
Author
jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#81 Post by jpeps »

amigo wrote:
Why can't you do a real upgrade of Puppy? Simply because the above has not been done. Why can't you easily remove a program or library that you don't need or want? Because the above has not been done. Why can't you 'replace' one program or lib with another which has superceded it? Because the above has not been done.
Well, all the above is easily done, but rarely needed. I once wrote a script experimenting with a modular approach that created an "extra.sfs" package that I could transfer files in and out of. Outside of the learning experience, it was generally a waste of time.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#82 Post by amigo »

I think not easily done on Puppy. Upgrading is not simply a matter of installing a new version over the old one. Maybe the new package contains different files -what happens to the old ones? -they are orphaned and you have no way of telling whether they belong to a package or not.
Waste of time? I think you had a glimpse of just how much work is needed to create a bit of sanity for your system Will the script you made work on any other version of puppy? I'll bet not, because the needed flexibility and fine-grained control are probably not in there -but that wouldn't be your fault -the base is disorderly so it would be impossible to add some order where none was intended.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#83 Post by jpeps »

I just scripted a kind of remaster-on-the-fly after separating out selected apps into an sfs module....but in the end...so what?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#84 Post by sunburnt »

File duplication conflicts occur because of the Unix legacy file system.
Static compiling solves conflicts for the included files.

But apps. don`t have to be legacy installed.
If apps. are loose files in a dir. or non-unioned image files they won`t
conflict because they don`t share any part of the file system tree.
Simple apps. do this well, complex ones not so much ( browsers & media ).
But Puppy already has problems with it`s browsers and media apps.

The idea... A sizable base of common shared libraries ( stripped Puppy? ).
Other not so common libs. and the small libs. are compiled into the apps.

Trouble begins when you want an app. that uses a different shared lib.
Now the problem starts all over again... Change it and other stuff breaks.
So perhaps the apps. and their libs. go as a set working group.?
This is a near impossibility of course as apps. cross use different libs.

# This makes an argument for static compiled apps., and damn the size.
Static to the point of the real core libs. of course. Linux, C, GTK, etc...
Where to draw the line is a tough decision... The smaller the better.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#85 Post by jpeps »

sunburnt wrote:
# This makes an argument for static compiled apps., and damn the size.
Static to the point of the real core libs. of course. Linux, C, GTK, etc...
Where to draw the line is a tough decision... The smaller the better.
well..for Puppy linux size matters. You spoke of mplayer...well, I just compiled one for exprimo. A static copy wouldn't have helped at all..it didn't work with my cpu, so needed to be compiled on my machine. The resulting binary is two files. ldd lists 54 libs. You want to include them in the package?
(and of course, we'll need to include ffmeg and deps in there as well :) )

The other issue of course is that unless you're doing constant remasters as described above, there's the issue of all the already existing duplicates. With the existing system, you can run something like PetCheck and simply build a small pet that doesn't duplicate anything. If you're not convinced, the other bonus is that the OS won't fail to boot if there's some dependency issue or typo (like it would in TC).

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#86 Post by sunburnt »

jpeps; I`ve noticed what you`re saying about TC.
Loose files and the union fs cause the remaster problem. Conflicting libs. Right?
That`s why I suggested a stripped Puppy with sfs lib. extras, easier to maintain.

Having a given lib. installed can only support the apps. that use it. Correct?
So multiple libs. are needed to support other apps. But how to separate?
If app`s. files and odd libs. are in a dir. or no-union sfs file, they`re separate.
Realistically there has to be some shared libs. Which ones? Statistics...

A media sfs with most libs., ffmpeg, codecs, and some common apps.
Smaller than Puppy`s sfs so easier to remaster. Auto built by the PC.?
Again, small apps. seem to work well, media is a real pain. A good fix?

And yes... My next focus was to be architecture.
Q: How many architectures are common? 386, i64, amd64 ( Is that about it? )
I assume i64 and amd64 are not interchangeable. How about Atom and VIA?

Build browsers and media apps. on the PC, this solves many problems.?
If they`re not part of the legacy fs, then conflicts can be fixed easier?
.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#87 Post by noryb009 »

I've finished my PPkg program (formally makepuppypkg), and I have created a separate thread for it.

sunburnt: I don't think an lib SFS would fix things - it would just add unneeded libraries to a puppy, making it bigger. It also means that a specification needs to be created (which would include a lot of arguing of size vs. if a library is needed aka. it would never be done), or each puplet would create it's own (which means we are back to the present state).

I think the best way to solve the library problem is to have packagers define dependencies, and have a package manager that manages dependencies. Neither of these are currently done, except for some of the packages in the official repository.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#88 Post by sunburnt »

noryb009; I`m sure you agree that shared installed libs. should be very
commonly used ones, if only an app. or two use it compile it with the app.

What about the libs. that Barry put into Puppy? I imagine some are not used.
Why are target groups of small sfs addon libs. any less valuable? ( Devx )

The gstreamer set of libs. are used in a number of media apps. I noticed.
I`m sure there`s other groups also, each sfs file is a dep. for it`s apps.

As I said... Taking satistics on lib. usage and size would be a good start.
Again, most apps. seem to work well, browsers and media are a problem.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#89 Post by noryb009 »

After reading the views in the thread, I've come up with a rough outline of a possible PET2 package format. I've attached it. Any views, ideas, or criticism is welcome.

I`m sure you agree that shared installed libs. should be very commonly used ones, if only an app. or two use it compile it with the app.
I agree with you, but I don't know how many libraries are only used a few times. Some stats would be interesting.
Attachments
PET2_spec_idea.htm.gz
(2.22 KiB) Downloaded 263 times

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#90 Post by sunburnt »

Here`s a list I made from Tiny Core Linux`s tree files.
The tree files are the lib. package`s groups of files ( 1 or more files ).
Looking at it, it`s kinda random, but it`s good for getting a ball park idea.
It doesn`t include Tiny Core Linux`s installed libraries.
But T.C.L. has far less than Puppy as it`s very small in size ( 10.5 MB ).
The installed libs. Tiny Core Linux has are undoubtedly very critical.
I`m sure they threw out any libs. not needed to boot bare bones T.C.L.

616 files, 250 files are used 1 time, 521 under 100 times, 590 under 1000.
Highest use is: glib2.tcz at: 70177 times.

What`s needed is the size of the libs. and the number of apps. that use it.
But who cares about seldom used apps.? So choose the app. list first.

Shared libs. = Large size and common use.
Static libs. = Small size and seldom used.

A real statistical analysis would be difficult to get, but well worth it...
Attachments
deps-tree_count.lst.gz
(3.2 KiB) Downloaded 276 times

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#91 Post by noryb009 »

sunburnt: That list is very interesting. I never knew so many libraries were used that little.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#92 Post by amigo »

It doesn't really matter that they are used just a few times. The thing is that *some* program *is* gonna need it -when it needs it. and what do you mean by used? That it actually linked and used by running a program, or that it is a run-time dependency of one, a few or many other progs/libs?

I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one. BTW, the current pet format is already 'pet2' -there was something possibly even more brain-dead in place before... and before that there were *.pup thingies.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#93 Post by noryb009 »

I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one.
Would you like to share what you think is a good package format, or how the proposal can be improved?
BTW, the current pet format is already 'pet2'
I'm pretty sure the current format is only "pet". The only change to the format I know of was the name of the specs file (pkgname-pkgver.specs -> pet.specs), which did not increase the package version.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#94 Post by sunburnt »

If you`re contemplating statically compiling them, common use is important.
Otherwise it doesn`t mean squat...
It doesn`t provide any solution to the package problem, just insight.

# Package problems:
Different library versions can`t coexist, no version control.
Libraries overwriting or overshadowing other libraries.
Tracking and removal of loose files.
( Any others that I missed? )

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#95 Post by amigo »

I don't mean that the current format is called 'pet2', but, as you say, the name of the specs file has changed -and the format of the contents. It's a small change, but even a small change which breaks the 'API' can be significant. src2pkg can create either old-style or new-style pets and of course, that means two separate routines for *that part* of the package-creation code. If ppm (or whatever tool) allows for backward-compatibility, then that is to its' credit -it must have been done deliberately or would not be so.

( Any others that I missed? ) Uh, lots still, but your thinking is certainly much further along than some other folks here.

Sorry, but I'm not really gonna hold any hands here -I've been studying hard for 10 years and it will take anyone else just as long. I won't hold myself back just so that others can catch up. Study other systems and try to figure it all out. Of course debian has the oldest and most elaborate system. It's very hard to beat, but still has a couple of drawbacks. We could all just use someone else's format and tools excpet for the perceived drawbacks. For me, the show-stopper for debian packages/packaging is the reliance on perl and the need for human-intervention in order to create *any* package. the show-stopper for rpm packages is the binary format which means you are alway stuck using some of their tools for installation and creation -even src2pkg uses 'rpmbuild' when creating rpm packages. The rpm 'API' has changed over the years, which just makes the problem worse.

disciple
Posts: 6984
Joined: Sun 21 May 2006, 01:46
Location: Auckland, New Zealand

#96 Post by disciple »

amigo wrote:Of course debian has the oldest and most elaborate system. It's very hard to beat
What do you think of the pacman system?

I think the real complaint about it from Q5sys earlier was this:
Those things listed above I wouldnt really consider issues with a package. They are from changes to the core system which require special action above and beyond normal updates. However a user has no idea about them until he gets a failed upgrade... or suddenly finds his system will not load properly after an upgrade. Sometimes I wish pacman had a feature where major changes like that would alert the user before installing.
I guess in theory pacman could have a feature that did that somehow, but I it wouldn't really be the 'Arch way'. And the point is that you are expected to read the mailing list to know about major changes that require user intervention or whatever.

==================
There were a lot of interesting thoughts in this thread, especially at the beginning. But I can't help thinking:

1) If you're building a distro from scratch, there is no point in reinventing the packaging wheel. A lot of smart people have spent a lot of time solving all the problems of a packaging system. Surely there is at least one mature package management system that pretty much meets your needs. Use it.
Or am I wrong? What is Debian's equivalent for dealing with things like "pacnew" files? Maybe no normal package manager is as "user friendly" for system upgrades as a Puppy live CD or frugal install, which automatically cleans up anything that would mess with the new version...

2) If you're building a distro from another distro's packages (woof) surely it is crazy not to use the other distro's package management system.

3) As Jemimah (I think) said earlier, the package format is not the real problem - the repository is. If this forum is the de facto package manager that is not sustainable. If only one person can put packages in the repository that is not sustainable. I'm not very familiar with the woof based puppies - how do they interact with the repositories of the "parent" distro? Can you just fire up the package manager and install something from them? That's what I thought was the intent, but I couldn't see how to when I had a quick look...
Do you know a good gtkdialog program? Please post a link here

Classic Puppy quotes

ROOT FOREVER
GTK2 FOREVER

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#97 Post by Aitch »

amigo

From my perspective you certainly seem the most knowledgeable that I've read, but rather than others proposing ideas which you then say are bad with reasons, couldn't you propose a package management scheme others could maybe implement [or did you, and I missed it?]

Personally, if it improves or modifies Puppy into a better distro, [or even forks like iguleder/sickgut are doing] I would forego backwards compatibility, if it meant packages from some standard repository could be used....or is this just a step too far to still be puppy development...?

At least this discussion seems to be drawing package management too a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?

Aitch :)

User avatar
Q5sys
Posts: 1105
Joined: Thu 11 Dec 2008, 19:49
Contact:

#98 Post by Q5sys »

disciple wrote: What do you think of the pacman system?

I think the real complaint about it from Q5sys earlier was this:
Those things listed above I wouldnt really consider issues with a package. They are from changes to the core system which require special action above and beyond normal updates. However a user has no idea about them until he gets a failed upgrade... or suddenly finds his system will not load properly after an upgrade. Sometimes I wish pacman had a feature where major changes like that would alert the user before installing.
I guess in theory pacman could have a feature that did that somehow, but I it wouldn't really be the 'Arch way'. And the point is that you are expected to read the mailing list to know about major changes that require user intervention or whatever.
Dont get me wrong I love the pacman system... it is just annoying to have to check the mailing lists for issues before i do an upgrade. And yes, those things only happen during core upgrades. However, since we dont seem to care about doing core upgrades though a package system in puppy (a wise move in my book), that shouldnt be an issue.
disciple wrote:3) As Jemimah (I think) said earlier, the package format is not the real problem - the repository is. If this forum is the de facto package manager that is not sustainable. If only one person can put packages in the repository that is not sustainable. I'm not very familiar with the woof based puppies - how do they interact with the repositories of the "parent" distro? Can you just fire up the package manager and install something from them? That's what I thought was the intent, but I couldn't see how to when I had a quick look...
Thats something thats been discussed before. And the issue with that is simply that most of the core devs in the puppylinux community are all working on their own pupplet versions. Asside from whoever is developing the offical release, everyone is off doing their work. Sure everyone helps on the offical releases, but the rest of the time time energy and effort are spent elsewhere. They are building their systems for their goals. This divergant focus is where our problem lies. Since each pupplet is remarkably different than another... compatibility of packages is in question.

To quote from here
jemimah wrote:
Q5sys wrote:
jemimah wrote:Ideally, we could just patch the existing ppm. I'm almost ready to work on something like this.

The hosting part should be fairly simple - just ftp access the way smokey01 has his site setup.

The difficult part is finding reliable people to be repo maintainers.
Well cant we start small and only increase in size as we have people on board to assist? We've got a pretty solid core group of developers. Each could work on their own little sliver, that way its not outsourced to random people who really dont care about the work. I dont really want to give them more work... but if they just simply approve or veto stuff... Im sure myself or some other maintainer can do the necessary tweaks here and there.
The problem is, nearly every one of our core devs has their own puplet. So we're all compiling the same packages, and they're generally not compatible.

For instance, in Saluki, I'm trying really hard to categorize things in a way that makes sense (to me anyway). So for each pet, I have to modify the .desktop file to make sure the categories are consistent, it has a full size icon, the name, comment, and generic name are set correctly. So even for stuff where the binaries are fine - I have to rebuilt the pet.

What would be most useful to the users, obviously, is if we could all agree on some sort of standard and have a central repo with packages that work on all puplets (at least the recent ones). This is quite possible for Racy/Wary but if you throw Slacko, Lupu, and Dpup into the mix, I don't think it'll succeed.

I think to start, we really need to figure out who is interested in being a repo maintainer and under what conditions they would be interested contributing.

If we can solve the social problem - I'm sure I can make the PPM support our solution.
So with that in mind I suggested we only work to build one for the official release. But again we come to the problem of who's going to do the work. With each of the core devs doing their own thing... are they going to be expected to build packages for the official release?
Read through that thread to get the jist of the discussion there. No reason for me to quote it all here. While that thread started off about a package site... it gets in the issue of package management a little.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#99 Post by sunburnt »

Diagnosing incompatibilities in new package builds can be a real pain.
Just statically compile anything that`s incompatible, conflicts are solved.
Not a perfect solution, but there`s few options with the current setup.

No matter how good Barry makes Puppy, one build can`t fit all needs.

But if a mostly complete set of dependency tree files could be had...
Then anyone trying to compile apps. would have a much easier time.
It`s a fairly easy thing to do ( time consuming ), and gives statistics.
And of course they would require updating as dependencies changed.

Q: Does anyone know how Slackware does tree files or handles deps.?

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#100 Post by noryb009 »

So with that in mind I suggested we only work to build one for the official release. But again we come to the problem of who's going to do the work. With each of the core devs doing their own thing... are they going to be expected to build packages for the official release?
That is one of the problems with Puppy's structure - everybody works on their own project alone, and doesn't share anything. I've created Ppkg to try to unite some the package creation process, but it still doesn't solve the high number of unmaintained puplets.
Diagnosing incompatibilities in new package builds can be a real pain.
Just statically compile anything that`s incompatible, conflicts are solved.
Not a perfect solution, but there`s few options with the current setup.
The "current setup" can be changed to something better - which is a topic of this thread.

Post Reply