The State of Package Management

What features/apps/bugfixes needed in a future Puppy

Should Puppy's package format be changed?

Yes, without backwards compatibility.
11
28%
Yes, with backwards compatibility.
10
26%
No, but the PET format should be standardized/stricter.
8
21%
No, the PET format works fine.
10
26%
 
Total votes: 39

Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#76 Post by sunburnt »

01micko; I made a similar utility years back.
It could make dependency tree files that would solve a lot of problems.

It`s not just the needed library, but the exact library that`s needed.
As amigo points out, change critical stuff and you need to redo everything.

# Gentoo Linux builds everything on the spot if I understand correctly.
This solves problems, but eventually you`d probably need to redo it also.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#77 Post by jpeps »

sunburnt wrote: It`s not just the needed library, but the exact library that`s needed.
As amigo points out, change critical stuff and you need to redo everything.
True, and the app isn't going to run if a dep is altered after it's loaded either. So why go through the hassle of moving to self-contained packages or some other cumbersome process? A good repository of libs would be a plus though (although they're generally fairly easy to find).

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#78 Post by amigo »

A central 'dump' of loose files is not very helpful -more often than not you are going to actually need more than one file -for example a 'real' library and the link that points to it. Most dependencies are really gonna need several files to work. A central repo of loose files is absolutely no help to the developer who tries to maintain it either. You need a way to have a system of grouping related things together.
That's why a list of *everything* is also useless -it remains a nightmare for users and for devs. Also, individual filenames don't always tell you anything about the version of that file, and they certainly don't give you any info about compile-options, etc.
No order is always gonna equal chaos.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#79 Post by sunburnt »

# Q:
But if a sfs of common libraries not in Puppy were built for each standard Puppy type,
and then Puppy`s apps. built upon them, they should be reliable and well constructed?

It`s short lived of course, as soon as a Barry makes a new Puppy, a new lib.
But this seems to be the best method of providing a solid base for Puppy?

As I said, I think media would be a big part of the needed libs. ( but not only )
I`ve tried gstreamer apps. with little success. And ffmpeg is a real pain!

Thinking in retrospect, my success with binaries has been with simple apps.
My attempts at media with binary and source files has been mostly failures.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#80 Post by amigo »

The best way to have a solid base is to deal with workable units in an orderly way -just what I've been saying all along. Each source triggers a new unit of installable material. They must be compiled in a certain order so that everything works as expected. In order to know which unit a certain dependency belongs to, you need to account for each file.

Creating and using sfs's or other mountable file-system images is fine, for what it is. But, the only rational way to construct them is by using individual packages -you install them into a subdir then make your sfs out of that. That would at least eliminate some of the disorderiness. But having a single sfs install the material from what is really several units starts causing problems of duplication. And what about this:
1. you create an sfs which includes a certain version of a lib.
2. I want to create an sfs which includes a later version of the same lib, or the same version, but compiled with different options included.

How do propose to resolve the conflict? If your sfs gets 'loaded' before mine, then my prog is gonna try to use your libs and will fail. Handling duplicate libs or separate versions of the same lib can be done of course, but involves other tricks besides simply relying on a certain load order.

If you were to create sfs's of each unit needed by the program you want to work, then it would be exactly the same problem as handling 'packages'. You'd still need a way to reolve the dependencies logically and orderly.

Why not simply 'install' packages on-the-fly? Some LiveCD systems used to handle it that way? Well, that's where your sfs's have an advantage because they don't need to use any RAM since they are simply mounted. The problem of which sfs's are needed and in what order they must be loaded remains though. Lists of files included in each 'unit', unit names which include both 'version' and 'build' information -plus architecture info if you are to ever have more than one arch supported. Units are most easily divided along the lines they come as -individual sources. Anything else is a mess.

Why can't you do a real upgrade of Puppy? Simply because the above has not been done. Why can't you easily remove a program or library that you don't need or want? Because the above has not been done. Why can't you 'replace' one program or lib with another which has superceded it? Because the above has not been done.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#81 Post by jpeps »

amigo wrote:
Why can't you do a real upgrade of Puppy? Simply because the above has not been done. Why can't you easily remove a program or library that you don't need or want? Because the above has not been done. Why can't you 'replace' one program or lib with another which has superceded it? Because the above has not been done.
Well, all the above is easily done, but rarely needed. I once wrote a script experimenting with a modular approach that created an "extra.sfs" package that I could transfer files in and out of. Outside of the learning experience, it was generally a waste of time.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#82 Post by amigo »

I think not easily done on Puppy. Upgrading is not simply a matter of installing a new version over the old one. Maybe the new package contains different files -what happens to the old ones? -they are orphaned and you have no way of telling whether they belong to a package or not.
Waste of time? I think you had a glimpse of just how much work is needed to create a bit of sanity for your system Will the script you made work on any other version of puppy? I'll bet not, because the needed flexibility and fine-grained control are probably not in there -but that wouldn't be your fault -the base is disorderly so it would be impossible to add some order where none was intended.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#83 Post by jpeps »

I just scripted a kind of remaster-on-the-fly after separating out selected apps into an sfs module....but in the end...so what?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#84 Post by sunburnt »

File duplication conflicts occur because of the Unix legacy file system.
Static compiling solves conflicts for the included files.

But apps. don`t have to be legacy installed.
If apps. are loose files in a dir. or non-unioned image files they won`t
conflict because they don`t share any part of the file system tree.
Simple apps. do this well, complex ones not so much ( browsers & media ).
But Puppy already has problems with it`s browsers and media apps.

The idea... A sizable base of common shared libraries ( stripped Puppy? ).
Other not so common libs. and the small libs. are compiled into the apps.

Trouble begins when you want an app. that uses a different shared lib.
Now the problem starts all over again... Change it and other stuff breaks.
So perhaps the apps. and their libs. go as a set working group.?
This is a near impossibility of course as apps. cross use different libs.

# This makes an argument for static compiled apps., and damn the size.
Static to the point of the real core libs. of course. Linux, C, GTK, etc...
Where to draw the line is a tough decision... The smaller the better.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#85 Post by jpeps »

sunburnt wrote:
# This makes an argument for static compiled apps., and damn the size.
Static to the point of the real core libs. of course. Linux, C, GTK, etc...
Where to draw the line is a tough decision... The smaller the better.
well..for Puppy linux size matters. You spoke of mplayer...well, I just compiled one for exprimo. A static copy wouldn't have helped at all..it didn't work with my cpu, so needed to be compiled on my machine. The resulting binary is two files. ldd lists 54 libs. You want to include them in the package?
(and of course, we'll need to include ffmeg and deps in there as well :) )

The other issue of course is that unless you're doing constant remasters as described above, there's the issue of all the already existing duplicates. With the existing system, you can run something like PetCheck and simply build a small pet that doesn't duplicate anything. If you're not convinced, the other bonus is that the OS won't fail to boot if there's some dependency issue or typo (like it would in TC).

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#86 Post by sunburnt »

jpeps; I`ve noticed what you`re saying about TC.
Loose files and the union fs cause the remaster problem. Conflicting libs. Right?
That`s why I suggested a stripped Puppy with sfs lib. extras, easier to maintain.

Having a given lib. installed can only support the apps. that use it. Correct?
So multiple libs. are needed to support other apps. But how to separate?
If app`s. files and odd libs. are in a dir. or no-union sfs file, they`re separate.
Realistically there has to be some shared libs. Which ones? Statistics...

A media sfs with most libs., ffmpeg, codecs, and some common apps.
Smaller than Puppy`s sfs so easier to remaster. Auto built by the PC.?
Again, small apps. seem to work well, media is a real pain. A good fix?

And yes... My next focus was to be architecture.
Q: How many architectures are common? 386, i64, amd64 ( Is that about it? )
I assume i64 and amd64 are not interchangeable. How about Atom and VIA?

Build browsers and media apps. on the PC, this solves many problems.?
If they`re not part of the legacy fs, then conflicts can be fixed easier?
.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#87 Post by noryb009 »

I've finished my PPkg program (formally makepuppypkg), and I have created a separate thread for it.

sunburnt: I don't think an lib SFS would fix things - it would just add unneeded libraries to a puppy, making it bigger. It also means that a specification needs to be created (which would include a lot of arguing of size vs. if a library is needed aka. it would never be done), or each puplet would create it's own (which means we are back to the present state).

I think the best way to solve the library problem is to have packagers define dependencies, and have a package manager that manages dependencies. Neither of these are currently done, except for some of the packages in the official repository.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#88 Post by sunburnt »

noryb009; I`m sure you agree that shared installed libs. should be very
commonly used ones, if only an app. or two use it compile it with the app.

What about the libs. that Barry put into Puppy? I imagine some are not used.
Why are target groups of small sfs addon libs. any less valuable? ( Devx )

The gstreamer set of libs. are used in a number of media apps. I noticed.
I`m sure there`s other groups also, each sfs file is a dep. for it`s apps.

As I said... Taking satistics on lib. usage and size would be a good start.
Again, most apps. seem to work well, browsers and media are a problem.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#89 Post by noryb009 »

After reading the views in the thread, I've come up with a rough outline of a possible PET2 package format. I've attached it. Any views, ideas, or criticism is welcome.

I`m sure you agree that shared installed libs. should be very commonly used ones, if only an app. or two use it compile it with the app.
I agree with you, but I don't know how many libraries are only used a few times. Some stats would be interesting.
Attachments
PET2_spec_idea.htm.gz
(2.22 KiB) Downloaded 263 times

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#90 Post by sunburnt »

Here`s a list I made from Tiny Core Linux`s tree files.
The tree files are the lib. package`s groups of files ( 1 or more files ).
Looking at it, it`s kinda random, but it`s good for getting a ball park idea.
It doesn`t include Tiny Core Linux`s installed libraries.
But T.C.L. has far less than Puppy as it`s very small in size ( 10.5 MB ).
The installed libs. Tiny Core Linux has are undoubtedly very critical.
I`m sure they threw out any libs. not needed to boot bare bones T.C.L.

616 files, 250 files are used 1 time, 521 under 100 times, 590 under 1000.
Highest use is: glib2.tcz at: 70177 times.

What`s needed is the size of the libs. and the number of apps. that use it.
But who cares about seldom used apps.? So choose the app. list first.

Shared libs. = Large size and common use.
Static libs. = Small size and seldom used.

A real statistical analysis would be difficult to get, but well worth it...
Attachments
deps-tree_count.lst.gz
(3.2 KiB) Downloaded 276 times

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#91 Post by noryb009 »

sunburnt: That list is very interesting. I never knew so many libraries were used that little.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#92 Post by amigo »

It doesn't really matter that they are used just a few times. The thing is that *some* program *is* gonna need it -when it needs it. and what do you mean by used? That it actually linked and used by running a program, or that it is a run-time dependency of one, a few or many other progs/libs?

I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one. BTW, the current pet format is already 'pet2' -there was something possibly even more brain-dead in place before... and before that there were *.pup thingies.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#93 Post by noryb009 »

I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one.
Would you like to share what you think is a good package format, or how the proposal can be improved?
BTW, the current pet format is already 'pet2'
I'm pretty sure the current format is only "pet". The only change to the format I know of was the name of the specs file (pkgname-pkgver.specs -> pet.specs), which did not increase the package version.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#94 Post by sunburnt »

If you`re contemplating statically compiling them, common use is important.
Otherwise it doesn`t mean squat...
It doesn`t provide any solution to the package problem, just insight.

# Package problems:
Different library versions can`t coexist, no version control.
Libraries overwriting or overshadowing other libraries.
Tracking and removal of loose files.
( Any others that I missed? )

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#95 Post by amigo »

I don't mean that the current format is called 'pet2', but, as you say, the name of the specs file has changed -and the format of the contents. It's a small change, but even a small change which breaks the 'API' can be significant. src2pkg can create either old-style or new-style pets and of course, that means two separate routines for *that part* of the package-creation code. If ppm (or whatever tool) allows for backward-compatibility, then that is to its' credit -it must have been done deliberately or would not be so.

( Any others that I missed? ) Uh, lots still, but your thinking is certainly much further along than some other folks here.

Sorry, but I'm not really gonna hold any hands here -I've been studying hard for 10 years and it will take anyone else just as long. I won't hold myself back just so that others can catch up. Study other systems and try to figure it all out. Of course debian has the oldest and most elaborate system. It's very hard to beat, but still has a couple of drawbacks. We could all just use someone else's format and tools excpet for the perceived drawbacks. For me, the show-stopper for debian packages/packaging is the reliance on perl and the need for human-intervention in order to create *any* package. the show-stopper for rpm packages is the binary format which means you are alway stuck using some of their tools for installation and creation -even src2pkg uses 'rpmbuild' when creating rpm packages. The rpm 'API' has changed over the years, which just makes the problem worse.

Post Reply