The State of Package Management

What features/apps/bugfixes needed in a future Puppy
Post Reply

Should Puppy's package format be changed?

Yes, without backwards compatibility.
11
28%
Yes, with backwards compatibility.
10
26%
No, but the PET format should be standardized/stricter.
8
21%
No, the PET format works fine.
10
26%
 
Total votes: 39

Message
Author
jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#106 Post by jpeps »

amigo wrote:2byte gets it! Very well put, fellow.
+1

I spent a few hours trying to connect listed builtin files with right pet.spec files in woof before giving up. I hadn't realized it's full of debs.

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#107 Post by Aitch »

Aitch wrote:At least this discussion seems to be drawing package management to a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?
This looks liked good progress -
Question: As devs do you not feel you have a right to ask Barry to help sort out the chaos he's created, or risk good devs forever leaving/forking, in an effort to create a good working, useable and updatable OS
I don't see it as insulting, as even he must realise the problems he's creating for you to try to sort out....he does have a natural responsibility for his creation which you guys don't, after all....?
Else Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....

Good discussion, guys'n'gals, thanks!

btw, I think jrb's pinstall.sh script idea needs more awareness, so here's a link

http://www.murga-linux.com/puppy/viewto ... 808#603808

Aitch :)

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#108 Post by jemimah »

Barry is very quick to accept bug fixes, and enhancements, but it seems unlikely that he'll want to rewrite woof.

Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.

What you should be asking for is a better, simpler, more user-friendly puppy builder.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#109 Post by amigo »

jpeps, the light gets brighter the closer you get to the end of the tunnel.
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.

"full of debs" -If your gonna have your own distro, surely you should be consistent about the content, where it comes from and how it is constructed -or ??

Aitch, "ask Barry to help sort out the chaos" -surely he would have done this if he thought it right or sensible. After all, it is *he* who would most benefit from it.

"good devs forever leaving/forking" Happens all the time. The best *have* spoken to BK about it without result.

"Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version" That's the most concise description of Puppy I have ever seen. Starts out broken, bugs don't get fixed, not up-gradeable -incapable of learning from its' own mistakes for the future benefit of both devs and users. I am skeptical, though, about the validity of this: "less broken later version"

Aitch, I *do* have a very workable, independent package format in place and working very well. Still undergoing some refinement, but already past the point of needing any changes which would affect backard or forward compatibility. Of course, the package format is supported by tools for the creation of compliant packages (src2pkg and others), and the tools for sanely installing, adding, removing, replacing or upgrading packages. There are also advanced tools which analyze the database created by during the above operations. There is still lots to do there, but new little things get added nearly every day. It certainly is functional enough for managing a completze distro with thousands of packages.

Unfortunately, writing an acceptable document which describes even the most basic concepts or specifications of the package format and tools would take me many, many hours of work. The tools do, of course, include '--help' functionality and man-pages, but they don't tell the whole story. Such documents will exist one day, but I'm still much to busy implementing advanced features (like automatic dependency *resolution*) to work on that now. It would be a waste of time to write a nice document when some aspects of the format and the tools are still changing -even though the changes are minor.

Note, while automatic dependency resolution is still not implement in the installer tools, the format already contains the needed 'slots' for the database files and they are generated automatically by src2pkg when building *.tpkg packages. The packages also already contain these files, as needed.

The 'resolver' itself will not be hugely complex and the algorithms will be completly separate from the normal installation tools. Even the tpkg-upgrade tool does not directly install the new package and remove the old one. It uses the individual tools written for each purpose. The same division of labor will be used for any GUI package management tools -they will rely on the underlying tools. This serves at least two purposes -it keeps our 'meaty' code for actually de-compressing archives, running install scripts and updating the database separate from 'window-dressing' code for a GUI. It also means that the GUI code is cleaner and less complex -read easier-to-write and easier to understand or maintain. A low-level tool for each individual operation, with higher-level tools serving as a front-end to them -the UNIX way, you know.

Since I'm 'rolling my own', I don't have to be concerned about getting anyone's 'approval'. If it works it will get adopted by others. Actually, I entertain no desire to have it used by Puppy, as by association it might be identified with Puppy. In fact, I don't even care if nobody else ever uses my format -I've done it to meet my own needs and for the pleasure of creating something that 'just works'.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#110 Post by amigo »

jemimah, I believe that a re-think of what comprises a 'frugal' installation would make package-management still possible. For instance, drop the 'frugal' part and simply install the whole system to a disk-image -no need for a save file, only needs a tiny initrd, and uses no RAM for parts of the system. Simply put, a loop-mounted system-in-file. Only the kernel and small initrd would be separate -just as now. I built just such a distro before -nooby would have loved it!
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#111 Post by noryb009 »

As devs do you not feel you have a right to ask Barry to help sort out the chaos he's created, or risk good devs forever leaving/forking, in an effort to create a good working, useable and updatable OS
I don't feel it's Barry's duty to do anything, but he should do what is best for Puppy, which would be to help.
Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....
That is one of the things a "real" package management system would hopefully fix.
Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.
Nobody is forcing you to upgrade, but it is much better then the current system, as Aitch called it, "being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....". You can currently overwrite woof-installed packages (and it usually works, if there aren't any extra dependencies needed).
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.
Are you talking about a package list inside a package repository? Package lists are needed for description searching and looking for package dependencies before downloading the package. Every distribution I know of that has a repository system has a package list - there is no reason why Puppy can't have one.

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#112 Post by jemimah »

amigo wrote:jemimah, I believe that a re-think of what comprises a 'frugal' installation would make package-management still possible. For instance, drop the 'frugal' part and simply install the whole system to a disk-image -no need for a save file, only needs a tiny initrd, and uses no RAM for parts of the system. Simply put, a loop-mounted system-in-file. Only the kernel and small initrd would be separate -just as now. I built just such a distro before -nooby would have loved it!
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.

My primary reason for running puppy is to run it from RAM on low spec machines. Since it's in RAM - compression is extremely desirable. Also running from a usb stick or rewritable CD with limited space makes the compression very helpful.

The existing puppy system works ok with uninstalling packages as it is, but since the system sfses are read-only, deleted files aren't really gone until you remaster. Running it like a normal distro just makes a huge save file which negates the reason to run puppy at all. I personally don't think it's worth the effort to try retrofit complete package management as there is very little actual benefit. Time is better spent on figuring out ways to make it easier to build a puplet.

In my opinion, if the compressed filesystems are not of importance to you, you'd be better off with a "real" distro. Puppy is fantastic for the niche it's designed for - but it's not great for anything else. Trying to fix that is just going to be an exercise in futility.

Just my opinion, carry on.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#113 Post by amigo »

"from RAM on low spec machines" I never have been able to understand this rationale. If a system is weak, whhy sould one want to occupy 'any' RAM which is not necessary?. Especially when you consider that, onve a program has been run or is running, there are 'two* copies of it in RAM. The only penalty which one pays for not running in RAM is the startup-latency which only occurs the *first* time the program is started. Plus, there are other things which can be done to improve those latency figures.

Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file. Does the save file only include config files and other highly-selected items, or can it also contain installed programs? Is the save file an sfs? If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session. Maybe this is the 'crux' the lenghty times to save on shutdown. I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc. With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.

My impression, taken from a couple of polls which have been done here, is that most Puppy users use a frugal installation, and that they do that because of the ease of manually installing it, if wanted, and/or because it allows them to have a Puppy installation co-reside with a pre-installed Windows OS without the need for any partitioning and less out of having to work with a low-spec machine. If that is the case, then the use of RAM as a hard-disk substitute would seem to be a bad choice. And the use of write-only save files -if that is the case.

The light-weightness or agility of Puppy is mostly due to the non-use of kde, gnome and other such bloated software. There is a significant and 'feelable' speed-up when starting programs for the first time which are in RAM instead of being read in from disk. But, they all have to be loaded into RAM during boot-up which is why it is difficult to make Puppy boot up faster. My kiss-5.0 starts, from grub-prompt to GUI login, in right at 10 seconds here (on a Pentium IV) -even though it has to read everything from disk, link it all, etc. I think it gets there faster than if I spent time loading it all into RAM and then load and link it from there. And I can still improve the boot-time a little, I think, without going to extremes.

At the very least, it is pretty obvious that most Puppy users *do* use a frugal install, so more effort should be put into making that go as smoothly and quickly as possible -perhaps by having an initrd dedicated to frugal installs, instead of wasting hundreds of lines of code and time, by checking for all the other possible way to run Puppy. Things like pfix=ram are nice and useful, but not often needed and mostly exist as a work-around to fix other design flaws -in my hardly-humble opinion, LOL. In this same category, I would put things like 'use a separate save-file for separate users', when the obvious solution was there all along -have a real multi-user-capable system just like everybody else. That capability had to be built *out* inetntionally, and makes it completley unwise to use Puppy as a server or for other mission-critical purposes. It also makes it difficult, if not impossible without patching or hacks, to use some software like Chrome browser, etc.

Every single drawback goes back to a concious design choice which was made. And every advantage which Puppy may have over any other distro, could be implemted in such a way to avoid the drawbacks. Wanna run as 'root', autologin as root, autologin as root straight to the desktop? No problem. It doesn't mean that you have to completely eliminate the choice of doing otherwise.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#114 Post by jpeps »

jemimah wrote: Puppy is fantastic for the niche it's designed for - but it's not great for anything else. Trying to fix that is just going to be an exercise in futility.

Just my opinion, carry on.
I think any system can be individually tailored; I use my own checks, uninstaller, etc... Remastering is simple, but rarely needed. The present system is light years ahead of something like TC.

Trying to adopt a sane approach is always subject to differing ideas and politics. I created the first update system for TC, and spent months perfecting checks, fixing bugs, etc., etc., only to have it continually sabotaged by RS because too many people liked it. So I've learned...roll your own, offer improvements, and stay out of the way.

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#115 Post by jemimah »

amigo wrote:"from RAM on low spec machines" I never have been able to understand this rationale. If a system is weak, whhy sould one want to occupy 'any' RAM which is not necessary?. Especially when you consider that, onve a program has been run or is running, there are 'two* copies of it in RAM. The only penalty which one pays for not running in RAM is the startup-latency which only occurs the *first* time the program is started. Plus, there are other things which can be done to improve those latency figures.

Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file. Does the save file only include config files and other highly-selected items, or can it also contain installed programs? Is the save file an sfs? If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session. Maybe this is the 'crux' the lenghty times to save on shutdown. I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc. With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.

My impression, taken from a couple of polls which have been done here, is that most Puppy users use a frugal installation, and that they do that because of the ease of manually installing it, if wanted, and/or because it allows them to have a Puppy installation co-reside with a pre-installed Windows OS without the need for any partitioning and less out of having to work with a low-spec machine. If that is the case, then the use of RAM as a hard-disk substitute would seem to be a bad choice. And the use of write-only save files -if that is the case.

The light-weightness or agility of Puppy is mostly due to the non-use of kde, gnome and other such bloated software. There is a significant and 'feelable' speed-up when starting programs for the first time which are in RAM instead of being read in from disk. But, they all have to be loaded into RAM during boot-up which is why it is difficult to make Puppy boot up faster. My kiss-5.0 starts, from grub-prompt to GUI login, in right at 10 seconds here (on a Pentium IV) -even though it has to read everything from disk, link it all, etc. I think it gets there faster than if I spent time loading it all into RAM and then load and link it from there. And I can still improve the boot-time a little, I think, without going to extremes.

At the very least, it is pretty obvious that most Puppy users *do* use a frugal install, so more effort should be put into making that go as smoothly and quickly as possible -perhaps by having an initrd dedicated to frugal installs, instead of wasting hundreds of lines of code and time, by checking for all the other possible way to run Puppy. Things like pfix=ram are nice and useful, but not often needed and mostly exist as a work-around to fix other design flaws -in my hardly-humble opinion, LOL. In this same category, I would put things like 'use a separate save-file for separate users', when the obvious solution was there all along -have a real multi-user-capable system just like everybody else. That capability had to be built *out* inetntionally, and makes it completley unwise to use Puppy as a server or for other mission-critical purposes. It also makes it difficult, if not impossible without patching or hacks, to use some software like Chrome browser, etc.

Every single drawback goes back to a concious design choice which was made. And every advantage which Puppy may have over any other distro, could be implemted in such a way to avoid the drawbacks. Wanna run as 'root', autologin as root, autologin as root straight to the desktop? No problem. It doesn't mean that you have to completely eliminate the choice of doing otherwise.
You want to run from ram if disk I/O is very slow or if disk writes are undesirable - like USBflash, CD, or old hard drives, or maybe if you want to be able to shut down your disk to save power.

The initrd in a normal puppy is pretty emtpy. All it does is find the system sfs and your save file, do the union mount, and switch root.

The save file is not an sfs, just a regular ext FS image - it doesn't get copied to ram, and yes you can install packages and copy whatever you want into it.

Copying the main sfs to RAM is optional. What gets copied to RAM depends on how much RAM you have and what media you are using.

I'm sure it's possible to build something similar that's less adhoc, but yeah, you'd want to start from scratch.

disciple
Posts: 6984
Joined: Sun 21 May 2006, 01:46
Location: Auckland, New Zealand

#116 Post by disciple »

Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file.
The save file or the main .sfs file (e.g. pup_411.sfs)?
Does the save file only include config files and other highly-selected items, or can it also contain installed programs?
The save file contains installed programs and every other change you make to the system.
Is the save file an sfs?
No, it is an ext2 or ext3 filesystem in a file. Maybe even ext4 these days.
If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session.
There is usually no copy of the save file - it is not loaded into RAM.
Maybe this is the 'crux' the lenghty times to save on shutdown.
The only shutdown problems I've heard of are specifically related to filesystems not being unmounted; lately I think just Samba mounts.
I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc.
That's what the save file is.
With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.
That sounds interesting. What would the performance be like on an old, slow machine?
The point in having the original Puppy files in an .sfs and putting anything new in a save file is:
1) for running from LiveCD (not usually multisession),
2) small backups, and
3) easy system upgrades (just replace a couple of files).
Do you know a good gtkdialog program? Please post a link here

Classic Puppy quotes

ROOT FOREVER
GTK2 FOREVER

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#117 Post by amigo »

Thanks for helping out there with the meaty answers. So, when you boot a 'frugal' installation, the intrid (a fat one?) is used, setting up the main sfs as the root partiton, then the save file is unioned above that? Is that about the flow of things?

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

Package management

#118 Post by 2byte »

Package management.

I think we all agree that full super duper pm is not possible with puppy. That doesn’t mean it can’t be made better.

Jemimah, I agree Barry should not rewrite woof to suit us, but I don’t think he would have to. I am not a woof expert by any means but there has to be a place during the build when the pets and packages are being added where a branch could be made to a separate routine that could assemble the package/file info for use in the database. If we presented a plugin bash routine that would do this surely he could call it from woof at the appropriate spot.

The plugin could produce a record of the package/pet with a complete list of files. The file paths, permissions, owner:group, size, time added, and md5 could be produced. Probably the package type and, if it’s a distro package, where it came from could also be recorded. If the package/pet has a dependency list, build script or info, or pinstall or doinst script then that could be recorded too. At the very least we could produce the package name and related file list.

This could be a base to build a better ppm from and it would cost little effort from Barry to implement.

amigo, how puppy works http://puppylinux.com/development/howpuppyworks.html


noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#119 Post by noryb009 »

We have kind of gotten off topic.

As I see it, there are 4 steps for making a new package format:
1) Decide the format (a new format, judging by the poll)
2) Write a package manager for the format (either from scratch, based on PPM, or based on another distribution's package manager)
3) Write a script based installer (or add the format to an existing one)
4) Add the format to woof as an option, then eventually the default

Steps 2-4 can't be done before step 1, so I suggest we start there. I have already posted an idea of a package format, would someone else like to either change it, or suggest a new format?

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#120 Post by jpeps »

noryb009 wrote:We have kind of gotten off topic.

As I see it, there are 4 steps for making a new package format:
1) Decide the format (a new format, judging by the poll)
You have a good sense of humor. What the poll indicates is that, with a very few rare exceptions, nobody really cares what's happening under the hood, and even fewer really understand it. That's actually a vote for leaving things pretty much alone.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#121 Post by noryb009 »

We at least took a sample* of the signed up "Suggestions" board readers. 16/25 people voted for a new format, which is 64% - almost two thirds of the users. You are welcome to put up a poll in a more viewed area, but until we get more data, we should to use the data we have - which is to create a new package format.

* I can't link directly on the forum as the link contains brackets. Real path is: http://en.wikipedia.org/wiki/Sample_(statistics)

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#122 Post by jpeps »

noryb009 wrote:We at least took a sample* of the signed up "Suggestions" board readers. 16/25 people voted for a new format, which is 64% - almost two thirds of the users. You are welcome to put up a poll in a more viewed area, but until we get more data, we should to use the data we have - which is to create a new package format.

* I can't link directly on the forum as the link contains brackets. Real path is: http://en.wikipedia.org/wiki/Sample_(statistics)
Simply incredible...

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#123 Post by jemimah »

amigo wrote:Thanks for helping out there with the meaty answers. So, when you boot a 'frugal' installation, the intrid (a fat one?) is used, setting up the main sfs as the root partiton, then the save file is unioned above that? Is that about the flow of things?
The "fat initrd" was pretty much a passing thing. 99% of the time a small initrd is used.

Image

This is old but mostly still accurate:
http://puppylinux.com/development/howpuppyworks.html

I'm curious what architecture you would design to solve the bootable usb problem. You've got to keep the system as small as possible to fit on a small drive and still leave space for the user's files, and you need to avoid disk i/o as much as possble, both because it is slow and because writes cause wear on the drive. As far as I know puppy is the only system that is really designed to work well under these circumstances.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#124 Post by jpeps »

jemimah wrote: This is old but mostly still accurate:
http://puppylinux.com/development/howpuppyworks.html
jemimah, 2byte just posted that a few responses ago :)

Maybe a poll for how many people read it...

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

Re: Package management

#125 Post by jemimah »

2byte wrote: Jemimah, I agree Barry should not rewrite woof to suit us, but I don’t think he would have to. I am not a woof expert by any means but there has to be a place during the build when the pets and packages are being added where a branch could be made to a separate routine that could assemble the package/file info for use in the database. If we presented a plugin bash routine that would do this surely he could call it from woof at the appropriate spot.

The plugin could produce a record of the package/pet with a complete list of files. The file paths, permissions, owner:group, size, time added, and md5 could be produced. Probably the package type and, if it’s a distro package, where it came from could also be recorded. If the package/pet has a dependency list, build script or info, or pinstall or doinst script then that could be recorded too. At the very least we could produce the package name and related file list.

I'm just not seeing what you think this DB is going to do for you. What functional enhancement is this going to provide?

You can pull the package name, dependencies, origin, and related file list out of puppy's existing DB.

Post Reply