Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 31 Oct 2014, 05:32
All times are UTC - 4
 Forum index » Taking the Puppy out for a walk » Suggestions
The State of Package Management
Moderators: Flash, Ian, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 8 of 15 Posts_count   Goto page: Previous 1, 2, 3, ..., 6, 7, 8, 9, 10, ..., 13, 14, 15 Next

Should Puppy's package format be changed?
Yes, without backwards compatibility.
28%
 28%  [ 11 ]
Yes, with backwards compatibility.
25%
 25%  [ 10 ]
No, but the PET format should be standardized/stricter.
20%
 20%  [ 8 ]
No, the PET format works fine.
25%
 25%  [ 10 ]
Total_votes : 39

Author Message
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Mon 20 Feb 2012, 03:22    Post_subject:  

amigo wrote:
2byte gets it! Very well put, fellow.


+1

I spent a few hours trying to connect listed builtin files with right pet.spec files in woof before giving up. I hadn't realized it's full of debs.
Back to top
View user's profile Send_private_message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Mon 20 Feb 2012, 10:48    Post_subject:  

Aitch wrote:
At least this discussion seems to be drawing package management to a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?


This looks liked good progress -
Question: As devs do you not feel you have a right to ask Barry to help sort out the chaos he's created, or risk good devs forever leaving/forking, in an effort to create a good working, useable and updatable OS
I don't see it as insulting, as even he must realise [I hope] the problems he's creating for you to try to sort out....he does have a natural responsibility for his creation which you guys don't, after all....?
Else Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....

Good discussion, guys'n'gals, thanks!

btw, I think jrb's pinstall.sh script idea needs more awareness, so here's a link

http://www.murga-linux.com/puppy/viewtopic.php?p=603808#603808

Aitch Smile
Back to top
View user's profile Send_private_message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Mon 20 Feb 2012, 13:01    Post_subject:  

Barry is very quick to accept bug fixes, and enhancements, but it seems unlikely that he'll want to rewrite woof.

Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.

What you should be asking for is a better, simpler, more user-friendly puppy builder.
Back to top
View user's profile Send_private_message Visit_website 
amigo

Joined: 02 Apr 2007
Posts: 2263

PostPosted: Mon 20 Feb 2012, 13:10    Post_subject:  

jpeps, the light gets brighter the closer you get to the end of the tunnel.
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.

"full of debs" -If your gonna have your own distro, surely you should be consistent about the content, where it comes from and how it is constructed -or ??

Aitch, "ask Barry to help sort out the chaos" -surely he would have done this if he thought it right or sensible. After all, it is *he* who would most benefit from it.

"good devs forever leaving/forking" Happens all the time. The best *have* spoken to BK about it without result.

"Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version" That's the most concise description of Puppy I have ever seen. Starts out broken, bugs don't get fixed, not up-gradeable -incapable of learning from its' own mistakes for the future benefit of both devs and users. I am skeptical, though, about the validity of this: "less broken later version"

Aitch, I *do* have a very workable, independent package format in place and working very well. Still undergoing some refinement, but already past the point of needing any changes which would affect backard or forward compatibility. Of course, the package format is supported by tools for the creation of compliant packages (src2pkg and others), and the tools for sanely installing, adding, removing, replacing or upgrading packages. There are also advanced tools which analyze the database created by during the above operations. There is still lots to do there, but new little things get added nearly every day. It certainly is functional enough for managing a completze distro with thousands of packages.

Unfortunately, writing an acceptable document which describes even the most basic concepts or specifications of the package format and tools would take me many, many hours of work. The tools do, of course, include '--help' functionality and man-pages, but they don't tell the whole story. Such documents will exist one day, but I'm still much to busy implementing advanced features (like automatic dependency *resolution*) to work on that now. It would be a waste of time to write a nice document when some aspects of the format and the tools are still changing -even though the changes are minor.

Note, while automatic dependency resolution is still not implement in the installer tools, the format already contains the needed 'slots' for the database files and they are generated automatically by src2pkg when building *.tpkg packages. The packages also already contain these files, as needed.

The 'resolver' itself will not be hugely complex and the algorithms will be completly separate from the normal installation tools. Even the tpkg-upgrade tool does not directly install the new package and remove the old one. It uses the individual tools written for each purpose. The same division of labor will be used for any GUI package management tools -they will rely on the underlying tools. This serves at least two purposes -it keeps our 'meaty' code for actually de-compressing archives, running install scripts and updating the database separate from 'window-dressing' code for a GUI. It also means that the GUI code is cleaner and less complex -read easier-to-write and easier to understand or maintain. A low-level tool for each individual operation, with higher-level tools serving as a front-end to them -the UNIX way, you know.

Since I'm 'rolling my own', I don't have to be concerned about getting anyone's 'approval'. If it works it will get adopted by others. Actually, I entertain no desire to have it used by Puppy, as by association it might be identified with Puppy. In fact, I don't even care if nobody else ever uses my format -I've done it to meet my own needs and for the pleasure of creating something that 'just works'.
Back to top
View user's profile Send_private_message 
amigo

Joined: 02 Apr 2007
Posts: 2263

PostPosted: Mon 20 Feb 2012, 13:22    Post_subject:  

jemimah, I believe that a re-think of what comprises a 'frugal' installation would make package-management still possible. For instance, drop the 'frugal' part and simply install the whole system to a disk-image -no need for a save file, only needs a tiny initrd, and uses no RAM for parts of the system. Simply put, a loop-mounted system-in-file. Only the kernel and small initrd would be separate -just as now. I built just such a distro before -nooby would have loved it!
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.
Back to top
View user's profile Send_private_message 
noryb009

Joined: 20 Mar 2010
Posts: 540

PostPosted: Mon 20 Feb 2012, 14:29    Post_subject:  

Quote:
As devs do you not feel you have a right to ask Barry to help sort out the chaos he's created, or risk good devs forever leaving/forking, in an effort to create a good working, useable and updatable OS

I don't feel it's Barry's duty to do anything, but he should do what is best for Puppy, which would be to help.

Quote:
Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....

That is one of the things a "real" package management system would hopefully fix.

Quote:
Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.

Nobody is forcing you to upgrade, but it is much better then the current system, as Aitch called it, "being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....". You can currently overwrite woof-installed packages (and it usually works, if there aren't any extra dependencies needed).

Quote:
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.

Are you talking about a package list inside a package repository? Package lists are needed for description searching and looking for package dependencies before downloading the package. Every distribution I know of that has a repository system has a package list - there is no reason why Puppy can't have one.
Back to top
View user's profile Send_private_message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Mon 20 Feb 2012, 14:32    Post_subject:  

amigo wrote:
jemimah, I believe that a re-think of what comprises a 'frugal' installation would make package-management still possible. For instance, drop the 'frugal' part and simply install the whole system to a disk-image -no need for a save file, only needs a tiny initrd, and uses no RAM for parts of the system. Simply put, a loop-mounted system-in-file. Only the kernel and small initrd would be separate -just as now. I built just such a distro before -nooby would have loved it!
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.



My primary reason for running puppy is to run it from RAM on low spec machines. Since it's in RAM - compression is extremely desirable. Also running from a usb stick or rewritable CD with limited space makes the compression very helpful.

The existing puppy system works ok with uninstalling packages as it is, but since the system sfses are read-only, deleted files aren't really gone until you remaster. Running it like a normal distro just makes a huge save file which negates the reason to run puppy at all. I personally don't think it's worth the effort to try retrofit complete package management as there is very little actual benefit. Time is better spent on figuring out ways to make it easier to build a puplet.

In my opinion, if the compressed filesystems are not of importance to you, you'd be better off with a "real" distro. Puppy is fantastic for the niche it's designed for - but it's not great for anything else. Trying to fix that is just going to be an exercise in futility.

Just my opinion, carry on.
Back to top
View user's profile Send_private_message Visit_website 
amigo

Joined: 02 Apr 2007
Posts: 2263

PostPosted: Mon 20 Feb 2012, 15:14    Post_subject:  

"from RAM on low spec machines" I never have been able to understand this rationale. If a system is weak, whhy sould one want to occupy 'any' RAM which is not necessary?. Especially when you consider that, onve a program has been run or is running, there are 'two* copies of it in RAM. The only penalty which one pays for not running in RAM is the startup-latency which only occurs the *first* time the program is started. Plus, there are other things which can be done to improve those latency figures.

Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file. Does the save file only include config files and other highly-selected items, or can it also contain installed programs? Is the save file an sfs? If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session. Maybe this is the 'crux' the lenghty times to save on shutdown. I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc. With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.

My impression, taken from a couple of polls which have been done here, is that most Puppy users use a frugal installation, and that they do that because of the ease of manually installing it, if wanted, and/or because it allows them to have a Puppy installation co-reside with a pre-installed Windows OS without the need for any partitioning and less out of having to work with a low-spec machine. If that is the case, then the use of RAM as a hard-disk substitute would seem to be a bad choice. And the use of write-only save files -if that is the case.

The light-weightness or agility of Puppy is mostly due to the non-use of kde, gnome and other such bloated software. There is a significant and 'feelable' speed-up when starting programs for the first time which are in RAM instead of being read in from disk. But, they all have to be loaded into RAM during boot-up which is why it is difficult to make Puppy boot up faster. My kiss-5.0 starts, from grub-prompt to GUI login, in right at 10 seconds here (on a Pentium IV) -even though it has to read everything from disk, link it all, etc. I think it gets there faster than if I spent time loading it all into RAM and then load and link it from there. And I can still improve the boot-time a little, I think, without going to extremes.

At the very least, it is pretty obvious that most Puppy users *do* use a frugal install, so more effort should be put into making that go as smoothly and quickly as possible -perhaps by having an initrd dedicated to frugal installs, instead of wasting hundreds of lines of code and time, by checking for all the other possible way to run Puppy. Things like pfix=ram are nice and useful, but not often needed and mostly exist as a work-around to fix other design flaws -in my hardly-humble opinion, LOL. In this same category, I would put things like 'use a separate save-file for separate users', when the obvious solution was there all along -have a real multi-user-capable system just like everybody else. That capability had to be built *out* inetntionally, and makes it completley unwise to use Puppy as a server or for other mission-critical purposes. It also makes it difficult, if not impossible without patching or hacks, to use some software like Chrome browser, etc.

Every single drawback goes back to a concious design choice which was made. And every advantage which Puppy may have over any other distro, could be implemted in such a way to avoid the drawbacks. Wanna run as 'root', autologin as root, autologin as root straight to the desktop? No problem. It doesn't mean that you have to completely eliminate the choice of doing otherwise.
Back to top
View user's profile Send_private_message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Mon 20 Feb 2012, 15:18    Post_subject:  

jemimah wrote:
Puppy is fantastic for the niche it's designed for - but it's not great for anything else. Trying to fix that is just going to be an exercise in futility.

Just my opinion, carry on.


I think any system can be individually tailored; I use my own checks, uninstaller, etc... Remastering is simple, but rarely needed. The present system is light years ahead of something like TC.

Trying to adopt a sane approach is always subject to differing ideas and politics. I created the first update system for TC, and spent months perfecting checks, fixing bugs, etc., etc., only to have it continually sabotaged by RS because too many people liked it. So I've learned...roll your own, offer improvements, and stay out of the way.
Back to top
View user's profile Send_private_message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Mon 20 Feb 2012, 15:39    Post_subject:  

amigo wrote:
"from RAM on low spec machines" I never have been able to understand this rationale. If a system is weak, whhy sould one want to occupy 'any' RAM which is not necessary?. Especially when you consider that, onve a program has been run or is running, there are 'two* copies of it in RAM. The only penalty which one pays for not running in RAM is the startup-latency which only occurs the *first* time the program is started. Plus, there are other things which can be done to improve those latency figures.

Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file. Does the save file only include config files and other highly-selected items, or can it also contain installed programs? Is the save file an sfs? If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session. Maybe this is the 'crux' the lenghty times to save on shutdown. I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc. With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.

My impression, taken from a couple of polls which have been done here, is that most Puppy users use a frugal installation, and that they do that because of the ease of manually installing it, if wanted, and/or because it allows them to have a Puppy installation co-reside with a pre-installed Windows OS without the need for any partitioning and less out of having to work with a low-spec machine. If that is the case, then the use of RAM as a hard-disk substitute would seem to be a bad choice. And the use of write-only save files -if that is the case.

The light-weightness or agility of Puppy is mostly due to the non-use of kde, gnome and other such bloated software. There is a significant and 'feelable' speed-up when starting programs for the first time which are in RAM instead of being read in from disk. But, they all have to be loaded into RAM during boot-up which is why it is difficult to make Puppy boot up faster. My kiss-5.0 starts, from grub-prompt to GUI login, in right at 10 seconds here (on a Pentium IV) -even though it has to read everything from disk, link it all, etc. I think it gets there faster than if I spent time loading it all into RAM and then load and link it from there. And I can still improve the boot-time a little, I think, without going to extremes.

At the very least, it is pretty obvious that most Puppy users *do* use a frugal install, so more effort should be put into making that go as smoothly and quickly as possible -perhaps by having an initrd dedicated to frugal installs, instead of wasting hundreds of lines of code and time, by checking for all the other possible way to run Puppy. Things like pfix=ram are nice and useful, but not often needed and mostly exist as a work-around to fix other design flaws -in my hardly-humble opinion, LOL. In this same category, I would put things like 'use a separate save-file for separate users', when the obvious solution was there all along -have a real multi-user-capable system just like everybody else. That capability had to be built *out* inetntionally, and makes it completley unwise to use Puppy as a server or for other mission-critical purposes. It also makes it difficult, if not impossible without patching or hacks, to use some software like Chrome browser, etc.

Every single drawback goes back to a concious design choice which was made. And every advantage which Puppy may have over any other distro, could be implemted in such a way to avoid the drawbacks. Wanna run as 'root', autologin as root, autologin as root straight to the desktop? No problem. It doesn't mean that you have to completely eliminate the choice of doing otherwise.


You want to run from ram if disk I/O is very slow or if disk writes are undesirable - like USBflash, CD, or old hard drives, or maybe if you want to be able to shut down your disk to save power.

The initrd in a normal puppy is pretty emtpy. All it does is find the system sfs and your save file, do the union mount, and switch root.

The save file is not an sfs, just a regular ext FS image - it doesn't get copied to ram, and yes you can install packages and copy whatever you want into it.

Copying the main sfs to RAM is optional. What gets copied to RAM depends on how much RAM you have and what media you are using.

I'm sure it's possible to build something similar that's less adhoc, but yeah, you'd want to start from scratch.
Back to top
View user's profile Send_private_message Visit_website 
disciple

Joined: 20 May 2006
Posts: 6449
Location: Auckland, New Zealand

PostPosted: Mon 20 Feb 2012, 15:59    Post_subject:  

Quote:
Honestly, I'm not exactly sure what all features are implemented by Puppy's 'frugal' installations. My impresion has been that some parts of the system are located in the initrd and may or may not be overlayed by the contents of the save file.

The save file or the main .sfs file (e.g. pup_411.sfs)?
Quote:
Does the save file only include config files and other highly-selected items, or can it also contain installed programs?

The save file contains installed programs and every other change you make to the system.
Quote:
Is the save file an sfs?

No, it is an ext2 or ext3 filesystem in a file. Maybe even ext4 these days.
Quote:
If so, I find that very wasteful in one sense -it means you have to always create a copy of it, in RAM or on-disk, to be able to then update at the end of your session.

There is usually no copy of the save file - it is not loaded into RAM.
Quote:
Maybe this is the 'crux' the lenghty times to save on shutdown.

The only shutdown problems I've heard of are specifically related to filesystems not being unmounted; lately I think just Samba mounts.
Quote:
I find the idea of using a normal FS-image using ext2/3/4 much more palatable -you can write to it directly and continuosuly, re-size it, etc.
That's what the save file is.
Quote:
With ext4 you can even have transparent on-the-fly compression. You can also achieve compression on ext2 or ext3 by using a kernel patch ('e2c' project on SourceForge). You can also use compression and even an alternative to union-mounts with the new brtfs file system. In short, the drawbacks to not using an 'sfs' can be worked around in several ways, if that is the bottleneck.

That sounds interesting. What would the performance be like on an old, slow machine?
The point in having the original Puppy files in an .sfs and putting anything new in a save file is:
1) for running from LiveCD (not usually multisession),
2) small backups, and
3) easy system upgrades (just replace a couple of files).

_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send_private_message 
amigo

Joined: 02 Apr 2007
Posts: 2263

PostPosted: Mon 20 Feb 2012, 16:55    Post_subject:  

Thanks for helping out there with the meaty answers. So, when you boot a 'frugal' installation, the intrid (a fat one?) is used, setting up the main sfs as the root partiton, then the save file is unioned above that? Is that about the flow of things?
Back to top
View user's profile Send_private_message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Mon 20 Feb 2012, 17:11    Post_subject: Package management  

Package management.

I think we all agree that full super duper pm is not possible with puppy. That doesn’t mean it can’t be made better.

Jemimah, I agree Barry should not rewrite woof to suit us, but I don’t think he would have to. I am not a woof expert by any means but there has to be a place during the build when the pets and packages are being added where a branch could be made to a separate routine that could assemble the package/file info for use in the database. If we presented a plugin bash routine that would do this surely he could call it from woof at the appropriate spot.

The plugin could produce a record of the package/pet with a complete list of files. The file paths, permissions, owner:group, size, time added, and md5 could be produced. Probably the package type and, if it’s a distro package, where it came from could also be recorded. If the package/pet has a dependency list, build script or info, or pinstall or doinst script then that could be recorded too. At the very least we could produce the package name and related file list.

This could be a base to build a better ppm from and it would cost little effort from Barry to implement.

amigo, how puppy works http://puppylinux.com/development/howpuppyworks.html

_________________

Back to top
View user's profile Send_private_message 
noryb009

Joined: 20 Mar 2010
Posts: 540

PostPosted: Mon 20 Feb 2012, 17:33    Post_subject:  

We have kind of gotten off topic.

As I see it, there are 4 steps for making a new package format:
1) Decide the format (a new format, judging by the poll)
2) Write a package manager for the format (either from scratch, based on PPM, or based on another distribution's package manager)
3) Write a script based installer (or add the format to an existing one)
4) Add the format to woof as an option, then eventually the default

Steps 2-4 can't be done before step 1, so I suggest we start there. I have already posted an idea of a package format, would someone else like to either change it, or suggest a new format?
Back to top
View user's profile Send_private_message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Mon 20 Feb 2012, 18:52    Post_subject:  

noryb009 wrote:
We have kind of gotten off topic.

As I see it, there are 4 steps for making a new package format:
1) Decide the format (a new format, judging by the poll)


You have a good sense of humor. What the poll indicates is that, with a very few rare exceptions, nobody really cares what's happening under the hood, and even fewer really understand it. That's actually a vote for leaving things pretty much alone.
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 8 of 15 Posts_count   Goto page: Previous 1, 2, 3, ..., 6, 7, 8, 9, 10, ..., 13, 14, 15 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Taking the Puppy out for a walk » Suggestions
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1581s ][ Queries: 14 (0.0132s) ][ GZIP on ]