Should Full HD install be faster than Frugal install?

[MU]

swapoff is good.
It is a button to turn swap off. So swap is used at moment.

I think you can not choose where to put the pup_save.3fs, Puppy does this automatically.
Maybe if you move it somewhere else by running a life-CD (Puppy1 or Puppy2 with the option pfix=ram or so), then Puppy might use the change?
Don't know that.
One thing good to know:
you can use several pup_save.3fs.
If you add a pup_save1.3fs, Puppy offers at boot a menu to choose from.

Mark

[fitzhugh]

Ok, good to know about the swapoff - I thought that it was telling me it was off and if I clicked it it would turn on. When I DO press it it turns to:



Note the small text that is barfed up kinda behind the 2nd partition (530M - told it 512 M since I have 256 M ram but it made it 530. not sure why). I can't turnit back on via MUT but I know there is a command, just haven't looked for it yet. In general I don't know why this will come up now that you explained this to me... I won't be turning it off.

Next: pup_save... yeah, I've used multiple pup_saves before but currently have only one. I think I'm missing something... I used grub to boot puppy in partition hda7 but it is using pup_save.sfs from hda1. So what I'm really seperating here are: partition on one level, then which COPY of the kernel and puppy_201.sfs I'm running by booting to different partitions, then which files via which pup_save.sfs I choose?
As I install libraries, applications etc. they get added to the pup_save that is currently being used, not to pup_201. So I could swap out pup_201 for an old pup_200 and it would continue to use all my normally/recently installed apps? (assuming they didn't require something missing in pup_200).

I noticed that now mplayer doesn't work, missing a library, though I've not tried it after booting from the original partition yet - will see if I even can boot from that partition when I post this.

The remaster liveCD: this creates a new pup_201 that includes all the changes, doesn't it? So if I want I can create a snapshot copy of my pup_save wrapped into a new pup_201 this way and put that on the other partitions? I know I can just copy pup_saves as well.

Basically I m stii trying to grasp exactly how to seperate my files from the applications from the os. These files are all over the place but I see them as a unified whole inside puppy? I've booted (so I think) the pup_201.sfs on hda7 but my symlinks out to the windows filesystem (My Documents file, for example) still work? Magic?

Reminds me: the devx_201.sfs contains everything to run the compiler. It can be added or removed without any installation just by either including it or not in the directory. Can other applications be packaged that way? THAT would be great! I want open office, I just drop openoffice_201.sfs into place. I want Barry's New Tiny All Purpose Everything Editor with Carrot Juicer in 1.3 Megs instead I just remove the openoffce_201.sfs and drop in BarrysEverythingEditor_201.sfs and boot. A modular system like that would be so much less intimidating to users than having to install. I can think of a few reasons why it might not work, but I gather it did with pup_more in puppy 1 to add open office (don't know any more than that it was done)' It would make configuring so much easier. What am I missing (a clue, I know, but what clue)?

fitzhugh

[fitzhugh]

Root vs rootnoverify in grub menu.1st? I read online about the two but don't understand. Any one have a simple explaination?

Thanks,
Fitzhugh

[MU]

what you say about openoffice works, yes.
search for mksquashfs for details.

http://www.puppyos.com/nfphpbb/search.php

Mark

[fitzhugh]

Ah, cool! I see how to convert the usr_more to usr_201 using mksquashfs, though I'm not clear on the general case... can this be done with any application I want? Is it possible to turn other things into *_201.sfs? That would simplify and speed up system building, sharing files, etc. If I understand, the squash filesystem is what layers one on top of another, so this is how it deals with the need open office might have for a file in a directory that is already in use by other applications - config files etc. I guess I'm wondering, if this is right, how do I scrape out just the files installed by an application and create a new .sfs? I am not looking for a long answer using lots of your time, just whether this is the right concept to chase down. I'll gladly spend the time to read about it.

If this is correct, what are the drawbacks? why do we use .pups instead?

One of my next projects is goingto be creating a .pup myself so I know what it invovles and can learn more about puppy. I'd like to learn about the above .sfs stuff concurrently.

Thanks!
Fitzhugh

[MU]

you can mount up to 4 such files I think.
So you should not use it for every single small app.

I think my megapup is a good example how to use it.
It contains a bunch of apps with its own launcher to start them.
http://dotpups.de/puppy-releases/2.01/K ... va-1.5.06/

Like this you have the base-system and complex apps seperated.
You easily can upgrade to Puppy 2.0x without downloading a full 600MB-iso (a megapup-Iso would have this size).

Also, an additional unionfs-layer always enhances the danger of errors.
The bug-list for unionfs was long, and it is a mere wonder, how good Barry got it working in Puppy2

To use Dotpups/unleashed packages:
examine /root/.packages/XXX.files , and make it of those listed there.
Unleashed and Dotpups created with my wizard register their files there.
Or extract them manually.
Unleashed packages are simple .tar.gz -files, so it is easy.
Dotpups are simple .zip with 1 or 2 .tar.gz in them, so it is easy too in most cases (except they use an additional script to set up things).
Mark

[fitzhugh]

*** EDIT *** I see that some of these questions are answered at the megapup site, so I'll return and mark the ones I answered myself... I just got all excited about this and jumped ahead. ***/EDIT***

you can mount up to 4 such files I think.
So you should not use it for every single small app.

So there is a limit built into puppy, or is it a limit in squashfs for the number of layers? Hadn't heard mention of a limit before but hadn't looked deeply yet.
Any reason why I couldn't make one for large productivity oriented programs like office, gimp, one for turning puppy into a lamp setup with an additional nice wiki, forum, etc all there, one for games, for the window manager / desktop I pick... up to four that I can add, or four including devx_20*?

What exactly is the naming convention? Is it ANY.sfs? I read an earlier thread you linked to about converting devx_more into devx_001, as Barry wrote (I think), but we now use devx_201 and devx_200 does NOT work with puppy 2.01. Is that because it needs to map to the puppy version number or because of internal incompatabilities?

I think my megapup is a good example how to use it.
It contains a bunch of apps with its own launcher to start them.
http://dotpups.de/puppy-releases/2.01/K ... va-1.5.06/

So, you add on these via squashfs? I'll check it out. I've wanted to try KDE for some time because I see things I like for it (like ksearch, or whatever it was called, and others). I'll see what you have done and how.

Like this you have the base-system and complex apps seperated.

wonderful! that is what I want.

You easily can upgrade to Puppy 2.0x without downloading a full 600MB-iso (a megapup-Iso would have this size).

Not quite following you here... you mean if you install megapup then when the next 2.02 etc. comes out just download that and use the .sfs that are there? Or do you mean something else? Also, regarding naming convention and such, what are the issues regarding these and upgrades, other than they need to be consistant with the underlying file structure and capabilities of the system?

Also, an additional unionfs-layer always enhances the danger of errors.
The bug-list for unionfs was long, and it is a mere wonder, how good Barry got it working in Puppy2
[/qoute]
yeah, I didn't realilze it wasn't more common, because it does a great job here and pulls of some pretty cool geek magic!

To use Dotpups/unleashed packages:
examine /root/.packages/XXX.files , and make it of those listed there.

Do you mean use what is in .packages/XXX.files to figure out where each current package puts what if I want to create a new .sfs version? That helps a bunch, but what about other things that I get - how do I know what is put where in order to re-create it as a .sfs?

I think I want to use this most to capture a 'portable' version of my most recent fully funnctioning setup. Being able to add something one piece at a time and make sure it works, add it to the appropriate .sfs, and then be able to move that to other copies of puppy would be wonderful.

Unleashed and Dotpups created with my wizard register their files there.
Or extract them manually.
Unleashed packages are simple .tar.gz -files, so it is easy.
Dotpups are simple .zip with 1 or 2 .tar.gz in them, so it is easy too in most cases (except they use an additional script to set up things).
Mark

This is really exciting... now I'll get even less sleep at night

[MU]

So there is a limit built into puppy, or is it a limit in squashfs for the number of layers? Hadn't heard mention of a limit before but hadn't looked deeply yet.

It is a Kernel-setting.
More make it slower, I think you can adress 256 in theory.
We could evaluate that later, first get common with it

What exactly is the naming convention? Is it ANY.sfs?

FreeName_Puppyversion.sfs
e.g. KDE_201.sfs
OpenOffice_201.sfs
for Puppy 2.01

So, you add on these via squashfs?

http://www.murga.org/~puppy/viewtopic.p ... hlight=kde

Not quite following you here... you mean if you install megapup then when the next 2.02 etc. comes out just download that and use the .sfs that are there?

Yes. just rename it from MP003_201.sfs to MP003_202.sfs then.
That's all

Do you mean use what is in .packages/XXX.files to figure out where each current package puts what if I want to create a new .sfs version? That helps a bunch, but what about other things that I get - how do I know what is put where in order to re-create it as a .sfs?

yes. For others it depends.
I sometimes compile apps, then I set the directory-listing in XFE so that it sorts by date of change.
Then I see quickly which folders changed and have the new files.
XFE: http://www.murga.org/~puppy/viewtopic.php?t=1922
You also could use Puppysearch or Puppys search-tools to search files by time of change.

This is really exciting... now I'll get even less sleep at night Smile

hehe...
Mark

[MU]

more loops:
http://www.murga.org/~puppy/viewtopic.php?p=40130#40130
Did not try it yet.

Ah yes, and devx_201.sfs is one of the 4.

That thread handles the details of loop-devices.
Puppy might not mount automatically more than 4 loops, even if the kernel supports more, in that case you had to mount them manually.
That is described there, too, though quite technical by nature.

It gets time to adapt my grafical tool for Puppy2:
http://www.murga.org/~puppy/viewtopic.php?t=7482

Mark

[fitzhugh]

Thanks! You also reminded me how I'd wanted to get xfe! I really miss a good (in my opinion) file manager, and it sounds a lot better.

I understand enough now to begin attempting this, both wrapping up stuff as seen in .packages into my own squashfs and compiling new stuff, seeing what was added, and re-creating it into a squashfs. I think I'll use a seperate puppy for these compile attempts. I also like the virtual drive solution you posted... can I alter the size of the 'drives'? Or add more? That pup doesn't create a .sfs, does it? Just curious - I understand it is just for puppy 1 so far.

There was a comment in that thread from Pizzasgood:

I use this concept for storing the dotpups I've downloaded and the configuration files for my wireless connection. That way I can start fresh when a new Puppy comes out and just mount it and reinstall everthing that I wan't, skipping things I don't want this particular time.

This touches on one of the core things I am aiming for: to modularize as much as possible. Here he is using one of your virtual drives for config files - How does the system know to find them? Don't they need to be in the right place, all mixed in with the config files from other things? What can and cannot be seperated out into its own seperate directory, partition or drive? If I remember correctly, the full java install I downloaded from sun is all under one directory and it said to uninstall just remove the directory, which is exactly what I want in one form or another for most things. Squashfs provides a way to overlay and therefor mix files in with others in existing directories, but still be able to exclude/include them all at once, but with java it is all in a seperate directory. Two different approaches to the same result: modularity. Am I getting this correct?
so, I understand how to start creating my own squashfs. Is it possible to do it like I think java does instead?

I've just been wondering this morning about loop devices since I think that is what I need to read about... I want to try accessing files in other pup_save*.sfs... in this case I just wanted my old pinboard because I edited the current one by hand, left in a typo, and when I restarted x it choked on the typo and actually erased the file! It decided it needed a new empty one and didn't save the bad one So, rather than reboot to my backup pup_save I wanted to just mount it... but I have not gotten to that yet. A lot of playing around just to avoid rebooting a couple times, but it is the resulting knowledge I'm really after. And this has come up a number of times. Sounds like the link to loops you posted is a good starting place - thanks!

One more question: I've only just started using forums when I started with puppy recently. I read the forum etiquette sticky post, but I'm not clear on when I should start a new thread and when I should add to an old one. I've been dividing up into seperate threads based on specific topic, as well as assuming that old threads on simiilar topics are dead, yet I've seen cases where dates of posts in threads show they were idle for a long time and then revived. I don't want to post a question in a thread no one will ever look at, but Idon't want to post so many different threads that it is a bother to others, plus it really helps people searching for similar answers when they are all in one thread.

A comment: I've been asking a LOT of questions. Please don't hesitate to answer with simple pointers toward more information. Usually I have looked for an answer and ended up in a dead end and just need a nudge down the right path, though details are great in other cases. All this help is fantastic and very appreciated, but I don't want to demand too much time from others. I'm starting with very little knowledge but am not limiting myself to just super-basic stuff. This means, of course, that I have a steep learning curve. Mastery of a domain usually depends more on knowing how to navigate the realm, how to find more info, than knowing everything possible. I'm getting there, slowly.

Fitzhugh

[fitzhugh]

Here is a major and simple concept I was entirely missing, which has added to my confusion greatly:
I was thinking, when I thought about it at all, that each partiton was seperate from the view of the filesystem, rather than understanding the concept of mounting. I didn't realize you would mount a partition to a directory inside the filesystem, and that it would then appear as a directory at that point from the filesystem. I thought to get to another partition you had to go all the way up to root, over, and down again - or traverse symlinks put in place. Doesn't solve everything for me, but it makes a lot more sense now. I undnerstood squashfs's ability to overlay, but had been working on the above incorrect assumption. I knew something[/] was wrong because it didn't make sense, but only just now had it hit me.

[sunburnt]

Here's a set of tests on most of the types of compressed fs & ALL of them are faster than raw files on the HD.
The HD being a mechanical device can't possably match the speed of the sys. memory & cpu as is shown.

http://kerneltrap.org/files/PERFORMANCE.README.txt

[fitzhugh]

sunburnt - thanks for posting that. I need a little help interpreting it, though, if you would...

I see that squashfs 2.1 is faster and smaller than those it is compared against, but I don't get how that would compare against the same tasks on the same data just sitting on an ext3 partition. Does my question here even make sense? You said they are faster than raw data on hdd since the hdd is a physical device and the others are in ram and cpu? Did I understand correctly? I guess I'm not understanding squashfs right: I thought it was a sort of virtual file system sitting in a file on a partition instead of directly on the partition, and assumed that there would be overhead in accessing that data. I also implicitly assumed (in other words had not considered it at all but included in my other assumptions) that it could then be pulled into ram disk as a seperate process, and that this could be done for any/most/many other kinds of filesystem, giving a large boost that was from not having to access a physical device. Clearly I seem to be missing sometihing, but I don't even know what question to ask... I'd appreciate any and all help, from urls to sites that might help to posts explaining what concepts I have wrong.

Thanks!
Fitzhugh

[sunburnt]

fitzhugh; Your confusion is quite understandable, it took me awhile to grasp what's going on.
In Puppy & Knoppix both, if there's enough memory to hold it, the compressed file is copied to the ramdisk.

This is NOT how the various compressed file systems work, the tests don't have the comp. files in memory.
The contense of the squash file (Puppy) or the cloop file (Knoppix) is never extracted, it's read directly.
Another words, the kernel module allows the file being read to be extracted on-the-fly, as it's used.
Your right, there's overhead in accessing a compressed file, but the files are smaller so they come off the HD faster.
On network mounted compressed files they operate faster also, same as HD, CD, or USB drives.
Newer PCs with faster cpu's do it in a micro second, faster than the HD can spool the data, understand?

Folks don't realize how slow a HD is compared to memory, BIG TIME, so the drives / network is the bottle neck.
Smaller files mean faster drive access, & most of the compression methods like gzip shrink files by 2.5 times!
So the drives / network is being accelerated by 2.5 times! This is a HUGE speed increase for data reading.
Most of what the computer does is reading data, not alot of writing goes on typically.

All this I've suspected for years, also what would be interesting to know would be where the brake even point is.
Another words, for a Pentium 200mhz is there no difference in compressed / uncomp. file access or not?
Or is the brake even point more K6 300, or Celeron 500, etc. (really it's academic, just curious though).

[rarsa]

(really it's academic, just curious though).

I've been following this thread and from the begining I had already reached this conclussion. This is a merely academic discussion. The difference is a teoretical difference.

I would say that unless you are working in a very odd environment such as a computer with lots of RAM and a very slow HDD or one with very little ram and a fast HDD you won't notice the difference.

[sunburnt]

rarsa; Actually the amount of ram wouldn't make any difference in the access speed between compressed files & non.
HD speed would... of course, & data access (app. loading) is a popular topic here, paticularly booting speed.
If 1MB of files must be read for a task & the HD only has to give out 400KB, it's gonna go faster, period.
HD latency will mute the speed up effects, especially for small files, like for HD storage small files waste space.

Ultimately most of what the PC does is read lots of small files (unfortunately), so both data rate & storage suffer.

[marksouth2000]

Actually the amount of ram wouldn't make any difference in the access speed between compressed files & non.

Sorry to disagree, sunburnt, but it does make a difference. Most operations on large data segments (like decompressing files) involve moving stuff around in RAM. If there's lots of it, then it's easy to find a space to move something to. Otherwise something else has to be moved first.

Sort of the way it's quicker to tidy a big apartment that has lots of space than a small one that is full of stuff. In the latter case you need to move a lot around in order to be able to tidy something else.

This is all even in the first edition of ast's book, IIRR.

This is also why overclocking is insane when RAM is relatively cheap. The same system can become faster by integer factors when RAM is added (with no other consequences), whereas overclocking by 50% gives speed gains of a few percent in exchange for reliability losses of several hundred percent.

Cheers,
Mark

[sunburnt]

Yes... but your now talking about system management & memory allocation, not just file access.
Paticularly in WinBlows, the OS shuffling data is tied directly to memory size.
But this has nothing to do with HD file access (strictly speaking as we were).
Memory isn't needed at all if the data is just tossed after it's read, but there's still the read speed.

[fitzhugh]

just getting back from a week's travelling so I'm playing catch up here...

Sunburnt: Thanks for the explaination - I really had been confused by that. As for access speed: I kept reading about super fast booting on older hardware, hardware even slower than the 1G PIII I was using up until a few days ago, but on that machine it took over a minute to boot a normal install (booting from CD but pup_save on HDD). Finally, and wholly by accident, I booted with the CD in the DVD drive instead of the CD-RW drive. Wow! MUCH faster! The whole time suck was scraping the pup_201 file off CD using a 4x (ouch) CD drive. The dvd was many many times faster (don't know the speed, but it went from ~45 seconds to load the pup_201 file to just 2-5 seconds). after I read here about moving the pup_201 file to HDD it loads that file so fast I can't even catch that part as it scrolls by. In other words, I can see how drives can make a difference.... think of Galileo sliding objects down ramps rather than dropping them to compare rates of acceleration - seeing the difference in CD loading speeds is like using the ramp, since differences in HDDs would be too small for me to notice in these cases (but not in terms of performance overall, just in this minimal case of loading a small file straight to memory - lots of repeated reads and/or writes would show the difference, I assume).

As for the issue of lots of tiny files: Are we really talking about lots of tiny files, or one big one? All our tiny files inside pup_save.3fs, but pup_save is one big file on the HDD. My brain is leaking I'm afraid. I'm sure it is simple, but I seem to be simpler.

Another thing: I've overlooked the fact that pup_save is .3fs and not .sfs - so it is not a squashfs like I thought. I thought it was and that that was how it was overlayed on top of the read-only stuff. I thought everything other than squashfs had to mount to a single point, rather than overlay the others, yet things we install go all over the place and are still there. I don't even know what questions to ask

Let me see if I understand correctly: the squashfs is its own filesystem sitting not directly on a partition but as a file inside another filesystem, ext3 in my case. If it is one large file to the underlying filesystem, how is it we can save small files as we go? Or are they just saved to the copy of the filesystem in memory and then written out by that puppysaved I see running all the time? Does it have to write the whole thing each time it actually saves to disk? OK - just found an explanation elsewhere that says it saves each file as you go, so I can't imagine it saving the whole thing: http://www.murga.org/~puppy/viewtopic.php?t=6606
It must really treat the "file" pup_save.3fs as a filesystem, as it is said to. I just don't understand how. Is it read once into ram, then modified in ram AND in the pup_save.3fs file each time you save something? as opposed to saving from ram to HDD periodically, as in USB flash drive installs (if I got that right, that is). I know I'm confused, and that is about all I know here!

So, using squashfs if faster simply because the overhead for uncompressing is less than the boost gained from faster access from disk due to smaller files, right? (not to say that must be the only issue). If it is all saved in memory doesn't that boost disappear, except when first reading? I would think that it would then be faster to have it all uncompressed in memory, assuming it fit, to avoid having to uncompress it (assuming also that here the tradeoff between access of files vs. decompression has reversed, with ram access being fast enough that uncompressing is the bottleneck, the opposite of when accessing from disk as in sunburnt's description). Aren't we also pretty much only reading once at the beginning, but saving whenever, well, whenever puppysaved is set to save? (see above confusion - this could be wrong).

What happens when the squashfs file is too big to fit in ram? That makes me all confused.

Thanks for helping me comprehend all this.
fitzhugh

[sunburnt]

Hi fitzhugh; A vacation... or business travel? Hope it went well!

Yep, drive performance is a big part of the whole PC picture because the OS is so dependant on it.
If you burn Puppy to a DVD the boot will go even faster, but still not as fast as newer HDs.

Compressed or not the big file is only mounted, the tiny files inside are read or written.
Win. / DOS mount all the drive partitions found at boot, Linux does it in boot code we can alter.
Linux also can mount files (compressed or not) that have a filesystem inside them.
When a HOME image is made, the next step is usually putting a ext2 filesystem on it.
Other filesystems can be put on it also, ext3, reiser, etc., probably even fat32.
Once the file with the FS on it is mounted, then it can be used just like any drive partition.

The "union filesystem" as it's called, joins the filesystems together so they overlay, mounting won't.
The union FS will overlay the same FS (ext2, reiser, etc.) at any point that it's told to do so.
The unioning is done in layers, anything in the top layer overshadows the rest, & on down the layers.
The save file isn't compressed so it can be written to, the squash file can only be read from.
So the save file oveshadows the squash file, allowing new files to be written over the ones in the squash.
The files in the squash are still there, but can't be seen as they're layered under the files in the save file.
Again... any files in ANY unioned layer will block out any files in lower layers with the same path/name.
The command for the unioning of filesystems is like this:
mount -t union (save file):(more file):(etc.):(squash file) /
This joins the ALREADY mounted files on "/", with the save file at the top & on down the layers.

There's a pic. that shows the Puppy boot layout, but everythings been changed so I can't find it.
Puppy looks on HDs, CDs/DVDs, for the save & squash files in a paticular order.
If enough memory, save & squash files are copied to it, if not the files stay put, slowing Puppy down.
If the save file is in memory, then it has to back it up to the storage drive at some point.

If the squash file's in memory, it's still compressed, however memory doesn't have drive latencies.
So the benefits we've been talking about probably wouldn't exist, it might even be slightly slower.
The tests I provided a link to don't show the performance of compressed files in memory.
Puppy is somewhat unique in the use of memory for the main storage access device.

I suggest that you Google for "linux union file system" (I did & learned alot), it's the crux of all this.
If you do, then real quick here you'll be telling me about some of the aspects of all of this...