Should Full HD install be faster than Frugal install?

Using applications, configuring, problems
Message
Author
User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

concept I was missing

#31 Post by fitzhugh »

Here is a major and simple concept I was entirely missing, which has added to my confusion greatly:
I was thinking, when I thought about it at all, that each partiton was seperate from the view of the filesystem, rather than understanding the concept of mounting. I didn't realize you would mount a partition to a directory inside the filesystem, and that it would then appear as a directory at that point from the filesystem. I thought to get to another partition you had to go all the way up to root, over, and down again - or traverse symlinks put in place. Doesn't solve everything for me, but it makes a lot more sense now. I undnerstood squashfs's ability to overlay, but had been working on the above incorrect assumption. I knew something[/] was wrong because it didn't make sense, but only just now had it hit me.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#32 Post by sunburnt »

Here's a set of tests on most of the types of compressed fs & ALL of them are faster than raw files on the HD.
The HD being a mechanical device can't possably match the speed of the sys. memory & cpu as is shown.

http://kerneltrap.org/files/PERFORMANCE.README.txt

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

#33 Post by fitzhugh »

sunburnt - thanks for posting that. I need a little help interpreting it, though, if you would...

I see that squashfs 2.1 is faster and smaller than those it is compared against, but I don't get how that would compare against the same tasks on the same data just sitting on an ext3 partition. Does my question here even make sense? You said they are faster than raw data on hdd since the hdd is a physical device and the others are in ram and cpu? Did I understand correctly? I guess I'm not understanding squashfs right: I thought it was a sort of virtual file system sitting in a file on a partition instead of directly on the partition, and assumed that there would be overhead in accessing that data. I also implicitly assumed (in other words had not considered it at all but included in my other assumptions) that it could then be pulled into ram disk as a seperate process, and that this could be done for any/most/many other kinds of filesystem, giving a large boost that was from not having to access a physical device. Clearly I seem to be missing sometihing, but I don't even know what question to ask... I'd appreciate any and all help, from urls to sites that might help to posts explaining what concepts I have wrong.

Thanks!
Fitzhugh

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#34 Post by sunburnt »

fitzhugh; Your confusion is quite understandable, it took me awhile to grasp what's going on.
In Puppy & Knoppix both, if there's enough memory to hold it, the compressed file is copied to the ramdisk.

This is NOT how the various compressed file systems work, the tests don't have the comp. files in memory.
The contense of the squash file (Puppy) or the cloop file (Knoppix) is never extracted, it's read directly.
Another words, the kernel module allows the file being read to be extracted on-the-fly, as it's used.
Your right, there's overhead in accessing a compressed file, but the files are smaller so they come off the HD faster.
On network mounted compressed files they operate faster also, same as HD, CD, or USB drives.
Newer PCs with faster cpu's do it in a micro second, faster than the HD can spool the data, understand?

Folks don't realize how slow a HD is compared to memory, BIG TIME, so the drives / network is the bottle neck.
Smaller files mean faster drive access, & most of the compression methods like gzip shrink files by 2.5 times!
So the drives / network is being accelerated by 2.5 times! This is a HUGE speed increase for data reading.
Most of what the computer does is reading data, not alot of writing goes on typically.

All this I've suspected for years, also what would be interesting to know would be where the brake even point is.
Another words, for a Pentium 200mhz is there no difference in compressed / uncomp. file access or not?
Or is the brake even point more K6 300, or Celeron 500, etc. (really it's academic, just curious though).

User avatar
rarsa
Posts: 3053
Joined: Sun 29 May 2005, 20:30
Location: Kitchener, Ontario, Canada
Contact:

#35 Post by rarsa »

sunburnt wrote:(really it's academic, just curious though).
I've been following this thread and from the begining I had already reached this conclussion. This is a merely academic discussion. The difference is a teoretical difference.

I would say that unless you are working in a very odd environment such as a computer with lots of RAM and a very slow HDD or one with very little ram and a fast HDD you won't notice the difference.
[url]http://rarsa.blogspot.com[/url] Covering my eclectic thoughts
[url]http://www.kwlug.org/blog/48[/url] Covering my Linux How-to

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#36 Post by sunburnt »

rarsa; Actually the amount of ram wouldn't make any difference in the access speed between compressed files & non.
HD speed would... of course, & data access (app. loading) is a popular topic here, paticularly booting speed.
If 1MB of files must be read for a task & the HD only has to give out 400KB, it's gonna go faster, period.
HD latency will mute the speed up effects, especially for small files, like for HD storage small files waste space.

Ultimately most of what the PC does is read lots of small files (unfortunately), so both data rate & storage suffer.

marksouth2000
Posts: 622
Joined: Wed 05 Apr 2006, 20:43

#37 Post by marksouth2000 »

sunburnt wrote:Actually the amount of ram wouldn't make any difference in the access speed between compressed files & non.
Sorry to disagree, sunburnt, but it does make a difference. Most operations on large data segments (like decompressing files) involve moving stuff around in RAM. If there's lots of it, then it's easy to find a space to move something to. Otherwise something else has to be moved first.

Sort of the way it's quicker to tidy a big apartment that has lots of space than a small one that is full of stuff. In the latter case you need to move a lot around in order to be able to tidy something else.

This is all even in the first edition of ast's book, IIRR.

This is also why overclocking is insane when RAM is relatively cheap. The same system can become faster by integer factors when RAM is added (with no other consequences), whereas overclocking by 50% gives speed gains of a few percent in exchange for reliability losses of several hundred percent.

Cheers,
Mark 8)

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#38 Post by sunburnt »

Yes... but your now talking about system management & memory allocation, not just file access.
Paticularly in WinBlows, the OS shuffling data is tied directly to memory size.
But this has nothing to do with HD file access (strictly speaking as we were).
Memory isn't needed at all if the data is just tossed after it's read, but there's still the read speed.

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

#39 Post by fitzhugh »

just getting back from a week's travelling so I'm playing catch up here...

Sunburnt: Thanks for the explaination - I really had been confused by that. As for access speed: I kept reading about super fast booting on older hardware, hardware even slower than the 1G PIII I was using up until a few days ago, but on that machine it took over a minute to boot a normal install (booting from CD but pup_save on HDD). Finally, and wholly by accident, I booted with the CD in the DVD drive instead of the CD-RW drive. Wow! MUCH faster! The whole time suck was scraping the pup_201 file off CD using a 4x (ouch) CD drive. The dvd was many many times faster (don't know the speed, but it went from ~45 seconds to load the pup_201 file to just 2-5 seconds). after I read here about moving the pup_201 file to HDD it loads that file so fast I can't even catch that part as it scrolls by. In other words, I can see how drives can make a difference.... think of Galileo sliding objects down ramps rather than dropping them to compare rates of acceleration - seeing the difference in CD loading speeds is like using the ramp, since differences in HDDs would be too small for me to notice in these cases (but not in terms of performance overall, just in this minimal case of loading a small file straight to memory - lots of repeated reads and/or writes would show the difference, I assume).

As for the issue of lots of tiny files: Are we really talking about lots of tiny files, or one big one? All our tiny files inside pup_save.3fs, but pup_save is one big file on the HDD. My brain is leaking I'm afraid. I'm sure it is simple, but I seem to be simpler.

Another thing: I've overlooked the fact that pup_save is .3fs and not .sfs - so it is not a squashfs like I thought. I thought it was and that that was how it was overlayed on top of the read-only stuff. I thought everything other than squashfs had to mount to a single point, rather than overlay the others, yet things we install go all over the place and are still there. I don't even know what questions to ask :?

Let me see if I understand correctly: the squashfs is its own filesystem sitting not directly on a partition but as a file inside another filesystem, ext3 in my case. If it is one large file to the underlying filesystem, how is it we can save small files as we go? Or are they just saved to the copy of the filesystem in memory and then written out by that puppysaved I see running all the time? Does it have to write the whole thing each time it actually saves to disk? OK - just found an explanation elsewhere that says it saves each file as you go, so I can't imagine it saving the whole thing: http://www.murga.org/~puppy/viewtopic.php?t=6606
It must really treat the "file" pup_save.3fs as a filesystem, as it is said to. I just don't understand how. Is it read once into ram, then modified in ram AND in the pup_save.3fs file each time you save something? as opposed to saving from ram to HDD periodically, as in USB flash drive installs (if I got that right, that is). I know I'm confused, and that is about all I know here!

So, using squashfs if faster simply because the overhead for uncompressing is less than the boost gained from faster access from disk due to smaller files, right? (not to say that must be the only issue). If it is all saved in memory doesn't that boost disappear, except when first reading? I would think that it would then be faster to have it all uncompressed in memory, assuming it fit, to avoid having to uncompress it (assuming also that here the tradeoff between access of files vs. decompression has reversed, with ram access being fast enough that uncompressing is the bottleneck, the opposite of when accessing from disk as in sunburnt's description). Aren't we also pretty much only reading once at the beginning, but saving whenever, well, whenever puppysaved is set to save? (see above confusion - this could be wrong).

What happens when the squashfs file is too big to fit in ram? That makes me all confused.

Thanks for helping me comprehend all this.
fitzhugh

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#40 Post by sunburnt »

Hi fitzhugh; A vacation... or business travel? Hope it went well!

Yep, drive performance is a big part of the whole PC picture because the OS is so dependant on it.
If you burn Puppy to a DVD the boot will go even faster, but still not as fast as newer HDs.

Compressed or not the big file is only mounted, the tiny files inside are read or written.
Win. / DOS mount all the drive partitions found at boot, Linux does it in boot code we can alter.
Linux also can mount files (compressed or not) that have a filesystem inside them.
When a HOME image is made, the next step is usually putting a ext2 filesystem on it.
Other filesystems can be put on it also, ext3, reiser, etc., probably even fat32.
Once the file with the FS on it is mounted, then it can be used just like any drive partition.

The "union filesystem" as it's called, joins the filesystems together so they overlay, mounting won't.
The union FS will overlay the same FS (ext2, reiser, etc.) at any point that it's told to do so.
The unioning is done in layers, anything in the top layer overshadows the rest, & on down the layers.
The save file isn't compressed so it can be written to, the squash file can only be read from.
So the save file oveshadows the squash file, allowing new files to be written over the ones in the squash.
The files in the squash are still there, but can't be seen as they're layered under the files in the save file.
Again... any files in ANY unioned layer will block out any files in lower layers with the same path/name.
The command for the unioning of filesystems is like this:
mount -t union (save file):(more file):(etc.):(squash file) /
This joins the ALREADY mounted files on "/", with the save file at the top & on down the layers.

There's a pic. that shows the Puppy boot layout, but everythings been changed so I can't find it.
Puppy looks on HDs, CDs/DVDs, for the save & squash files in a paticular order.
If enough memory, save & squash files are copied to it, if not the files stay put, slowing Puppy down.
If the save file is in memory, then it has to back it up to the storage drive at some point.

If the squash file's in memory, it's still compressed, however memory doesn't have drive latencies.
So the benefits we've been talking about probably wouldn't exist, it might even be slightly slower.
The tests I provided a link to don't show the performance of compressed files in memory.
Puppy is somewhat unique in the use of memory for the main storage access device.

I suggest that you Google for "linux union file system" (I did & learned alot), it's the crux of all this.
If you do, then real quick here you'll be telling me about some of the aspects of all of this...

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

Great help - thanks (+ more never ending questions...)

#41 Post by fitzhugh »

A vacation... or business travel? Hope it went well!

Vacation, through and through. Had a wonderful time - a friend and her two great danes from online I'd not met in person drove up from southern California and picked me and my wolfhound Abigail up (Berkeley), then drove us to the redwoods along the northern coast of California for camping. Had almost as much fun as Abi and the danes.

Thank you for getting me back (mostly) on track! Your detailed answers were a huge help in not only telling me how it works but also in seeing what I was wrong about all along.
Compressed or not the big file is only mounted, the tiny files inside are read or written.
I has wrongly assumed that you had to write a whole file at a time, rather than being able to write parts... now I gather you can write parts IF it is mounted or, um, unioned? Mounting sticks it in one spot in the file hierarchy, unioning (if that is the verb) lays it over whatever parts it structurally overlaps [please let me know if any statements I make are wrong - they are how I understand things currently]
Win. / DOS mount all the drive partitions found at boot, Linux does it in boot code we can alter.
Ok, makes sense. This gives us the ability to stick them wherever we want in the hierarchy, while in windows they are each a different drive.
Linux also can mount files (compressed or not) that have a filesystem inside them.
This is a crucial piece that was right in front of me but that I didn't really recognize. I knew this was happening, but I thought it was only done with squashfs files, though I don't know where I came up with that nonsense.
OH, I had merged unionfs and squashfs into one thing, dropping the term unionfs from memory, so I was left with the misconception that squashfs was both compressed and the only way to union mount filesystems, and that you could only union mount one squashfs onto another (or other multiple) squashfs. Suddenly it all snaps much closer into place.
When a HOME image is made, the next step is usually putting a ext2 filesystem on it. Other filesystems can be put on it also, ext3, reiser, etc., probably even fat32. Once the file with the FS on it is mounted, then it can be used just like any drive partition.
Sorry, what is a HOME image? Can't find anything to tell me via google. I understand an image is the file that holds a filesystem, like an .iso cd image file or, I assume, the various .sfs or .3fs files we use.

The union FS will overlay the same FS (ext2, reiser, etc.) at any point that it's told to do so.
I understood the concept of layering union filesystems into one single filesystem as far as the user is concerned, with only the top layer of any particular file, directory etc. showing, but had thought it was only a squashfs thing. That right there was a source of MAJOR confusion for me. I had thought that was the whole point of squashfs, rather than compression as I think it is now. Again, no idea how I got this wrong idea stuck in my thick head ;)

Now, are you saying here (above) that it can only overlay one filesystem onto another of the same type (ext2, etc.)? Don't we overlay pup_save.3fs and pup_201.sfs? I know .sfs is squashfs, but what is 3fs? ext3? What happens if they don't all support something important like symbolic linking?

Hmmm, perhaps what I have wrong is thinking that the ability to be layered through union mounting is an attribute of the original filesystems, rather than the resulting filesystem being different, a unionfs. Is it that unionfs is the resulting filesystem, and other filesystems can be layered into it?

Can you have any (supported) filesystem type inside a file on any (supported) filesystem type partition? ext3 image on a partition formatted as fat32 etc?

Ok, a really silly useless question that I'm curious about: what happens if you try to hold one image inside another, and then mount or union mount them all together?

Uh oh... my confusion regarding union filesystems also had me confused about the overhead associated with a compressed filesystem vs. whatever overhead there is in a union filesystem. How much overhead is there for union? I read we can only use what, 5 layers currently? That doesn't sound right, since doesn't the multisession stack every single one of them saved so far as it starts? (If so, doesn't that slow it down after a while, say after booting and saving many many times? What happens when you get too big for one cd? Does it get squished into one layer, freeing up all that space taken by "old" or lower level versions? Still, could run out over time... even faster on a CD than a DVD of course) Sorry, back to # of layers: We can only use 5(?) .sfs files without recompiling the kernel or perhaps a boot parameter; I figured all along that was due to a limit in # of layers. Is that not the reason? I think I'm getting something all wrong again.



If the save file is in memory, then it has to back it up to the storage drive at some point.
This has had me a bit confused. I understand the concept but am not sure how it is implemented. We're (potentially) doing everything in ram, so we need to save to HDD or other media or we lose whatever changes we make. So far so good. but according to the posts here by Rarsa
http://www.murga.org/~puppy/viewtopic.php?t=9047
this isn't the case... it is written on each change. I can understand how it could work either way: reading once at the beginning into ram, then making changes to ram along the way and saving periodically and at end, OR reading once at beginning into ram and saving each change as it goes right onto HDD. You'd still get the ram resident performance boost with either from the lack of reading from disk, since ram would always hold current version, but would lose the write performance benefit in the latter since it writes every change upon happening. Which does it use? It it is periodical, I would really like to be able to control it and know when it happens: a "throttle" control in the taskbar to control how often it saves to HDD, for example I really don't want it saving all that often when just browsing since it will slow me down, or if I were into games, while gaming... if it crashes I don't need it to have just saved and don't want the resources spent on that instead of generating monsters for me to kill on the screen. However, if I'm in the middle of a huge coding project I will be totally freaked if it crashes and loses all my work. Also, an indicator light to show it is happening now, which, when not indicating a current read/write (ok, two lightes) it would simply display time since last save and/or time till next. Lastly, a Save Now button, as I think exists for multisession?
If the squash file's in memory, it's still compressed, however memory doesn't have drive latencies. So the benefits we've been talking about probably wouldn't exist, it might even be slightly slower. The tests I provided a link to don't show the performance of compressed files in memory. Puppy is somewhat unique in the use of memory for the main storage access device.
Makes good sense.
I suggest that you Google for "linux union file system" (I did & learned alot), it's the crux of all this.
Saddly, I actualy had tried rather hard to figure this out on my own, but I was entirely missing the term "union file system" since, as stated above, I kept looking for unionfs info in squashfs searches, obviously not going to find the info I was after that way! :roll: I've since found answers to most of my other questions:
http://www.fsl.cs.sunysb.edu/docs/union ... nionfs.pdf but not all, plus at 25 pages it unwieldy.

Thank you!
Fitzhugh

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#42 Post by sunburnt »

I went through the Redwoods once & it was a glorious venture, as was Columbia falls near Portland.

Obviously any changes to the files inside the the save file also change the save file (of course).
Any file system must be mounted in Linux to read or write it, but FS are disk checked unmounted.

HOME in Linux is where the user's data is stored, docs., emails, etc., the user's unique work space.
In Puppy the save file overlays at the top level, & it's used for HOME, & to save settings & installed apps.

I beleave union will overlay any FS that the Linux kernel can mount (I might be wrong about this).
The squash FS is the compress file not the FS in it (I think), & it's an image of the FS it's made from.
An uncompressed image file can be disk checked & has used & free space just like any partition.
A uncompressed image file is a file of given size, & like a drive partition when it's full, that's it.
The squash file is made with the utility file: mksquashfs, it compresses the dir. given to a file with no empty space.
Being as it can't be written to there's no need for empty space that would make the file bigger.

Again, the drives, partitions, image & squash files must be mounted the normal way.
Union then layers the mounts in a overlaying stack blending them together, in Puppy2 the file: /sbin/init does this, in Puppy1 the file: /etc/rc.d/rc.sysinit does this (have a look at it, you'll learn alot).
Once a FS is mounted the FS type is irrelevant, the FS is attached to the rest of the FS.
To view it properly, mounting JOINS FS, union OVERLAYS the mounts (the point the FS is attached).

Yes, any image file with any FS can be on any other type of FS on any type of storage.
This is why Puppy can work on a fat32 FS, the kernel, initrd.gz, & pup_003.sfs are just files.
But inside the files are Linux FS (ext2, ext3, etc.), so all that's needed is a way to start the kernel.
The kernel is loaded into memory, initrd.gz is decompressed into a ramdisk, it has the base FS.
pup_003.sfs & the save files are then mounted to the base FS & then unioned to "mix" them.
When mounted they're accessable only at the mount point, when unioned they're accessable there & also the point at which they were mixed with the other FS, blending them together.
In Puppy1 this union (blending) point was /root, in Puppy2 it's /.

An image file inside another image file is no problem of course, & it should be possable to mount & union it.
It's an interesting idea, I'm not sure there's a point to it though, but it does open up possabilities.


It seems to me that the unionfs is just a file allocation table that the kernel uses to view the FS structure.
So there wouldn't be much overhead in using it, but there's always a little more work to do.
I can't remember how many mounted FS can be unioned, I thought it was 8 & the new union's more.
You may be confusing the number of unions possable with the number of loop devices Puppy had.
"the multisession stack" I'm not following you, but... I think only the changed files are saved.
Other wise the CD would fill up in no time at all, I don't use CDs so I vauge on their implimentation.

I like many others here only use the frugal setup, not even the full HD install.
One big advantage to this is the 3 main file are only read, so can't be corrupted by written data.
The save file of course is writable, so it like any mounted drives & the internet is a bad data source.
But if the PC crashes from bad data or just the random crash, rebooting will restore the OS.
That's why when you get Puppy the way you want it, you should make a backup of the save file.
If anything bad happens, just copy the backup over the corrupt one & it's good to go again.

Ram isn't used if there's not enough of it to hold the files & still run the OS.
If Puppy is on CD or a flash drive the save file isn't written back every time there's a write.
This would wear out the flash in no time, so like the CD it's written back at shutdown.
This means if Puppy crashes with these setups then the current session's data & changes are lost.
If the setup is that the save file is only on the HD, then it's written to directly, no backup & restore.
Puppy2 has a set time to backup the file if it's in the ramdisk, I don't know much about all that.
This is a big source of confusion as there are so many different ways Puppy can arrange itself.
Folks can't find the save file, Puppy can't find the squash file, etc., there's no way to control alot of this.
I suggested a config. file that would "overrule" Puppy's auto. setup & allow locating ALL of the files.

In addition to being faster on drives, a compressed FS like squashfs takes approx. 1/2.5 less space.
This means the average pup_003.sfs file at 60MB would expand to 150MB of drive or memory.
So there's an obvious advantage to the squash files, reliability & security, & 2.5 times less space used.
Considering MU's Openoffice, Java, Wine, etc. "more" squash file (300MB I think), it would be ~ 750MB!

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

Ok, it is almost all clear now :)

#43 Post by fitzhugh »

Thanks so much for your patience and effort to get my head untangled here!
sunburnt wrote:Any file system must be mounted in Linux to read or write it, but FS are disk checked unmounted.
Sorry, not sure what you mean here. First part makes sense, but "FS are disk checked unmounted"? Not sure what you are saying.

HOME in Linux is where the user's data is stored, docs., emails, etc., the user's unique work space.
Ok, now I understand what you meant.
In Puppy the save file overlays at the top level, & it's used for HOME, & to save settings & installed apps.
Yeah, this is one thing I did understand.
The squash FS is the compress file not the FS in it (I think), & it's an image of the FS it's made from.
Now that makes sense.
An uncompressed image file can be disk checked & has used & free space just like any partition.
A uncompressed image file is a file of given size, & like a drive partition when it's full, that's it.
The squash file is made with the utility file: mksquashfs, it compresses the dir. given to a file with no empty space.
Being as it can't be written to there's no need for empty space that would make the file bigger.
This makes good sense. However, I'm thinking that to get a read time boost (due to smaller reads from disk for same files once compressed) it can't just be removing empty space at the end but also real compression of the data.

Union then layers the mounts in a overlaying stack blending them together, in Puppy2 the file: /sbin/init does this, in Puppy1 the file: /etc/rc.d/rc.sysinit does this (have a look at it, you'll learn alot).
/sbin/init in Puppy 2.01 is a symlink to busybox's init, so I don't think that is what you meant to type...? The script that does that would be a very good read for me.

Once a FS is mounted the FS type is irrelevant, the FS is attached to the rest of the FS.
To view it properly, mounting JOINS FS, union OVERLAYS the mounts (the point the FS is attached).
...pup_003.sfs & the save files are then mounted to the base FS & then unioned to "mix" them.
pup_003.sfs was the Puppy 1 equivalent of pup_save, right? I only ran Puppy 1 for about a week before Puppy 2 was released, so I never learned the details. I'm not aware of a pup_003 in Puppy 2 so I'm guessing it was what was used in Puppy 1.
When mounted they're accessable only at the mount point, when unioned they're accessable there & also the point at which they were mixed with the other FS, blending them together.
In Puppy1 this union (blending) point was /root, in Puppy2 it's /.
Does this mean I can see the individual layers by looking at their mount point, rather than seeing the blend while looking all around the rest of the resulting unioned filesystem? That could be interesting... I'll take a look and see if that is the case, or if I'm confused about this.

An image file inside another image file is no problem of course, & it should be possable to mount & union it.
It's an interesting idea, I'm not sure there's a point to it though, but it does open up possabilities.
Yeah, um... interesting is probably a generous way to put it ;) It was truly not a practical question but just another piece of the puzzle to help me grasp all this. I see the potential for spasdic recursion :)
It seems to me that the unionfs is just a file allocation table that the kernel uses to view the FS structure.
So there wouldn't be much overhead in using it, but there's always a little more work to do.
I can't remember how many mounted FS can be unioned, I thought it was 8 & the new union's more.
You may be confusing the number of unions possable with the number of loop devices Puppy had.
Actually I think you're right about that... the problem of limits to the number of layers of union mounted filesystems was mentioned by (I think) Mark (MU), could be wrong about that and don't recall which thread... I'd asked about using squashfs files to provide a super user friendly way to add and remove programs: either the .sfs file is there, providing the functionality of whatever is inside it (like the old user_more.sfs providing open office, I gather, or MU's megapuppy providing a large range of things in a .sfs file).... or the .sfs file is NOT there, and all that functionality is gone... would be much friendlier than installing and having it scattered all over - it still would be, but could be removed easily. I was trying to think of a way to get true modularity. I was told that there is a limit to the number of separate ones that can be loaded... but it was refered to as a loop device limit of maybe 5 (??) that could be increased with a boot param or when compiling the kernel. So, I'm guessing (correct me if wrong) that all the various layers going to make up the primary/original union FS use one loop device (??), while each additional one added to be recognized at boot takes up another, so it is more like each collection. No, that doesn't make sense - I am certainly confused about the origin of the limit.

"the multisession stack" I'm not following you...
I meant that on the cd the files have to be stored in different FS files and then layered back into one for use by the user... just what we're talking about all through here, only on CD or DVD multisession there can be a LOT of different files that need to be stacked together, oldest on bottom, newer versions layered over them, hiding earlier versions... I was saying that this seems to be no problem, which proved there was a hole in my understanding since that meant the limit of five union layers could not be right, as you pointed out. Now I'm wondering what the limit really is and why.
Oh, by stack I just meant the layers of older files still on the cd but covered by current versions. I believe there is one layer per session on live CD
, but... I think only the changed files are saved.
Other wise the CD would fill up in no time at all, I don't use CDs so I vauge on their implimentation.
That is my understanding as well.

I like many others here only use the frugal setup, not even the full HD install.
One big advantage to this is the 3 main file are only read, so can't be corrupted by written data.
The save file of course is writable, so it like any mounted drives & the internet is a bad data source.
But if the PC crashes from bad data or just the random crash, rebooting will restore the OS.
....That's why when you get Puppy the way you want it, you should make a backup of the save file.
If anything bad happens, just copy the backup over the corrupt one & it's good to go again.
Yup, did that last week, but found afterward I had configured things off a little so now when I restore I have to clean up each time... silly not to have checked it first!
Ram isn't used if there's not enough of it to hold the files & still run the OS.
So, I gather then that increasing the size of pup_save, as can be done from the menu, runs the risk of making the whole file too large for memory, in which case NONE of pup_save gets the boost from residing in memory... is that right? I tend to store all documents and such outside on another partition anyhow, since I know I will occasionally hairball my pup_save file.
Folks can't find the save file, Puppy can't find the squash file, etc., there's no way to control alot of this.
I suggested a config. file that would "overrule" Puppy's auto. setup & allow locating ALL of the files.
Yeah, that sounds like a wise idea... just too much flexability for moving things all over the place and getting into trouble.
In addition to being faster on drives, a compressed FS like squashfs takes approx. 1/2.5 less space.
This means the average pup_003.sfs file at 60MB would expand to 150MB of drive or memory.
So there's an obvious advantage to the squash files, reliability & security, & 2.5 times less space used.
Considering MU's Openoffice, Java, Wine, etc. "more" squash file (300MB I think), it would be ~ 750MB!
That is a HUGE savings!

Thanks again,
Fitzhugh

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#44 Post by sunburnt »

If you run: fsck on a mounted drive you'll get the message: running a filesystem check on a mounted partition can cause damage, do you want to continue?
Drives, partitions, image files, etc. are typically checked for integrity unmounted.

Yes squash uses something like gzip for it's compression, it's been suggested that for the new version "squash 3" that they use lhaz compression (I think) for greater compression (like RAR gets).

I'm sorry, I forget to fill in all the details sometimes, yes... after booting the /sbin/init file is a link.
You have to ungzip the initrd.gz file to look at the file I'm talking about, it's replaced with a link.

No, pup_003.sfs is the Puppy 2 squash file (sfs = Squash File System), it's size is about 70MB I think.
The Puppy 1 save file is pup001, or really anything that is put in the kernels boot argument.
The kernel argument for Puppy 1 looks like this: PFILE=pupSRV-none-262144.
The save file is made the first bootup, an empty file that it copies the contense of image.gz or initrd.gz to.

Yep, a single drive or partition can even be mounted several times & viewed from each of the mount points.
In LanPuppy the client PCs (up to 240 of them) all mount & use the same squash file, as it doesn't change.
But each client has to have it's own save file if it's going to be able to be setup with user preferences.
But if the clients were say Keosks or Info terminals, they could all be the same & use one save file.
In LanPuppy the client's mount Samba shares on PupServer & then mount the squash file in the share.

I sure the unionfs limit is what the developers found safe to use, & there's some reason for that limit.
The loop devices are used to mount image & squash files, the unionfs doesn't use them at all.
I had hoped that Puppy2 would make possable exactly what you described, a new way to package apps.
But upon looking at it I realized that like MU's MegaPup, an install file would usually still be needed.
So the "just drop the .sfs file in place & go" idea usually wouldn't work (like the usr_devx.sfs file does).
For anything that doesn't have to replace already existing files in the union blend it would work.
But to replace existing files, they'd have to be copied to the top layer (the save file) to overshadow other files.
Also as MU pointed out, the unionfs is limited in the number layers, so you want each file to pack alot.
What your hitting at is what's done by a distro. called Morphix that's based on Knoppix > based on Debian.
Morphix has a base module (most of the common stuff, Xwin & all), & various minimodules to customize.
I found it very interesting, but it's a LiveCD based distro. so I didn't know enough to make it a HD distro.
I'll be taking another look at it soon, as I've learned alot in the last 6 months... http://morphix.org/

For one layer per session there'd be one save file per session, & that'd be WAY too many to union.
As I said, I think only the changed files are saved, like a standard backup would save & restore.
I don't do CDs so I might be wrong, but it only makes sense this way, & only 2 - 256MB saves on a CD.

Yep, I use external storage for my HOME work space, I & others have suggested a separate HOME file.
So the user has a clean work space not cluttered with the config. stuff, saves from noob accidents.
In LanPuppy the save file is only 8 - 16MB as that's all that's needed to store the settings & preferences.
The user's HOME space is the Samba HOME share, so it's not an image file at all, just a share on the server.

An idea for Puppy 2 would be a selection of smaller .sfs files to larger ones that contain several small ones.
Sort of like a tree whose branches join to larger files to the trunk that'd be a huge file of everything.
This'd limit the number of unions while allowing a large selection of apps. to be added & removed.
LanPuppy's clients all using only one file, could have the "everything .sfs file", like Knoppix's 3GB DVD.

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

#45 Post by fitzhugh »

I'll have to read through again when truly awake... just on the computer now because the neighbors were banging around and now I can't sleep :P
For now just one question: you mention using a seperate HOME, as I did. I'd like to hear how you have it set up - I imagine there is a better way. Oh, duh... there is one, at least, and it is the obvious: I'm currently mounting all the partitions as /mnt/hda* and then symlinking in the directories from the other partition into home:
ln -s /mnt/hda7/my-documents /root/my-documents
ln -s /mnt/hda7/my-roxapps /root/my-roxapps
ln -s /mnt/hda7/my-applications /root/my-applications
etc. I did it this way because I didn't understand about mounting in other places, rather I was stuck in a spot only partway into the linux mindset from that other place of nonstop crashes, viri, bugs, expenses, "features" like malicious disabling of competing software...
See, I was seeing mounting as just turning on or off a disk drive, in essense, and assumed that they always mounted to /mnt/* in much the same way windows 'mounts' [wrong term I gather now, but it was how I saw it] each drive as A, B, C... instead of understanding the implications of being able to mount anywhere - I'd read that but not thought about it at all - in one ear and out the other. So, I'll just mount hda7 as something under root instead of linking it it, and go from there. I'd like to hear how others are doing it - I clearly tend to miss the obvious.

Thanks,
Fitzhugh

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#46 Post by Sit Heel Speak »

My, but this thread is a wonderful education!
sunburnt wrote:...pup_003.sfs is the Puppy 2 squash file (sfs = Squash File System),...
pup_003.sfs was the Puppy 2 squash file in one of the alpha versions, I forget which one. The squash file in the final release of Puppy 2.00 is pup_200.sfs, in 2.01r2 is pup_201.sfs.

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

#47 Post by fitzhugh »

Sit Heel Speak wrote:My, but this thread is a wonderful education!
sunburnt wrote:...pup_003.sfs is the Puppy 2 squash file (sfs = Squash File System),...
pup_003.sfs was the Puppy 2 squash file in one of the alpha versions, I forget which one. The squash file in the final release of Puppy 2.00 is pup_200.sfs, in 2.01r2 is pup_201.sfs.
Thanks - that is another piece of the puzzle! I jumped from 1.08 to 2.00, not trying the alpha, so I was unaware of that file.
And yes - I'm learning a LOT here :) Try doing that in the windows world! Sure, there are the occasional, um, oddities who are passionate about xp, but in my experience they tend to be misanthrops and quite unlikely to help someone with less knowledge than them, and certainly not to the degree I see (and personally benefit from, as this thread shows) with Puppy Linux.

User avatar
fitzhugh
Posts: 217
Joined: Fri 16 Jun 2006, 02:58
Location: Berkeley

#48 Post by fitzhugh »

Sit Heel Speak wrote:My, but this thread is a wonderful education!
sunburnt wrote:...pup_003.sfs is the Puppy 2 squash file (sfs = Squash File System),...
pup_003.sfs was the Puppy 2 squash file in one of the alpha versions, I forget which one. The squash file in the final release of Puppy 2.00 is pup_200.sfs, in 2.01r2 is pup_201.sfs.
Thanks - that is another piece of the puzzle! I jumped from 1.08 to 2.00, not trying the alpha, so I was unaware of that file.
And yes - I'm learning a LOT here :) Try doing that in the windows world! Sure, there are the occasional, um, oddities who are passionate about xp, but in my experience they tend to be misanthropes and quite unlikely to help someone with less knowledge than them, and certainly not to the degree I see (and personally benefit from, as this thread shows) with Puppy Linux.

User avatar
Wolfen
Posts: 86
Joined: Tue 18 Jul 2006, 09:34
Location: Behind you..

#49 Post by Wolfen »

Is there somebody who can sum this up, and tell how to install GRUB, boot grub, etc. in a way that's not so complicated?
I like people telling things straight forward, instead of making things so complicated.

Questions:
How to install GRUB
Just how should the "menu.lst" look
How to boot GRUB

My comuter has one HDD, with C:\ D:\ E:\ or hda1 hda5 hda6, WinME installed,
and on hda6 is pup_save.3fs , pupswap.swp and pup_202.sfs (to boot faster)
Puppy 2.02 is on live-CD
Wolfen

PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#50 Post by PaulBx1 »

Much of what is discussed here is covered in How Puppy Works, although it's not a bad idea to rehash from another perspective (I've learned a lot too).

As Barry mentions there, if pup_save.3fs is on flash, the copy in memory is saved to flash periodically (every 30 minutes, not 30 seconds - that was a mistake). Also at shutdown. If it is on hard drive then I'm guessing the updating is just done via the normal disk cache algorithms, likely handled down at the kernel level. Or?

Post Reply