Understanding mounts/memory/RAM & filesystems/_save file

Booting, installing, newbie
Message
Author
User avatar
canbyte
Posts: 264
Joined: Sat 10 Jan 2009, 20:20
Location: Hamilton, Canada

Understanding mounts/memory/RAM & filesystems/_save file

#1 Post by canbyte »

[Edit] please add any comments before i set about editing/transferring to wiki - link to other threads, fill in on gui issues + whatever [/edit]


How can i kick junk out of ram so seamonkey & all the pages i open can run as fast as possible? ie. manage memory to ensure no encroachment on seamonkey's ability to handle multiple (18+) flash heavy (business) pages.

I download lots of junk (articles, pdf files etc) and see no reason to hold it in memory so i save it to a 2nd flash drive - but there it is. Even though i unmount the 2nd flash drive, the free ram indicator on bottom right stays the same - so its still in memory?

Whole buncha questions come up.

I also think this info should be in puppy's manual or the wiki and if you can help me understand what's going on, i'll be glad to make a little effort to get that info edited & into the wiki or manual, whichever you think best (or i have permission for).

So......

Are all the files on the mounted drive put entirely in memory, or just the file system/structure/info or whatever you call it so it can be accessed? or

This windows refugee is confused by the same object being in two places/ two name at once:
sdb1 also exists as mnt/home,
sdc1 also exists as mnt/sdc1.
They all exist in memory. I'm getting confused as to which flash is which. Hard to tell even by watching their little lights. Can i relabel them to say whiteflash, blackflash, etc?

How can some files be removed from memory but still remain accessible?

What exactly is that little free ram meter reading?
Have a look at this film
http://www.informationliberation.com/?id=26614
About an hour in, a message flashed up to the effect that there is only 2mb left in ram, then 1 then 0. Crash. The ram meter never deviated from 443mb free. Can anybody detect the size of this film. PCU meter also rises to 6 or 7 lines at time of crash.

What exactly is the red graphical indicator beside it calibrated when various lines show up? (I used to worry at the third line, now only at the 5th on the Dell)

Why do the dates not change on the pup_save files? ie seems to remain at the created date and not the last modified date. Can this be fixed - its very confusing and hard to keep track of my changes (despite adding a suffix to the name). Now i have 2 files dated at the same instant! Help!!

Newbies note that a 1G flash drive will not hold a backup of your 512mB pup_save file!!! Mine got truncated & unusable. No warning.

Having put all my junk on the 2nd flash drive, can windows grab files and change them without risking puppy's ability to use them?

What is the purpose of swap files? How big should a swap file be/ what logic? My guess is that swap files are used to compensate for too little ram. While puppy seems to be able to run in 128mb ram, I really could not use seamonkey (heavily) until I went to 378mb. The Dell (256mb) is ok as long as i'm careful what/how many i load. CPU speed is also a factor.

Having installed Devx to get my printer working, can i now unload it to free up memory? How? How much memory does it consume? If i can only accomplish that by stepping back to an older pup_save file, how can i keep the changes that i want (plus all the info junk)?

ie. i keep adding bookmarks & don't want to lose them. Can i save bookmarks to a separate file? Ok, i see that under bookmark manager l tools l export - Wiki needs to record that!

Thanks ahead
Last edited by canbyte on Mon 09 Mar 2009, 16:32, edited 2 times in total.
[color=orange]1. Dell Dimension E521, AMD Athln 64, 2 GHz 1.93GB ram,
Puppy 533 on CD, accesses flash drive only,
FFox Nightly12.0
2. Compaq P3 733Hz 375RAM
Printer: Oki C3400 > LAN [/color]

Bruce B

Re: How to optimize memory/RAM & _save file?

#2 Post by Bruce B »

canbyte wrote: I download lots of junk (articles, pdf files etc) and see no reason to hold it in memory so i save it to a 2nd flash drive - but there it is. Even though i unmount the 2nd flash drive, the free ram indicator on bottom right stays the same - so its still in memory?
Right off the bat we have nomenclature problems.

To me memory means RAM

To Puppy memory means in most cases 'physical storage space' and some times might mean RAM.

Addressing some questions

1) mounted devices typically don't use RAM, but they could, if you set them up that way. The general rule is they are where they are physically located and get mounted in order to be accessible.
sdb1 also exists as mnt/home,
sdc1 also exists as mnt/sdc1.
They all exist in memory.
2) /mnt is a traditional parent directory for mounting devices in child directories of /mnt

conventionally mounts don't exist in ram and there is nothing in the 'mount' command which would move the device physically.

3) no change on the pup_save file date / time stamp because Puppy doesn't utilize it as a flat file character device, rather as a mounted block device, not treating it differently than it would a partition.

4) a swap file should be a bigger than it needs to be. by monitoring your swap file usage over a period of time, you can determine the optimal size. On this computer, it is 80mb. If the usage ever approached 80mb I'd make it larger.

-------------------

This could clear up a portion of your questions I hope.

Bruce

DMcCunney
Posts: 889
Joined: Tue 03 Feb 2009, 00:45

Re: How to optimize memory/RAM & _save file?

#3 Post by DMcCunney »

canbyte wrote:How can i kick junk out of ram so seamonkey & all the pages i open can run as fast as possible? ie. manage memory to ensure no encroachment on seamonkey's ability to handle multiple (18+) flash heavy (business) pages.
You may need to adjust your expectations. I don't like to have that many open tabs on my Windoze box, where I have a processor three times as fast as my Puppy box and 2GB of RAM as opposed to 256MB on Puppy.

Personally, I wouldn't try to have that many open tabs to flash heavy sites all at once. Why is it necessary for you to do so?
I download lots of junk (articles, pdf files etc) and see no reason to hold it in memory so i save it to a 2nd flash drive - but there it is. Even though i unmount the 2nd flash drive, the free ram indicator on bottom right stays the same - so its still in memory?
It might help to talk about what memory is on a Linux machine.

Linux and Windows are both virtual memory systems. Total virtual memory is the sum of installed RAM and the swap file on disk. (On Linux, swap is normally a separate partition. On Windows, swap is normally pagefile.sys in the root directory of the boot drive , though you can specify a different location.)

The OS divides available physical RAM into 4K pages, and keeps track of what pages are used by what processes. When more physical RAM is required by something than is available, the OS will swap out pages in RAM that have not been accessed recently to make room. If something tries to access a page that was swapped out, a "page fault" occurs, and the requested page is swapped back into RAM. This happens transparently to the user and to any applications running.

Applications think they have the total virtual memory to play in.

For both Linux and Windows, more RAM is better. Memory is an order of magnitude faster than disk, and the best way to improve performance is to reduce the need for paging.

In general, Linux handles memory management for you. You can make sure you aren't running any other applications besides Seamonkey to reduce demands on RAM, but beyond that Linux handles it all.
Are all the files on the mounted drive put entirely in memory, or just the file system/structure/info or whatever you call it so it can be accessed?
It depends upon the installation. Puppy is capable of running entirely in RAM if enough exists, but for most Linux systems, what is in RAM will be pointers to the stuff on disk.
This windows refugee is confused by the same object being in two places/ two name at once:
sdb1 also exists as mnt/home,
sdc1 also exists as mnt/sdc1.
They all exist in memory. I'm getting confused as to which flash is which. Hard to tell even by watching their little lights. Can i relabel them to say whiteflash, blackflash, etc?
Linux handles file systems differently than Windows.

On Windows, when you have more than one file system (like partitions on a single drive, or more than one physical drive), each file system is seen as a drive, and identified with a drive letter in My Computer. It is also possible to specify a volume name that will be shown with the drive letter.

Linux doesn't do that. If there is more than one file system, other file systems are mounted off the main one. It's common in Linux installations, for example, to have a root file system starting at /, and having separate partitions for user home directories and for third-party applications. This is set up when Linux is first installed. These are likely to be mounted off of root as /home and /opt respectively, though other mount points can be specified. Once the initial setup has been done, Linux handles that for you. In most cases you don't need to know or care that /home and /opt are in fact separate partitions. Linux makes them appear to be parts of one large unified file system.

In Linux, a directory entry does not point to a file. It points to a construct called an inode which is maintained by the kernel. The inode stores things like the files name, size, owner, owner's group, permissions, and creation/access times, and pointers to where the file lives on disk.

This makes several interesting things possible. Because directory entries point to inodes, not files, it's possible to have more than one directory entry pointing to the same inode. This lets you have the same file appear in more than one directory, or the same file appearing under several different names in the same directory. For example, Unix systems ship with a standard full screen editor called vi. There is also a line editor called ex, and a read-only file viewer called view. They are three different names for the same program, and all point to the same underlying executable. The program knows which name it was called by and behaves differently depending upon which name was specified.

There are two kinds of links, created by the ln command. The vi example above is a "hard" link. Hard links simply provide additional directory entries pointing at a file on disk. Hard links have the limitation that they must all be on the same file system.

Symbolic links are similar in concept to but more powerful than Windows shortcuts. They are pointers to a file. Linux follows the pointer and accesses the file. This makes it possible to do things like have a program that actually resides on a separate partition appear to be in a system directory.

NTFS 5 in Windows 2K and XP supports hard links, though the functionality is not exposed by default, and you need a Microsoft utility or a third party tool to make them. Windows Vista adds support for full symbolic links.

When you see file systems appearing in more than one place, you are seeing symbolic links. /mnt/sdb1 is the actual mount point in the file system where the sdb1 file system is attached. /mnt/home is a symbolic link to the mount.

I plug flash drives into a USB 2.0 add-on card on my Puppy machine, and simply note what mount point is assigned when I plug one in. I've never had a problem keeping track of which was which. One thing might do is simply create a file on your flash drive with a desired name, like _whitespace. When you mount the drive and access the contents, you'll see the name. (I use a leading underscore in the file name because file names are sorted alphabetically by default, and the "_" appears before alpha chars.)
How can some files be removed from memory but still remain accessible?
See above.
What exactly is that little free ram meter reading?
Total system load, which is not the same things as RAM usage. See http://www.murga-linux.com/puppy/viewtopic.php?t=28483 for an extensive discussion.
Newbies note that a 1G flash drive will not hold a backup of your 512mB pup_save file!!! Mine got truncated & unusable. No warning.
I have no idea what actually happened, but I don't believe a Puppy limitation was involved. The save file is simply one big file being written to a storage medium. If the write fails and the file is truncated, the usual reason is either that there wasn't enough free space on the medium to store the entire file, or the media has a problem. The first thing I would check is that the 1GB flash drive did in fact have more than 512MB free. The second thing I would do is access it under Windows and run CHKDSK. Flash drives are format with the FAT file system, as either FAT16 if < 2GB in size, or FAT32 if > 2GB. It's possible for them to have file system corruption just like a real FAT formatted disk drive, which will mightily confuse things accessing them.
Having put all my junk on the 2nd flash drive, can windows grab files and change them without risking puppy's ability to use them?
What files, and what would you be doing with them? In general, yes you can, but it depends on the files and what you do in Windows.
What is the purpose of swap files? How big should a swap file be/ what logic? My guess is that swap files are used to compensate for too little ram.
See the above discussion about virtual memory.

As a rule of thumb, I make a swap partition as large as installed RAM, then monitor usage and resize after I've seen what actually gets used in practice.
While puppy seems to be able to run in 128mb ram, I really could not use seamonkey (heavily) until I went to 378mb. The Dell (256mb) is ok as long as i'm careful what/how many i load. CPU speed is also a factor.
Puppy has been reported to run in as little as 64MB, or even 48MB. If you try to do that, you really need a swap file.
Having installed Devx to get my printer working, can i now unload it to free up memory? How? How much memory does it consume? If i can only accomplish that by stepping back to an older pup_save file, how can i keep the changes that i want (plus all the info junk)?
Dexv includes the Gnu compiler and associated utilities. Once you have compiled something and put it in place, it should be possible to remove Devx.
ie. i keep adding bookmarks & don't want to lose them. Can i save bookmarks to a separate file? Ok, i see that under bookmark manager l tools l export - Wiki needs to record that!
Under Puppy, your SM bookmarks are stored in your SM profile, which will be in /root/.mozilla/default/<something>.slt/bookmarks.html

I believe this stuff should be in your pup-save file, reloaded the next time you boot Puppy.
______
Dennis
Last edited by DMcCunney on Sun 08 Mar 2009, 21:31, edited 2 times in total.

User avatar
canbyte
Posts: 264
Joined: Sat 10 Jan 2009, 20:20
Location: Hamilton, Canada

#4 Post by canbyte »

Thanks Dennis and BB, what a great tutorial.
I may have some questions after i reread and let things sink in for a bit. Mostly, i guess i'll just have to get used to things in linux - a run in period of 6 months or so may end these points that i now find confusing. Otherwise i'll be looking for workarounds like your great idea to put a _descriptorFileName on each flash drive. Glad Wallmart is selling them for just 5 bucks each but it can get confusing!

Do you/ someone have some good links for the wiki entry such as swapfiles, uninstalling (howto), methods of optimizing setup/memory, especially where you indicated "It depends upon the installation." I suppose, one wants to include all/some of the main installation types

One thing to clarify for us newbies is the gui representation of these issues. Here is what i understand, maybe someone can fill in the rest.

green dot on folder = its mounted
grey dot = not mounted
yellow dot = ??

bolded folder or file = ?
a box with an arrow = ?

Cheers for now, i'll edit later as needed
[color=orange]1. Dell Dimension E521, AMD Athln 64, 2 GHz 1.93GB ram,
Puppy 533 on CD, accesses flash drive only,
FFox Nightly12.0
2. Compaq P3 733Hz 375RAM
Printer: Oki C3400 > LAN [/color]

User avatar
`f00
Posts: 807
Joined: Thu 06 Nov 2008, 19:13
Location: the Western Reserve

#5 Post by `f00 »

The 'gospel' to link to would be Barry's pages (install, etc), they are diligently updated to both current versions and noted for older work that may still be of import.

yellow or gold dot overlay on a drive icon means mounted for boot or "locked" (note also - if a .sfs file is mounted the drive or partition that the .sfs is on will also be locked until the .sfs is unmounted (if I recall correctly there is no static graphical representation of this locked state in the case of temporary mounting such as when one simply clicks on an .sfs to open it, when there is a green 'success' message box .. the drive will simply not unmount until the .sfs is first unmounted by another click which then shows a nice orange 'unmounting' message box).

bolded labels generally mean that the file or folder has been recently accessed or modified.

"box with an arrow" may be a compressed file or utility for dealing with compression .. this can get confusing to some as different graphical representations are often used (folder with zipper on it, a screwpress, a c-clamp etc)

A very good local reference to the Rox file system is available on Puppy distros by clicking the life-preserver icon on the Rox filer toolbar. It's amazing how often this jewel is overlooked by new users.

Hope this helps a trifle and good luck with the writing, canbyte

DMcCunney
Posts: 889
Joined: Tue 03 Feb 2009, 00:45

Re: How to optimize memory/RAM & _save file?

#6 Post by DMcCunney »

canbyte wrote:
Newbies note that a 1G flash drive will not hold a backup of your 512mB pup_save file!!! Mine got truncated & unusable. No warning.
I have no idea what actually happened, but I don't believe a Puppy limitation was involved. The save file is simply one big file being written to a storage medium. If the write fails and the file is truncated, the usual reason is either that there wasn't enough free space on the medium to store the entire file, or the media has a problem. The first thing I would check is that the 1GB flash drive did in fact have more than 512MB free. The second thing I would do is access it under Windows and run CHKDSK. Flash drives are formatted with the FAT file system, as either FAT16 if < 2GB in size, or FAT32 if > 2GB. It's possible for them to have file system corruption just like a real FAT formatted disk drive, which will mightily confuse things accessing them.
Thinking about this a bit more, were you trying to store two pup-save files on the 1GB drive? That might have bitten you.

The issue is that 1GB is the nominal total size of the media. The actual usable size will be less, because there is overhead taken by the formatting. I have a 512MB SD card plugged into my Puppy box at the moment, along with a 4GB card. (The 4GB card is where I am storing standard stuff I always want in Puppy. The 512MB card gets stuff from my Windows system to the Puppy system, since the Windows box doesn't want to read the 4GB card in my existing SD card reader.)

Puppy thinks there are 488MB total space on the 512MB card, with 100MB currently free, and 3.75GB total space on the 4GB card, with 2.3GB free.

If you have two 512MB pup_save files, and try to store both on a 1GB flash drive, one will get trashed, as there isn't enough space to hold both.
Windows will complain and abort the transfer if there isn't enough space on the target media to hold what you are trying to store.

Puppy (if you do it through Rox) will copy as much as it can, but not abort the transfer. It will simply copy what will fit and stop. It will throw an error message that the copy was not successful, but it's possible to miss that and find out the hard way things didn't work when you try to access the file.
______
Dennis

User avatar
canbyte
Posts: 264
Joined: Sat 10 Jan 2009, 20:20
Location: Hamilton, Canada

#7 Post by canbyte »

Great info Dennis, Foo. Before i browse to see what i should respond to, ie before i forget, i think this thread is a good place to discuss some disk oriented issues such as fragmentation or lack of it in linux. Will my browser performance deteriorate to the point of unusability over time as it did in Win98? I've suspected a little slowness in startup of the browser lately but haven't got any definitive evidence. Do we have to do periodic defragging, if not, why not?

I've noticed in some posts, a concern about flashdrive lifespan - some kind of deterioration of something physical? What should one do about this if anything - ie. reorder files? How would anyone notice - or how would one diagnose this problem?

I've come at this thread from my viewpoint of crash prevention, stability, understanding the limits of function vis a vis ram/memory. Other folks might want to delve into these same topics to achieve greater speed or other objectives (games, multimedia) i'm not familiar with. Please feel free to explore/explain any tangent you think users should know about.

I've also come at things as a newbie, noob to be exact. So i'll be trying to put the main issues up front /top with the details techies appreciate later on. The file foo pointed me to is a good example of the tech aspects involved, what i want to do is distill out a few critical items newbies should absorb asap.

I didn't think of it before but our discussion of the file system also affects how one approaches the issue of backup. On windows, i periodically make a cd copy of My Documents and a few other things like mail, addresses, etc. Here is where we need to understand what is inside the pup_save file and what is outside. If/when the time comes to install 4.2, will the pup_save file be portable to that system or will i have to import all my stuff, rebuild my printer driver, etc. So do folks typically just keep backups of their _save file or do they keep backups of their separated files. Well, i guess if you're a pack rat like me, you gotta keep data files separately so pup_save is really just saving settings, bookmarks, special drivers, etc.

I've got the idea that newbies can ignore filesystems unless we interact with windows systems. Any difference between ext2 and ext3 doesn't affect us and is handled automatically by linux.

One of the nicer aspects of puppy that is an unexpected energy(?) bonus - the pc's hard drive goes to sleep for long periods of time. At least on the Compaq. This means very quiet operation. At times it seems a bit noisier so i assume the hd came to life and was wondering if/why it would do that (no blinking light to guide me). Anyone have any thoughts as to this aspect, what to expect, how to encourage the sleep mode? Same thing for the monitor with puppy's blank screen saver.

One area of curiosity / mystery to me is this kernel thing which seems to be central to linux and would be nice to give a thumbnail explanation of for newbies. Using my limited investigation skills, i asked Pfind to look for 'kernel', whereupon 7 files showed up from win98 on the hd! Some were quite large (700k) Puppy has 25 such files, some being about 300kb. I guess this is the OS guts but i also wonder about all the drives needed for the zillions of cards, devices, motherboards, memory/ cpu/ types, and all the legacy protocols. Has to be huge database in there somewhere. You'd wonder why the kernel isn't more of a custom thing, downloaded to suit each/any combination of hardware.

Specific points from above:
Dennis, you hit tthe nail on the head - the 1G usbdrive was trying to save 2 _save files, 512 each! Interesting explanation of result.
The truncated file ended up being about 80 mb instead of 512.

I jus discovered that one can change the label on the sdb1 icon - using right click l edit. Makes life easier.

What Bruce mentioned still bothers me, i wonder if there is a way to change it to be time stamped every day.
3) no change on the pup_save file date / time stamp because Puppy doesn't utilize it as a flat file character device, rather as a mounted block device, not treating it differently than it would a partition.
Thanks again all
[color=orange]1. Dell Dimension E521, AMD Athln 64, 2 GHz 1.93GB ram,
Puppy 533 on CD, accesses flash drive only,
FFox Nightly12.0
2. Compaq P3 733Hz 375RAM
Printer: Oki C3400 > LAN [/color]

User avatar
rjbrewer
Posts: 4405
Joined: Tue 22 Jan 2008, 21:41
Location: merriam, kansas

#8 Post by rjbrewer »

Canbyte:

If you want to cavort with the Bulls and Bears of Wallstreet, you
need heavy equipment. Your puny little Puppy boxes don't
stand a chance of handling the tasks you're giving them.

Puppy can not make the wretched excesses of Lassez Faire
Capitalism go away. "Crashing" is the name of the game in the
foreseeable future.

Personally; if I were into that stuff; I'd have a few dozen little
computers and monitors to keep track of "the financial sky
falling". Just for fun of course.

Hang in there.

rjb

Inspiron 700m, Pent.M 1.6Ghz, 1Gb ram.
Msi Wind U100, N270 1.6>2.0Ghz, 1.5Gb ram.
Eeepc 8g 701, 900Mhz, 1Gb ram.
Full installs

Bruce B

#9 Post by Bruce B »

canbyte wrote: What Bruce mentioned still bothers me, i wonder if there is a way to change it to be time stamped every day.
3) no change on the pup_save file date / time stamp because Puppy doesn't utilize it as a flat file character device, rather as a mounted block device, not treating it differently than it would a partition.
Thanks again all
I liken Puppy to an Erector Set, I build it how I want. If you want pup_save time stamped, that's easy.

How about updating the date / time on shutdown?

We use the touch command.

To see what a command does, generally you can say cmdname --help

Let's see what happens if we:

touch --help

Code: Select all

[~] busybox touch --help
BusyBox v1.8.2 (2007-11-30 09:20:42 GMT-8) multi-call binary

Usage: touch [-c] FILE [FILE ...]

Update the last-modified date on the given FILE[s]

Options:
        -c      Do not create any files

[~] 
If I ran this command

touch /initrd/mnt/dev_save/dingo/pup_save.3fs it would change the (modified) time stamp on the pup_save file

If I put the same command in /etc/rc.d/rc.shutdown it would update the time stamp on my pup_save.3fs every time I rebooted or shutdown.

You could put it as the first command in /etc/rc.d/rc.shutdown. Only your command would be different in terms of path and pup_save extension, most pup_save using .2fs and mine using .3fs and your path directory is likely different also. And Linux is stricter than an English teacher with her hair in a bun.

.

DMcCunney
Posts: 889
Joined: Tue 03 Feb 2009, 00:45

#10 Post by DMcCunney »

canbyte wrote:Great info Dennis, Foo. Before i browse to see what i should respond to, ie before i forget, i think this thread is a good place to discuss some disk oriented issues such as fragmentation or lack of it in linux. Will my browser performance deteriorate to the point of unusability over time as it did in Win98? I've suspected a little slowness in startup of the browser lately but haven't got any definitive evidence. Do we have to do periodic defragging, if not, why not?
Linux file systems are far more resistant to fragmentation than FAT file systems are. It's not a concern. Most Unix systems are up 24/7 for years (save scheduled maintenance windows) and never have a problem.
I've noticed in some posts, a concern about flashdrive lifespan - some kind of deterioration of something physical? What should one do about this if anything - ie. reorder files? How would anyone notice - or how would one diagnose this problem?
It's not a huge concern. The flash media used in things like thumbdrives has a limit to the number of writes it can handle. The limit is about 100,000. But the circuitry in the drive transparently remaps failed sectors and marks them unusable, so what you would see is a graceful degradation. You are likely to upgrade to larger, faster flash drives long before you would see any actual effects from disk sectiors becoming unusable.
I didn't think of it before but our discussion of the file system also affects how one approaches the issue of backup. On windows, i periodically make a cd copy of My Documents and a few other things like mail, addresses, etc. Here is where we need to understand what is inside the pup_save file and what is outside. If/when the time comes to install 4.2, will the pup_save file be portable to that system or will i have to import all my stuff, rebuild my printer driver, etc.
The pup_save file is a squash file system, automounted by Puppy. The changes in 4.2 should not make an existing pup_save unusabe, as the SFS file format hasn't changed, nor has the way Puppy handles it. (If you think about it, changes that made Puppy 4.2 incompatible with existing pup_save files would be profoundly stupid, and while the Puppy devs are occasionally mistaken, they aren't dumb.)

(NB: I use a Full install, so I'm not expert on pup_save files and what does/doesn't get included.)
I've got the idea that newbies can ignore filesystems unless we interact with windows systems. Any difference between ext2 and ext3 doesn't affect us and is handled automatically by linux.
Pretty much. There are third-party Windows utilities for reading ext2 file systems, but in general, Windows sees the pup_save file as just another big file in its file system. It neither knows nor cares what's in it.
One area of curiosity / mystery to me is this kernel thing which seems to be central to linux and would be nice to give a thumbnail explanation of for newbies. Using my limited investigation skills, i asked Pfind to look for 'kernel', whereupon 7 files showed up from win98 on the hd! Some were quite large (700k) Puppy has 25 such files, some being about 300kb. I guess this is the OS guts but i also wonder about all the drives needed for the zillions of cards, devices, motherboards, memory/ cpu/ types, and all the legacy protocols. Has to be huge database in there somewhere. You'd wonder why the kernel isn't more of a custom thing, downloaded to suit each/any combination of hardware.
The kernel is vmlinuz in the root directory. It's stored in a compressed form and is uncompressed and loaded at boot. You can see the messages about it when you boot.

Drivers are loaded and linked to the running kernel at boot time. There is a large database of drivers to handle possible hardware on which the system will be run, but which drivers are actually used is determined when Linux is initially installed. They aren't all loaded and active at once.

The kernel is the core of the OS. It sits between you, your programs, and the hardware. It handles memory management, process management, and access to the hardware through drivers. Properly speaking, Linux is the kernel. The rest of the stuff shipped with a Linux distribution tends to be Gnu versions of Unix utilities, and are available on lots of different systems. Mac OS/X includes them, for example. The fact that shipping Linux systems include the Gnu utilities is why they are often called Gnu/Linux systems.

(I run a Windows package called Cygwin. Cygwin is a project to port the Gnu toolchain to Windows. Instead of rewriting every utility to use Windows system calls instead of Unix system calls, the Cygwin developers wrote a POSIX compatibility layer containing the most frequently used Unix system calls translated to Win32 equivalents. Lots of Unix code builds under Cygwin because it links against the Cygwin1.dll, and sees the routines it expects to see. Cygwin includes the Gnu Compiler Suite and most of the other standard Unix/Linux utilities, including the vi editor and the bash shell.)

It's less popular now, but one of the common tasks a Linux user would perform was to recompile the kernel. Linux distributions ship with a generic kernel intended to run on the widest possible range of systems, and includes stuff a particular system may not need. Users would get the source and recompile the kernel to include only the stuff their particular system required to make it smaller and more efficient.

Modern hardware tends to be powerful enough that there is less need for that sort of optimization (and the results might not be significant if you did.)
Dennis, you hit the nail on the head - the 1G usbdrive was trying to save 2 _save files, 512 each! Interesting explanation of result.
The truncated file ended up being about 80 mb instead of 512.
Use a 2GB card. You ought to be able to store three pup_save files.
______
Dennis

DMcCunney
Posts: 889
Joined: Tue 03 Feb 2009, 00:45

#11 Post by DMcCunney »

rjbrewer wrote:Puppy can not make the wretched excesses of Lassez Faire Capitalism go away. "Crashing" is the name of the game in the
foreseeable future.
"Laissez-faire Capitalism" is irrelevant to the equation.

Hardware gets steadily smaller, faster, and more powerful. More powerful hardware permits applications that weren't possible on older systems. It also means that expectations must be adjusted, as things possible on newer hardware can't be done on older systems, as they lack the power to perform the tasks in an acceptable manner. Programs may not run at all, or run so slowly as to be unusable.

Puppy Linux does a good job of getting the most out of older hardware, but can't perform miracles.

Hardware is cheap, and getting cheaper. The scarce resource is programmer time. It's faster, cheaper, and simpler to just get more powerful hardware than to try to optimize code to run on older hardware.

This is a consequence of computer development, and will be true regardless of the economic system in place.
______
Dennis

User avatar
rjbrewer
Posts: 4405
Joined: Tue 22 Jan 2008, 21:41
Location: merriam, kansas

#12 Post by rjbrewer »

DMcCunney

You missed the point of my "tongue in cheek" message
to Canbyte.

Inspiron 700m, Pent.M 1.6Ghz, 1Gb ram.
Msi Wind U100, N270 1.6>2.0Ghz, 1.5Gb ram.
Eeepc 8g 701, 900Mhz, 1Gb ram.
Full installs

User avatar
canbyte
Posts: 264
Joined: Sat 10 Jan 2009, 20:20
Location: Hamilton, Canada

#13 Post by canbyte »

Hi folks, Unfortunately got derailed in my reply when Dell crashed this am.

About that tongue crashing in cheek thing, I have to guess that the folks here would be about the least vulnerable to this meltdown, or lets hope so.

RJB said
Puppy can not make the wretched excesses of Lassez Faire Capitalism go away.
Well!! Perhaps this pic-that-tells-thousand-words will make folks sit up and take notice of our little beast! You never know.
http://finance.yahoo.com/q/bc?t=1y&l=on ... &c=&s=msft

The hardware issue of course is important - i have avoided the more intensive involvement in something called Level 2 & 3 systems, dual monitors and the like. I just use normal web base systems that Puppy seems to handle ok. The reason for so many windows is just habit, inclination and sometimes, the press of time. One learns by experience what works and adjusts accordingly. So far, very good.

You've given me lots to chew on. But first, a few new items.

I tried the partition thing on the dell - unsuccessfully. Then, Using the 'graphical disk map' item on the menu, i discovered to my surprise that there is 234mb on 'file system', (incl. 170 on 'usr'. I had thought puppy was only 80mb or so - no wonder the dell crashed with only a little going on. How did it ever start with only 127mb ram ( and showing 53free)???
Finally got swap going with these easy instructions from purple ghost.
http://www.murga-linux.com/puppy/viewtopic.php?p=279708

Free mem went from about 200mb (smaller save file) to 645 (partition activated) then 495 once the pup_save file got rebooted at the larger size. I wonder why it dropped?

More on swap files.
This link discusses swap file instead of swap partition & large sizes
http://www.murga-linux.com/puppy/viewtopic.php?t=28174

This link has interesting dissertation on life of sticks - confirming no worry.
http://www.murga-linux.com/puppy/viewto ... 6&start=15

This link has discussion of swap security issues
http://www.murga-linux.com/puppy/viewtopic.php?t=36856

THis link gives strategies to use for lots of junk (photos) on limited (frugal) systems. Go to last 5 posts on page. Involves moving myDocs folder
http://www.murga-linux.com/puppy/viewto ... 0b6746714f
rcrsn51 says
Think of the savefile of a frugal install as being like the glove compartment of your car. It's more organized, but it has limited size.

Think of the rest of your hard drive as being like the back seat of your car. It's less organized, but you have way more space.

In a frugal install, anything you save through the "file" icon goes into the savefile. Anything you save through /mnt/sda1 goes into the rest of the partition outside of the savefile.
Good rule of thumb.

I guess one can surmise that using different puplets will affect the memory/swap/speed thing. Perhaps someone can make up a chart that gives all the functions & memory requirements of various puplets.

Dennis wrote:
The second thing I would do is access it under Windows and run CHKDSK.
A caution for newbies here: do not let windows fix the problem, correct?? ----
since you said later that the flash drives essentially fix themselves.

I guess this thread should be called something like "memory management, requirements and optimization". I'm not sure we are there yet but perhaps we're getting close.

Cheers
[color=orange]1. Dell Dimension E521, AMD Athln 64, 2 GHz 1.93GB ram,
Puppy 533 on CD, accesses flash drive only,
FFox Nightly12.0
2. Compaq P3 733Hz 375RAM
Printer: Oki C3400 > LAN [/color]

User avatar
rjbrewer
Posts: 4405
Joined: Tue 22 Jan 2008, 21:41
Location: merriam, kansas

#14 Post by rjbrewer »

A little tip Canbyte:

Life with Puppy is at least a whole order of magnitude easier
using "full install" instead of "frugal". No worries over errant
pup-save files or mysteries about the numbers in the free-mem
applet.
I did download and do a cd-less frugal install of 4.2rc yesterday;
then used that frugal install to make a full install without needing
a cd. Deleted the frugal.

Takes 38 seconds for full to boot on my laptop....about 60 secs.
for the frugal.

The movie played fine on my setup. Interesting; but it's best to
be cautious when dealing with "lunatic fringe Alex Jones" type
stuff.

rjb

Inspiron 700m, Pent.M 1.6Ghz, 1Gb ram.
Msi Wind U100, N270 1.6>2.0Ghz, 1.5Gb ram.
Eeepc 8g 701, 900Mhz, 1Gb ram.
Full installs

DMcCunney
Posts: 889
Joined: Tue 03 Feb 2009, 00:45

#15 Post by DMcCunney »

canbyte wrote:Perhaps this pic-that-tells-thousand-words will make folks sit up and take notice of our little beast! You never know.
http://finance.yahoo.com/q/bc?t=1y&l=on ... &c=&s=msft
MS would be having problems these days in any case.

MS is the quintessential "growth" company. For many years, it handily beat analyst's expectations, and recorded double-digit increases in revenues and profits. As a consequence, it got a stock price in the stratosphere. It didn't originally pay dividends, but investors made out in capital gains: the value of their holdings increased.

The problem is maintaining growth. At this point, just about everything in the US that can run Windows and Office, does. Revenue comes from upgrades of existing installations, and not new sales. (Blame Vista getting getting released before the extant hardware was really up to running it on MS's need to get a new product out the door and generate a revenue stream from it.) Where will the new sales growth come from? The EU is resistant to MS. MS has so far failed to significantly penetrate the Indian or Chinese markets. MS's online initiatives have never been successful. The Xbox is only beginning to contribute to MS's bottom line.

MS is in transition from a "growth" company to a "mature" company that dominates its market. Mature companies throw off enormous amounts of cash, but don't have stock proces in the stratosphere. Bill Gates picked an apt time to retire. He left a winner, with Microsoft at a peak. It's now Ken Ballmer's job to support the price of the stock, and it's a good question as to whether he can in the absence of significant new sales of MS products to the rest of the world.
Dennis wrote:
The second thing I would do is access it under Windows and run CHKDSK.
A caution for newbies here: do not let windows fix the problem, correct?? ----
since you said later that the flash drives essentially fix themselves.
No, because you are conflating two different kinds of problems. A few words about drives might help.

All drives, whether they are actual disks with spinning platters or flash media that looks like a disk share common characteristics and problems. Problems can be hardware or software related.

The usual hardware problem is "bad blocks". These are actual sectors on the disk (or memory cells in flash media) that cannot be successfully read/written. Hard drives ship from the manufacturer with a bad block table, and blocks in the table are marked as unusable and not accessed when data is read from or written to the drive. If bad blocks develop on a drive in a Windows machine, the usual fix is to run CHKDSK with a flag that tells it to do a surface scan. It will scan all disk sectors, and if it finds bad ones it will mark them as unusable and add them to the bad block table.

As previously discussed, flash media has limits inherent in the technology on the number of times it can be written to. But as mentioned, the controller circuitry in flash drives transparently maps bad cells as unusable and copies the data elsewhere, so what you would see if you saw anything was a gradual degradation as more cells grew bad. In practice, you are likely to replace the flash drive with a bigger faster one long before you would actually notice any issues.

Those are hardware issues. The more common problem is file system corruption, which is fundamentally software. Flash drives come formatted with the MS-DOS FAT file system. There are several variants of FAT in existence.

The smallest unit of disk space that will be read from/written to in one operation is the cluster. Each cluster must have a unique address. The version of FAT used by MS=DOS PCs and Windows 3.X machines was FAT16. It used a 16 bit cluster address, and therefore permitted a maximum of 65,536 (2^16) clusters on a drive. How large a cluster was depended on the size of the drive, with the maximum possible size of a FAT16 volume being 2GB. This gave rise to a phenomenon called "slack space". Since the smallest unit of disk that could be used was the cluster, and since a cluster could not contain more than one file, you could have cases where a one line text file totaling 100 bytes could actually take up much more space on the drive, because the rest of the space in the cluster it occupied could not be used for anything else and would be wasted. The waste space was called "slack space", and various utilities existed to tell you how much slack space there was on a given volume.

As drives got progressively larger, it became necessary to partition them into multiple logical drives of 2GB or less so MS-DOS and Windows could use them. MS came out with an updated version of FAT called FAT32, whch used 32 bit addressing, permitting many more clusters and much larger volumes.

Flash drives are shipped formatted as FAT, using either FAT16 if the drive is <= 2GB, or FAT32 if it's > 2GB. FAT is well known and widely implemented, and just about everything understands how to access a FAT file system, which is why it is used in flash drives.

The common problem with FAT file systems is "lost clusters". It's possible for clusters to be marked as "in use" in the File Allocation Table without actually being owned by any files. The usual cause for this is a program or Windows itself crashing before the FAT table was properly updated. These are known as "lost clusters", and mightily confuse the OS and programs. On MS-DOS and Windows, the solution is to run CHKDSK (or ScanDisk on Win9.X machines). CHKDSK examines the file system and looks for lost clusters. If it finds them, it allocates them to files it creates with names like FILE000.CHK, and puts those files in a directory called FOUND.000 where they may be examined and (usually) deleted, restoring the file system to sanity. (CHKDSK has no idea what it in the clusters - it may be program code, data, text - it simply maps them to files, and the contents may or may not be useful.)

I use SD cards in my Palm OS PDA. I've occasionally had to pop one into a USB card reader and run CHKDSK on it to repair lost clusters, because they confused the FAT driver the PDA used to access the card. You can assume Puppy will be similarly confused if lost clusters exist.

If the flash drive is used to store pup_save files, it's problematic. Lost clusters may be part of a pup_save file, and deleting them will make the file unusable, but if you have lost clusters, you're going to have problems accessing the pup_save file in any case. Your best option may be to copy the contents of the drive elsewhere, reformat the drive, then copy the contents back, but there's no guarantee the pup_save files will be usable.

You can check a drive for problems by just running CHKDSK with no arguments. CHKDSK will do a read-only scan looking for problems. If it finds any it will report that it found errors and can't continue in read only mode.

I use flash drives with my Puppy machine, but haven't had problems with them so far. As mentioned, the cause of lost clusters is a program or OS crash that prevented the file system from getting updated. Puppy doesn't crash on me. Nor does XP and my applications on the desktop. (People who have that sort of problem are probably still using Win98/ME, and really need to move away from them. Win2K and XP are an order of magnitude more robust and stable.)
______
Dennis

User avatar
canbyte
Posts: 264
Joined: Sat 10 Jan 2009, 20:20
Location: Hamilton, Canada

#16 Post by canbyte »

Interesting perspective on MSFT - this link compares MSFT to linux
http://finance.yahoo.com/q/bc?s=NOVL&t= ... &c=rh.msft
Too bad we can't bet on Puppy! Better than an electric car imo.

Haruumph! Why is html off?? My preference is set 'html on' All the links are dead. Sorry :-(

RJB:
best to be cautious when dealing with "lunatic fringe Alex Jones" type stuff.
Compared to this mainstream stuff
http://finance.yahoo.com/q/bc?t=5y&l=on ... &a=&c=&s=c
i'd say the lunatic fringe is looking better all the time:
http://finance.yahoo.com/q/bc?s=GLD&t=5 ... z=m&q=l&c=
Who is the crazy really??!!

I'm surprised at the longer startup time for frugal, seems counterintuitive - any thoughts why?

Your sequential install method sounds interesting/ practical - should post in the (underused) wiki.

Curious- what is rc in 4.2rc. What does 4.2 fix/add cf 412?
---
I tried to create a swap file on a flash drive without any success, at least using Gparted. Would accept the command but made error during operation. I'm hoping that setting the flash back to original fat16 will not lead to future errors (appears to work). Thread here confirms flash swap is not a good idea
http://www.murga-linux.com/puppy/viewtopic.php?t=29755
http://www.murga-linux.com/puppy/viewtopic.php?t=19176

I'm now wondering about the advisability of a swap File as described here. http://www.murga-linux.com/puppy/viewto ... 0b6746714f
and
http://www.murga-linux.com/puppy/viewtopic.php?t=28174
for very large swap files

This wiki is a bit old (v2) but may still be a good description of the operation? Quite technical, needs 'rule extraction' for us newbies.
How Puppy Works
http://www.puppylinux.com/development/h ... works.html
Barry says
"However, PCs with less than 128M really do need a swap partition for Puppy to work properly. Even though Puppy can boot up from live-CD on a PC with only 32M RAM, some of the applications are memory hogs -- Mozilla for example. Mozilla is not viable on a PC with less than 128M RAM. However, as mentioned above, a swap partition increases the effective size of the ramdisk, so you can get Mozilla to work on PCs with very little RAM, albiet slowly. It is also a good idea to add a swap file, if you have some spare space on the hard drive."
Q. Is v4 = v2 for our purposes?

I've sorta run out of questions other than still wondering what the real size of puppy is - does it really vary between 80 and 234mb depending on space available? Can other folks have a look at their ram with Filesystem>Gdmap and record their results here?

Finally, to complete an earlier thought - summary for file symbols
Maybe this has been done elsewhere???
Mostly i'm browsing and guessing so please add others & change/fill in type
open brown box = zip file
open box blue = dotpup?
hand under folder = shared
eyeball icon = jpg
meadphones = sound file
typed page = text file
typed page + A = (Adobe) pdf file
printer? (.ps) = ??
binary page (.swf))= compiled binary file
color film (.avi,.mov,.mpg)= multimedia
film abc =
console page = script (unix)
gear = ?
i encircled = help
wrench in monitor = wizard
target = save to usb
dard drive icon = drive drivers
cd symbol =
plug symbol =
! in orange triangle = core?
=
=


Cheers
Last edited by canbyte on Wed 11 Mar 2009, 18:27, edited 1 time in total.
[color=orange]1. Dell Dimension E521, AMD Athln 64, 2 GHz 1.93GB ram,
Puppy 533 on CD, accesses flash drive only,
FFox Nightly12.0
2. Compaq P3 733Hz 375RAM
Printer: Oki C3400 > LAN [/color]

User avatar
rjbrewer
Posts: 4405
Joined: Tue 22 Jan 2008, 21:41
Location: merriam, kansas

#17 Post by rjbrewer »

Canbyte;

About the size of puppy: Using live cd its' compressed image is
under 100mb and can run in 128mb ram.
With a full install the image decompresses on the hard drive to
about 220 or so mb.
Hard drives are very slow compared to ram; swap files or par-
titions help a little, but not much. Doesn't hurt to have some.

A swap partition the same size as ram is also used for hibernation
or suspend to ram by some Linux distros for laptops.

One swap partition will be used by all the O.S. installed on the
system; swap files need to be on each partition AFAIK.

Puppy 4.2 boots a little slower; has a lot of "widget" action
going on. It really is beautiful though.

I'm putting together a little piece for the "how to" section on
my newly discovered install method. Haven't seen anything
exactly like it before.

Bill Gates is buying 500,000 shares a day of a waste manage-
ment company.

edit
When you copy a link to the post, highlight it before you
push the URL button.
I still haven't mastered the "short-link" version.

rjb





rjb

Inspiron 700m, Pent.M 1.6Ghz, 1Gb ram.
Msi Wind U100, N270 1.6>2.0Ghz, 1.5Gb ram.
Eeepc 8g 701, 900Mhz, 1Gb ram.
Full installs

Bruce B

#18 Post by Bruce B »

Puppy's core files

vmlinux
initrd.gz
pup_xxx.sfs
zdrv_xxx.sfs ( not used after 4.00 )
boot files

overview of the core frugal files

vmlinux

is the Linux kernel, which is expanded and loaded into memory. it is read from the boot media, which could be a variety of media types such as cdrom disc, usb stick, hard drive and etc.

once the kernel has been read, it is no longer needed during that session.


initrd.gz

it has the script and files necessary to determine your session type, additional parameters and give you the working operating system. could be conceptualized as the 'glue' to put the necessary parts together. once used, it is not needed or used again during that session

pup_xxx.sfs

where xxx corresponds to your version number. the vast majority of Puppy's files are contained in pup_xxx.sfs. unlike vmlinux and initrd.gz; pup_xxx.sfs is in continual use during your session and can not be removed. the session depends on this file being mounted and available


it is mounted as a squash file system

the mount point is typically /initrd/pup_ro2

it is mounted read-only

it is not decompressed and is used as a compressed file system

when files are needed from it, they are copied out of it (and decompressed) into memory, after the program and files have been closed by the user, they might remain in memory cache. if they are and Linux needs the memory, it will drop various caches to make room for the next program you want to run

if you have sufficient RAM, Puppy allows pup_xxx.sfs to be copied in its entirety to RAM, thereby freeing the dependency on the original pup_xxx.sfs file. if the original was on a CD Disc, by copying it to RAM the CD Disc can be freed as no files, (vmlinuz, initrd.gz and pup_xxx.sfs) are needed any longer, everything necessary to run Puppy is in RAM. The CD disc can be removed

boot files

there can be different kinds of boot files, the only thing I wish to say is Linux is not self-booting and depends on an external boot process to get anything up and running. Depending on the install type, the user has a lot of say regarding what system he uses to boot Linux

=====================

Looking back at vmlinuz, initrd.gz and pup_xxx.sfs, please note that none of these files are written to or writable in normal operations. they are all read only.

If we want to write any information at all it will not be to these files. This is where the convience of the pup_save file comes into play.

pup_save.2fs

it doesn't come with Puppy and is not essential to its operation. it is a user option for the purpose of being able to 'write' and 'keep' changes.

the pup_save is simply a file which has been formatted as an ext2 Linux format (the default format anyway)

it is not compressed

it is mounted as a read/write filesystem at /initrd/pup_rw

it is not copied to RAM or moved

in the boot process initrd.gz locates it and 'glues' it in as part of the total operating system

when using a pup_save file the user sees a layered hodgepodge of pup_xxx.sfs, (other user mounted devices) and the pup_save file, although it may be a literal hodgepodge, it appears as a nicely organized directory tree and files

unbreakable rules

a session doesn't write to our three core files ( a user could )

we cannot have duplicate files or directories

we cannot for example; have two files, one file and one directory, or two directories of the same name under the same parent directory

but we do have many duplicates and we can't. we can't have two /root directories for example but we do

layered filesystems

considering we break unbreakable rules, the way it is done is by layering the various filesystems, directory trees and files.

naming conflicts of which there are many are resolved in favor of the contents of the pup_save file.

example: pup_xxx.sfs has /usr/bin/leafpad and that is the file we are using, (the file inside pup_xxx.sfs), which is the source of the file we use. if we install a newer version of /usr/bin/leafpad . . .

. . . we introduce a naming conflict . . .

. . . we do not make an overwrite, the first /usr/bin/leafpad is in a readonly filesystem. we can't write to it. we can't delete it

we can add another /usr/bin/leafpad to the pup_save file, but that breaks the rule. you cannot have two files of the same name in the same place, which in this case is /usr/bin/leafpad

when using prioritized layered filesystems, giving top priority to pup_save we can break the rule. the /usr/bin/leafpad in the pup_xxx.sfs is made invisible and inaccessible and unknown, pup_save wins the naming conflict and the leafpad we are using is the one we put inside the pup_save file

====================

some conclusions

while you can't modify the core files in normal operation, except pup_save, it's still an erector set of sorts, one, you can take apart, and put back together as you wish.

User avatar
potchan
Posts: 109
Joined: Sat 05 Apr 2008, 11:46
Location: Pilots' Height Tel Aviv - Yaffo, Israel לינוקס_פותחן
Contact:

Potchan's view and search for enlightnment on mem' stuff

#19 Post by potchan »

Hi Dennis, Bruce B and Team,

Take a look at this:

http://www.murga-linux.com/puppy/viewtopic.php?t=37660

Any help for you ? :roll:
Timmo', the potchan (=opener) team at - [url]http://potchan.org[/url]. Taste it, love it, code in it.

Bruce B

Re: Potchan's view and search for enlightnment on mem' stuff

#20 Post by Bruce B »

potchan wrote:Hi Dennis, Bruce B and Team,

Take a look at this:

http://www.murga-linux.com/puppy/viewtopic.php?t=37660

Any help for you ? :roll:
Only I noticed amigo referenced the swapon -s switch and
it doesn't exist in Puppy's swapon. I've attached the full
featured swapon if you want it.

Code: Select all

Usage:
 swapon -a [-e] [-v] [-f]             enable all swaps from /etc/fstab
 swapon [-p priority] [-v] [-f] <special>  enable given swap
 swapon -s                            display swap usage summary
 swapon -h                            display help
 swapon -V                            display version

The <special> parameter:
 {-L label | LABEL=label}             LABEL of device to be used
 {-U uuid  | UUID=uuid}               UUID of device to be used
 <device>                             name of device to be used
 <file>                               name of file to be used

~
Attachments
swapon.gz
(19.52 KiB) Downloaded 2033 times

Post Reply