"Swap finder" - request for script

[greengeek]

Sorry if this is in the wrong section - I couldn't find a better place to put it.

I would like some info / pointers / coding assistance in finding a method to make it possible/easy for an inexperienced user in a live session of Puppy to go hunting for a swap partition.
EDIT : ..and then I want to use that info to automatically help the user set up a swap on an external usb drive or HDD if some other swap is not currently available

Why?

Imagine that you are using a live session (no savefile and no /home partition) and you want to do something (like remastering) that is going to consume more RAM than you have available. As far as I know, all you need to do to increase the effective working storage area is to add 'swapon' and give the system a location to use and everything will be ok.

I'm working on an automatic remastering method (in live session) for novices who won't know whether they need extra swap or not, and won't know how to make a swap partition or set swapon.

I don't care about processing speed so all I am wanting to do is allow the user to plug in a slow usb stick (even in a usb 1.1 port) and allocate that usb stick as swap.

But how to achieve this for a new user who has no idea which sdx device it will be and has no knowledge of the swapon syntax?

In the situation where I hope to use this script the user will not have any prior knowledge of what is required - they will just see a message on screen telling them RAM is insufficient and they need to plug in a blank usb stick.

I need the system to detect this most recently plugged stick (ignoring existing plugged drives) and maybe bring up a rox window so they can identify that it contains no critical data.
(This could be problematic because I currently have disabled desktop drive icons as I don't want the novice user affecting existing hard drive data)

Once the drive is visible to the user and confirmed as safe to use I guess then It would be necessary to reformat the usb stick as swap, and allocate swapon, before returning to the original remaster (or whatever) script.

Another example of when such a process might be useful could be other RAM intensive things like video processing in live session.

Any criticisms, advice or code examples much appreciated!

[rcrsn51]

cat /proc/swaps

[greengeek]

cat /proc/swaps

Thank you - I was not aware of that command. That will be useful.

I've just run that command and it tells me two swap partitions are available on this machine but that neither of them is in use in this non standard Live pup I've built. Is it normal for all live pups to ignore swap partitions? Or does it vary depending on what each dev set up?

I guess I previously assumed all Linuxes used whatever swap was available, but assumptions never help much.

[gcmartin]

Is it normal for all live pups to ignore swap partitions?

No! Only a couple PUPs do so. Most every PUP (and many Linux distros) will detect a SWAP partition and incorporate it into the running system.

Or does it vary depending on what each dev set up?

Couple of devs choose not to make SWAP available to the system at boot. These dev's expect users to be intelligent Linux users.

Hope this helps

[greengeek]

Thanks GC - any idea where the code is that makes Puppy find and use existing swaps? Any idea if it's in the initrd.gz or main sfs?
cheers

[RSH]

This here is really interesting, but it confuses me even more as I'm already confused about the swap partitions.

My PC has 2GB of RAM and 1 swap partition around 4GB.

When hovering the mouse pointer over the freememapplet_tray in the sys-tray, a popup tells me that I do have 3GB of available memory in my save file (running in RAM, PupMode 5, no save file in use) and the terminal gives this output:

# cat /proc/swaps
Filename Type______Size______Used__Priority
/dev/sdc4 partition_4194300_0______-1
#

Underscores included to format text for the forum (it's eating multiple spaces)

So is my swap in use?
Why there is only 1GB added to the 2GB of RAM and shown as the combined 3GB available space in save file?

Edit:

Thanks GC - any idea where the code is that makes Puppy find and use existing swaps? Any idea if it's in the initrd.gz or main sfs?
cheers

The init script in intrd.gz (and rc.sysinit) is searching for the swap.

Code: Select all

#there are technical problems with loading a swap partition/file before the union
#is created, so not doing it until rc.sysinit runs. however, if a tmpfs needs to be
#created here, set it's size in anticipation of a swap being loaded...
EXTRAALLOCK=0
SWAPPART="`echo "$PCPARTSALL" | grep '|swap|' | head -n 1`"
[ "$SWAPPART" ] && SWAPPARTSIZE=`echo -n "$SWAPPART" | cut -f 3 -d '|'`
[ $SWAPPARTSIZE ] && EXTRAALLOCK=`expr $SWAPPARTSIZE \/ 2`
if [ $EXTRAALLOCK -eq 0 ];then
 [ -f ${SMNTPT}/pupswap.swp ] && SWAPFILESIZEBYTES=`stat -c %s ${SMNTPT}/pupswap.swp`
 [ $SWAPFILESIZEBYTES ] && EXTRAALLOCK=`expr $SWAPFILESIZEBYTES \/ 2048` #use half. 110521 also convert bytes to kb.
fi

[greengeek]

...My PC has 2GB of RAM and 1 swap partition around 4GB. When hovering the mouse pointer over the freememapplet_tray in the sys-tray, a popup tells me that I do have 3GB of available memory in my save file.... So is my swap in use?
Why there is only 1GB added to the 2GB of RAM and shown as the combined 3GB available space in save file?

I would say that it is suggesting the swap is not in use. So the question is where did the extra 1GB of free RAM come from? Must be a motherboard special 'freebie' for good customers

Actually, I see a similar pattern on my machine - I only have 1GB ram, apparently no swap loaded, but showing 1.2GB of 'personal storage free space'.

EDIT : No, wait - I'm horribly wrong. When I saw "used=0" I thought it meant "0" as a boolean - and I was expecting to see "1". I just went back and did the cat /proc/swaps again and this time I saw "used=656" so that suggests it is a byte count (yep, I'm dumb), which of course means my swap MUST be in use. I wonder if you could pull out half your ram and do some video editing and see what "used" value it shows on your system then?

The init script in intrd.gz (and rc.sysinit) is searching for the swap.

Code: Select all

#there are technical problems with loading a swap partition/file before the union
#is created, so not doing it until rc.sysinit runs. however, if a tmpfs needs to be
#created here, set it's size in anticipation of a swap being loaded...
......

Thanks, I will check out rc.sysinit for the swap code.

[rcrsn51]

The "free" command also has swap information.

[greengeek]

The "free" command also has swap information.

Thanks. I have attached a comparison of both sets of info.

With regard to the info offered by cat /proc/swaps - it definitely seems to be relevant to swap partitions that are currently in use (rather than "available to be used if you want to set them up"). If I type:

Code: Select all

#swapoff /dev/sdb3
#swapoff /dev/sda3

(to turn off both of my swap partitions), then cat /proc/swaps returns blank information. (I might have expected that it would still reveal them but mark them as disabled but thats not the case)

Question: If the swap partitions were not available/activated during boot - is there any way to make the system discover them automatically if I plug in the usb based swap partition after boot?
(EDIT : - if I plug in a new swap usb stick after boot it seems as if the system finds the swap partition itself and all I need to do is force a swapon manually. Except for a couple of times it didnt detect the swap at all. Not sure why. More testing to do)

I am assuming it will be necessary to see what code is in rc.sysinit and try to emulate/expand it for use after manual insertion of the swap usb.

[saintless]

..and then I want to use that info to automatically help the user set up a swap on an external usb drive or HDD if some other swap is not currently available

Maybe this will help you (do not download the archive from there - it will not work in Puppy):
http://murga-linux.com/puppy/viewtopic. ... 004#788004

Attached the same make-swap GUI for Puppy (Yad lines replaced with xmessage and echo command - could be done better with gtkdialog progress bar). What make-swap offers is a choice to create swap file. After creating the program mounts (swapon) this swap file. You can create and use multiple swap files. The free command will increase the swap size information with every new swap file added.

Question: If the swap partitions were not available/activated during boot - is there any way to make the system discover them automatically if I plug in the usb based swap partition after boot?

This is a script from Puppy we use to mount swap partition manually (if it is available but not mounted on boot). link in /root/Startup will automate the searching after X starts, but I don't know how to make it auto-start when usb stick is inserted:

Code: Select all

#!/bin/sh

if grep -qv noswap /proc/cmdline; then
  echo "LOAD SWAP"
  for ONESWAP in `fdisk -l | grep ' Linux swap' | cut -f 1 -d ' ' | tr '\n' ' '`
  do
    echo -n "Loading swap partition $ONESWAP..." >/dev/console
    swapon $ONESWAP

    [ $? -eq 0 ] && SWAPON="yes"
  done
fi

It should work in Puppy without changes. I don't think grep -qv noswap /proc/cmdline will make troubles in Puppy but needs testing.

[greengeek]

Hi Toni, thanks for the info and script. What is the purpose of looking at /proc/cmdline - is that checking to see if loading swap has been forbidden by a kernel parameter issued at boot time?

[gcmartin]

When there is adequate RAM for system needs, SWAP will not get used. But, as system needs increase, SWAP will begin to be used to the system's advantage as it protect for good overall performance.

One area you might want to pay attention to when using SWAP partitions. Its a little dangerous to have 2 different SWAP partitions or files active at the same time across several devices. Reason: all devices do NOT exhibit the exact same characteristics. So, for example, one USB works at one I/O rate and a 2nd will operate at a different I/O rate. When SWAP use is spread across the 2, system performance will suffer because of the slower device.

There is a good reason for having multiple SWAPs; namely redundancy. But the performance penalty is nothing to ignore.

For redundancy, it may be better to insure that the devices are twins with serial numbers in the same batch if redundancy is needed. There's a greater chance of equal performance and redundancy would be achieved.

Sometime, although I have not used this in a personal home system, SWAP can be added to a running system when one might notice SWAP and RAM becoming totally consumed. This would achieve some relief until the system in brought back into a "normal"operating range. But, excepting for datacenters I have never seen this employed.

If anyone gets that pesky message of system space getting consumed, it might be worth a try to add additional SWAP via the command line to see if that message goes away versus closing down applications. Hmmm???

Hope this helps

[saintless]

Hi Toni, thanks for the info and script. What is the purpose of looking at /proc/cmdline - is that checking to see if loading swap has been forbidden by a kernel parameter issued at boot time?

Yes, this is check for noswap parameter we use in DebianDog. The script is taken from loadswap_func function inside Puppy /etc/rc.d/rc.sysinit. You can check this loadswap_func and use part of it instead this script.
I think you will not find noswap parameter in puppy boot code and this script could be changed to simple version to work with Puppy - something like:

Code: Select all

#!/bin/sh

  for ONESWAP in `fdisk -l | grep ' Linux swap' | cut -f 1 -d ' ' | tr '\n' ' '`
  do
    echo -n "Loading swap partition $ONESWAP..." >/dev/console
    swapon $ONESWAP
  done

If there is swap partition not in use - it will be mounted.
If there is swap partition in use - it will give message it is already in use.

Toni

[greengeek]

If anyone gets that pesky message of system space getting consumed, it might be worth a try to add additional SWAP via the command line to see if that message goes away versus closing down applications. Hmmm???

Yes, that is the sort of situation I am thinking of - times when puppy is running solely in ram and we know that it must be running short of available memory. It would be nice to plug in a dedicated external storage device that would alleviate the shortage of ram 'on-the-fly'.

I'm not clear on exactly when puppy DOES tell you it is running low on ram, and when it doesn't tell you. I've had some remastering failures which I assumed were probably unflagged ram shortages - but I guess Puppy sometimes is too busy to let me know the problem at the time...

and this script could be changed to simple version to work with Puppy - something like:

If there is swap partition not in use - it will be mounted.
If there is swap partition in use - it will give message it is already in use.

Thanks Toni, this will be very handy. I plan to combine this with a dialog telling the user how to prepare an appropriate usb stick as 'swap' device before they proceed with the remastering (or video editing) process.
.

[greengeek]

On a vaguely related topic I'd like to ask the following question:

If a user had a laptop containing something like a 1GHz cpu (ie enough to run a modern pup) but only 256MB ram (a bit too small to do much with) could they simply plug in a usb stick formatted as swap and have it act as extended ram? Would such an arrangement allow a live session (running in ram only) to behave as if the machine had 1GHz cpu + 1GB of ram? (other than the considerations of usb bus speed versus normal ram speed).

What if you plugged in a USB2 pcmcia card with the 'usb swap' plugged into that - would the system be able to use that as a means of increasing overall ram?

[RSH]

From my experience a SFS Module is not loaded into RAM, if there isn't enough free RAM available.

Usually when it comes to the point, to load the main sfs at boot up, it takes a few seconds until it is loaded and boot process is continuing.

In my beginning days I've worked on a remaster of Lucid Puppy 525 (my first go) that in the end has had reached a compressed size of appr. 1GB. At this time my PC has had only 1.5GB RAM available, so the main sfs of this remastered Lucid was just somehow touched at boot up (took less than a second) and the boot process did continue and was successfully finished each time.

I assume loading SFS Modules by sfs_load -or similar technic- will act equally to this.

[saintless]

On a vaguely related topic I'd like to ask the following question:

If a user had a laptop containing something like a 1GHz cpu (ie enough to run a modern pup) but only 256MB ram (a bit too small to do much with) could they simply plug in a usb stick formatted as swap and have it act as extended ram? Would such an arrangement allow a live session (running in ram only) to behave as if the machine had 1GHz cpu + 1GB of ram? (other than the considerations of usb bus speed versus normal ram speed).

What if you plugged in a USB2 pcmcia card with the 'usb swap' plugged into that - would the system be able to use that as a means of increasing overall ram?

No, swap partition/file is used when the amount of RAM is full. Swap does not replace physical memory.
I really hope someone can answer yes to this question and give working method to copy to RAM main sfs module 512Mb on boot on computer with 256Mb RAM + 1Gb Swap partition but I doubt it is possible.
And maybe it will net be faster or better than booting from flash drive without copy to RAM.

[technosaurus]

This is how to go about it.
Swap partitions:
guess_fstype or blkid on all filesystems in /proc/partitions
Swap files:
use find on / to look for puppy swap files (what is the extension? I forget)
Fallback:
create an empty 1Gb swap file and bzip it to include as a fallback if none is found (it compresses to only a few kb with bzip -9 ... its even better than xz for empty files)

For 256Mb ram - use Akita... usb will likely be slower than using the disk and a swap partition/file, but it will at least prevent OOM errors

Ideally your swap partition should be on a separate drive (preferably the fastest one and preferably a separate disk controller... for IDE drives, that means on a separate cable)

Note: newer kernels have zswap which compresses the swap and can actually speed it up ... I think tmpfs (used for /dev/shm and sometimes /tmp) has a similar feature now too.

[gcmartin]

... Note: newer kernels have zswap which compresses the swap and can actually speed it up ...

"ZSWAP" is the technology that I fail to grasp. Maybe someone has a different view.

Using ANY compressor on data to/from the SWAP device would tax the CPU. In the days when drives were small 100MB or 600MB or etc. this was a consideration. But, even most USBs are multi-gigabyte in size where size of SWAP is not the primary issue. (USBs, different from HDDs/SDDs bring a number of I/O operations to the table versus size.)

The primary issue is to load/unload/reload RAM slots as quickly as possible via the I/O subsystem when the processor has identified candidates for SWAP. To add an additional burden on the processor which at this point is probably approaching saturation seems not a worthwhile condition for consideration. I would be cautious in wanting to add that burden to the processor.

But, there is one advantage that can be derived in use of such. Time to move a RAM frame to the drive's destination will be reduced. But, that time is only apparent when the I/O operation's buffer has been loaded for SWAP I/O operations to proceed with its roll-in/roll-out, which usually is a background operation; whereas any compressor would be a foreground operation. Thus, unless we are considering some massive datacenter with Mainframes or LargeMinis running Linux, I have a difficult position to see a ZSWAP benefit when used in a desktop PC for a single user. This does NOT mean its not a nice parm. I only purport consideration as to how this SWAP compressor would benefit a 'single-user' system.

Lastly, if one decides to use it as on a USB drive which, we all know, is limited in the number of un-failed I/O operations MTBFs when compared with a HDD, then even though many USBs are slower, reducing the number of I/Os via a compressor can/will extend the life of a USB. But, again, this will be at the expense of the CPU and the value of proceeding on USB for this is questionable in overall system benefit. We must be understanding that in many cases, SWAP is ONLY going to become a factor as the system's saturation exceeds a RAM threshold where Linux is trying to insure good system performance.

Potentially, their is a silver lining in this somewhere. Others are invited to offer their ideas on ZSWAP. We all can learn in discussion.

Here to help
Edited some typos

[greengeek]

I'm not too much concerned with speed or cpu overhead. The roadblock I am trying to bypass is the problem that occurs when trying to remaster a 200MB puppy when running RAM-only in a machine that has only 256MB ram.

The puppy I am building has the ability to 're-clone' itself and burn itself to a new cd or dvd automatically but I need sufficient ram to run the actual puppy as well as additional space to juggle the cloned iso at the same time.

Similarly this puppy allows building a new version of itself incorporating specific personalisations and can juggle the new personalised iso in ram if there is room...

The trick is how to warn the user that their resources are inadequate, but also coach them through the steps to overcome the problem by plugging in a 'swap formatted' usb. I think I've got the right building blocks here to make some progress now...