"Swap finder" - request for script

[RSH]

From my experience a SFS Module is not loaded into RAM, if there isn't enough free RAM available.

Usually when it comes to the point, to load the main sfs at boot up, it takes a few seconds until it is loaded and boot process is continuing.

In my beginning days I've worked on a remaster of Lucid Puppy 525 (my first go) that in the end has had reached a compressed size of appr. 1GB. At this time my PC has had only 1.5GB RAM available, so the main sfs of this remastered Lucid was just somehow touched at boot up (took less than a second) and the boot process did continue and was successfully finished each time.

I assume loading SFS Modules by sfs_load -or similar technic- will act equally to this.

[saintless]

On a vaguely related topic I'd like to ask the following question:

If a user had a laptop containing something like a 1GHz cpu (ie enough to run a modern pup) but only 256MB ram (a bit too small to do much with) could they simply plug in a usb stick formatted as swap and have it act as extended ram? Would such an arrangement allow a live session (running in ram only) to behave as if the machine had 1GHz cpu + 1GB of ram? (other than the considerations of usb bus speed versus normal ram speed).

What if you plugged in a USB2 pcmcia card with the 'usb swap' plugged into that - would the system be able to use that as a means of increasing overall ram?

No, swap partition/file is used when the amount of RAM is full. Swap does not replace physical memory.
I really hope someone can answer yes to this question and give working method to copy to RAM main sfs module 512Mb on boot on computer with 256Mb RAM + 1Gb Swap partition but I doubt it is possible.
And maybe it will net be faster or better than booting from flash drive without copy to RAM.

[technosaurus]

This is how to go about it.
Swap partitions:
guess_fstype or blkid on all filesystems in /proc/partitions
Swap files:
use find on / to look for puppy swap files (what is the extension? I forget)
Fallback:
create an empty 1Gb swap file and bzip it to include as a fallback if none is found (it compresses to only a few kb with bzip -9 ... its even better than xz for empty files)

For 256Mb ram - use Akita... usb will likely be slower than using the disk and a swap partition/file, but it will at least prevent OOM errors

Ideally your swap partition should be on a separate drive (preferably the fastest one and preferably a separate disk controller... for IDE drives, that means on a separate cable)

Note: newer kernels have zswap which compresses the swap and can actually speed it up ... I think tmpfs (used for /dev/shm and sometimes /tmp) has a similar feature now too.

[gcmartin]

... Note: newer kernels have zswap which compresses the swap and can actually speed it up ...

"ZSWAP" is the technology that I fail to grasp. Maybe someone has a different view.

Using ANY compressor on data to/from the SWAP device would tax the CPU. In the days when drives were small 100MB or 600MB or etc. this was a consideration. But, even most USBs are multi-gigabyte in size where size of SWAP is not the primary issue. (USBs, different from HDDs/SDDs bring a number of I/O operations to the table versus size.)

The primary issue is to load/unload/reload RAM slots as quickly as possible via the I/O subsystem when the processor has identified candidates for SWAP. To add an additional burden on the processor which at this point is probably approaching saturation seems not a worthwhile condition for consideration. I would be cautious in wanting to add that burden to the processor.

But, there is one advantage that can be derived in use of such. Time to move a RAM frame to the drive's destination will be reduced. But, that time is only apparent when the I/O operation's buffer has been loaded for SWAP I/O operations to proceed with its roll-in/roll-out, which usually is a background operation; whereas any compressor would be a foreground operation. Thus, unless we are considering some massive datacenter with Mainframes or LargeMinis running Linux, I have a difficult position to see a ZSWAP benefit when used in a desktop PC for a single user. This does NOT mean its not a nice parm. I only purport consideration as to how this SWAP compressor would benefit a 'single-user' system.

Lastly, if one decides to use it as on a USB drive which, we all know, is limited in the number of un-failed I/O operations MTBFs when compared with a HDD, then even though many USBs are slower, reducing the number of I/Os via a compressor can/will extend the life of a USB. But, again, this will be at the expense of the CPU and the value of proceeding on USB for this is questionable in overall system benefit. We must be understanding that in many cases, SWAP is ONLY going to become a factor as the system's saturation exceeds a RAM threshold where Linux is trying to insure good system performance.

Potentially, their is a silver lining in this somewhere. Others are invited to offer their ideas on ZSWAP. We all can learn in discussion.

Here to help
Edited some typos

[greengeek]

I'm not too much concerned with speed or cpu overhead. The roadblock I am trying to bypass is the problem that occurs when trying to remaster a 200MB puppy when running RAM-only in a machine that has only 256MB ram.

The puppy I am building has the ability to 're-clone' itself and burn itself to a new cd or dvd automatically but I need sufficient ram to run the actual puppy as well as additional space to juggle the cloned iso at the same time.

Similarly this puppy allows building a new version of itself incorporating specific personalisations and can juggle the new personalised iso in ram if there is room...

The trick is how to warn the user that their resources are inadequate, but also coach them through the steps to overcome the problem by plugging in a 'swap formatted' usb. I think I've got the right building blocks here to make some progress now...

[technosaurus]

Zswap doesnt use heavy compression methods like xz, bz2 or even gz. Instead it uses lzop (and now lz4) which sacrifices some compression for speed. In fact, it can compress/decompress faster than most disk io.

I have been using lz4 for my kernel compression and it boots a bit faster than uncompressed.

[greengeek]

So do you have to activate the compression feature by recompiling the kernel or does it just 'happen' as a default?

[gcmartin]

Microsoft has been using I/O subsystem compression for about 15 years in NTFS. When it was concieved, it is an LZ type compression technique used. In datacenter setups, I have consistently deployed "compressed NTFS" for data drives. The low overhead is ideal and in these environments, there is ample CPU cycles to address the use needs.

I still do today for home systems even in Linux home systems. But, I have never placed Paging I/O (or a SWAP file) on compressed NTFS drives/partitions because of the potential problems whenever system becomes constrained.

In Linux, as of yet, there is no setup that addresses compression in a partition's filesystem. To my knowledge, Microsoft is the only OS vendor that has done so. In Linux, though, Linux's NTFS support knows how to access/read/write to an compressed NTFS partition and has been doing so for years.

ZSWAP is describe with clarity here.
ZRAM is describe with clarity here.

@Technosaurus may have a good point. It would be nice to see some performance measurements somewhere which shows that in constrains, this technology allows greater workload thruput with this compression's added burden on the CPU.

In normal operations at low CPU utilizations, I agree that more data can be packed into buffers than when uncompressed yielding great benefits. This is quite evident in some datacenter drives and controllers where the compression in handled outside of the CPU and the CPU never sees it. In this environment, the I/O "literally flies" in comparison to drives/controllers without the hardware assists (most times on the power of 10s in terms of data thruput). But in a non-compressed HDD without these hardware advancements as is the case for all home PCs, the CPU has to do all the work because datacenter technology does not exist. In fact, I have never been able to find a vendor who manufactured home drives which reportedly utilizes compression technology outside of the computer in the drive itself or in its cache on the I/O channel as seen in data centers. (And, I certainly cannot afford the cost of these specializations that business buy to handle their extremely high transaction rates.)

Also, @Technosaurus may be showing that there are/could be some changes in the overall behavior of the Linux system during contraints when SWAP is utilized where benefit are achieved thru some redesign and use of ZWAP to move buffers frames in/out of the system which exceeds past methods for SWAP. An intelligent design change, so to speak.

But, I cannot find any reports showing workload "thruput" comparisons where the higher CPU utilization resulting from ZWAP increases in compression during roll-in/roll-out operations yields higher workload thru the system.

Make no mistake, I am in favor of such a benefit. But, I would like to see some reports of this or see some new performance tools designed/redesigned to produce a constrained environment and report workload thresholds.

Here to help

[technosaurus]

There are only a few situations where disk i/o and heavy computation go hand in hand such as transcoding a video from 1 format to another. This is why (if you look in the kernel fs code) some filesystems with compression actually look at file extensions and do not compress certain types (for 1 its also not very effective to recompress something that already is) Either way lz4 is so computationally light that it will benefit 99% of the time.

If you want an example of high CPU utilization and disk io, you need look no further than boot speed comparisons where lzop/lz4 typically beats uncompressed kernels. This only shows decompression though.
.
One side note is that if you are using zram _and_ zswap where the 2 compression types are the same, a copy_in 'should' be a direct copy, but I don't think the kernel devs have noticed this yet. This could be of great benefit where files are going into swap. Presumably if they are coming out of swap they will need to be decompressed anyhow, so only copy_in is needed.

[gcmartin]

To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps

[technosaurus]

To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps

I've been using plan9 tools too much (where everything is a file) ... yes pages ... basically when a "dirty" page is accessed the kernel pops it from swap. The juggling act is quite impressive, predicting when a page should come from swap based on the activity of its associated groups/processes and vice versa going to swap based on inactivity. Adding a zram layer with the same compression type as the zswap in between could speed things up significantly... this way predicted cache misses could go to zram and predicted unused pages could go to zswap.

I become more and more convinced that a ZISC computer could be orders of magnitude faster than conventional CPUs (they could atomically swap entire 4096 byte pages in one instruction)

[Karl Godt]

Code: Select all

#!/bin/sh

trap exit INT TERM KILL

while [ running ]
do
sleep 2

 _ALL_SWAPS_=`blkid | grep -i swap | cut -f1 -d':'`

 oldIFS="$IFS"
 IFS=$'\n'
 for oneSWAP in $_ALL_SWAPS_
 do
 grep -qw "$oneSWAP" /proc/swaps || swapon "$oneSWAP"
 done
 IFS="$oldIFS"

done

blkid /dev/sd* works for me to omit probing of floppy drive fd0

[greengeek]

Just adding a note to compare notes with another thread that covers some similar issues:
http://www.murga-linux.com/puppy/viewtopic.php?t=95235