"Swap finder" - request for script

Using applications, configuring, problems
Message
Author
User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#21 Post by technosaurus »

Zswap doesnt use heavy compression methods like xz, bz2 or even gz. Instead it uses lzop (and now lz4) which sacrifices some compression for speed. In fact, it can compress/decompress faster than most disk io.

I have been using lz4 for my kernel compression and it boots a bit faster than uncompressed.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#22 Post by greengeek »

So do you have to activate the compression feature by recompiling the kernel or does it just 'happen' as a default?

gcmartin

#23 Post by gcmartin »

Microsoft has been using I/O subsystem compression for about 15 years in NTFS. When it was concieved, it is an LZ type compression technique used. In datacenter setups, I have consistently deployed "compressed NTFS" for data drives. The low overhead is ideal and in these environments, there is ample CPU cycles to address the use needs.

I still do today for home systems even in Linux home systems. But, I have never placed Paging I/O (or a SWAP file) on compressed NTFS drives/partitions because of the potential problems whenever system becomes constrained.

In Linux, as of yet, there is no setup that addresses compression in a partition's filesystem. To my knowledge, Microsoft is the only OS vendor that has done so. In Linux, though, Linux's NTFS support knows how to access/read/write to an compressed NTFS partition and has been doing so for years.

ZSWAP is describe with clarity here.
ZRAM is describe with clarity here.

@Technosaurus may have a good point. It would be nice to see some performance measurements somewhere which shows that in constrains, this technology allows greater workload thruput with this compression's added burden on the CPU.

In normal operations at low CPU utilizations, I agree that more data can be packed into buffers than when uncompressed yielding great benefits. This is quite evident in some datacenter drives and controllers where the compression in handled outside of the CPU and the CPU never sees it. In this environment, the I/O "literally flies" in comparison to drives/controllers without the hardware assists (most times on the power of 10s in terms of data thruput). But in a non-compressed HDD without these hardware advancements as is the case for all home PCs, the CPU has to do all the work because datacenter technology does not exist. In fact, I have never been able to find a vendor who manufactured home drives which reportedly utilizes compression technology outside of the computer in the drive itself or in its cache on the I/O channel as seen in data centers. (And, I certainly cannot afford the cost of these specializations that business buy to handle their extremely high transaction rates.)

Also, @Technosaurus may be showing that there are/could be some changes in the overall behavior of the Linux system during contraints when SWAP is utilized where benefit are achieved thru some redesign and use of ZWAP to move buffers frames in/out of the system which exceeds past methods for SWAP. An intelligent design change, so to speak.

But, I cannot find any reports showing workload "thruput" comparisons where the higher CPU utilization resulting from ZWAP increases in compression during roll-in/roll-out operations yields higher workload thru the system.

Make no mistake, I am in favor of such a benefit. But, I would like to see some reports of this or see some new performance tools designed/redesigned to produce a constrained environment and report workload thresholds.

Here to help

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#24 Post by technosaurus »

There are only a few situations where disk i/o and heavy computation go hand in hand such as transcoding a video from 1 format to another. This is why (if you look in the kernel fs code) some filesystems with compression actually look at file extensions and do not compress certain types (for 1 its also not very effective to recompress something that already is) Either way lz4 is so computationally light that it will benefit 99% of the time.

If you want an example of high CPU utilization and disk io, you need look no further than boot speed comparisons where lzop/lz4 typically beats uncompressed kernels. This only shows decompression though.
.
One side note is that if you are using zram _and_ zswap where the 2 compression types are the same, a copy_in 'should' be a direct copy, but I don't think the kernel devs have noticed this yet. This could be of great benefit where files are going into swap. Presumably if they are coming out of swap they will need to be decompressed anyhow, so only copy_in is needed.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

gcmartin

#25 Post by gcmartin »

To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#26 Post by technosaurus »

gcmartin wrote:To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps
I've been using plan9 tools too much (where everything is a file) ... yes pages ... basically when a "dirty" page is accessed the kernel pops it from swap. The juggling act is quite impressive, predicting when a page should come from swap based on the activity of its associated groups/processes and vice versa going to swap based on inactivity. Adding a zram layer with the same compression type as the zswap in between could speed things up significantly... this way predicted cache misses could go to zram and predicted unused pages could go to zswap.

I become more and more convinced that a ZISC computer could be orders of magnitude faster than conventional CPUs (they could atomically swap entire 4096 byte pages in one instruction)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#27 Post by Karl Godt »

Code: Select all

#!/bin/sh

trap exit INT TERM KILL

while [ running ]
do
sleep 2

 _ALL_SWAPS_=`blkid | grep -i swap | cut -f1 -d':'`

 oldIFS="$IFS"
 IFS=$'\n'
 for oneSWAP in $_ALL_SWAPS_
 do
 grep -qw "$oneSWAP" /proc/swaps || swapon "$oneSWAP"
 done
 IFS="$oldIFS"

done
blkid /dev/sd* works for me to omit probing of floppy drive fd0
«Give me GUI or Death» -- I give you [[Xx]term[inal]] [[Cc]on[s][ole]] .
Macpup user since 2010 on full installations.
People who want problems with Puppy boot frugal :P

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#28 Post by greengeek »

Just adding a note to compare notes with another thread that covers some similar issues:
http://www.murga-linux.com/puppy/viewtopic.php?t=95235

Post Reply