Zswap doesnt use heavy compression methods like xz, bz2 or even gz. Instead it uses lzop (and now lz4) which sacrifices some compression for speed. In fact, it can compress/decompress faster than most disk io.
I have been using lz4 for my kernel compression and it boots a bit faster than uncompressed.
"Swap finder" - request for script
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
Microsoft has been using I/O subsystem compression for about 15 years in NTFS. When it was concieved, it is an LZ type compression technique used. In datacenter setups, I have consistently deployed "compressed NTFS" for data drives. The low overhead is ideal and in these environments, there is ample CPU cycles to address the use needs.
I still do today for home systems even in Linux home systems. But, I have never placed Paging I/O (or a SWAP file) on compressed NTFS drives/partitions because of the potential problems whenever system becomes constrained.
In Linux, as of yet, there is no setup that addresses compression in a partition's filesystem. To my knowledge, Microsoft is the only OS vendor that has done so. In Linux, though, Linux's NTFS support knows how to access/read/write to an compressed NTFS partition and has been doing so for years.
ZSWAP is describe with clarity here.
ZRAM is describe with clarity here.
@Technosaurus may have a good point. It would be nice to see some performance measurements somewhere which shows that in constrains, this technology allows greater workload thruput with this compression's added burden on the CPU.
In normal operations at low CPU utilizations, I agree that more data can be packed into buffers than when uncompressed yielding great benefits. This is quite evident in some datacenter drives and controllers where the compression in handled outside of the CPU and the CPU never sees it. In this environment, the I/O "literally flies" in comparison to drives/controllers without the hardware assists (most times on the power of 10s in terms of data thruput). But in a non-compressed HDD without these hardware advancements as is the case for all home PCs, the CPU has to do all the work because datacenter technology does not exist. In fact, I have never been able to find a vendor who manufactured home drives which reportedly utilizes compression technology outside of the computer in the drive itself or in its cache on the I/O channel as seen in data centers. (And, I certainly cannot afford the cost of these specializations that business buy to handle their extremely high transaction rates.)
Also, @Technosaurus may be showing that there are/could be some changes in the overall behavior of the Linux system during contraints when SWAP is utilized where benefit are achieved thru some redesign and use of ZWAP to move buffers frames in/out of the system which exceeds past methods for SWAP. An intelligent design change, so to speak.
But, I cannot find any reports showing workload "thruput" comparisons where the higher CPU utilization resulting from ZWAP increases in compression during roll-in/roll-out operations yields higher workload thru the system.
Make no mistake, I am in favor of such a benefit. But, I would like to see some reports of this or see some new performance tools designed/redesigned to produce a constrained environment and report workload thresholds.
Here to help
I still do today for home systems even in Linux home systems. But, I have never placed Paging I/O (or a SWAP file) on compressed NTFS drives/partitions because of the potential problems whenever system becomes constrained.
In Linux, as of yet, there is no setup that addresses compression in a partition's filesystem. To my knowledge, Microsoft is the only OS vendor that has done so. In Linux, though, Linux's NTFS support knows how to access/read/write to an compressed NTFS partition and has been doing so for years.
ZSWAP is describe with clarity here.
ZRAM is describe with clarity here.
@Technosaurus may have a good point. It would be nice to see some performance measurements somewhere which shows that in constrains, this technology allows greater workload thruput with this compression's added burden on the CPU.
In normal operations at low CPU utilizations, I agree that more data can be packed into buffers than when uncompressed yielding great benefits. This is quite evident in some datacenter drives and controllers where the compression in handled outside of the CPU and the CPU never sees it. In this environment, the I/O "literally flies" in comparison to drives/controllers without the hardware assists (most times on the power of 10s in terms of data thruput). But in a non-compressed HDD without these hardware advancements as is the case for all home PCs, the CPU has to do all the work because datacenter technology does not exist. In fact, I have never been able to find a vendor who manufactured home drives which reportedly utilizes compression technology outside of the computer in the drive itself or in its cache on the I/O channel as seen in data centers. (And, I certainly cannot afford the cost of these specializations that business buy to handle their extremely high transaction rates.)
Also, @Technosaurus may be showing that there are/could be some changes in the overall behavior of the Linux system during contraints when SWAP is utilized where benefit are achieved thru some redesign and use of ZWAP to move buffers frames in/out of the system which exceeds past methods for SWAP. An intelligent design change, so to speak.
But, I cannot find any reports showing workload "thruput" comparisons where the higher CPU utilization resulting from ZWAP increases in compression during roll-in/roll-out operations yields higher workload thru the system.
Make no mistake, I am in favor of such a benefit. But, I would like to see some reports of this or see some new performance tools designed/redesigned to produce a constrained environment and report workload thresholds.
Here to help
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
There are only a few situations where disk i/o and heavy computation go hand in hand such as transcoding a video from 1 format to another. This is why (if you look in the kernel fs code) some filesystems with compression actually look at file extensions and do not compress certain types (for 1 its also not very effective to recompress something that already is) Either way lz4 is so computationally light that it will benefit 99% of the time.
If you want an example of high CPU utilization and disk io, you need look no further than boot speed comparisons where lzop/lz4 typically beats uncompressed kernels. This only shows decompression though.
.
One side note is that if you are using zram _and_ zswap where the 2 compression types are the same, a copy_in 'should' be a direct copy, but I don't think the kernel devs have noticed this yet. This could be of great benefit where files are going into swap. Presumably if they are coming out of swap they will need to be decompressed anyhow, so only copy_in is needed.
If you want an example of high CPU utilization and disk io, you need look no further than boot speed comparisons where lzop/lz4 typically beats uncompressed kernels. This only shows decompression though.
.
One side note is that if you are using zram _and_ zswap where the 2 compression types are the same, a copy_in 'should' be a direct copy, but I don't think the kernel devs have noticed this yet. This could be of great benefit where files are going into swap. Presumably if they are coming out of swap they will need to be decompressed anyhow, so only copy_in is needed.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
I've been using plan9 tools too much (where everything is a file) ... yes pages ... basically when a "dirty" page is accessed the kernel pops it from swap. The juggling act is quite impressive, predicting when a page should come from swap based on the activity of its associated groups/processes and vice versa going to swap based on inactivity. Adding a zram layer with the same compression type as the zswap in between could speed things up significantly... this way predicted cache misses could go to zram and predicted unused pages could go to zswap.gcmartin wrote:To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.
Hope this helps
I become more and more convinced that a ZISC computer could be orders of magnitude faster than conventional CPUs (they could atomically swap entire 4096 byte pages in one instruction)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
Code: Select all
#!/bin/sh
trap exit INT TERM KILL
while [ running ]
do
sleep 2
_ALL_SWAPS_=`blkid | grep -i swap | cut -f1 -d':'`
oldIFS="$IFS"
IFS=$'\n'
for oneSWAP in $_ALL_SWAPS_
do
grep -qw "$oneSWAP" /proc/swaps || swapon "$oneSWAP"
done
IFS="$oldIFS"
done
«Give me GUI or Death» -- I give you [[Xx]term[inal]] [[Cc]on[s][ole]] .
Macpup user since 2010 on full installations.
People who want problems with Puppy boot frugal :P
Macpup user since 2010 on full installations.
People who want problems with Puppy boot frugal :P
Just adding a note to compare notes with another thread that covers some similar issues:
http://www.murga-linux.com/puppy/viewtopic.php?t=95235
http://www.murga-linux.com/puppy/viewtopic.php?t=95235