Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 23 Sep 2014, 20:45
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
"Swap finder" - request for script
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 2 [28 Posts]   Goto page: Previous 1, 2
Author Message
RSH


Joined: 05 Sep 2011
Posts: 2420
Location: Germany

PostPosted: Sat 02 Aug 2014, 17:10    Post subject:  

From my experience a SFS Module is not loaded into RAM, if there isn't enough free RAM available.

Usually when it comes to the point, to load the main sfs at boot up, it takes a few seconds until it is loaded and boot process is continuing.

In my beginning days I've worked on a remaster of Lucid Puppy 525 (my first go) that in the end has had reached a compressed size of appr. 1GB. At this time my PC has had only 1.5GB RAM available, so the main sfs of this remastered Lucid was just somehow touched at boot up (took less than a second) and the boot process did continue and was successfully finished each time.

I assume loading SFS Modules by sfs_load -or similar technic- will act equally to this.

_________________
LazY Puppy
RSH's DNA
SARA B.
Back to top
View user's profile Send private message 
saintless


Joined: 11 Jun 2011
Posts: 2465
Location: Bulgaria

PostPosted: Sun 03 Aug 2014, 02:26    Post subject:  

greengeek wrote:
On a vaguely related topic I'd like to ask the following question:

If a user had a laptop containing something like a 1GHz cpu (ie enough to run a modern pup) but only 256MB ram (a bit too small to do much with) could they simply plug in a usb stick formatted as swap and have it act as extended ram? Would such an arrangement allow a live session (running in ram only) to behave as if the machine had 1GHz cpu + 1GB of ram? (other than the considerations of usb bus speed versus normal ram speed).

What if you plugged in a USB2 pcmcia card with the 'usb swap' plugged into that - would the system be able to use that as a means of increasing overall ram?

No, swap partition/file is used when the amount of RAM is full. Swap does not replace physical memory.
I really hope someone can answer yes to this question and give working method to copy to RAM main sfs module 512Mb on boot on computer with 256Mb RAM + 1Gb Swap partition but I doubt it is possible.
And maybe it will net be faster or better than booting from flash drive without copy to RAM.

_________________
Farewell, Nooby, you will be missed...
Back to top
View user's profile Send private message MSN Messenger 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Mon 04 Aug 2014, 03:46    Post subject:  

This is how to go about it.
Swap partitions:
guess_fstype or blkid on all filesystems in /proc/partitions
Swap files:
use find on / to look for puppy swap files (what is the extension? I forget)
Fallback:
create an empty 1Gb swap file and bzip it to include as a fallback if none is found (it compresses to only a few kb with bzip -9 ... its even better than xz for empty files)

For 256Mb ram - use Akita... usb will likely be slower than using the disk and a swap partition/file, but it will at least prevent OOM errors

Ideally your swap partition should be on a separate drive (preferably the fastest one and preferably a separate disk controller... for IDE drives, that means on a separate cable)

Note: newer kernels have zswap which compresses the swap and can actually speed it up ... I think tmpfs (used for /dev/shm and sometimes /tmp) has a similar feature now too.

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4277
Location: Earth

PostPosted: Mon 04 Aug 2014, 16:17    Post subject:  

technosaurus wrote:
... Note: newer kernels have zswap which compresses the swap and can actually speed it up ...
"ZSWAP" is the technology that I fail to grasp. Maybe someone has a different view.

Using ANY compressor on data to/from the SWAP device would tax the CPU. In the days when drives were small 100MB or 600MB or etc. this was a consideration. But, even most USBs are multi-gigabyte in size where size of SWAP is not the primary issue. (USBs, different from HDDs/SDDs bring a number of I/O operations to the table versus size.)

The primary issue is to load/unload/reload RAM slots as quickly as possible via the I/O subsystem when the processor has identified candidates for SWAP. To add an additional burden on the processor which at this point is probably approaching saturation seems not a worthwhile condition for consideration. I would be cautious in wanting to add that burden to the processor.

But, there is one advantage that can be derived in use of such. Time to move a RAM frame to the drive's destination will be reduced. But, that time is only apparent when the I/O operation's buffer has been loaded for SWAP I/O operations to proceed with its roll-in/roll-out, which usually is a background operation; whereas any compressor would be a foreground operation. Thus, unless we are considering some massive datacenter with Mainframes or LargeMinis running Linux, I have a difficult position to see a ZSWAP benefit when used in a desktop PC for a single user. This does NOT mean its not a nice parm. I only purport consideration as to how this SWAP compressor would benefit a 'single-user' system.

Lastly, if one decides to use it as on a USB drive which, we all know, is limited in the number of un-failed I/O operations MTBFs when compared with a HDD, then even though many USBs are slower, reducing the number of I/Os via a compressor can/will extend the life of a USB. But, again, this will be at the expense of the CPU and the value of proceeding on USB for this is questionable in overall system benefit. We must be understanding that in many cases, SWAP is ONLY going to become a factor as the system's saturation exceeds a RAM threshold where Linux is trying to insure good system performance.

Potentially, their is a silver lining in this somewhere. Others are invited to offer their ideas on ZSWAP. We all can learn in discussion.

Here to help
Edited some typos

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile

Last edited by gcmartin on Mon 04 Aug 2014, 18:55; edited 1 time in total
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2551
Location: New Zealand

PostPosted: Mon 04 Aug 2014, 18:31    Post subject:  

I'm not too much concerned with speed or cpu overhead. The roadblock I am trying to bypass is the problem that occurs when trying to remaster a 200MB puppy when running RAM-only in a machine that has only 256MB ram.

The puppy I am building has the ability to 're-clone' itself and burn itself to a new cd or dvd automatically but I need sufficient ram to run the actual puppy as well as additional space to juggle the cloned iso at the same time.

Similarly this puppy allows building a new version of itself incorporating specific personalisations and can juggle the new personalised iso in ram if there is room...

The trick is how to warn the user that their resources are inadequate, but also coach them through the steps to overcome the problem by plugging in a 'swap formatted' usb. I think I've got the right building blocks here to make some progress now...
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Mon 04 Aug 2014, 21:02    Post subject:  

Zswap doesnt use heavy compression methods like xz, bz2 or even gz. Instead it uses lzop (and now lz4) which sacrifices some compression for speed. In fact, it can compress/decompress faster than most disk io.

I have been using lz4 for my kernel compression and it boots a bit faster than uncompressed.

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2551
Location: New Zealand

PostPosted: Mon 04 Aug 2014, 22:35    Post subject:  

So do you have to activate the compression feature by recompiling the kernel or does it just 'happen' as a default?
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4277
Location: Earth

PostPosted: Tue 05 Aug 2014, 01:03    Post subject:  

Microsoft has been using I/O subsystem compression for about 15 years in NTFS. When it was concieved, it is an LZ type compression technique used. In datacenter setups, I have consistently deployed "compressed NTFS" for data drives. The low overhead is ideal and in these environments, there is ample CPU cycles to address the use needs.

I still do today for home systems even in Linux home systems. But, I have never placed Paging I/O (or a SWAP file) on compressed NTFS drives/partitions because of the potential problems whenever system becomes constrained.

In Linux, as of yet, there is no setup that addresses compression in a partition's filesystem. To my knowledge, Microsoft is the only OS vendor that has done so. In Linux, though, Linux's NTFS support knows how to access/read/write to an compressed NTFS partition and has been doing so for years.

ZSWAP is describe with clarity here.
ZRAM is describe with clarity here.

@Technosaurus may have a good point. It would be nice to see some performance measurements somewhere which shows that in constrains, this technology allows greater workload thruput with this compression's added burden on the CPU.

In normal operations at low CPU utilizations, I agree that more data can be packed into buffers than when uncompressed yielding great benefits. This is quite evident in some datacenter drives and controllers where the compression in handled outside of the CPU and the CPU never sees it. In this environment, the I/O "literally flies" in comparison to drives/controllers without the hardware assists (most times on the power of 10s in terms of data thruput). But in a non-compressed HDD without these hardware advancements as is the case for all home PCs, the CPU has to do all the work because datacenter technology does not exist. In fact, I have never been able to find a vendor who manufactured home drives which reportedly utilizes compression technology outside of the computer in the drive itself or in its cache on the I/O channel as seen in data centers. (And, I certainly cannot afford the cost of these specializations that business buy to handle their extremely high transaction rates.)

Also, @Technosaurus may be showing that there are/could be some changes in the overall behavior of the Linux system during contraints when SWAP is utilized where benefit are achieved thru some redesign and use of ZWAP to move buffers frames in/out of the system which exceeds past methods for SWAP. An intelligent design change, so to speak.

But, I cannot find any reports showing workload "thruput" comparisons where the higher CPU utilization resulting from ZWAP increases in compression during roll-in/roll-out operations yields higher workload thru the system.

Make no mistake, I am in favor of such a benefit. But, I would like to see some reports of this or see some new performance tools designed/redesigned to produce a constrained environment and report workload thresholds.

Here to help

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Tue 05 Aug 2014, 14:07    Post subject:  

There are only a few situations where disk i/o and heavy computation go hand in hand such as transcoding a video from 1 format to another. This is why (if you look in the kernel fs code) some filesystems with compression actually look at file extensions and do not compress certain types (for 1 its also not very effective to recompress something that already is) Either way lz4 is so computationally light that it will benefit 99% of the time.

If you want an example of high CPU utilization and disk io, you need look no further than boot speed comparisons where lzop/lz4 typically beats uncompressed kernels. This only shows decompression though.
.
One side note is that if you are using zram _and_ zswap where the 2 compression types are the same, a copy_in 'should' be a direct copy, but I don't think the kernel devs have noticed this yet. This could be of great benefit where files are going into swap. Presumably if they are coming out of swap they will need to be decompressed anyhow, so only copy_in is needed.

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4277
Location: Earth

PostPosted: Tue 05 Aug 2014, 14:26    Post subject:  

To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Tue 05 Aug 2014, 15:05    Post subject:  

gcmartin wrote:
To my knowledge, SWAP is frame related, not file related in any way. Paging subsystems have no interest in files. They exist to insure running programs have "space" for execution. They perform the juggling act behind the scenes of program execution.

Hope this helps
I've been using plan9 tools too much (where everything is a file) ... yes pages ... basically when a "dirty" page is accessed the kernel pops it from swap. The juggling act is quite impressive, predicting when a page should come from swap based on the activity of its associated groups/processes and vice versa going to swap based on inactivity. Adding a zram layer with the same compression type as the zswap in between could speed things up significantly... this way predicted cache misses could go to zram and predicted unused pages could go to zswap.

I become more and more convinced that a ZISC computer could be orders of magnitude faster than conventional CPUs (they could atomically swap entire 4096 byte pages in one instruction)

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 3972
Location: Kiel,Germany

PostPosted: Tue 05 Aug 2014, 20:25    Post subject:  

Code:
#!/bin/sh

trap exit INT TERM KILL

while [ running ]
do
sleep 2

 _ALL_SWAPS_=`blkid | grep -i swap | cut -f1 -d':'`

 oldIFS="$IFS"
 IFS=$'\n'
 for oneSWAP in $_ALL_SWAPS_
 do
 grep -qw "$oneSWAP" /proc/swaps || swapon "$oneSWAP"
 done
 IFS="$oldIFS"

done

blkid /dev/sd* works for me to omit probing of floppy drive fd0

_________________
«Give me GUI or Death» -- I give you [[Xx]term[inal]] [[Cc]on[s][ole]] .
Macpup user since 2010 on full installations.
People who want problems with Puppy boot frugal Razz
Back to top
View user's profile Send private message Visit poster's website 
greengeek

Joined: 20 Jul 2010
Posts: 2551
Location: New Zealand

PostPosted: Wed 13 Aug 2014, 19:43    Post subject:  

Just adding a note to compare notes with another thread that covers some similar issues:
http://www.murga-linux.com/puppy/viewtopic.php?t=95235
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 2 [28 Posts]   Goto page: Previous 1, 2
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1069s ][ Queries: 13 (0.0130s) ][ GZIP on ]