Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 29 Aug 2014, 10:43
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
"Swap finder" - request for script
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [28 Posts]   Goto page: 1, 2 Next
Author Message
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Thu 31 Jul 2014, 16:26    Post subject:  "Swap finder" - request for script  

Sorry if this is in the wrong section - I couldn't find a better place to put it.

I would like some info / pointers / coding assistance in finding a method to make it possible/easy for an inexperienced user in a live session of Puppy to go hunting for a swap partition.
EDIT : ..and then I want to use that info to automatically help the user set up a swap on an external usb drive or HDD if some other swap is not currently available

Why?

Imagine that you are using a live session (no savefile and no /home partition) and you want to do something (like remastering) that is going to consume more RAM than you have available. As far as I know, all you need to do to increase the effective working storage area is to add 'swapon' and give the system a location to use and everything will be ok.

I'm working on an automatic remastering method (in live session) for novices who won't know whether they need extra swap or not, and won't know how to make a swap partition or set swapon.

I don't care about processing speed so all I am wanting to do is allow the user to plug in a slow usb stick (even in a usb 1.1 port) and allocate that usb stick as swap.

But how to achieve this for a new user who has no idea which sdx device it will be and has no knowledge of the swapon syntax?

In the situation where I hope to use this script the user will not have any prior knowledge of what is required - they will just see a message on screen telling them RAM is insufficient and they need to plug in a blank usb stick.

I need the system to detect this most recently plugged stick (ignoring existing plugged drives) and maybe bring up a rox window so they can identify that it contains no critical data.
(This could be problematic because I currently have disabled desktop drive icons as I don't want the novice user affecting existing hard drive data)

Once the drive is visible to the user and confirmed as safe to use I guess then It would be necessary to reformat the usb stick as swap, and allocate swapon, before returning to the original remaster (or whatever) script.

Another example of when such a process might be useful could be other RAM intensive things like video processing in live session.

Any criticisms, advice or code examples much appreciated!

Last edited by greengeek on Thu 31 Jul 2014, 18:12; edited 1 time in total
Back to top
View user's profile Send private message 
rcrsn51


Joined: 05 Sep 2006
Posts: 9059
Location: Stratford, Ontario

PostPosted: Thu 31 Jul 2014, 16:41    Post subject:  

cat /proc/swaps
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Thu 31 Jul 2014, 18:07    Post subject:  

rcrsn51 wrote:
cat /proc/swaps

Thank you - I was not aware of that command. That will be useful.

I've just run that command and it tells me two swap partitions are available on this machine but that neither of them is in use in this non standard Live pup I've built. Is it normal for all live pups to ignore swap partitions? Or does it vary depending on what each dev set up?

I guess I previously assumed all Linuxes used whatever swap was available, but assumptions never help much.
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4235
Location: Earth

PostPosted: Thu 31 Jul 2014, 18:33    Post subject:  

greengeek wrote:
Is it normal for all live pups to ignore swap partitions?
No! Only a couple PUPs do so. Most every PUP (and many Linux distros) will detect a SWAP partition and incorporate it into the running system.
greengeek wrote:
Or does it vary depending on what each dev set up?
Couple of devs choose not to make SWAP available to the system at boot. These dev's expect users to be intelligent Linux users.

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Thu 31 Jul 2014, 19:06    Post subject:  

Thanks GC - any idea where the code is that makes Puppy find and use existing swaps? Any idea if it's in the initrd.gz or main sfs?
cheers
Back to top
View user's profile Send private message 
RSH


Joined: 05 Sep 2011
Posts: 2412
Location: Germany

PostPosted: Thu 31 Jul 2014, 19:09    Post subject:  

This here is really interesting, but it confuses me even more as I'm already confused about the swap partitions.

My PC has 2GB of RAM and 1 swap partition around 4GB.

When hovering the mouse pointer over the freememapplet_tray in the sys-tray, a popup tells me that I do have 3GB of available memory in my save file (running in RAM, PupMode 5, no save file in use) and the terminal gives this output:
cat /proc/swaps wrote:
# cat /proc/swaps
Filename Type______Size______Used__Priority
/dev/sdc4 partition_4194300_0______-1
#

Underscores included to format text for the forum (it's eating multiple spaces)

So is my swap in use?
Why there is only 1GB added to the 2GB of RAM and shown as the combined 3GB available space in save file?

Edit:

greengeek wrote:
Thanks GC - any idea where the code is that makes Puppy find and use existing swaps? Any idea if it's in the initrd.gz or main sfs?
cheers

The init script in intrd.gz (and rc.sysinit) is searching for the swap.
Code:
#there are technical problems with loading a swap partition/file before the union
#is created, so not doing it until rc.sysinit runs. however, if a tmpfs needs to be
#created here, set it's size in anticipation of a swap being loaded...
EXTRAALLOCK=0
SWAPPART="`echo "$PCPARTSALL" | grep '|swap|' | head -n 1`"
[ "$SWAPPART" ] && SWAPPARTSIZE=`echo -n "$SWAPPART" | cut -f 3 -d '|'`
[ $SWAPPARTSIZE ] && EXTRAALLOCK=`expr $SWAPPARTSIZE \/ 2`
if [ $EXTRAALLOCK -eq 0 ];then
 [ -f ${SMNTPT}/pupswap.swp ] && SWAPFILESIZEBYTES=`stat -c %s ${SMNTPT}/pupswap.swp`
 [ $SWAPFILESIZEBYTES ] && EXTRAALLOCK=`expr $SWAPFILESIZEBYTES \/ 2048` #use half. 110521 also convert bytes to kb.
fi

_________________
LazY Puppy
RSH's DNA
SARA B.
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Thu 31 Jul 2014, 19:53    Post subject:  

RSH wrote:
...My PC has 2GB of RAM and 1 swap partition around 4GB. When hovering the mouse pointer over the freememapplet_tray in the sys-tray, a popup tells me that I do have 3GB of available memory in my save file.... So is my swap in use?
Why there is only 1GB added to the 2GB of RAM and shown as the combined 3GB available space in save file?

I would say that it is suggesting the swap is not in use. So the question is where did the extra 1GB of free RAM come from? Must be a motherboard special 'freebie' for good customers Smile

Actually, I see a similar pattern on my machine - I only have 1GB ram, apparently no swap loaded, but showing 1.2GB of 'personal storage free space'.

EDIT : No, wait - I'm horribly wrong. When I saw "used=0" I thought it meant "0" as a boolean - and I was expecting to see "1". I just went back and did the cat /proc/swaps again and this time I saw "used=656" so that suggests it is a byte count (yep, I'm dumb), which of course means my swap MUST be in use. I wonder if you could pull out half your ram and do some video editing and see what "used" value it shows on your system then?

Quote:
The init script in intrd.gz (and rc.sysinit) is searching for the swap.
Code:
#there are technical problems with loading a swap partition/file before the union
#is created, so not doing it until rc.sysinit runs. however, if a tmpfs needs to be
#created here, set it's size in anticipation of a swap being loaded...
......

Thanks, I will check out rc.sysinit for the swap code.
Back to top
View user's profile Send private message 
rcrsn51


Joined: 05 Sep 2006
Posts: 9059
Location: Stratford, Ontario

PostPosted: Thu 31 Jul 2014, 21:15    Post subject:  

The "free" command also has swap information.
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Thu 31 Jul 2014, 22:08    Post subject:  

Quote:
The "free" command also has swap information.

Thanks. I have attached a comparison of both sets of info.

With regard to the info offered by cat /proc/swaps - it definitely seems to be relevant to swap partitions that are currently in use (rather than "available to be used if you want to set them up"). If I type:
Code:
#swapoff /dev/sdb3
#swapoff /dev/sda3

(to turn off both of my swap partitions), then cat /proc/swaps returns blank information. (I might have expected that it would still reveal them but mark them as disabled but thats not the case)

Question: If the swap partitions were not available/activated during boot - is there any way to make the system discover them automatically if I plug in the usb based swap partition after boot?
(EDIT : - if I plug in a new swap usb stick after boot it seems as if the system finds the swap partition itself and all I need to do is force a swapon manually. Except for a couple of times it didnt detect the swap at all. Not sure why. More testing to do)

I am assuming it will be necessary to see what code is in rc.sysinit and try to emulate/expand it for use after manual insertion of the swap usb.
Swaps.jpg
 Description   
 Filesize   22.13 KB
 Viewed   225 Time(s)

Swaps.jpg

Back to top
View user's profile Send private message 
saintless


Joined: 11 Jun 2011
Posts: 2373
Location: Bulgaria

PostPosted: Fri 01 Aug 2014, 00:56    Post subject: Re: "Swap finder" - request for script  

greengeek wrote:
..and then I want to use that info to automatically help the user set up a swap on an external usb drive or HDD if some other swap is not currently available

Maybe this will help you (do not download the archive from there - it will not work in Puppy):
http://murga-linux.com/puppy/viewtopic.php?p=788004#788004

Attached the same make-swap GUI for Puppy (Yad lines replaced with xmessage and echo command - could be done better with gtkdialog progress bar). What make-swap offers is a choice to create swap file. After creating the program mounts (swapon) this swap file. You can create and use multiple swap files. The free command will increase the swap size information with every new swap file added.

Quote:
Question: If the swap partitions were not available/activated during boot - is there any way to make the system discover them automatically if I plug in the usb based swap partition after boot?

This is a script from Puppy we use to mount swap partition manually (if it is available but not mounted on boot). link in /root/Startup will automate the searching after X starts, but I don't know how to make it auto-start when usb stick is inserted:
Code:
#!/bin/sh

if grep -qv noswap /proc/cmdline; then
  echo "LOAD SWAP"
  for ONESWAP in `fdisk -l | grep ' Linux swap' | cut -f 1 -d ' ' | tr '\n' ' '`
  do
    echo -n "Loading swap partition $ONESWAP..." >/dev/console
    swapon $ONESWAP

    [ $? -eq 0 ] && SWAPON="yes"
  done
fi

It should work in Puppy without changes. I don't think grep -qv noswap /proc/cmdline will make troubles in Puppy but needs testing.
make-swap.zip
Description 
zip

 Download 
Filename  make-swap.zip 
Filesize  978 Bytes 
Downloaded  13 Time(s) 

_________________
Farewell, Nooby, you will be missed...
Back to top
View user's profile Send private message MSN Messenger 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Fri 01 Aug 2014, 03:59    Post subject:  

Hi Toni, thanks for the info and script. What is the purpose of looking at /proc/cmdline - is that checking to see if loading swap has been forbidden by a kernel parameter issued at boot time?
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4235
Location: Earth

PostPosted: Fri 01 Aug 2014, 04:18    Post subject: Careful in defining 2 SWAP locations in a running system  

When there is adequate RAM for system needs, SWAP will not get used. But, as system needs increase, SWAP will begin to be used to the system's advantage as it protect for good overall performance.

One area you might want to pay attention to when using SWAP partitions. Its a little dangerous to have 2 different SWAP partitions or files active at the same time across several devices. Reason: all devices do NOT exhibit the exact same characteristics. So, for example, one USB works at one I/O rate and a 2nd will operate at a different I/O rate. When SWAP use is spread across the 2, system performance will suffer because of the slower device.

There is a good reason for having multiple SWAPs; namely redundancy. But the performance penalty is nothing to ignore.

For redundancy, it may be better to insure that the devices are twins with serial numbers in the same batch if redundancy is needed. There's a greater chance of equal performance and redundancy would be achieved.

Sometime, although I have not used this in a personal home system, SWAP can be added to a running system when one might notice SWAP and RAM becoming totally consumed. This would achieve some relief until the system in brought back into a "normal"operating range. But, excepting for datacenters I have never seen this employed.

If anyone gets that pesky message of system space getting consumed, it might be worth a try to add additional SWAP via the command line to see if that message goes away versus closing down applications. Hmmm???

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
saintless


Joined: 11 Jun 2011
Posts: 2373
Location: Bulgaria

PostPosted: Fri 01 Aug 2014, 04:47    Post subject:  

greengeek wrote:
Hi Toni, thanks for the info and script. What is the purpose of looking at /proc/cmdline - is that checking to see if loading swap has been forbidden by a kernel parameter issued at boot time?

Yes, this is check for noswap parameter we use in DebianDog. The script is taken from loadswap_func function inside Puppy /etc/rc.d/rc.sysinit. You can check this loadswap_func and use part of it instead this script.
I think you will not find noswap parameter in puppy boot code and this script could be changed to simple version to work with Puppy - something like:
Code:
#!/bin/sh

  for ONESWAP in `fdisk -l | grep ' Linux swap' | cut -f 1 -d ' ' | tr '\n' ' '`
  do
    echo -n "Loading swap partition $ONESWAP..." >/dev/console
    swapon $ONESWAP
  done

If there is swap partition not in use - it will be mounted.
If there is swap partition in use - it will give message it is already in use.

Toni

_________________
Farewell, Nooby, you will be missed...
Back to top
View user's profile Send private message MSN Messenger 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Fri 01 Aug 2014, 16:37    Post subject: Re: Careful in defining 2 SWAP locations in a running system  

gcmartin wrote:
If anyone gets that pesky message of system space getting consumed, it might be worth a try to add additional SWAP via the command line to see if that message goes away versus closing down applications. Hmmm???

Yes, that is the sort of situation I am thinking of - times when puppy is running solely in ram and we know that it must be running short of available memory. It would be nice to plug in a dedicated external storage device that would alleviate the shortage of ram 'on-the-fly'.

I'm not clear on exactly when puppy DOES tell you it is running low on ram, and when it doesn't tell you. I've had some remastering failures which I assumed were probably unflagged ram shortages - but I guess Puppy sometimes is too busy to let me know the problem at the time...

saintless wrote:
and this script could be changed to simple version to work with Puppy - something like:

If there is swap partition not in use - it will be mounted.
If there is swap partition in use - it will give message it is already in use.

Thanks Toni, this will be very handy. I plan to combine this with a dialog telling the user how to prepare an appropriate usb stick as 'swap' device before they proceed with the remastering (or video editing) process.
.
Back to top
View user's profile Send private message 
greengeek

Joined: 20 Jul 2010
Posts: 2500
Location: New Zealand

PostPosted: Sat 02 Aug 2014, 16:47    Post subject:  

On a vaguely related topic I'd like to ask the following question:

If a user had a laptop containing something like a 1GHz cpu (ie enough to run a modern pup) but only 256MB ram (a bit too small to do much with) could they simply plug in a usb stick formatted as swap and have it act as extended ram? Would such an arrangement allow a live session (running in ram only) to behave as if the machine had 1GHz cpu + 1GB of ram? (other than the considerations of usb bus speed versus normal ram speed).

What if you plugged in a USB2 pcmcia card with the 'usb swap' plugged into that - would the system be able to use that as a means of increasing overall ram?
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [28 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1171s ][ Queries: 12 (0.0090s) ][ GZIP on ]