Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 16 Jul 2018, 20:29
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Ideal ratio between size of pupsave and size of RAM?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 3 [35 Posts]   Goto page: 1, 2, 3 Next
Author Message
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Wed 11 Apr 2018, 02:11    Post subject:  Ideal ratio between size of pupsave and size of RAM?
Subject description: (Could be a save folder as well.)
 

Hello all.

As the title says:
What is the ideal ratio between the size of a pupsave (or pupfolder)
and the size of the RAM on the user's computer?


TIA.

I know that Puppy can detect if there is not enough RAM and then leave
the puppy_xxx.sfs on disk.

I know also that the "nocopy" parm can be used to NOT run Puppy from
RAM.

~~~~~~~~~~~~~~~

This is the case / example that I have in mind:

Puppyist has a huge pupsave folder (let's say, 7.5 Gb in total) on a 8 Gb
USB key.

Puppyist wants to make room on USB key and decides to squash huge
pupfolder as an adrv_xxx.sfs archive.

Puppyist proceeds with mksquashfs, and the result is an ~ 3 Gb adrv.

Puppyist reboots with big adrv_xxx.sfs, and there is a "kernel panic".

... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv­ sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)

~~~~~~~~~~~~~~

What do we tell Puppyists to avoid such situations ?

TIA.

_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
RetroTechGuy


Joined: 15 Dec 2009
Posts: 2892
Location: USA

PostPosted: Wed 11 Apr 2018, 12:45    Post subject: Re: Ideal ratio between size of pupsave and size of RAM?
Subject description: (Could be a save folder as well.)
 

musher0 wrote:
Hello all.

As the title says:
What is the ideal ratio between the size of a pupsave (or pupfolder)
and the size of the RAM on the user's computer?


TIA.

I know that Puppy can detect if there is not enough RAM and then leave
the puppy_xxx.sfs on disk.

I know also that the "nocopy" parm can be used to NOT run Puppy from
RAM.

~~~~~~~~~~~~~~~

This is the case / example that I have in mind:

Puppyist has a huge pupsave folder (let's say, 7.5 Gb in total) on a 8 Gb
USB key.

Puppyist wants to make room on USB key and decides to squash huge
pupfolder as an adrv_xxx.sfs archive.

Puppyist proceeds with mksquashfs, and the result is an ~ 3 Gb adrv.

Puppyist reboots with big adrv_xxx.sfs, and there is a "kernel panic".

... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv­ sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)

~~~~~~~~~~~~~~

What do we tell Puppyists to avoid such situations ?

TIA.


Y'know, I don't think that I've run into that... But I've never found a reason to make the Pupsave larger than 1GB. I store all sorts of junk outside, somewhere on /mnt/home/ (in particular, crap like temporary cache folders, and the like). My email folder also sits outside -- and is often set up to be shared with multiple Puppy versions (I use Tbird, and as long as they are the same version, it doesn't cause a ruckus).

_________________
Add swapfile
WellMinded Search
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Wed 11 Apr 2018, 15:29    Post subject:  

Thanks, RetroTechGuy.
_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
mikeslr


Joined: 16 Jun 2008
Posts: 2468
Location: 500 seconds from Sol

PostPosted: Wed 11 Apr 2018, 19:34    Post subject:  

Hi musher0,

"... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv­ sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)".

I don't think so. But, having no idea how to experiment to find out, I only know what I've been told. Which amounts to this:

In the absence of the boot command "copy*", a Puppy does not copy an entire SFS into RAM. Rather, when a Frugal Puppy (using a merge-file system) boots it copies some files into RAM and creates in RAM an inodes list which points to where the balance of the files are. At least that's my "short-hand" take from the following posts:

http://murga-linux.com/puppy/viewtopic.php?p=282162#282162
http://murga-linux.com/puppy/viewtopic.php?p=827458#827458
http://www.linfo.org/inode.html

* The one experiment I ran using the copy boot command did result in more ram being used on bootup. But I did not make a note of how much, nor to what extent files were being cached/buffered.

This experiment may tell you more about the relationship between RAM usage if SFSes are loaded: http://murga-linux.com/puppy/viewtopic.php?p=686093#686093. I still don't have a firm handle on the relationship between cache-and-buffer usage** and the amount of RAM which remains available. http://murga-linux.com/puppy/viewtopic.php?p=686093#686093.

mikesLr

** cached where? buffered where? What physical action are those terms telling us has taken place.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Wed 11 Apr 2018, 20:47    Post subject:  

Thanks, mikeslr.
_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 897

PostPosted: Wed 11 Apr 2018, 22:25    Post subject:  

I boot my fatdog with it's base sfs loaded to RAM in decompressed state. (By giving giving "basesfs=expand" kernel parameter)
This uses more ram, the 350Mb sfs gets expanded to around 1.2Gb in ram.
I have 6Gb ram.

Few days ago I created a base sfs by combining 32bit lib sfs and devx sfs in original basesfs.

While booting it tried to expand the sfs to ram but gave me error of insufficient space.
Then it booted to bulldog, a command line version of fatdog with only core utilities.

I didn't try booting with "basesfs=ram" option, which only copies the sfs to ram without expanding it.

I will test it out later when I am home.

I don't know if this helps you.
Fatdog does have few different tricks than puppy.

I don't know how much but some ram should be reserved initially to facilitate loading of other things to ram.
Maybe if you go through initrd you may find something. Like creation of initial ramdisk or something....
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Wed 11 Apr 2018, 23:01    Post subject:  

Thanks for this different point of view, drunkjedi.
_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
mikeslr


Joined: 16 Jun 2008
Posts: 2468
Location: 500 seconds from Sol

PostPosted: Thu 12 Apr 2018, 09:13    Post subject:  

Hi musher0,

drunkjedi's post juggled my synapses. I have a vague recollection that when computers typically had 256 Mbs of RAM or less there had been written into Puppies (in initrd?) a limit as to how much of the OS and applications "in storage" would be copied on bootup into RAM in order to preserve some RAM available for manipulation of files. (a) That may be a false memory. (b) To what extent it continues in current Puppies and (c) what, if any, limit may now exist is unknown to me.

Except for priority, we've been assuming adrv is handled like other SFSes. Is that assumption correct?

mikesLr
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 897

PostPosted: Thu 12 Apr 2018, 10:53    Post subject:  

So I tried booting with the big base sfs (987 Mb) I created with only "basesfs=ram" kernel parameter.
It copied the base sfs to ram nicely and booted fine.

I again tried to boot with "basesfs=expand" kernel parameter.
It gave following error, which I forgot to mention last time,
"base2ram: Not expanding, need 4149 MB but only 2931 MB is available."
4149 Mb is the size of that big base sfs if I uncompress it with unsquashfs command.

So out of my 6Gb of ram only almost 3Gb is available to uncompress that sfs.
Hope this helps.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Thu 12 Apr 2018, 14:28    Post subject:  

drunkjedi wrote:
So I tried booting with the big base sfs (987 Mb) I created with only "basesfs=ram" kernel parameter.
It copied the base sfs to ram nicely and booted fine.

I again tried to boot with "basesfs=expand" kernel parameter.
It gave following error, which I forgot to mention last time,
"base2ram: Not expanding, need 4149 MB but only 2931 MB is available."
4149 Mb is the size of that big base sfs if I uncompress it with unsquashfs command.

So out of my 6Gb of ram only almost 3Gb is available to uncompress that sfs.
Hope this helps.
It does. Spot on information. Thank you very much.
_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
8Geee


Joined: 12 May 2008
Posts: 1595
Location: N.E. USA

PostPosted: Thu 12 Apr 2018, 20:55    Post subject:  

We tell puppyist...

1.) RAM should be the maximum size of all iso's + all save-files. As long as RAM is larger everything is OK.

2.) Do not use f2fs UNLESS the USB drive will house only the one puppy.

3.) Use ext3 as the file systems for all partitions and saves (.3fs). Journalling CAN shrink a .3fs file if data is removed.


4.) When selecting a partition size, consider the following calculation... 3*iso, then make the save file 1/3 of that total (round up!). For slacko5.7-nonPAE 3*180 = 540 + 256 is about 800Mb. If this final answer is MORE that your RAM, you must use pfix=nocopy.

JMHO
8Geee

PS: my personal set up of slacko5.7-2018A (nonPAE) uses a 768Mb partition. RAM = 2048Mb. With 116Mb of the save used the partition shows 480Mb used of 768Mb. It looks like filling the save-file (140Mb more) will bring the partition to 620Mb. That will allow for an expansion with caveat that its time to remaster.

_________________
Linux user #498913

Some people need to reimagine their thinking.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12395
Location: Gatineau (Qc), Canada

PostPosted: Thu 12 Apr 2018, 21:32    Post subject:  

Thanks 8Gee. You pretty much summarize my thinking.

I was thinking of including this part as safeguard in my pupsave to adrv
script. What you guys think ? (This is just a rough draft.)
TIA
Code:

ls -1 /initrd | grep pup >liste
>liste2
while read line
     do du -h -c /initrd/$line | grep total | awk '$1 ~ /M/ {print $1}' >> liste2
done < liste

Tot=0
while read line;do # we add
     Tot="`expr $Tot + "${line%M*}"`"
done < liste2
MEV="`free -m | awk '$1 ~ /Mem/ { print $2}'`" # echo $MEV
if [ "$MEV" -gt "$Tot" ];then
      case "${LANG:0:2}" in fr) echo -e "\n\e[33m\e[4mOn peut continuer."
      ;; # && echo ok || echo "pas ok" # exit.
      en|*)echo -e "\n\e[33m\e[4mWe can continue."
      ;;
# Probably conservative calculation, but will prevent bad surprises.

_________________
musher0
~~~~~~~~~~
Siempre será canción nueva... (V. Jara, Manifiesto)
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4822
Location: Kingwood, TX

PostPosted: Fri 13 Apr 2018, 01:10    Post subject:  

There are so many variations of init in the various puplets, the only way to know is to read the /init in your initrd. IIRC, though most of them reserve half of the available RAM for /tmp and/or /dev/shm. Some newer pups also set up some of the ram as compressed RAM.

@drunkjedi, FWIW if you are going to expand the squashfs into RAM, you may as well put it into an initramfs/initrd (lz4hc compression seems to load fastest) ... You may want/need to modify /init but it should be ~5x speedup.

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
drunkjedi


Joined: 24 May 2015
Posts: 897

PostPosted: Fri 13 Apr 2018, 15:09    Post subject:  

technosaurus wrote:
@drunkjedi, FWIW if you are going to expand the squashfs into RAM, you may as well put it into an initramfs/initrd (lz4hc compression seems to load fastest) ... You may want/need to modify /init but it should be ~5x speedup.
You mean using a humongous initrd?
Like what fatdog already comes with?
I have found (as jamesbond suggested to me few years back) that running with small initrd and then loading the base sfs boots faster because of superior Linux kernel taking over the task of loading to ram from bios (or bootloader I don't remember which, I may have to search old posts).
Easier would be to present you with boot times of my experiments
I do remember having reduction of almost 15 sec of boot time back then.
Which is big because my desktop boots fatdog now in flat 17sec.
Give me few days to give you a report, my hands are full now. TTYL
Back to top
View user's profile Send private message 
gyro

Joined: 28 Oct 2008
Posts: 1527
Location: Brisbane, Australia

PostPosted: Mon 16 Apr 2018, 08:01    Post subject:  

Here's a clue:
Enter "df" as a command in a console.
Look for "tmpfs" in the first column and then it's mount point in the last column.
This is typically "/initrd/mnt/tmpfs"
Now have a look at the files in that directory.
You should see a directory called "tmp", as well as all the sfs's that have been "copied to ram", identical to the corresponding files on disk.

So how much ram does a 1GiB sfs use when it is "copied to ram"?
Answer, 1GiB.

Don't be worried too much by the "size" column in the output from "df", because a "tmpfs" consumes ram dynamically, the "size" is just the maximum it is allowed to grow to.

Neither a savefile nor a savefolder is ever copied into ram, they are always accessed directly on disk.
That is why there are no partitions mounted in a running pupmode=5 Puppy, but the partition containing the savefile/savefolder is always mounted in a running pupmmode=12 Puppy.

If you have concerns about your ram size, ensure that you have a swap partition that is equal to the pyhsical size of your ram.

And if you have concerns about savefile size, use a savefolder.

gyro
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 3 [35 Posts]   Goto page: 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0773s ][ Queries: 12 (0.0151s) ][ GZIP on ]