Ideal ratio between size of pupsave and size of RAM?

Message

musher0 · #1 Post by **musher0** » Wed 11 Apr 2018, 06:11

Hello all.

As the title says:
What is the ideal ratio between the size of a pupsave (or pupfolder)
and the size of the RAM on the user's computer?

TIA.

I know that Puppy can detect if there is not enough RAM and then leave
the puppy_xxx.sfs on disk.

I know also that the "nocopy" parm can be used to NOT run Puppy from
RAM.

~~~~~~~~~~~~~~~

This is the case / example that I have in mind:

Puppyist has a huge pupsave folder (let's say, 7.5 Gb in total) on a 8 Gb
USB key.

Puppyist wants to make room on USB key and decides to squash huge
pupfolder as an adrv_xxx.sfs archive.

Puppyist proceeds with mksquashfs, and the result is an ~ 3 Gb adrv.

Puppyist reboots with big adrv_xxx.sfs, and there is a "kernel panic".

... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)

~~~~~~~~~~~~~~

What do we tell Puppyists to avoid such situations ?

TIA.

RetroTechGuy · #2 Post by **RetroTechGuy** » Wed 11 Apr 2018, 16:45

musher0 wrote:Hello all.

As the title says:
What is the ideal ratio between the size of a pupsave (or pupfolder)
and the size of the RAM on the user's computer?

TIA.

I know that Puppy can detect if there is not enough RAM and then leave
the puppy_xxx.sfs on disk.

I know also that the "nocopy" parm can be used to NOT run Puppy from
RAM.

~~~~~~~~~~~~~~~

This is the case / example that I have in mind:

Puppyist has a huge pupsave folder (let's say, 7.5 Gb in total) on a 8 Gb
USB key.

Puppyist wants to make room on USB key and decides to squash huge
pupfolder as an adrv_xxx.sfs archive.

Puppyist proceeds with mksquashfs, and the result is an ~ 3 Gb adrv.

Puppyist reboots with big adrv_xxx.sfs, and there is a "kernel panic".

... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)

~~~~~~~~~~~~~~

What do we tell Puppyists to avoid such situations ?

TIA.

Y'know, I don't think that I've run into that... But I've never found a reason to make the Pupsave larger than 1GB. I store all sorts of junk outside, somewhere on /mnt/home/ (in particular, crap like temporary cache folders, and the like). My email folder also sits outside -- and is often set up to be shared with multiple Puppy versions (I use Tbird, and as long as they are the same version, it doesn't cause a ruckus).

musher0 · #3 Post by **musher0** » Wed 11 Apr 2018, 19:29

Thanks, RetroTechGuy.

mikeslr · #4 Post by **mikeslr** » Wed 11 Apr 2018, 23:34

Hi musher0,

"... Puppyist had forgotten (s)he only had 1.5 Gb of RAM... whereas live
adrv sfs still needs 7.5 Gb of RAM (Am I correct in supposing this?)".

I don't think so. But, having no idea how to experiment to find out, I only know what I've been told. Which amounts to this:

In the absence of the boot command "copy*", a Puppy does not copy an entire SFS into RAM. Rather, when a Frugal Puppy (using a merge-file system) boots it copies some files into RAM and creates in RAM an inodes list which points to where the balance of the files are. At least that's my "short-hand" take from the following posts:

http://murga-linux.com/puppy/viewtopic. ... 162#282162
http://murga-linux.com/puppy/viewtopic. ... 458#827458
http://www.linfo.org/inode.html

* The one experiment I ran using the copy boot command did result in more ram being used on bootup. But I did not make a note of how much, nor to what extent files were being cached/buffered.

This experiment may tell you more about the relationship between RAM usage if SFSes are loaded: http://murga-linux.com/puppy/viewtopic. ... 093#686093. I still don't have a firm handle on the relationship between cache-and-buffer usage** and the amount of RAM which remains available. http://murga-linux.com/puppy/viewtopic. ... 093#686093.

mikesLr

** cached where? buffered where? What physical action are those terms telling us has taken place.

musher0 · #5 Post by **musher0** » Thu 12 Apr 2018, 00:47

Thanks, mikeslr.

drunkjedi · #6 Post by **drunkjedi** » Thu 12 Apr 2018, 02:25

I boot my fatdog with it's base sfs loaded to RAM in decompressed state. (By giving giving "basesfs=expand" kernel parameter)
This uses more ram, the 350Mb sfs gets expanded to around 1.2Gb in ram.
I have 6Gb ram.

Few days ago I created a base sfs by combining 32bit lib sfs and devx sfs in original basesfs.

While booting it tried to expand the sfs to ram but gave me error of insufficient space.
Then it booted to bulldog, a command line version of fatdog with only core utilities.

I didn't try booting with "basesfs=ram" option, which only copies the sfs to ram without expanding it.

I will test it out later when I am home.

I don't know if this helps you.
Fatdog does have few different tricks than puppy.

I don't know how much but some ram should be reserved initially to facilitate loading of other things to ram.
Maybe if you go through initrd you may find something. Like creation of initial ramdisk or something....

musher0 · #7 Post by **musher0** » Thu 12 Apr 2018, 03:01

Thanks for this different point of view, drunkjedi.

mikeslr · #8 Post by **mikeslr** » Thu 12 Apr 2018, 13:13

Hi musher0,

drunkjedi's post juggled my synapses. I have a vague recollection that when computers typically had 256 Mbs of RAM or less there had been written into Puppies (in initrd?) a limit as to how much of the OS and applications "in storage" would be copied on bootup into RAM in order to preserve some RAM available for manipulation of files. (a) That may be a false memory. (b) To what extent it continues in current Puppies and (c) what, if any, limit may now exist is unknown to me.

Except for priority, we've been assuming adrv is handled like other SFSes. Is that assumption correct?

mikesLr

drunkjedi · #9 Post by **drunkjedi** » Thu 12 Apr 2018, 14:53

So I tried booting with the big base sfs (987 Mb) I created with only "basesfs=ram" kernel parameter.
It copied the base sfs to ram nicely and booted fine.

I again tried to boot with "basesfs=expand" kernel parameter.
It gave following error, which I forgot to mention last time,
"base2ram: Not expanding, need 4149 MB but only 2931 MB is available."
4149 Mb is the size of that big base sfs if I uncompress it with unsquashfs command.

So out of my 6Gb of ram only almost 3Gb is available to uncompress that sfs.
Hope this helps.

musher0 · #10 Post by **musher0** » Thu 12 Apr 2018, 18:28

drunkjedi wrote:So I tried booting with the big base sfs (987 Mb) I created with only "basesfs=ram" kernel parameter.
It copied the base sfs to ram nicely and booted fine.

I again tried to boot with "basesfs=expand" kernel parameter.
It gave following error, which I forgot to mention last time,
"base2ram: Not expanding, need 4149 MB but only 2931 MB is available."
4149 Mb is the size of that big base sfs if I uncompress it with unsquashfs command.

So out of my 6Gb of ram only almost 3Gb is available to uncompress that sfs.
Hope this helps.

It does. Spot on information. Thank you very much.

8Geee · #11 Post by **8Geee** » Fri 13 Apr 2018, 00:55

We tell puppyist...

1.) RAM should be the maximum size of all iso's + all save-files. As long as RAM is larger everything is OK.

2.) Do not use f2fs UNLESS the USB drive will house only the one puppy.

3.) Use ext3 as the file systems for all partitions and saves (.3fs). Journalling CAN shrink a .3fs file if data is removed.

4.) When selecting a partition size, consider the following calculation... 3*iso, then make the save file 1/3 of that total (round up!). For slacko5.7-nonPAE 3*180 = 540 + 256 is about 800Mb. If this final answer is MORE that your RAM, you must use pfix=nocopy.

JMHO
8Geee

PS: my personal set up of slacko5.7-2018A (nonPAE) uses a 768Mb partition. RAM = 2048Mb. With 116Mb of the save used the partition shows 480Mb used of 768Mb. It looks like filling the save-file (140Mb more) will bring the partition to 620Mb. That will allow for an expansion with caveat that its time to remaster.

musher0 · #12 Post by **musher0** » Fri 13 Apr 2018, 01:32

Thanks 8Gee. You pretty much summarize my thinking.

I was thinking of including this part as safeguard in my pupsave to adrv
script. What you guys think ? (This is just a rough draft.)
TIA

Code: Select all

ls -1 /initrd | grep pup >liste
>liste2
while read line
     do du -h -c /initrd/$line | grep total | awk '$1 ~ /M/ {print $1}' >> liste2
done < liste

Tot=0
while read line;do # we add
     Tot="`expr $Tot + "${line%M*}"`"
done < liste2
MEV="`free -m | awk '$1 ~ /Mem/ { print $2}'`" # echo $MEV
if [ "$MEV" -gt "$Tot" ];then
      case "${LANG:0:2}" in fr) echo -e "\n\e[33m\e[4mOn peut continuer."
      ;; # && echo ok || echo "pas ok" # exit.
      en|*)echo -e "\n\e[33m\e[4mWe can continue."
      ;;
# Probably conservative calculation, but will prevent bad surprises.

technosaurus · #13 Post by **technosaurus** » Fri 13 Apr 2018, 05:10

There are so many variations of init in the various puplets, the only way to know is to read the /init in your initrd. IIRC, though most of them reserve half of the available RAM for /tmp and/or /dev/shm. Some newer pups also set up some of the ram as compressed RAM.

@drunkjedi, FWIW if you are going to expand the squashfs into RAM, you may as well put it into an initramfs/initrd (lz4hc compression seems to load fastest) ... You may want/need to modify /init but it should be ~5x speedup.

drunkjedi · #14 Post by **drunkjedi** » Fri 13 Apr 2018, 19:09

technosaurus wrote:@drunkjedi, FWIW if you are going to expand the squashfs into RAM, you may as well put it into an initramfs/initrd (lz4hc compression seems to load fastest) ... You may want/need to modify /init but it should be ~5x speedup.

You mean using a humongous initrd?
Like what fatdog already comes with?
I have found (as jamesbond suggested to me few years back) that running with small initrd and then loading the base sfs boots faster because of superior Linux kernel taking over the task of loading to ram from bios (or bootloader I don't remember which, I may have to search old posts).
Easier would be to present you with boot times of my experiments
I do remember having reduction of almost 15 sec of boot time back then.
Which is big because my desktop boots fatdog now in flat 17sec.
Give me few days to give you a report, my hands are full now. TTYL

gyro · #15 Post by **gyro** » Mon 16 Apr 2018, 12:01

Here's a clue:
Enter "df" as a command in a console.
Look for "tmpfs" in the first column and then it's mount point in the last column.
This is typically "/initrd/mnt/tmpfs"
Now have a look at the files in that directory.
You should see a directory called "tmp", as well as all the sfs's that have been "copied to ram", identical to the corresponding files on disk.

So how much ram does a 1GiB sfs use when it is "copied to ram"?
Answer, 1GiB.

Don't be worried too much by the "size" column in the output from "df", because a "tmpfs" consumes ram dynamically, the "size" is just the maximum it is allowed to grow to.

Neither a savefile nor a savefolder is ever copied into ram, they are always accessed directly on disk.
That is why there are no partitions mounted in a running pupmode=5 Puppy, but the partition containing the savefile/savefolder is always mounted in a running pupmmode=12 Puppy.

If you have concerns about your ram size, ensure that you have a swap partition that is equal to the pyhsical size of your ram.

And if you have concerns about savefile size, use a savefolder.

gyro

hamoudoudou · #16 Post by **hamoudoudou** » Mon 16 Apr 2018, 18:38

just to clarify how Ram is loaded at boot

gyro · #17 Post by **gyro** » Mon 16 Apr 2018, 23:23

hamoudoudou wrote:just to clarify how Ram is loaded at boot

Sorry, the diagram shows the structure of the aufs/unionfs stack that gets mounted in an underdog Puppy.
It does not show which elements actually reside in ram.

But please don't take my word for it. Read the source, it's found in "/initrd/init".

gyro

bigpup · #18 Post by **bigpup** » Tue 17 Apr 2018, 03:51

Simply watch the boot process messages.
They tell you what gets loaded and when.

hamoudoudou · #19 Post by **hamoudoudou** » Thu 19 Apr 2018, 02:37

The Puppy Linux Trade mark is to be fast because it runs full in RAM.
"Puppy Linux is yet another Linux distribution. What's different here is that Puppy is extraordinarily small, yet quite full-featured. Puppy boots into a ramdisk and, unlike live CD distributions that have to keep pulling stuff off the CD, it loads into RAM. " Xenialpup 7.5, Distrowatch

Searching and asking at boot which pupsaves to load, then immediately loading one of them. We are used to that... and that is the reason why a Puppy must be tiny..
Sure you can run without pupsave, and load on the fly SFS just needed for the session.. by a click on it.. easy.
If the 'click on it' process is too easy, sure experts can write scripts, (re-write the script hidden behind GUI's, located in sbin).

gyro · #20 Post by **gyro** » Thu 19 Apr 2018, 04:46

@hamoudoudou,
Unfortunately the "Loading" you refer to in your image means that the savefile/savefolder is being loaded into the aufs stack.
Not that it is being copied into ram.

I recognise only 1 authority as to what the "init" script does, the source code of the "init" script.

gyro

(old)Puppy Linux Discussion Forum

(old)Puppy Linux Discussion Forum