Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 23 Aug 2014, 09:47
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Copying files to large ramdisk
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [26 Posts]   Goto page: 1, 2 Next
Author Message
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Sun 10 Aug 2014, 07:12    Post subject:  Copying files to large ramdisk
Subject description: cp or sfs?
 

Hi all

I'm using Puppy to build virtual organs. The sound generation is using Linuxsampler and the samples are streamed from disk when playing. I think that I would get a performance improvement by moving the samples to a ramdisk and streaming from there.

I can create the ramdisk without problems - it needs to be 3100MB because that is the total sample size.

My question is really about getting the files in there at startup.

Would it be best just to use cp to get the files from a folder on dev_save to /tmp/ramdisk...

or can I create an sfs (using dir2sfs) with the files in /tmp/ramdisk inside my working folder and use my ramdisk script to load the sfs once the ramdisk is created.

If both methods would work, which would potentially be faster?

Does anyone have any other suggestions?

Regards
GrahamW
Back to top
View user's profile Send private message 
Flash
Official Dog Handler


Joined: 04 May 2005
Posts: 11019
Location: Arizona USA

PostPosted: Sun 10 Aug 2014, 08:19    Post subject:  

Puppy also has rsync, possibly another way to do what you want. I have no idea which would be the best choice. With Puppy it's easy enough to try them all so you can compare them.
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 351
Location: London

PostPosted: Sun 10 Aug 2014, 10:23    Post subject:  

may i see your ramdisk script? I used to include a ramdisk with auto sync to a backup folder on hdd. The ramdisk script is created during boot and uloads all files from the ramdisksave folder to the rmdisk which is mounted on /ramdisk in he running os. It worked a charm.
Back to top
View user's profile Send private message MSN Messenger 
gcmartin

Joined: 14 Oct 2005
Posts: 4220
Location: Earth

PostPosted: Sun 10 Aug 2014, 14:20    Post subject: Re: Copying files to large ramdisk
Subject description: cp or sfs?
 

Hello GrahamW
moonskin wrote:
... The sound generation is using Linuxsampler and the samples are streamed from disk when playing. I think that I would get a performance improvement by moving the samples to a ramdisk and streaming from there.
If your PUP is NOT a full-install, there is no need to do this. Puppy, when run from Live disc or Frugal is a RAM system. This means that, excepting those HDD/USB mounted partitions, EVERYTHING is carried out in RAM!

As such, you ONLY need to move your samples to folders you create in, say, either /root (kept at shutdown) or /tmp (thrown away at shutdown) and you are done. Thus no complicated creations are even necessary.

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Mon 11 Aug 2014, 05:56    Post subject:  

Hi stemsee
stemsee wrote:
may i see your ramdisk script?


I found Pizzasgood's ramdisk_gui script from http://www.murga-linux.com/puppy/viewtopic.php?t=74663 but just used it as a guide to the basic lines I needed. No originality on my part.

I have since found an interesting article at http://www.observium.org/wiki/Persistent_RAM_disk_RRD_storage which suggests using an fstab entry to create the ramdisk.

Interesting also is the tar option mentioned for speeding up the loading process. That would suit me as the files I am using never change.

Cheers
Graham
Back to top
View user's profile Send private message 
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Mon 11 Aug 2014, 06:09    Post subject:  

Hi gcmartin

Thanks for your reply.

Not really familiar with the way a savefile works in a frugal install. I know the main sfs gets loaded into ram but I thought the savefile remained on disk so that it can be written to as you go.

Am I wrong on this? Is it loaded into RAM and synced to the drive?

I did have a look at the "ramboot" idea in the forum but that seems quite old and I gather it would not be implemented in my puppy. That would be ideal even though it would mean a big savefile. I've slimmed the samples down to 2GB but that is as low as I can go.

(I've remastered XPrecise2.4 - my favourite puppy by far - specifically for this purpose with specialised reverbs and virtual-organ software. It's even tempted some Windows users to swap over.)

Cheers
GrahamW
Back to top
View user's profile Send private message 
mikeb


Joined: 23 Nov 2006
Posts: 8181

PostPosted: Mon 11 Aug 2014, 06:49    Post subject:  

What format are the samples.... would be worth testing the compressibility using mksquashfs. As for Wav it may work well, but if a compressed format like ogg it may not gain much. You can also make an uncompressed sfs for convenience. I was thinking like you in doing what I do for additional sfs files... copy to a ramdisk (tmpfs) and mount from there... does give faster access inspite of the decompression overhead which is designed to be low for squashfs IF you are getting a decent space saving from compression so reducing copy time.. tar by itself is uncompressed... gzip would be one option on top..another fast decompression option though I am not sure of the ram usage of the latter.

Otherwise make yer tmpfs/ramdisk and copy the files in as is....

Note sfs on the fly is supposed to increase ramdisk and copy an sfs if enough room is available so might be useable as is though I cannot 100% validate its behaviour.

mike

ps not sure of the effect of a ramdisk within the layered filesystem so might be worth doing it outside..eg /initrd/mnt
Back to top
View user's profile Send private message 
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Mon 11 Aug 2014, 07:28    Post subject:  

Hi Mike

The samples are in Gigasampler format. I made an sfs with xz compression and the folder came down to 1.2GB. (tar.gz gives 1.5GB)

After reading some more, my next question becomes..

If I load an sfs at boot using the bootmanager to set it up, is that sfs expanded into RAM and the disk not used again?

I suppose another alternative is to remaster with my samples folder in root. It would make the main sfs over 2GB in size. Is that a problem? (given that I don't need an iso, just the files for a frugal install.)

Cheers
GrahamW
Back to top
View user's profile Send private message 
mikeb


Joined: 23 Nov 2006
Posts: 8181

PostPosted: Mon 11 Aug 2014, 08:35    Post subject:  

I don't believe the standard puppies load additional sfs to ram unless its a multisession cd... but don't quote me...so doing it yourself is the only guaranteed way...sorry I should have mentioned that.
Also you don't have to have the sfs as part of the layered file system (ie the files are not constrained by location)...only mounted by your description...Ie you point the programs at wherever the samples are if I understand correctly.

xz does have higher overhead..I notice it starting up libreoffice for example. You can opt for the standard gz + squashfs instead..larger but faster if its performance you are after... one to experiment with.
Again its a balance of copy time and ram gain over decompression...but you get the picture.

If a sfs is loaded to ram then yes the hard drive is free and decompression is done on the fly so there is no expansion of the sfs as such only of currently read data.

All fun and games

Mike
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4220
Location: Earth

PostPosted: Mon 11 Aug 2014, 13:42    Post subject:  

Hello @MoonSkin. Hope you saw the "Supermoon" last night.

I should have mentioned that I recommend that "if you consider creating a folder in /tmp or /root with your Frugal system", where you will copy your music, that you have a SWAP partition that at the very least is as large as your RAM. This is easily created using gParted (from Menu) and it will be automatically used whenever your distro boots.

This is "highly" recommended for overall consistent behavior and performance reasons as well as system's stability.

ALSO
Oh, to answer an earlier concern by you; YES, those files will exist in your running system RAM operations; thus, not something involved in disk activity. And, YES, if the directory you create is in /root, the save-session subsystem WILL maintain that information for your subsequent boots.

These benefits are architected for you. Nothing to install. Just some blind understanding.

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 351
Location: London

PostPosted: Mon 11 Aug 2014, 15:02    Post subject:  

moonskin wrote:

I found Pizzasgood's ramdisk_gui script from http://www.murga-linux.com/puppy/viewtopic.php?t=74663 but just used it as a guide to the basic lines I needed. No originality on my part.


That is a useful gui app, but it does not offer any persistence (save on exit). I intend to make a gui for my own ramdisk system, which doesn't use fstab. But offers full persistence and maintainence, with or without a savefile.
Back to top
View user's profile Send private message MSN Messenger 
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Tue 12 Aug 2014, 10:41    Post subject:  

Thank you everyone for your input.

I think this is what I will start with. (Not tested yet.)

Code:
#!/bin/sh

MOUNTPOINT="/tmp/ranks"
SIZE="2200"

# check for free memory
free_mem="$(free -mt | grep Mem | awk '{print $4}')"

if (("$free_mem" < "$SIZE")); then

   #let the user know there is not enough
   Xdialog --title "Ranks" --ok-label "OK" --msgbox "Sorry!\n\nThere was not enough free memory\n to load the samples\nThey will be streamed from disk." 0 0

   # create symlink in /tmp to point to the "ranks" folder
   ln -s /mnt/home/ranks /tmp/ranks
   
else

#Creates a $SIZE ramdisk at /tmp/ranks and copies files from "ranks" folder on HDD

#make sure the mountpoint exists
mkdir -p "$MOUNTPOINT"

#mount the ramdisk
mount -t tmpfs -o size=${SIZE}m tmpfs "$MOUNTPOINT"

#Open a Terminal to let rsync show progress & copy the files
Terminal -e "rsync -av --progress /mnt/home/ranks/ /tmp/ranks/"
fi



Cheers
Graham
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 351
Location: London

PostPosted: Tue 12 Aug 2014, 12:51    Post subject:  

This looks like a very neat and efficient piece of code to me. Do you mind if I use it? I would add this at the start.

size could be input.
Code:
echo "Enter the size of the ramdisk you wish to have. eg 2000  (= 2Gb)."
read size


To get input from the user, instead of one fixed size.
Back to top
View user's profile Send private message MSN Messenger 
moonskin

Joined: 18 Oct 2008
Posts: 38
Location: Wollongong, Australia

PostPosted: Tue 12 Aug 2014, 19:53    Post subject:  

stemsee wrote:
This looks like a very neat and efficient piece of code to me. Do you mind if I use it?


I don't claim any ownership. It's just cobbled together from bits and pieces Wink
Quote:
I would add this at the start.

size could be input.
Code:
echo "Enter the size of the ramdisk you wish to have. eg 2000  (= 2Gb)."
read size


To get input from the user, instead of one fixed size.


Fair enough. I'll eventually remove the dialog as my system is completely unattended. Switch on power and the system boots, scripts start all the apps and connect them together. The organist just plays the organ as normal via the organ console which sends MIDI to the computer. Once done, I treat it as a sort of embedded system.

Cheers
GrahamW
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4220
Location: Earth

PostPosted: Tue 12 Aug 2014, 21:46    Post subject:  

Question
  • Does the program that will use this folder expect something that looks like an HDD with a filesystem?
  • Or is it just looking for a folder with files?

  • And is the mount command simply a means of carving out a slice of RAM set aside permanently for Music which does NOT swap versus allowing the folder to be dynamic or mounted to backstore in SWAP when not in use? For example
    Code:
    #mount the ramdisk
    mount -F tmpfs -o size=${SIZE}m SWAP "$MOUNTPOINT"
Curious

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [26 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0979s ][ Queries: 11 (0.0104s) ][ GZIP on ]