Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 21 Dec 2014, 11:58
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
new save script
Moderators: Flash, Ian, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Author Message
Leachim

Joined: 27 May 2007
Posts: 229

PostPosted: Mon 07 Jan 2008, 20:37    Post_subject:  new save script
Sub_title: extreme speed improvement
 

I'm currently rewriting some of the main Puppy scripts. I now have a completely new init-script and a completely new save-script.

The results of the modified save-script are really astonishing!

Here are the main differences:

First of all I use aufs instead of unionfs which allows for moving files between layers without remounting.

The save script neither copy or move a file if not necessary. I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents. Because reading of flash memory is so much faster than writing I first compare the files and copy or move the file only if it is different to the version stored in the save-file (or if it is a new file).

If a file is not opened by any process it is just moved to the save-file otherwise it is copied.

I also fixed a bug in the original script which made files reappear after reboot under some circumstances. (A file in the squashfs-layer reappeared if the file also existed in the save-file and was deleted.)

Now the benchmark results:

For an average work load the old script took about 1 minute to save the changes. This time increased for each new run of the script (every half an hour) because the ramdisk was never cleared.

The new script takes about 10 seconds for the first save and - if rerun - about 2 seconds ...

I will test the new script in everyday work during the next days (making more backups than usually ^^). If everything works as expected I will post the script here. The script should be 100%-compatible with the existing one - I use aufs in unionfs-compatibility-mode.
Back to top
View user's profile Send_private_message 
Dougal


Joined: 19 Oct 2005
Posts: 2505
Location: Hell more grotesque than any medieval woodcut

PostPosted: Sat 12 Jan 2008, 08:55    Post_subject: Re: new save script
Sub_title: extreme speed improvement
 

Leachim wrote:
I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.

I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?

_________________
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Back to top
View user's profile Send_private_message 
tlchost

Joined: 05 Aug 2007
Posts: 1741
Location: Baltimore, Maryland USA

PostPosted: Sat 12 Jan 2008, 10:13    Post_subject: Re: new save script
Sub_title: extreme speed improvement
 

Dougal wrote:
Leachim wrote:
I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.

I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?


I've tried every version of 3.x I can find...and none of them are as easy to remaster a LiveCD as 2.17.1 was.

Hopefully someone can write a script to do it.....

What you did for 2.17.1 saves so much time and frustration.

Thanks
Back to top
View user's profile Send_private_message Visit_website 
Dougal


Joined: 19 Oct 2005
Posts: 2505
Location: Hell more grotesque than any medieval woodcut

PostPosted: Sun 13 Jan 2008, 08:06    Post_subject:  

I actually have 3.01 now, so I can go over it and update my remaster script.
_________________
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Back to top
View user's profile Send_private_message 
tlchost

Joined: 05 Aug 2007
Posts: 1741
Location: Baltimore, Maryland USA

PostPosted: Sun 13 Jan 2008, 08:26    Post_subject:  

Dougal wrote:
I actually have 3.01 now, so I can go over it and update my remaster script.


That would be super.

When I use the current remaster script, it appears that none of the scripts I save in /root, things in /root/.gaim, in /etc/ are not saved during the remastering.

Thanks in advance for your script.
Back to top
View user's profile Send_private_message Visit_website 
ttuuxxx


Joined: 05 May 2007
Posts: 10847
Location: Ontario Canada,Sydney Australia

PostPosted: Sun 13 Jan 2008, 09:24    Post_subject:  

Dougal yes your old remaster script, kills vesa in the 3.0 series which as we all know is a bad thing, It would be nice if you could have a erase cdrw/dvdrw before burning option.
its been a while since i used your remaster script. I get the 2 confused, I know the puppy remaster scrip misses the root/.mozilla , so you don't get the themes or extension and you manually have to move it, also it forgets the rox settings in the .icons folder and few others, about 5 settings in /root.
A new remaster script would be better then sliced bread. At least I think so Smile
Keep up the excellent work Dougal
ttuuxxx

_________________
http://audio.online-convert.com/ <-- excellent site
http://samples.mplayerhq.hu/A-codecs/ <-- Codec Test Files
http://html5games.com/ <-- excellent HTML5 games Smile
Back to top
View user's profile Send_private_message Visit_website 
Flash
Official Dog Handler


Joined: 04 May 2005
Posts: 11181
Location: Arizona USA

PostPosted: Sun 13 Jan 2008, 11:54    Post_subject:  

Leachim, does your save script also work on a multisession DVD?
_________________
Puppy Help 101 - an interactive tutorial for Lupu 5.25
Back to top
View user's profile Send_private_message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Sun 13 Jan 2008, 18:33    Post_subject:  

Quote:
I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?
I know that whenever you load a module, if it's in the zdrv_xxx.sfs file it gets copied into the pup_save.2fs file to avoid having to use zdrv_xxx.sfs next time. Maybe the same code is applying to situations where it isn't actually needed?
_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib

Back to top
View user's profile Send_private_message Visit_website 
Leachim

Joined: 27 May 2007
Posts: 229

PostPosted: Sun 13 Jan 2008, 19:03    Post_subject:  

I had less time than expected this week. But today I rewrote the script again - there were some problems with folders. It seems that one cannot move a folder to the save-file if just a single file within the folder is held open by some process. I also changed some minor options in mounting aufs.

I will test again myself this week - I don't want to run somebody into frustration if the save-script fails - it's crucial!

The save-script is meant for pen-drives only - sorry no cd-seesions are created.

The update-time goes down to 0.15 seconds if nothing has to be done.

Here are some performance results:

I tested Firefox, Opera and Seamonkey.

The following values specify the run-times (including sync) of the save-script:

after start of the browser / repeated run of save-script / after shut-down of the browser

Firefox: 8,9s / 8,1s / 9,1s

Opera: 3,4s / 1,4s / 12,2s

Seamonkey: 4,0s / 3,1s / 11,3s

This values are hardware-specific, but their relative measurements should hold for other hardware, too.

The values for "repeated run of save-script" are a measurement for the amount of files held open by the browser. It would not make sense to remove open-files from the ramdisk. (I think it's also impossible to do so.)

Currently I have all 3 browsers open. Here is my ramdisk-usage (all values in KB):

0.5 drwx 2008-01-13 20:11 dev
0.0 drwx 2008-01-13 21:15 etc
0.0 drwx 2008-01-12 12:01 opt
2357.0 drwx 2008-01-13 20:23 root
0.0 drwx 2008-01-13 20:17 sbin
148.0 drwx 2008-01-13 23:45 tmp
0.0 drwx 2008-01-13 19:44 usr
396.0 drwx 2007-11-10 02:35 var

Remember that /var and /tmp are not copied back to the save-file!

/root contains some open files, so the remained in the ramdisk.
Back to top
View user's profile Send_private_message 
Leachim

Joined: 27 May 2007
Posts: 229

PostPosted: Tue 15 Jan 2008, 12:14    Post_subject:  

While rewriting the script this weekend I forgot to implement the "check before copy"-optimization.

I now added it again and have save- or better check-times of about 0.6 seconds even with lots of open applications.

So this awful forced pausing each half hour will have an end! Smile

I will offer a complete (Puppy based) distribution including the new init- and save-scripts, dual-core-support, accelerated graphics and much more for download sometime next week.
Back to top
View user's profile Send_private_message 
zigbert


Joined: 29 Mar 2006
Posts: 5814
Location: Valåmoen, Norway

PostPosted: Tue 15 Jan 2008, 12:22    Post_subject:  

Sounds like it is time to invest in an usb-pen. Very Happy
_________________
Stardust resources
Back to top
View user's profile Send_private_message Visit_website 
linuxcbon

Joined: 09 Aug 2007
Posts: 799

PostPosted: Thu 24 Jan 2008, 20:30    Post_subject:  

Is it only for usb ? can the new dingo benefit from these improvements ?
Back to top
View user's profile Send_private_message 
Leachim

Joined: 27 May 2007
Posts: 229

PostPosted: Thu 24 Jan 2008, 21:14    Post_subject:  

The current version is for USB (flash) only, but should work with slight modifications with other aufs-layer schemes, too. I myself want to replace the ext2-save files with on the fly created sfs-files - which are sequentially written ant that is much more suitable for flash drives. So instead of creating a newly inserted sfs-layer it should make no big difference to create a new session on a CD.
_____________

I encountered some trouble with the save-script this week- maybe due to a bug in the aufs release I used.

After discussion with Junjiro Okajima, the author of aufs, I now use the newest version and till now the strange behaviour did not show up again.
Back to top
View user's profile Send_private_message 
linuxcbon

Joined: 09 Aug 2007
Posts: 799

PostPosted: Fri 25 Jan 2008, 12:59    Post_subject:  

Leachim can you please send the tips to Barry so everybody will enjoy the new speed in the next puppy Smile Thank you !
Back to top
View user's profile Send_private_message 
Leachim

Joined: 27 May 2007
Posts: 229

PostPosted: Fri 25 Jan 2008, 23:03    Post_subject:  

I will post everything after a week of finding no more bugs ...

Last night (I had to install and patch the new 2.6.24 kernel as soon as it was available ...) I found (and fixed) a severe bug handling device nodes.

The good news: aufs and the save script work well with the new kernel, too. As I understand, Barry is going to use the new kernel for Dingo. I'm now using the aufs-build from last monday and lost no more files since then due to strange aufs behaviour. (Files were never really lost - the just disappeared in the current session and were all back after a reboot.)
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0867s ][ Queries: 11 (0.0033s) ][ GZIP on ]