new save script

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

new save script

#1 Post by Leachim »

I'm currently rewriting some of the main Puppy scripts. I now have a completely new init-script and a completely new save-script.

The results of the modified save-script are really astonishing!

Here are the main differences:

First of all I use aufs instead of unionfs which allows for moving files between layers without remounting.

The save script neither copy or move a file if not necessary. I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents. Because reading of flash memory is so much faster than writing I first compare the files and copy or move the file only if it is different to the version stored in the save-file (or if it is a new file).

If a file is not opened by any process it is just moved to the save-file otherwise it is copied.

I also fixed a bug in the original script which made files reappear after reboot under some circumstances. (A file in the squashfs-layer reappeared if the file also existed in the save-file and was deleted.)

Now the benchmark results:

For an average work load the old script took about 1 minute to save the changes. This time increased for each new run of the script (every half an hour) because the ramdisk was never cleared.

The new script takes about 10 seconds for the first save and - if rerun - about 2 seconds ...

I will test the new script in everyday work during the next days (making more backups than usually ^^). If everything works as expected I will post the script here. The script should be 100%-compatible with the existing one - I use aufs in unionfs-compatibility-mode.

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

Re: new save script

#2 Post by Dougal »

Leachim wrote:I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.
I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

tlchost
Posts: 2057
Joined: Sun 05 Aug 2007, 23:26
Location: Baltimore, Maryland USA
Contact:

Re: new save script

#3 Post by tlchost »

Dougal wrote:
Leachim wrote:I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.
I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?
I've tried every version of 3.x I can find...and none of them are as easy to remaster a LiveCD as 2.17.1 was.

Hopefully someone can write a script to do it.....

What you did for 2.17.1 saves so much time and frustration.

Thanks

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#4 Post by Dougal »

I actually have 3.01 now, so I can go over it and update my remaster script.
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

tlchost
Posts: 2057
Joined: Sun 05 Aug 2007, 23:26
Location: Baltimore, Maryland USA
Contact:

#5 Post by tlchost »

Dougal wrote:I actually have 3.01 now, so I can go over it and update my remaster script.
That would be super.

When I use the current remaster script, it appears that none of the scripts I save in /root, things in /root/.gaim, in /etc/ are not saved during the remastering.

Thanks in advance for your script.

User avatar
ttuuxxx
Posts: 11171
Joined: Sat 05 May 2007, 10:00
Location: Ontario Canada,Sydney Australia
Contact:

#6 Post by ttuuxxx »

Dougal yes your old remaster script, kills vesa in the 3.0 series which as we all know is a bad thing, It would be nice if you could have a erase cdrw/dvdrw before burning option.
its been a while since i used your remaster script. I get the 2 confused, I know the puppy remaster scrip misses the root/.mozilla , so you don't get the themes or extension and you manually have to move it, also it forgets the rox settings in the .icons folder and few others, about 5 settings in /root.
A new remaster script would be better then sliced bread. At least I think so :)
Keep up the excellent work Dougal
ttuuxxx
http://audio.online-convert.com/ <-- excellent site
http://samples.mplayerhq.hu/A-codecs/ <-- Codec Test Files
http://html5games.com/ <-- excellent HTML5 games :)

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#7 Post by Flash »

Leachim, does your save script also work on a multisession DVD?
[url=http://www.murga-linux.com/puppy/viewtopic.php?t=69321][color=blue]Puppy Help 101 - an interactive tutorial for Lupu 5.25[/color][/url]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#8 Post by Pizzasgood »

I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?
I know that whenever you load a module, if it's in the zdrv_xxx.sfs file it gets copied into the pup_save.2fs file to avoid having to use zdrv_xxx.sfs next time. Maybe the same code is applying to situations where it isn't actually needed?
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#9 Post by Leachim »

I had less time than expected this week. But today I rewrote the script again - there were some problems with folders. It seems that one cannot move a folder to the save-file if just a single file within the folder is held open by some process. I also changed some minor options in mounting aufs.

I will test again myself this week - I don't want to run somebody into frustration if the save-script fails - it's crucial!

The save-script is meant for pen-drives only - sorry no cd-seesions are created.

The update-time goes down to 0.15 seconds if nothing has to be done.

Here are some performance results:

I tested Firefox, Opera and Seamonkey.

The following values specify the run-times (including sync) of the save-script:

after start of the browser / repeated run of save-script / after shut-down of the browser

Firefox: 8,9s / 8,1s / 9,1s

Opera: 3,4s / 1,4s / 12,2s

Seamonkey: 4,0s / 3,1s / 11,3s

This values are hardware-specific, but their relative measurements should hold for other hardware, too.

The values for "repeated run of save-script" are a measurement for the amount of files held open by the browser. It would not make sense to remove open-files from the ramdisk. (I think it's also impossible to do so.)

Currently I have all 3 browsers open. Here is my ramdisk-usage (all values in KB):

0.5 drwx 2008-01-13 20:11 dev
0.0 drwx 2008-01-13 21:15 etc
0.0 drwx 2008-01-12 12:01 opt
2357.0 drwx 2008-01-13 20:23 root
0.0 drwx 2008-01-13 20:17 sbin
148.0 drwx 2008-01-13 23:45 tmp
0.0 drwx 2008-01-13 19:44 usr
396.0 drwx 2007-11-10 02:35 var

Remember that /var and /tmp are not copied back to the save-file!

/root contains some open files, so the remained in the ramdisk.

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#10 Post by Leachim »

While rewriting the script this weekend I forgot to implement the "check before copy"-optimization.

I now added it again and have save- or better check-times of about 0.6 seconds even with lots of open applications.

So this awful forced pausing each half hour will have an end! :-)

I will offer a complete (Puppy based) distribution including the new init- and save-scripts, dual-core-support, accelerated graphics and much more for download sometime next week.

User avatar
zigbert
Posts: 6621
Joined: Wed 29 Mar 2006, 18:13
Location: Valåmoen, Norway
Contact:

#11 Post by zigbert »

Sounds like it is time to invest in an usb-pen. :D

linuxcbon
Posts: 1312
Joined: Thu 09 Aug 2007, 22:54

#12 Post by linuxcbon »

Is it only for usb ? can the new dingo benefit from these improvements ?

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#13 Post by Leachim »

The current version is for USB (flash) only, but should work with slight modifications with other aufs-layer schemes, too. I myself want to replace the ext2-save files with on the fly created sfs-files - which are sequentially written ant that is much more suitable for flash drives. So instead of creating a newly inserted sfs-layer it should make no big difference to create a new session on a CD.
_____________

I encountered some trouble with the save-script this week- maybe due to a bug in the aufs release I used.

After discussion with Junjiro Okajima, the author of aufs, I now use the newest version and till now the strange behaviour did not show up again.

linuxcbon
Posts: 1312
Joined: Thu 09 Aug 2007, 22:54

#14 Post by linuxcbon »

Leachim can you please send the tips to Barry so everybody will enjoy the new speed in the next puppy :-) Thank you !

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#15 Post by Leachim »

I will post everything after a week of finding no more bugs ...

Last night (I had to install and patch the new 2.6.24 kernel as soon as it was available ...) I found (and fixed) a severe bug handling device nodes.

The good news: aufs and the save script work well with the new kernel, too. As I understand, Barry is going to use the new kernel for Dingo. I'm now using the aufs-build from last monday and lost no more files since then due to strange aufs behaviour. (Files were never really lost - the just disappeared in the current session and were all back after a reboot.)

tlchost
Posts: 2057
Joined: Sun 05 Aug 2007, 23:26
Location: Baltimore, Maryland USA
Contact:

#16 Post by tlchost »

Leachim wrote:I will post everything after a week of finding no more bugs ...
Do you think the script will work with a Live-CD ?

thanks

Post Reply