Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 20 May 2013, 13:46
All times are UTC - 4

 Forum index » Off-Topic Area » Programming
Speeding up the SnapMerge
Post new topic   Reply to topic View previous topic :: View next topic
Page 6 of 6 [90 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6
Author Message
01micko


Joined: 11 Oct 2008
Posts: 7018
Location: qld
PostPosted: Sun 13 Feb 2011, 00:28    Post subject:  
Hi Jamesbond

I tried s17 on my xo-1 olpc laptop and it does make a big speed difference. No problem so far after installing a couple of pets and deleting a few files.

Note that I am not having mavrothal's issue of devices not unmounting. Patriot's code is indeed in the rc.shutdown script. These are peculiar machines so it could be anything.

Cheers
_________________
keep the faith Cool ..
Back to top
View user's profile Send private message Visit poster's website 
mavrothal


Joined: 24 Aug 2009
Posts: 1058
PostPosted: Sun 13 Feb 2011, 02:53    Post subject:  
jamesbond wrote:

Quote:
Now if someone can make puppy to umount the savelayer cleanly too... Wink
I thought Patriot has done this, and I'm sure that at least Fluppy, later version of Puppeee and Fatdog has this included. I'm not sure whether it makes it to the base puppy (woof) though.


The code is there (as Micko said) but it does a lazy umount. So if you have an ext3/4 formatted media you see that every time it recovers the journal (even if you add some time after the lazy umount for things to be written).
I think Mick is using vfat, ext2 media thus does not see the problem, since as far as puppy is concerned the fsckme.err/.flg is removed already.
_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 1058
PostPosted: Sun 13 Feb 2011, 05:33    Post subject: Mplayer conflict?  
Tried s-17 (in XOpup).
I do not know if this a bug or a feature Confused but when gnome-mplayer is open snapmergepuppy freezes in the initial "merge requested" dialog.
Even if mplayer is paused or stopped or even without a file loaded.
Resumes when mplayer closes.

There is nothing in messages or xerrs.log indicative of a problem other than a process can not be found to be stopped and then restarted, at the "kill -STOP ${allprocess[$i]}" and "kill -CONT $i" step respectively. But this message has nothing to do with gnome-mplayer running or not.

For comparison, running a flash movie in the browser pauses no problem in snapmerge (and only as small freeze in flash).
_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL
PostPosted: Sun 13 Feb 2011, 12:55    Post subject:  
mavrothal wrote:
jamesbond wrote:

Quote:
Now if someone can make puppy to umount the savelayer cleanly too... Wink
I thought Patriot has done this, and I'm sure that at least Fluppy, later version of Puppeee and Fatdog has this included. I'm not sure whether it makes it to the base puppy (woof) though.


The code is there (as Micko said) but it does a lazy umount. So if you have an ext3/4 formatted media you see that every time it recovers the journal (even if you add some time after the lazy umount for things to be written).
I think Mick is using vfat, ext2 media thus does not see the problem, since as far as puppy is concerned the fsckme.err/.flg is removed already.


You probably have an old version of the shutdown script.

Patriot's code looks like this and does work:

Code:
if [ "$SAVE_LAYER" ];then
 sync
 SAVEDEV="`mount | grep "/initrd${SAVE_LAYER}" | cut -f 1 -d ' '`"
 SAVEFS="`mount | grep "/initrd${SAVE_LAYER}" | cut -f 5 -d ' '`"
 #100615 Patriot: suggested this code to enable save-layer to remount ro...
 uniFS=$(awk '/unionfs/ {print $3}' /proc/mounts) #ex: aufs
 if [ "$uniFS" == "aufs" -a "$SAVE_LAYER" == "/pup_rw" ]; then
  #i think only work if prepended dir is a separate f.s...
  if [ "`mount | grep '^tmpfs on /tmp '`" != "" ];then #created in initrd.
   mkdir -p /tmp/unrootfs
   busybox mount -o remount,prepend:/tmp/unrootfs,xino=/tmp/unrootfs/xino -t $uniFS / /
   sync
  fi
 fi
 busybox mount -t $SAVEFS -o remount,ro $SAVEDEV /initrd${SAVE_LAYER} 2>/dev/null
 umount-FULL -i -n -l /initrd${SAVE_LAYER} 2>/dev/null #-l is lazy unmount.
fi
Back to top
View user's profile Send private message Visit poster's website 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL
PostPosted: Sun 13 Feb 2011, 14:25    Post subject:  
jamesbond wrote:
Didn't incorporate your suggestion yet - I was working on it, finished it and then see your post.

Anyway - here is s17 - modularised version of s16.

Changes -
1) warn user based on size of tmpfs and number of files in tmpfs (I included both because I'm not sure which steps causes the slow down)

2) modularised the functions

It still works as before (ie pauses the whole process), but I've separated the freeze function, snapmerge function, and cleanup function. It will be easy to change so that it will only freeze during cleanup. Look at line 204 - just move the freeze_processes in-between snapmerge and cleanup.

If we can confirm that the slowdown is because of tmpfs deletion, then there's not point in doing this. But if the slowdown is because rsync - ok perhaps it's worthwhile to take additional risks for shorter freeze time.

Let me know.

Oh yes the warn_user limit is arbitrary - it really depends on how slow things are. I'll take your feedback on this. Currently I set the limit to 64MB and 1000 files.

cheers!


I'm going to start a new save file and try to run in pupmode 13 for a few weeks and see how it goes.
_________________
http://saluki-linux.com
Back to top
View user's profile Send private message Visit poster's website 
mavrothal


Joined: 24 Aug 2009
Posts: 1058
PostPosted: Sun 13 Feb 2011, 15:34    Post subject:  
[quote="jemimah"]
mavrothal wrote:
You probably have an old version of the shutdown script.

Patriot's code looks like this and does work:

Actually this the the exact code I have and it works in the sense that shuts down the XO. The "problem" is that is doing it after a lazy umount of the save layer.
The "busybox mount -t $SAVEFS..." step reports "mounting /dev/mmcblk0p1 on /initrd/pup_rw failed: Device or resource busy"

Oh well, maybe is an XO specific issue Confused
_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL
PostPosted: Sun 13 Feb 2011, 16:10    Post subject:  
It should mount the save layer read-only and then the lazy unmount. If the read-only step is successful, then the save file will be clean.
_________________
http://saluki-linux.com
Back to top
View user's profile Send private message Visit poster's website 
mavrothal


Joined: 24 Aug 2009
Posts: 1058
PostPosted: Sun 13 Feb 2011, 16:29    Post subject:  
jemimah wrote:
It should mount the save layer read-only

That's the exact step that fails with "Device or resource busy"
_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Sun 13 Feb 2011, 19:42    Post subject: Re: Mplayer conflict?  
mavrothal wrote:
Tried s-17 (in XOpup).
I do not know if this a bug or a feature Confused but when gnome-mplayer is open snapmergepuppy freezes in the initial "merge requested" dialog.
Even if mplayer is paused or stopped or even without a file loaded.
Resumes when mplayer closes.

There is nothing in messages or xerrs.log indicative of a problem other than a process can not be found to be stopped and then restarted, at the "kill -STOP ${allprocess[$i]}" and "kill -CONT $i" step respectively. But this message has nothing to do with gnome-mplayer running or not.

For comparison, running a flash movie in the browser pauses no problem in snapmerge (and only as small freeze in flash).
Hmm that's odd. How about this - open gnome-mplayer, and then open terminal, and run "snapmergepuppy" from terminal and tell me what you see. If everything works, the problem is elsewhere (snapmergepuppy is just the last chain of command that do the actual work, there are other scripts that work). If snapmergepuppy never finishes, (and you can press Ctrl-C to break out of it) - hmmm I need to think harder Very Happy
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Sun 13 Feb 2011, 19:43    Post subject:  
jemimah wrote:
I'm going to start a new save file and try to run in pupmode 13 for a few weeks and see how it goes.
I can have a break for a while Smile
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
mavrothal


Joined: 24 Aug 2009
Posts: 1058
PostPosted: Mon 14 Feb 2011, 14:42    Post subject: Re: Mplayer conflict?  
jamesbond wrote:
Hmm that's odd. How about this - open gnome-mplayer, and then open terminal, and run "snapmergepuppy" from terminal and tell me what you see. If everything works, the problem is elsewhere (snapmergepuppy is just the last chain of command that do the actual work, there are other scripts that work). If snapmergepuppy never finishes, (and you can press Ctrl-C to break out of it) - hmmm I need to think harder Very Happy


You are actually right.
Is pup_event_frontend_d that does not allow merges by default when mplayer (and other apps) are open. Embarassed However, save2flash does run and you end up with a hung warning screen.
Snapmergepuppy is fine
Get your rest Very Happy
_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Thu 24 Feb 2011, 19:25    Post subject:  
s18 - fix for those running bash version 4 (e.g. fatdog). For those not using bash4, there is no need to upgrade.

cheers!
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13

Last edited by jamesbond on Tue 01 Mar 2011, 07:06; edited 1 time in total
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Tue 01 Mar 2011, 07:05    Post subject:  
s19 - use --force option for rsync to make sure that it we replace a directory with a symlink at the top layer, this will be rsync-ed down to the save layer.

I'm cleaning up and removing all the old versions.
snapmerge-19.pet
Description 
pet

 Download 
Filename  snapmerge-19.pet 
Filesize  97.78 KB 
Downloaded  185 Time(s) 

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Fri 04 Mar 2011, 09:23    Post subject:  
Been eating my own dogfood for a week now, running this on 1GB pupsave file on top of 16GB ext3-formatted flash. I bumped up the size of the warning from 32MB in the script to 128MB as I don't notice a delay during merge.

So far looks okay, no data loss, no odd behaviour, etc.

cheers!
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1534
Location: The Blue Marble
PostPosted: Wed 23 Mar 2011, 05:37    Post subject:  
The story three weeks later - so far so good. The save is almost always instantaneous - for 200MB worth of data (this is on a 4GB machine - so it appears that the kernel manage to cache most of it - different machines may fare differently). The delay however is apparent when we do "sync".
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 6 of 6 [90 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.0855s ][ Queries: 12 (0.0055s) ][ GZIP on ]