Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 21 Oct 2014, 05:32
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
sfs_load-2.3 on-the-fly
Post new topic   Reply to topic View previous topic :: View next topic
Page 4 of 20 [295 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, ..., 18, 19, 20 Next
Author Message
jamesbond

Joined: 26 Feb 2007
Posts: 2226
Location: The Blue Marble

PostPosted: Thu 03 Feb 2011, 22:32    Post subject:  

jamesbond wrote:
2. As in my earlier comment, if you don't mind adding "non-persistent" loading of SFS beyond the original 6 SFS limit.

Testing this - this is currently works partially:
a) Adding SFS beyond the maximum 6 works. The script will create the necessary mountpoint in /initrd (e.g. /initrd/pup_ro10). But it uses the wrong loop device - instead of using loop10, it uses loop2 instead.
b) It still adds into SFSLIST (obviously because this is how it's designed for right now).
c) Removing it (within the same session as adding it) seems to work (success message), but actually fails - the SFS is still mounted. But because it's no longer in SFSLIST, sfs_load doesn't see it anymore.

Well, I know this is probably outside your original scope of design when you started, so if you don't want to hear this, just tell me to shut up and I'll do Very Happy

cheers!

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2226
Location: The Blue Marble

PostPosted: Fri 04 Feb 2011, 00:12    Post subject:  

jamesbond wrote:
Just FYI (and a puzzle) - this works with when busybox mount is used. When mount-FULL is used, it will bring down the system Shocked I wonder what's the difference?
Interesting - even calling mount.aufs directly still kills Shocked this busybox mount must have a power we all don't know Twisted Evil

EDIT: mount.aufs is just a wrapper which will call the full mount - and thus it kills. busybox mount make the mount syscall directly.

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread

Last edited by jamesbond on Fri 04 Feb 2011, 06:04; edited 1 time in total
Back to top
View user's profile Send private message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Fri 04 Feb 2011, 01:26    Post subject:  

jamesbond wrote:
jamesbond wrote:
2. As in my earlier comment, if you don't mind adding "non-persistent" loading of SFS beyond the original 6 SFS limit.

Testing this - this is currently works partially:


Why not just create a simple merge utility?

EDIT:

http://murga-linux.com/puppy/viewtopic.php?p=492619#492619
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2226
Location: The Blue Marble

PostPosted: Fri 04 Feb 2011, 06:07    Post subject:  

jpeps wrote:
jamesbond wrote:
jamesbond wrote:
2. As in my earlier comment, if you don't mind adding "non-persistent" loading of SFS beyond the original 6 SFS limit.

Testing this - this is currently works partially:


Why not just create a simple merge utility?

EDIT:

http://murga-linux.com/puppy/viewtopic.php?p=492619#492619

Because this is what OTF SFS loader is for ? Smile

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
shinobar


Joined: 28 May 2009
Posts: 2631
Location: Japan

PostPosted: Fri 04 Feb 2011, 08:31    Post subject: sfs_load-0.4  

Thanks to all testers.
But please test again with the new version, sfs_load-0.4.
Mainly improved loading SFS from CD with RAM mode.
It offers COPYtoHDD/RAM/NOCOPY options for the live CD without pupsave.
You can see how it works with the multilingual Wary-500m07.

Doesn't check the number of extra SFS. I am not sure what happens when it exceeds 6...
sfs_load_whereto.png
 Description   Load from CD without pupsave
 Filesize   6.77 KB
 Viewed   1169 Time(s)

sfs_load_whereto.png


_________________
Google Chrome portable
Downloads for Puppy Linux http://shino.pos.to/linux/downloads.html

Last edited by shinobar on Fri 04 Feb 2011, 11:12; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
jamesbond

Joined: 26 Feb 2007
Posts: 2226
Location: The Blue Marble

PostPosted: Fri 04 Feb 2011, 11:03    Post subject:  

All my tests have been done from Fatdog64 511 with frugal install with pfix=noram option (/pup_rw is save file, and /pup_ro2 is main.sfs on disk).

sfs load and unload ok. But there is a subtle issue with unloading sfs out of order - e.g. you have sfs1, sfs2, sfs3, sfs4, sfs5, sfs6 loaded (in that order) - it's ok if we unload sfs6, unload sfs5, and then load sfs51 and sfs61.
But if (assuming original order) we tried to unload sfs5, and then tried to load sfs51, problem happens --- because unloading sfs5 will free up /dev/loop8, but when we try to load the next sfs, this free loop device is not detected - because /dev/loop9 is still in use (by sfs6). What happens after that:
- sfs51 will be loaded in /dev/loop2
- but mount point assigned to /initrd/pup_ro10
and all hell breaks loose after that (ie you can no longer unload sfs51). Problem fixes itself after reboot.
This is particularly easy to spot when one is running at full capacity (6 SFS) but it will happen sooner or later no matter how many SFS are being loaded.
It's a bit complicated to explain but I hope you get the idea.

Solution: a few ways to solve, the obvious one is probably to make a smarter function to find free loop device - ie to detect gaps (if loop5 is used and loop7 is used but loop6 is free, we can still use loop6 instead of using loop8).

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
Q5sys


Joined: 11 Dec 2008
Posts: 1073

PostPosted: Fri 04 Feb 2011, 18:47    Post subject:  

shinobar wrote:
sc0ttman wrote:
2. Does this allow users to change the maximum number of loops available - so that users can choose how many SFSs are loaded on the fly?
(Very useful!)

No. The sfs_loader keeps the compatibility with the traditional bootmanager and has same limit.
Barry thinks too many layers to the unionfs slows down the performance.


Has anyone ever gone about testing this? I'm using a P4 1.4ghz laptop with 2GB ram running Lighthouse Pup and have ~ 25 SFS files (about 1.7gb worth of SFS files) running and I dont perceive any noticeable slowing of the system... granted booting takes a few seconds more, than when I have 1 or 2 loaded. Granted on a slower system with less CPU and RAM, I can understand the concept of it slowing things down, but how slow of a CPU would you have to have for the filesystem to start slowing down the overall system?

_________________



Back to top
View user's profile Send private message Visit poster's website 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Fri 04 Feb 2011, 20:53    Post subject:  

Here is some code from the snapmerge script. Adding more layers makes it slower because each whiteout file needs to be checked on each layer. This script is already painfully slow and the main reason I don't want to add more layers,

I supposed it's worth experimenting and seeing what how much difference it makes.

Code:
while read N
do
 BN="`basename "$N"`"
 DN="`dirname "$N"`"
 [ "$BN" = ".wh.aufs" ] && continue #w003 aufs has file .wh..wh.aufs in /initrd/pup_rw.
 [ "$DN" = "." ] && continue
 if [ "$BN" = "__dir_opaque" ];then #w003
  #'.wh.__dir_opaque' marks ignore all contents in lower layers...
  rm -rf "${BASE}/${DN}/*" 2>/dev/null #wipe anything in save layer.
  #also need to save the whiteout file to block all lower layers (may be readonly)...
  touch "${BASE}/${DN}/.wh.__dir_opaque" 2>/dev/null
  rm -f "$SNAP/$DN/.wh.__dir_opaque" #should force aufs layer "reval".
  continue
 fi
 #comes in here with the '.wh.' prefix stripped off, leaving actual filename...
 rm -rf "$BASE/$N"
 #if file exists on a lower layer, have to save the whiteout file...
 BLOCKFILE=""
 [ -e "/initrd/pup_ro1/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro2/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro3/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro4/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro5/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro6/$N" ] && BLOCKFILE="yes"
 [ -e "/initrd/pup_ro7/$N" ] && BLOCKFILE="yes" #v424
 [ -e "/initrd/pup_ro8/$N" ] && BLOCKFILE="yes" #v424
 [ -e "/initrd/pup_ro9/$N" ] && BLOCKFILE="yes" #v424
 [ "$BLOCKFILE" = "yes" ] && touch "${BASE}/${DN}/.wh.${BN}"
 rm -f "$SNAP/$DN/.wh.$BN" #remove whiteout file. should force aufs layer "reval".
done
Back to top
View user's profile Send private message Visit poster's website 
shinobar


Joined: 28 May 2009
Posts: 2631
Location: Japan

PostPosted: Sat 05 Feb 2011, 01:42    Post subject: whiteout  

jemimah wrote:
Here is some code from the snapmerge script. Adding more layers makes it slower because each whiteout file needs to be checked on each layer. This script is already painfully slow

Yes, jemimah. It is so slow.

Code:
#also need to save the whiteout file to block all lower layers

I wonder why we need to check them. Why not unconditionally copy all the file in pup_rw...?
I also wonder what the rc.update does...? Rolling Eyes

_________________
Google Chrome portable
Downloads for Puppy Linux http://shino.pos.to/linux/downloads.html

Last edited by shinobar on Sat 05 Feb 2011, 01:49; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
shinobar


Joined: 28 May 2009
Posts: 2631
Location: Japan

PostPosted: Sat 05 Feb 2011, 01:45    Post subject: sfs_load-0.5  

OK, now available Smile See the top post of this topic.

UPDATE: 5 Feb 2011 v0.5: fix was the SFS removed from the list even unload failed, search smaller number if pup_roN is not available (thanks to jamesbond)

_________________
Google Chrome portable
Downloads for Puppy Linux http://shino.pos.to/linux/downloads.html

Last edited by shinobar on Sat 05 Feb 2011, 03:03; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Sat 05 Feb 2011, 02:47    Post subject: Re: whiteout  

shinobar wrote:

Code:
#also need to save the whiteout file to block all lower layers

I wonder why we need to check them. Why not unconditionally copy all the file in pup_rw...?
I also wonder what the rc.update does...? Rolling Eyes


Say I create a new file then delete it and a white out file gets saved to the save file. Then later I add an sfs containing a file of the same name. The file will not appear because the white out file is there blocking it. I believe there is code in the init script to check for this condition and delete the interefering white outs when you add an SFS, but I know from experience that even that doesn't always work.

But that's an interesting thought - maybe the whiteout checking code in snapmerge is redundant and can be removed. However, It may be an error condition in AUFS to have a whiteout file with no file below it. I know for sure UnionFS is really picky about that, but I think AUFS is more tolerant.

However, I think the real bottleneck in the script is checking for free space in the save file for every single file copied down. That could be omitted in the case where your save file has more free space than the size of the files in RAM - but otherwise I think you need to do it.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2385
Location: UK

PostPosted: Sat 05 Feb 2011, 05:55    Post subject: Re: whiteout  

jemimah wrote:
I think the real bottleneck in the script is checking for free space in the save file for every single file copied down. That could be omitted in the case where your save file has more free space than the size of the files in RAM - but otherwise I think you need to do it.

Would it not be enough to check the free space is larger than the sfs to be loaded on the fly?
(At least twice as big as the sfs file, for example..)

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2226
Location: The Blue Marble

PostPosted: Sat 05 Feb 2011, 10:46    Post subject:  

shinobar, sorry to hijack your thread. I'll move off your lawn very quickly after this.

Been thinking about it too ... I'm comparing the situation that requires snapmergepuppy and the one where /pup_rw is mounted directly on pupsave file. In this case, no management of whiteout files is done (as shinobar said) - and yet things will work correctly.

In the specific PUPMODE where merge script is required, these are the conditions:
a) there are, effectively, two pupsave files - the tmpfs layer, and the real pupsave (mounted ro by aufs)
b) we want to create the impression that this two pupsaves work as one
c) we don't want to duplicate items from pupsave to tmpfs
d) optionally, tmpfs and pupsave is allowed to have different size

a) & b) is rather easy to accomplish, it's c) & d) which causes the most headache and the need for merge script. Actually, c) is also the cause of problem if your real pupsave file is almost full, yet the tmpfs is empty (ie fresh boot). One can keep adding things without knowing that one cannot save the stuff anymore. Kinda like vmware thin provisioning, but without enough backing storage Shocked

If it's only a) & b) - easy - just load pupsave to tmpfs at start, and then rsync everything to pupsave during shutdown (or during merge). The real pupsave don't even need to be part of the branch.

But we need to do c) and d) since that's the agreed design criteria for now. Based on the above, I think the only check needed is as follows, for a combination of a "real file" and its corresponding whiteout file:

1. whiteout file exists in tmpfs, real file exists in pupsave
Cause ==> the file has just been deleted during user session.
Action ==> delete real file in pupsave & create the whiteout file (to prevent any file from lower layer getting exposed).
Then delete the whiteout in tmpfs.

2. whiteout file exist in tmpfs, real file doesn't exist in pupsave
Cause ==> whiteout is for a file in lower layer
Action ==> create whiteout file in pupsave
Then delete the whiteout in tmpfs.

3. real file exists in tmpfs, whiteout exist in pupsave
Cause ==> new file created over previously deleted file (from previous session)
Action ==> copy file from tmpfs to pupsave, and delete whiteout in pupsave
Then delete the real file in tmpfs.

4. real file exists in tmpfs, whiteout doesn't exist in pupsave
Cause ==> new file created in this session
Action ==> copy file from tmpfs to pupsave,
Then delete the real file in tmpfs.

5. real file exist in tmpfs, real file also exist in pupsave
Cause ==> file is updated in this session
Action ==> copy file from tmpfs to pupsave,
Then delete the real file in tmpfs.

Of course when I say "file" it also applies to directories.

I think that should handle 90% of the cases. We skip corner cases of "we only save the whiteout files only if the lower layer SFS have the real files" - I don't really see why this is necessary.

If the slowness comes from checking all those files in the SFS layers, then by dealing only with tmpfs and pupsave, this delay should be greatly reduced. If it's not, then the above may not help. In fact, I'm doubting the need to have c) and d) in the first place ... I mean, you have that very important big file you need to save, you can always save it in /mnt/home (ie the real storage).

Ok, I'm off - jemimah we can start another thread on this if you want to.

Shinobar, thanks for the update, I'll test it and get back to you.

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Sat 05 Feb 2011, 15:40    Post subject:  

Yes let's discuss this further. I was about to attempt optimizing it on my own anyway. Here is a new thread.

http://murga-linux.com/puppy/viewtopic.php?p=493055#493055
Back to top
View user's profile Send private message Visit poster's website 
seaside

Joined: 11 Apr 2007
Posts: 887

PostPosted: Sun 06 Feb 2011, 11:30    Post subject:  

I thought this might be another aspect to throw in for consideration.

http://murga-linux.com/puppy/viewtopic.php?p=493294#493294

Thanks for this work.

Cheers,
s
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 4 of 20 [295 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, ..., 18, 19, 20 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1432s ][ Queries: 12 (0.0370s) ][ GZIP on ]