Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 22 Oct 2014, 18:01
All times are UTC - 4
 Forum index » Taking the Puppy out for a walk » Suggestions
AUFS - Perfect replacement for Unionfs
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [27 Posts]   Goto page: 1, 2 Next
Author Message
MU


Joined: 24 Aug 2005
Posts: 13642
Location: Karlsruhe, Germany

PostPosted: Wed 04 Jun 2008, 13:19    Post subject:  AUFS - Perfect replacement for Unionfs
Subject description: penetration test gives 100% quality seal
 

Trouble with Unionfs is quite common, 2 threads concerning this are:
http://murga-linux.com/puppy/viewtopic.php?t=27407
and
http://murga-linux.com/puppy/viewtopic.php?p=203675#203675

Minisys Developer Stefan now created a penetration test, that runs several million file-operations.
It simulates approx. 4 years of typical usage.

When we use AUFS instead of Unionfs, we have NO errors at all Exclamation

Myself, I run Muppy since 9 hours now, and the most problematic firefox still runs without problems.

This fixed also the issue, that copying thousand files can cause requests, if they shall be overwritten (though they do not exist yet). Strangely, this also happens in Suse 10.0, that does not use unionfs (so is there another bug involved?).

Stefan sais: with AUFS everything now is PERFECT, and now we can recommend Muppy for crital usage as server-system.

I strongly recommend, to think about replacing Unionfs with AUFS.

If you have concerns about speed:
Aufs might be around 3% slower than unionfs.
So you will not recognize it.
This value must not be seen as "exact" value, as buffering and other influence will give wrong results here.

Mark

_________________
my recommended links
Back to top
View user's profile Send private message Visit poster's website 
kirk

Joined: 11 Nov 2005
Posts: 1419
Location: florida

PostPosted: Wed 04 Jun 2008, 21:45    Post subject:  

Mark,

Puppy4 with the 2.6.25 kernel uses a newer version of Unionfs. Unionfs has had a lot of changes since it was included in the -mm tree. The test you ran on Muppy, can that easily be ran on Puppy4? (or 4.1alpha2 in a couple days)
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7047
Location: Perth, Western Australia

PostPosted: Thu 05 Jun 2008, 07:17    Post subject:  

I dropped aufs awhile back (sometime late in 2007 I think) and returned to unionfs. I did mention why in the blog, I think. I can't recall exact details right now, but there was a situation where aufs failed to create the union, and unionfs succeeded.
I think it was when I was building the humongous Puppy, which has the pup_xxx.sfs inside the initrd.gz. I'm guessing a bit, but I think that I had to move the pup_xxx.sfs to /pup_rw before making the union, but as the pup_xxx.sfs itself forms one layer of the union (/pup_ro1 or /pup_ro2), aufs threw a fit and spat out an error that it couldn't do it. Unionfs on the other hand, just went ahead and did it, and and it worked fine.

That was the moment when I decided to go back to unionfs. I had already been considering doing so because unionfs is now in the -mm branch of the kernel, and is also being very actively improved, some interesting ideas they are working on.

_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
MU


Joined: 24 Aug 2005
Posts: 13642
Location: Karlsruhe, Germany

PostPosted: Thu 05 Jun 2008, 08:22    Post subject:  

Kirk:
Stefan will see, if he can provide a package with such a test.
Then it could be run on any system that uses Minisys (which is easily installed/removed, as all is in just one folder).
We are very busy atm, so I can not promise a release-date.

Barry:
Thanks for the info, why you switched back to unionfs.
Depending on the results we get with Puppy 4, it might be important though, to raise the unionfs issue again.
Knoppix and Slax moved to Aufs, because it just works, while there were problems with Unionfs.
I have read, that already 2007, the unionfs maintainers tried to get it in the Kernel, though it was buggy, and aufs was known to be stable.
So just the fact, that it is in the Kernel now, does not give me trust. Tests are required.

If aufs is incompatible with the humugenious initrd, unionfs might be used for this case only, aufs in all other cases.

The problems we had with Unionfs in Muppy, made Muppy almost unusable as server system for usage in a company.
Just with some dirty hacks like a second savefile (not layered) we could tweak it so far, that we could use it for our special purpose (Minisys).
But that is no general solution for other programs.

Also Firefox and Wine are very popular, so it was very bad, that they ran unstable using unionfs, when mounted from a huge SFS.

So I think this issue is not trivial, and now that we know, that aufs works great, it would be important to check, if Dingo has similar issues if it is used in an extended environment (using huge SFS extensions with Firefox ans so on).

Mark

_________________
my recommended links
Back to top
View user's profile Send private message Visit poster's website 
raffy

Joined: 25 May 2005
Posts: 4776
Location: Manila

PostPosted: Thu 05 Jun 2008, 19:26    Post subject: humongous initrd  

Mark, I've been watching with interest Muppy-Embryo, which can be used as base of humongous initrd. The key to using it is in the init scripts that load pup_xxx.sfs (and possibly zdrv_xxx.sfs too) from /, see the humongous initrd creation instructions by Barry here:
http://murga-linux.com/puppy/viewtopic.php?t=23615
(I reproduced it there because the blog URL was no longer working.)

Barry's notes about AUFS problems are in http://puppylinux.com/news/news216-217.htm:
Quote:
I've gone back to Aufs, as Unionfs2 was causing applications to crash. The Unionfs developers have responded to my bug report by updatng the patch for the 2.6.21.5 kernel, however it is an invasive patch that requires a complete kernel recompile, and I would have to go through the entire all-day process again.

_________________
Puppy user since Oct 2004. Want FreeOffice? Get the sfs (English only).
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sun 08 Jun 2008, 14:33    Post subject:  

The experience that NathanF had with AUFS showed the problem with it & UnionFS.
A layered file system requires lots of memory to work good & fast.
At the very least, reducing the size of the file system that is unioned helps lots.
Partly why Puppy-1xx worked so well & on older & smaller capability PCs.
It only unioned part of the file system, not the entire root of the FS.
And it also would Union & unUnion SFS files with no trouble what so ever!.
I haven't worked with this in over a year, but I'm guessing newer Puppies still won't do it.
The correct solution is to do away with unioning the FS, there's other ways to do it...
First I thought a union FS was great, now I see it as complex, troublesome, & unnecessary.
A Full install has none of these problems & is wicked fast, & Server certified.

Last edited by sunburnt on Mon 09 Jun 2008, 00:41; edited 1 time in total
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7047
Location: Perth, Western Australia

PostPosted: Sun 08 Jun 2008, 21:05    Post subject:  

sunburnt,
yes, perhaps a writable compressed f.s. like zfs would be a better proposition later on. Like you, I have often wondered what have we got into with this layered f.s. on '/'.

_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Sun 08 Jun 2008, 22:26    Post subject:  

Barry; Storage space isn't anything to worry about, even with USB flash.
A compressed file system like ZFS isn't really needed & isn't more secure.
The best part about Squash files is that they're static, so are fairly reliable.
A full install is far more open to hacking & infection by viruses.

How about breaking the single large main Puppy SFS file into smaller parts?
Small SFS files can be rebuilt quickly, appending's the same, big file or small.

initramfs has the writable dirs. & links to the static dirs. that're in SFS files.
initramfs is /, & the real files can be of any type & arranged anyway you want.

pup500_base.sfs (console) is all static dirs. in "/" = /bin, /dev, /lib, & /sbin,
pup500_X11R7.sfs (X), pup500_usr.sfs (apps.), & pup500_zdrv.sfs.
A save file could be substituted for any of these partial SFS files of course.
The idea though is to quickly install directly to the SFS files by appending,
or to uninstall apps. a new SFS file's needed, so smaller SFS files are better.
Persistance for /etc & other dirs. is a small save file written at shutdown.

I suggested to MU that extra SFS files could be mounted but not unioned.
He thought it was a good idea, so I've been working on a setup for it all.
I wrote a SFS file loader, it adds to & removes from $PATH for the SFS files.

It seems to me that /bin & /sbin is moot in Puppy as everyone is root user,
so combining all the /bin & /sbin dirs. would save on all the paths in $PATH.
And combining all the /lib dirs. would save on paths in $LD_LIBRARY_PATH.

I posted an idea in "Truly off-topic" for an "advanced multi. dir. link".

Two ideas presented here really, rearranging Puppy & attaching SFS files.
If any of this interests you, let me know... Terry
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7047
Location: Perth, Western Australia

PostPosted: Mon 09 Jun 2008, 19:58    Post subject:  

sunburnt,
One of the fundamental reasons that we went for a layered f.s. on '/' was so that everything becomes writable. We considered this was good for installing packages for example.

Do you have any way worked out, if someone wants to install a package that goes into /usr for example, but you have a sfs file mounted on /usr -- do you envisage a "install" script that doesn't install directly, perhaps modifies the sfs files -- but then, they would have to be remounted. Yes, this looks interesting... I would like to know your further thoughts on this, maybe by pm.
(or another thread if others are interested in following the argument and maybe trying some experiments)

I think you mentioned also, why even do a switch_root, just stay in the initramfs. That could be a simple and very reliable configuration. This basic idea has occurred to me in the past, but I was not sure how to do it, so never followed it up.

_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7047
Location: Perth, Western Australia

PostPosted: Mon 09 Jun 2008, 20:13    Post subject:  

sunburnt,
A further note, one thing that I considered awhile back was something called 'cowloop'. This makes a directory with a read-only f.s. mounted on it writable -- to a special file.

_________________
http://bkhome.org/news/
Back to top
View user's profile Send private message Visit poster's website 
kirk

Joined: 11 Nov 2005
Posts: 1419
Location: florida

PostPosted: Mon 09 Jun 2008, 23:32    Post subject:  

Quote:
One of the fundamental reasons that we went for a layered f.s. on '/' was so that everything becomes writable. We considered this was good for installing packages for example.


Amen to that! I sure don't miss that from Puppy 1.x. Also, having all of the changes in one file (pup_save) is a good thing, for many reasons. The only real problem we have now is not being able to unmount the loops at shutdown.
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Thu 12 Jun 2008, 17:50    Post subject:  

kirk; I think the unmount problem is indicative of the union problem (maybe).
AUFS may have recognized the unioning on / problem & refused to do it, but no error!
UnionFS allows it, but no warning about crashes when using unionctl on the branches.
I beleave I read a UnionFS warning about unioning on / & the probem with using unionctl.
I think something shouldn't be inside the union, maybe /mnt, or maybe something else.

Barry; Sorry for my delayed answer... My father-in-law passed away & it's been a hectic week.
But then it also gave everyone else the chance to pop-in a word here.
As no one said anything, I'll PM you so as not to totally mess up Mark's forum post.
Too late... (sorry Mark) Rolling Eyes
Back to top
View user's profile Send private message 
MU


Joined: 24 Aug 2005
Posts: 13642
Location: Karlsruhe, Germany

PostPosted: Thu 12 Jun 2008, 19:23    Post subject:  

Quote:
Too late... (sorry Mark)


no prob.
I find it good, that the unionfs-issues are in discussion again.
We meanwhile checked the server and our other AUFS installations, and they ran very reliable.
I now definately will continue using AUFS.

Concerning the test-scripts I can't tell a releasedate yet, as we have more than 10-12 hours of work every day at moment.

Myself I will have next week to upgrade some desktop-tools, then again install servers.

Mark

_________________
my recommended links
Back to top
View user's profile Send private message Visit poster's website 
sunburnt


Joined: 08 Jun 2005
Posts: 5037
Location: Arizona, U.S.A.

PostPosted: Thu 12 Jun 2008, 20:23    Post subject:  

MU; If you want an idea as to my install to SFS files...
It's mount the appended or newly made SFS file over the original SFS file's mount point.
So the same SFS file that's been appended to shows its contense in place of the orignal.
A newly made SFS file is also mounted-over the orignal in the same manner... Both work !
This way there's no problem with trying to unmount an SFS file that's being used !
The new SFS file & it's mount-over work fine until shutdown, next boot uses the new one.
A shutdown script renames a newly made SFS file, but an appended one boots as it is.

I also suggested ideas for a new Puppy layout that has initramfs as /, & multi. SFS files.
It goes with the idea above "install to SFS with no image file", so no large Save file's needed.
I also suggested mounting extra SFS files but not unioning them, as you & I discussed.
Back to top
View user's profile Send private message 
kirk

Joined: 11 Nov 2005
Posts: 1419
Location: florida

PostPosted: Thu 12 Jun 2008, 22:28    Post subject:  

Quote:
AUFS may have recognized the unioning on / problem & refused to do it, but no error!


AUFS will do that. Barry used it in 2.17 and it was optional in 3.x. The problem is when you try to unmount the loops they're always busy. I noticed the AUFS author hasn't been sitting still ether. Lots of new work on AUFS.

I'm beginning to doubt Unionfs too. I've built with T2 a few times in the past using Dingo alpha2 with no extra sfs file and had no problems. Probably had the computer running for a month. I've been using Dingo 3.99 with a large sfs file (fatdog) to build with T2 again. Several times I've had it seem to run out of memory. I say seem to, because free shows RAM available but when you try to execute somthing it says there's no space left and nothing much will work. Reboot fixes it. May not be a Unionfs problem, have to do more checking.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [27 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Taking the Puppy out for a walk » Suggestions
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0935s ][ Queries: 11 (0.0041s) ][ GZIP on ]