AUFS - Perfect replacement for Unionfs

What features/apps/bugfixes needed in a future Puppy
Message
Author
User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

AUFS - Perfect replacement for Unionfs

#1 Post by MU »

Trouble with Unionfs is quite common, 2 threads concerning this are:
http://murga-linux.com/puppy/viewtopic.php?t=27407
and
http://murga-linux.com/puppy/viewtopic. ... 675#203675

Minisys Developer Stefan now created a penetration test, that runs several million file-operations.
It simulates approx. 4 years of typical usage.

When we use AUFS instead of Unionfs, we have NO errors at all :!:

Myself, I run Muppy since 9 hours now, and the most problematic firefox still runs without problems.

This fixed also the issue, that copying thousand files can cause requests, if they shall be overwritten (though they do not exist yet). Strangely, this also happens in Suse 10.0, that does not use unionfs (so is there another bug involved?).

Stefan sais: with AUFS everything now is PERFECT, and now we can recommend Muppy for crital usage as server-system.

I strongly recommend, to think about replacing Unionfs with AUFS.

If you have concerns about speed:
Aufs might be around 3% slower than unionfs.
So you will not recognize it.
This value must not be seen as "exact" value, as buffering and other influence will give wrong results here.

Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#2 Post by kirk »

Mark,

Puppy4 with the 2.6.25 kernel uses a newer version of Unionfs. Unionfs has had a lot of changes since it was included in the -mm tree. The test you ran on Muppy, can that easily be ran on Puppy4? (or 4.1alpha2 in a couple days)

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#3 Post by BarryK »

I dropped aufs awhile back (sometime late in 2007 I think) and returned to unionfs. I did mention why in the blog, I think. I can't recall exact details right now, but there was a situation where aufs failed to create the union, and unionfs succeeded.
I think it was when I was building the humongous Puppy, which has the pup_xxx.sfs inside the initrd.gz. I'm guessing a bit, but I think that I had to move the pup_xxx.sfs to /pup_rw before making the union, but as the pup_xxx.sfs itself forms one layer of the union (/pup_ro1 or /pup_ro2), aufs threw a fit and spat out an error that it couldn't do it. Unionfs on the other hand, just went ahead and did it, and and it worked fine.

That was the moment when I decided to go back to unionfs. I had already been considering doing so because unionfs is now in the -mm branch of the kernel, and is also being very actively improved, some interesting ideas they are working on.
[url]https://bkhome.org/news/[/url]

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#4 Post by MU »

Kirk:
Stefan will see, if he can provide a package with such a test.
Then it could be run on any system that uses Minisys (which is easily installed/removed, as all is in just one folder).
We are very busy atm, so I can not promise a release-date.

Barry:
Thanks for the info, why you switched back to unionfs.
Depending on the results we get with Puppy 4, it might be important though, to raise the unionfs issue again.
Knoppix and Slax moved to Aufs, because it just works, while there were problems with Unionfs.
I have read, that already 2007, the unionfs maintainers tried to get it in the Kernel, though it was buggy, and aufs was known to be stable.
So just the fact, that it is in the Kernel now, does not give me trust. Tests are required.

If aufs is incompatible with the humugenious initrd, unionfs might be used for this case only, aufs in all other cases.

The problems we had with Unionfs in Muppy, made Muppy almost unusable as server system for usage in a company.
Just with some dirty hacks like a second savefile (not layered) we could tweak it so far, that we could use it for our special purpose (Minisys).
But that is no general solution for other programs.

Also Firefox and Wine are very popular, so it was very bad, that they ran unstable using unionfs, when mounted from a huge SFS.

So I think this issue is not trivial, and now that we know, that aufs works great, it would be important to check, if Dingo has similar issues if it is used in an extended environment (using huge SFS extensions with Firefox ans so on).

Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

raffy
Posts: 4798
Joined: Wed 25 May 2005, 12:20
Location: Manila

humongous initrd

#5 Post by raffy »

Mark, I've been watching with interest Muppy-Embryo, which can be used as base of humongous initrd. The key to using it is in the init scripts that load pup_xxx.sfs (and possibly zdrv_xxx.sfs too) from /, see the humongous initrd creation instructions by Barry here:
http://murga-linux.com/puppy/viewtopic.php?t=23615
(I reproduced it there because the blog URL was no longer working.)

Barry's notes about AUFS problems are in http://puppylinux.com/news/news216-217.htm:[quote]I've gone back to Aufs, as Unionfs2 was causing applications to crash. The Unionfs developers have responded to my bug report by updatng the patch for the 2.6.21.5 kernel, however it is an invasive patch that requires a complete kernel recompile, and I would have to go through the entire all-day process again.[/quote]
Puppy user since Oct 2004. Want FreeOffice? [url=http://puppylinux.info/topic/freeoffice-2012-sfs]Get the sfs (English only)[/url].

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#6 Post by sunburnt »

The experience that NathanF had with AUFS showed the problem with it & UnionFS.
A layered file system requires lots of memory to work good & fast.
At the very least, reducing the size of the file system that is unioned helps lots.
Partly why Puppy-1xx worked so well & on older & smaller capability PCs.
It only unioned part of the file system, not the entire root of the FS.
And it also would Union & unUnion SFS files with no trouble what so ever!.
I haven't worked with this in over a year, but I'm guessing newer Puppies still won't do it.
The correct solution is to do away with unioning the FS, there's other ways to do it...
First I thought a union FS was great, now I see it as complex, troublesome, & unnecessary.
A Full install has none of these problems & is wicked fast, & Server certified.
Last edited by sunburnt on Mon 09 Jun 2008, 04:41, edited 1 time in total.

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#7 Post by BarryK »

sunburnt,
yes, perhaps a writable compressed f.s. like zfs would be a better proposition later on. Like you, I have often wondered what have we got into with this layered f.s. on '/'.
[url]https://bkhome.org/news/[/url]

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#8 Post by sunburnt »

Barry; Storage space isn't anything to worry about, even with USB flash.
A compressed file system like ZFS isn't really needed & isn't more secure.
The best part about Squash files is that they're static, so are fairly reliable.
A full install is far more open to hacking & infection by viruses.

How about breaking the single large main Puppy SFS file into smaller parts?
Small SFS files can be rebuilt quickly, appending's the same, big file or small.

initramfs has the writable dirs. & links to the static dirs. that're in SFS files.
initramfs is /, & the real files can be of any type & arranged anyway you want.

pup500_base.sfs (console) is all static dirs. in "/" = /bin, /dev, /lib, & /sbin,
pup500_X11R7.sfs (X), pup500_usr.sfs (apps.), & pup500_zdrv.sfs.
A save file could be substituted for any of these partial SFS files of course.
The idea though is to quickly install directly to the SFS files by appending,
or to uninstall apps. a new SFS file's needed, so smaller SFS files are better.
Persistance for /etc & other dirs. is a small save file written at shutdown.

I suggested to MU that extra SFS files could be mounted but not unioned.
He thought it was a good idea, so I've been working on a setup for it all.
I wrote a SFS file loader, it adds to & removes from $PATH for the SFS files.

It seems to me that /bin & /sbin is moot in Puppy as everyone is root user,
so combining all the /bin & /sbin dirs. would save on all the paths in $PATH.
And combining all the /lib dirs. would save on paths in $LD_LIBRARY_PATH.

I posted an idea in "Truly off-topic" for an "advanced multi. dir. link".

Two ideas presented here really, rearranging Puppy & attaching SFS files.
If any of this interests you, let me know... Terry

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#9 Post by BarryK »

sunburnt,
One of the fundamental reasons that we went for a layered f.s. on '/' was so that everything becomes writable. We considered this was good for installing packages for example.

Do you have any way worked out, if someone wants to install a package that goes into /usr for example, but you have a sfs file mounted on /usr -- do you envisage a "install" script that doesn't install directly, perhaps modifies the sfs files -- but then, they would have to be remounted. Yes, this looks interesting... I would like to know your further thoughts on this, maybe by pm.
(or another thread if others are interested in following the argument and maybe trying some experiments)

I think you mentioned also, why even do a switch_root, just stay in the initramfs. That could be a simple and very reliable configuration. This basic idea has occurred to me in the past, but I was not sure how to do it, so never followed it up.
[url]https://bkhome.org/news/[/url]

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#10 Post by BarryK »

sunburnt,
A further note, one thing that I considered awhile back was something called 'cowloop'. This makes a directory with a read-only f.s. mounted on it writable -- to a special file.
[url]https://bkhome.org/news/[/url]

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#11 Post by kirk »

One of the fundamental reasons that we went for a layered f.s. on '/' was so that everything becomes writable. We considered this was good for installing packages for example.
Amen to that! I sure don't miss that from Puppy 1.x. Also, having all of the changes in one file (pup_save) is a good thing, for many reasons. The only real problem we have now is not being able to unmount the loops at shutdown.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#12 Post by sunburnt »

kirk; I think the unmount problem is indicative of the union problem (maybe).
AUFS may have recognized the unioning on / problem & refused to do it, but no error!
UnionFS allows it, but no warning about crashes when using unionctl on the branches.
I beleave I read a UnionFS warning about unioning on / & the probem with using unionctl.
I think something shouldn't be inside the union, maybe /mnt, or maybe something else.

Barry; Sorry for my delayed answer... My father-in-law passed away & it's been a hectic week.
But then it also gave everyone else the chance to pop-in a word here.
As no one said anything, I'll PM you so as not to totally mess up Mark's forum post.
Too late... (sorry Mark) :roll:

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#13 Post by MU »

Too late... (sorry Mark)
no prob.
I find it good, that the unionfs-issues are in discussion again.
We meanwhile checked the server and our other AUFS installations, and they ran very reliable.
I now definately will continue using AUFS.

Concerning the test-scripts I can't tell a releasedate yet, as we have more than 10-12 hours of work every day at moment.

Myself I will have next week to upgrade some desktop-tools, then again install servers.

Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#14 Post by sunburnt »

MU; If you want an idea as to my install to SFS files...
It's mount the appended or newly made SFS file over the original SFS file's mount point.
So the same SFS file that's been appended to shows its contense in place of the orignal.
A newly made SFS file is also mounted-over the orignal in the same manner... Both work !
This way there's no problem with trying to unmount an SFS file that's being used !
The new SFS file & it's mount-over work fine until shutdown, next boot uses the new one.
A shutdown script renames a newly made SFS file, but an appended one boots as it is.

I also suggested ideas for a new Puppy layout that has initramfs as /, & multi. SFS files.
It goes with the idea above "install to SFS with no image file", so no large Save file's needed.
I also suggested mounting extra SFS files but not unioning them, as you & I discussed.

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#15 Post by kirk »

AUFS may have recognized the unioning on / problem & refused to do it, but no error!
AUFS will do that. Barry used it in 2.17 and it was optional in 3.x. The problem is when you try to unmount the loops they're always busy. I noticed the AUFS author hasn't been sitting still ether. Lots of new work on AUFS.

I'm beginning to doubt Unionfs too. I've built with T2 a few times in the past using Dingo alpha2 with no extra sfs file and had no problems. Probably had the computer running for a month. I've been using Dingo 3.99 with a large sfs file (fatdog) to build with T2 again. Several times I've had it seem to run out of memory. I say seem to, because free shows RAM available but when you try to execute somthing it says there's no space left and nothing much will work. Reboot fixes it. May not be a Unionfs problem, have to do more checking.

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#16 Post by MU »

Kirk, yes, your "no space left" problem is a known issue with unionfs:
http://murga-linux.com/puppy/viewtopic.php?t=29707
Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#17 Post by kirk »

It's more than just a white-out problem. I removed my big sfs file (207MB) and now it's been compiling for 14 hours or so with no problems. Before I removed my big sfs file, I tried killing any unneeded processes, even the tray applets. Nothing contained in the big sfs file was running. Still died. I'm planning to make a puplet from 4.1, just compiled the latest kernel with squash-lzma and AUFS. Once I get it together I'll try a t2 build and see how it goes.

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#18 Post by BarryK »

kirk,
hah hah, I see that you have already noticed my blog post!
Thanks for the info about the splice patch, I'll see if I can find it.
[url]https://bkhome.org/news/[/url]

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#19 Post by sunburnt »

Barry, & everyone else...; What thoughts have you on eliminating the unions in Puppy all together?

Puppy running / in initramfs with "writable" / dirs. in /, & "static" / dirs. would be links to partition(s), image file(s), or sfs file(s) on HD, CD-DVD, or USB.
The layout would look something like this:

/bin = Link
/dev = Dir.
/etc = Dir.
/lib = Link
/mnt = Dir.
/opt = Link
/proc = Dir.
/root = Dir.
/sbin = Link
/sys = Dir.
/tmp = Dir.
/usr = Link
/var = Dir.

A few of the / Dirs. may need to be Links instead. /usr may be a Dir. with Links in it.
/root/my-applications & /root/my-documents could be links to dirs. (with new shorter names) on a storage device.

Many different OS layouts can be setup this way, & unions could be used or not used.
Puppy has gotten rather complex & this may be a good way to simplify it, some might call it more complex yet.
But if the sfs files are fairly small, then they can be installed to quickly & no union's needed.
All of this is straight forward & easily do able with no doubts at all about it working.

What do you think guys?

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#20 Post by BarryK »

sunburnt,
Yes, I agree that it is definitely worth looking into. Not for 4.1 though, as I don't want to make any further architectural changes for that.
But afterward, an experimental release without any layered f.s. is definitely high on my list to look into. Of course, if anyone else wants to try building something right now, go for it.
[url]https://bkhome.org/news/[/url]

Post Reply