using SFS with full installs

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

using SFS with full installs

#1 Post by mikeb »

After recently being introduced by amigo to a method of running an application in a layered fashion without touching the base file system my mind wandered into potential uses.

One then popped in and that was as a way of using sfs files with a full install which is a common request and usually involves extracting, or find a pet , or elaborate methods of linking, or other such methods.

Amigo's chrooted system seems to fit the bill nicely and a quick test in a virtual machine looks promising...I used an Xchat sfs and it ran normally.

Basic overview...
make folders... 'union', 'write' and 'app' somewhere.
mount the sfs to app.
make a fuse unionfs to 'union' using 'write', 'app' and '/'
fakechroot or chroot union /path/to/app
done.
In effect a unionfs is made using the full install as the bottom read only layer, the mounted sfs is next and a read/write layer on top.

It seems simple , does not care where the layers are located which is handy if confusing to me, and indeed solves a problem.

Don't expect replies...just posting for posterity and to keep notes.

mike

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#2 Post by sunburnt »

Mike; I think the newer Puppies will union an SFS file over a Full install.

Back in the early days I wondered why this didn`t work. Probably unionfs...
But I use Frugal installs so this wasn`t a priority.
Thinking about it, what`s the difference between unioning a mounted SFS file or /.?
In both cases you union on another dir. and chroot into it.
.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#3 Post by mikeb »

Mike; I think the newer Puppies will union an SFS file over a Full install.
hmm maybe so...but I failed with aufs a while ago. Would be nice to know one way or the other though seems odd if the question still gets asked.

What I could do was union to a folder of a full install ..eg /usr..

Well I have a full install of lucid running at the moment..I will test that...if anyone has anything newer would be of interest.

mike

User avatar
L18L
Posts: 3479
Joined: Sat 19 Jun 2010, 18:56
Location: www.eussenheim.de/

using SFS with full installs

#4 Post by L18L »

mikeb wrote:...if anyone has anything newer would be of interest.
Quirky 6.0
log of loading a devx sfs....

# make
bash: /usr/bin/make: Datei oder Verzeichnis nicht gefunden
# LANGUAGE=en sfs_load
sfs_load: Your Puppy is running in a mode without extra SFS support. But this program can install SFSes using pseudo-PET. It is not exactly the same as loading SFS but works well in most cases.
WARNING: Using this under 'PUPMODE=2' is EXPERIMENTAL.

(gtkdialog4:12646): GLib-GObject-WARNING **: gsignal.c:3081: signal name `depressed' is invalid for instance `0x8507878'
/usr/sbin/sfs_load: line 774: 12586 Terminated $GTKDIALOG -p DIALOG -c > /dev/null
cp: cannot create symbolic link `/usr/include/gphoto2/gphoto2/gphoto2': No such file or directory


gtk-update-icon-cache: Cache file created successfully.
Generating /root/.icewm/menu...
Generating /root/.jwmrc...
/usr/sbin/sfs_load: line 369: 12958 Terminated $GTKDIALOG -p DIALOG -c > /dev/null
/usr/sbin/sfs_load: line 2502: 10050 Terminated $GTKDIALOG -p DIALOG -c > /dev/null
sfs_load: --stop
#
#
# make
make: *** No targets specified and no makefile found. Stop.
#

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#5 Post by mikeb »

Thanks for the input.

So that just demonstrates the extracting of an sfs to a full install...which is what i assume is normally offered. The idea is not to add to or interfere with the full install...plus its a bit on the clumsy side.

Wanted some confirmation that a sfs can be somehow unioned to / for a full install.

tested in lucid...union of sfs and / to /...well that failed

tried using a aufs mount in a similar fashion...ie union to 'union' including '/' as the bottom layer...failed also. So really in this case aufs refuses to include '/' as a layer ...question is can later kernels do this.?

By the way has no problem adding the sfs to /usr with
busybox mount -t aufs -o udba=reval,diropq=w,dirs=/root/app=ro:/usr=ro unionfs /usr
in other words you can union to one of the layers but obviously no use for the hundreds of sfs files built to run on '/'

mike

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#6 Post by mikeb »

busybox mount -t aufs -o udba=reval,diropq=w,dirs=/=rw:/root/app=ro unionfs /
where /root/app has had the sfs mounted already.

that sort of thing...but apart from the upsidedown layering would / be ok for the read/write layer (something would have to be ) ? Also if it did succeed its unlikely it could be unmounted...my /usr mount certainly would not.

mike

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#7 Post by sunburnt »

You can`t union on / of course, so it would have to be a boot process thing probably, not an add-on.

Like Puppy, make the initial union mount on /U with / as the top Save layer and chroot into /U.
Now you can add and subtract SFS files with sfs_load. ( I wish it had been made to do any image file )

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#8 Post by mikeb »

You can`t union on / of course, so it would have to be a boot process thing probably, not an add-on.
your first post seemed to suggest otherwise...not to worry..I double checked anyway.
Like Puppy, make the initial union mount on /U with / as the top Save layer and chroot into /U.
Now you can add and subtract SFS files with sfs_load. ( I wish it had been made to do any image file )
a layered variation...not the purpose here.

This is a way of using an sfs with a standard (not just puppy really) full install. It appears to work... seems simple enough ...in fact the only fiddly bit I can see is finding a way to create a menu entry that would be for a universal script rather than a one script per app situation.... Also note 'app' can be a normal file tree, or an image file or whatever...

here's the steps in bash... and unionfs-fuse attached for anyone interested in testing it out....used xchat as example sfs. and testing in /root . note how the unionfs-fuse command wants full paths. originally fakechroot was used though it seems to work with just chroot...I am sure there are reasons to use the latter but this seems ok for testing purposes...means the only binary to add is unionfs-fuse making it applicable to any puppy.

Code: Select all

mkdir -p union
mkdir -p app
mkdir -p write

mount -o loop xchat.sfs app
unionfs-fuse -o nonempty -o allow_root -o cow /root/write=RW:/root/app=RO:/=RO /root/union

chroot union xchat
mike
Attachments
unionfs-fuse.tar.gz
place in /usr/bin
(16.5 KiB) Downloaded 604 times

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#9 Post by sunburnt »

Yes, I`ve made union versions of my AppPkg.

My early post I ment unioning a SFS file or / ( as layers ), not unioning on / ( Sorry...).

If you mount on a dir that`s not empty it`s contents disappears.
unionfs-fuse will do this, your example uses the option but it`s not needed as the dir is empty.

I found I had to bind mount some of Linux`s mounted dirs to get it to work ( as amigo said ).
I also found that mhddfs worked very easily with no fussing or binding. It uses libfuse.
But as amigo pointed out, it`s not made for this purpose. But if it works... Go with it.!

You can make a chroot env. after booting and add/remove SFS files.
Each of the SFS apps running in the chroot, the rest of the Full install running outside.
I`ve never tried to add/remove SFS files using unionfs-fuse, but I`d think it`d do it.
.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#10 Post by jamesbond »

mikeb wrote:busybox mount -t aufs -o udba=reval,diropq=w,dirs=/=rw:/root/app=ro unionfs /
where /root/app has had the sfs mounted already.
Great idea but wrong implementation.
1. Aufs will fail because "/" is already aufs (aufs can't have another aufs as its branch, unless you force it to do so in kernel-config - but I reckon this isn't a good idea anyway).
2. But you can insert (or delete) another branch into existing aufs mounts (including /): mount -t aufs -o append:/root/app=ro unionfs /.
This line, in fact, is the magic how various implementations of "SFS-On-The-Fly" works.
that sort of thing...but apart from the upsidedown layering would / be ok for the read/write layer (something would have to be ) ? Also if it did succeed its unlikely it could be unmounted...my /usr mount certainly would not.
Yes it can be unmounted if you can ensure that no applications are holding a file on the layer you want to unmount. Aufs also has the option to *remove* layers, and once the layer is removed you can unmount the sfs.
3. mount -t aufs -o del:/root/app unionfs / (and then umount /root/app)

Unionfs-fuse doesn't have the same limitation, and it can be used as non-root. But the "chroot" needs root privilege anyway (unless if you use fakechroot - but fakechroot isn't 100% foolproof). Whichever works, I suppose.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#11 Post by amigo »

I used fakechroot with unionfs-fuse with a mind to non-root users. unionfs-fuse has a 'chroot' option but I have not played with it all. Some of you youngsters might see how that goes.

A full implementation with mount-binded dirs would still need 'bindfs' which is another FUSE tool which implements 'mount --bind' capability.

Yes, aufs will restrict you -and rightfully so, I believe. The title says full install, though. Does that really mean there is no aufs being used?

If not, then you can do whatever you like with either aufs or uniofs-fuse. The idea with the chrome app using unionfs-fuse/fakechroot was to come up with a solution att eh app level, so that it could be used anywhere and would self-load. The effect of sandboxing by using chroot is just an extra possible benefit.

You could use unionfs-fuse to union '/' if you wanted to. It's just that that has to be done from within an initrd -before that 'real', but layered, root is mounted. No user would ever be aware of any I/O latency, since it is still faster than disk I/O.

The last thing is security. Yeah, there are ways to break out of a chroot, but this implementation minimizes the possibility of that since the whole chroot is done in one command. Compare with setting up a union manually, and then chrooting in there with:
'chroot union-dir /bin/bash'
Now, that one is 'dangerous', if you like. But the end result can be no worse than allowing a mortal to simply run as root...

You can easily (well, here at least) use a layer for a shared /usr or similar setup. Or you can create a completely different distro all under a subdir somewhere and just chroot in there for a completely different runtime. In such a case, you should probably unmount both /proc and /sys before entering the chroot and then re-mount them within the chroot.

Also, I want to stress again, mhddfs is fundamentally different from aufs or unionfs-fuse. With mhddfs, *all* layers are presumed to be writable. mhddfs works like a distributed volume. When new files are added you cannot control directly which part of the volume they will be written to. It will always write to the 'layer' which has the most free space. It does have a simpler syntax, but is not really the right tool for this kind of application.

In the script for the chrome app there are notes about the mount locations, which you said you were still confused about. Since I have a true multi-user system I pay attention to these things... I avoided using /tmp completely because anything there is accessible by anyone. Our mortal wants a bit of privacy. So, the only other place we can assume that a mortal has write privileges is under his/her $HOME directory(see POSIX specs). What I did for the chrome app was create a dir under their $HOME, but not directly under HOME. We create a dir like would be had for user configuration files for any other program, and then do our mounting under there -as well as putting the write layer there -that keeps everything under one dir. Sine the chrome thing was packaged as a Rox AppDir, I used $HOME/Choices as the top subdir. But one could use:
$HOME/.config/program-name
Avoid other locations for best compatibility and to keep non-angry
users.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#12 Post by mikeb »

Great idea but wrong implementation.
1. Aufs will fail because "/" is already aufs (aufs can't have another aufs as its branch, unless you force it to do so in kernel-config - but I reckon this isn't a good idea anyway).
2. But you can insert (or delete) another branch into existing aufs mounts (including /): mount -t aufs -o append:/root/app=ro unionfs /.
This line, in fact, is the magic how various implementations of "SFS-On-The-Fly" works.
That was a semi joke funny as it was suggested that the union could be made from / to / .... just to confirm that this need has not already been satisfied with some kernel update.

I have a sfs on the fly script that's a fraction of the size everyone else is using and layers in the correct order. It also unmounts on the fly. ..fast and reliable with no problems with files being hidden. Based on the slax way rather than puppy.
This is not really a post about aufs...that was a side track...and multiple sfs layering for frugal distros is something I have been using for years...so lets drop that one. :)

nonempty I assume it for the possibly persistant write layer.

chroot on puppy since its running as root....fake if not...that clarifies that one.

My tests yesterday needed no bind mounting ...just ran as is.
....that was with a full install... perhaps doing it on an already unioned filesystem is another matter.

@amigo... thought you might drop in at some point...just as I was writing this in fact :D
I started this as your union-fuse method seemed a neat way of using sfs when a full install is involved... and yes no aufs whatsoever. As mentioned no binding seems necessary ... just need a neat way of launching required.
I was not thinking particularly in terms of security or privacy
but living in home and fakechroot would indeed do the job. Otherwise for puppy and its open legs, /tmp could be used for a total clean up after use. My only thought was a persistant 'write' would be more convenient.

Apart from utilising sfs files it could be used for running software over a network.

Its really about protecting those vunerable full installs and speed. Something to chew on so a fresh post with your info seemed appropriate.... someone might take up the idea.

mike

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#13 Post by amigo »

Well, if you are not using aufs otherwise, it would certainly do the job and fits well if you have/want a managed system. I like the idea of individual chroots and it's a good selling point.
Oh, about fakechroot, if your /bin/sh points to busybox you'll need to patch the getopt out of the wrapper script.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#14 Post by mikeb »

Oh, about fakechroot, if your /bin/sh points to busybox you'll need to patch the getopt out of the wrapper script
yes discovered that one :D

mike

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#15 Post by amigo »

I thought I had patched it out, but I sure don't find that right now.... If you did, be a good fellow and post a patch.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#16 Post by mikeb »

Patched and in a zip...well I did it on Windows :D
Will redo later...

note for users... fakechroot -- chroot {executable}

edit...now a tar.gz.... I have complied....

mike
Attachments
fakechroot.tar.gz
(41.07 KiB) Downloaded 382 times
Last edited by mikeb on Tue 24 Dec 2013, 18:27, edited 1 time in total.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#17 Post by jamesbond »

mikeb wrote:well I did it on Windows :D
Heresy !!! :lol: That being said 7zip will create tgz archive on Windows too :wink:
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#18 Post by mikeb »

That being said 7zip will create tgz archive on Windows too
You would think so but it only offered tar with no compression options...so zip it was.

mike

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#19 Post by jamesbond »

mikeb wrote:You would think so but it only offered tar with no compression options...so zip it was.
It does, but it's a two step process (tar first, then gz/bz2/xz). (You know, I have to boot into my Windows VM to confirm this ;) ) Agree that zip is more convenient.

Anyway, back on topic, if you're running full installs (which apparently missed my attention in my first reply) - the if you run root you can easily mount "/" as a branch of aufs. Of course, unionfs-fuse is probably better in this regard because it doesn't require patched kernels, so no contest if you're running other distros, and you get all the benefit.

Amigo thanks for bringing bindfs to my attention.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#20 Post by mikeb »

hmm could not be asked... zip appears to be the same compression as gzip but with an index too so i prefer it for large archives/backups as I can rescue a single file quickly and update the archive without a full rebuild.
the if you run root you can easily mount "/" as a branch of aufs.
so you have created a union using '/' as a layer using aufs? that was a query earlier. Lucid refused to do this. If so why is this not the method used in puppy for running an sfs?

mike

Post Reply