Bind mounts do the same job as links. Links are better.!

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

Bind mounts do the same job as links. Links are better.!

#1 Post by sunburnt »

Many Linux type O.S.s do some remapping of the root file system.
Links are usually used, I prefer them to bind mounts ( no mounting to do ).

I don`t see anything that a bind mount does that a link can`t do also.
In fact bind mount acts just like mount does, multiple and stacked mounts.
Both will mount a fs and image file, but bind mount will mount a dir.
Both will mount an item in many places, and stack mount many items in one.

Now if a bind mount could act like mhddfs and merge two dirs. into one.
mhddfs works very well, but is a user space program and so is rather slow.

### Q: Is there anything inherently better about bind mounts than links?
.
Last edited by sunburnt on Tue 22 Jan 2013, 03:38, edited 1 time in total.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#2 Post by amigo »

Of course. bind mounts are directories which are made visible in two places. The binding is able to make dirs visible even where links would not work correctly -like in a chroot. I still don't understand why you dislike mounting things -especially union mounts which do what you want. If you union-mount /etc/ and $APP_DIR/etc you get exactly what you want. The method you are using still forces you to still create/destroy links in the real rootfs and even have to backup/restore config files which you want to override. The union mount *is* the simple solution you seek. A 'multi-link' which you dream of, would not really provide a solution -you'd still need a way to tell/query the link the priorities to use. A union mount does this by letting you order the mounts any way you like and lets you use as many layers as you like.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#3 Post by sunburnt »

amigo; Why did I know it would be you to answer this post? :D

A junction link is just the current link that already holds a path in it now.
It just needs to be recognized by the kernel, Bash, or FS ( not sure which ).
Try this: ln -s /bin:/sbin:/usr/bin:/usr/sbin /root/000
Now look at it by hovering the mouse over it in the ROX-Filer.

A path is a priority or hierarchy. So the junction would work properly.
Read the path, first to last priority, and write to the first path only.
If the first path was read-only or not enough space then a write error.
This is it`s simplest form, more complex write behavior could be added.

Links surely use less cpu and ram resources than anything else.

### BUT... Back to reality as we know it now. :x

Using aufs means a union layer each for: /etc, /var, /opt , /usr/share.
These are the dirs. that I`m linking in now. No others seem to be needed.

This would be nice for /opt and /usr/share as it would require no clean up.
But the links in /opt and /usr/share are easy to create and remove.

The config. files in /etc and messages in /var are kept in the AppPkg.
Being as the files can be in sub dirs., managing the paths is a bit complex.
Here a union ( or junction ) would make life a lot easier.

If I understand, mount AppPkg/etc on top of /etc, but isn`t it a conflict?
Any other process writing to /etc actually writes to AppPkg/etc ( right? ).
Doesn`t any union effect all processes writing to the unioned FS?
This is a real problem I think ( or am I not perceiving this properly? ).

But setting "$PATHs" only effect the current shell and die with it ( nice...).

Unions will most likely never be accepted into the kernel.
However a junction as I`ve described it could be... Just an enhanced link.
Here`s a good article about unions: http://lwn.net/Articles/324291/

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#4 Post by amigo »

"Why did I know it would be you to answer this post?" Because you are bright -and because no one else showed up to debate or provide feedback when you posted about this before...

"Any other process writing to /etc actually writes to AppPkg/etc ( right? ).
Doesn`t any union effect all processes writing to the unioned FS?"
You are getting brighter! Your appraisal is correct -and I'll admit that I hadn't really thought about that when touting a union. The above is so because of what you might call 'the point of intervention'. As mentioned in the article you link to, a union *mount* is not the same thing as a *union file system*, because the intervention point is different. The kernel has its own internal file system, the VFS, which is a *complete view* of what is 'available'. Each file system has to approach the VFS using a universal interface. The concepts of mounts and file systems lie *outside* the VFS -these things must be processed before talking to the VFS.

symbolic links also have their own special point of interpretation -at the inode level -which means they are below the file system/mount level -but before the VFS. The VFS canot deal with any multi-references -doing so would add complexity to the VFS and hence the resistance of the kernel devs to implement such a thing. For the same reason, they don't want unioning code in the VFS. All these multi-reference/layering/(over)mounting things should be handled in user space -otherwise the VFS code would become slower and less maintainable.

A symbolic link should point to a specific inode. Think about it this way: you are driving and come to a sign which says 'this way', but instead of a single arrow pointing some direction, there are two arrows pointing in two directions. Is the sign really helpful to you? No, you must drive first in one direction and if that is not the 'right' place, then you must go back to where you started and try the other one. And what happens if the second also doesn't bring you to your destination. In that case you must backup to the *last* such junction you met... So, the VFS would have lots to do -at least some of this would have to be done for every access to the VFS. Each section of a path would have to be buffered -think about what would happen when a 'multi-link' pointed to another 'multi-link', which, in turn, pointed to another...

So, implementing a multi-link would amount to the same complexity as a multi-layered mount or PATH -definitely work which should be done in userspace -not in the kernel. Think about it this way: the kernel only deals with 'reality' -all sorts of 'possibilities' which require lots of extra work, should be delegated to userspace.

Now, again, I'll try to be helpful with practical solutions. What has happened is that, by seeking to implement several/many of these 'single-dir' applications, you have run into some of the limitations. These limitations are exactly why the concept has not taken off everywhere. You'll say: "But what about Android apps?". They only work because they all depend on just one thing, java. And because they are written from scratch to work within the Android environment -if they need to over-ride config files from the main system, then the path to the new files is explicitly coded to do so.

As you know, I've created lots of ROX-Filer APPDIRs, and even a framework for compiling them from scratch -which works lots of times *but not always*. You are this same point where you see that the same solution does not work for all programs.

If the sources use autotools (autoconf & Co.), then you can easily set prefix=/path/to/this/app so that sysconfdir, sharedstatedir and others are all set automatically *at compile time*. An app compiled in such a manner will work properly when installed to the location which matches what you specified as 'prefix'. You can also, but less easily, tell sources which use cmake to use such a prefix. With both autotools and cmake, you can also individually set sysconfdir, and the others.

The real problems start when trying to build sources which use other configuration methods or those which use none at all(just a Makefile). In these last cases, you must usually intervene at the source level by patching the sources so that any reference to conf files, state files, etc are hard-coded into the binary. Another possibility is using relative paths. *Some* sources will already work this way. For such sources, an APPDIR can be created using 'prefix=.'. Some other sources can be modified to allow this, as well.

Do you see now why this is so hard to implement on any typical distro? Do you understand why Android makes this pretty easy? I'll point out that BSD systems also make this somewhat easier, because BSD is distributed as a complete OS. There the devs have complete control of what is included and where it is. This is not the case for any Linux distro. The best you can do is make plenty of assumptions about the use case. You can only reliably create such an app for a specific distro and version of it. *Good practice dictates that you do this for any and every program or library package/non-package you create anyway*.

What the above paragraph boils down to is this: If your app needs extra libs which are not available on the base system you are building for, then they should be built into your program (statically-compiled), if it is not likely that other apps would need them. If other apps are going to need them and they are largish, then they should be installed as a separate package -in the normal places and using the normal package format. If you try to put them into another non-installed single-dir thingy, then ate compile time you will have to juggle lots of things so that the compiler finds them -and then have to juggle LD_LIBRARY_PATH at runtime so that the linker finds them. LD_LIBRARY_PATH and PATH are easy enough to manipulate. As you have seen, it's these other paths/etc, /var..) which are harder to work with at runtime.

The more assumptions you make about the base system, the easier your job. I really loved the idea of a self-compiling APP_DIR, but it quickly reaches limits when implemented in any sort of universal way.

Being a program maintainer can be a very hard job, indeed. It is only possible for a single person to keep up with a limited number of packages. The more you must modify the sources or manipulate the build, the less of them you will be able to handle -period. This is why my super-duper package-creation software still does not create APP_DIRs automatically -too many variables in their creation. I am capable of doing a few of them, but doing it for many, many apps is just more than is possible. Instead, I use normally-packaged-and-located programs and libs along with proper dependency management. And dependency-management can only be done at the *package level* and *at compile-time*. What I mean is that dependency information must be generated at compile-time for later use by the package manager.

Gawd, I haven't written so much in at least a year! But do feel free to extend this if I haven't(still?) covered your questions -both asked and unasked. You have potential or I wouldn't take the time...

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#5 Post by sunburnt »

Thanks amigo, high praise from one of the few I`d consider as an authority.
Albert Einstein: I have always despised any form of authority.
To punish me for my contempt,
the universe has seen fit to make me myself an authority.
True, I usually ask system level Qs that are rather base or cutting edge.

So I think you`re saying that a junction ( Karl Godt offered the term ) link
would be integrated into the file system drivers?

The JLink would make a generic path for anything, in this case /etc, /var, /opt, /usr/share.
If Linux had PATHs for configs. and such this wouldn`t be needed.
It would do what`s needed here, it couldn`t replace a union of course ( too simple ).

# A link target is a dir./file, just like an exec. that`s searched for through the PATH.

Are you sure a PATH is as complex to manage as a same level layered FS?
A union is very cpu and ram intensive, a path seems relatively simple.
The Q is mount vs link complexity, a tough Q for sure. I like a links simplicity.
Follow the PATH`s paths in precedence until the dir. or file is found.
A union has a precedence ( path ) of it`s mounted layers to follow.

# Android is another beast all together... Now if we could "clean" Linux...

Yes, an AppPkg builder will work for "good apps.", odd apps. ( many ) are still manual labor.
The app. folks seem to love to obfuscate their packages. Anarchists.!

I chose to build with Ubuntu binaries, they do the work to assure a good tool chain.
So no compiling`s needed, just a good method of utilization that works most of the time.
Should be no lib. conflicts, so completely library-less AppPkgs should be possible.

Dependency info... What if Ubuntu just had complete dep. files with the URL minus the mirror?
I know this just makes too much sense, so it`ll never happen because of them.
But it would eliminate much of this dependency hell mainly due to package management.!!!
It`s the "Uncle Bill" syndrome of: "Do it my way or not at all!!!" Planned incompatibility.

# Q to myself is; Have the pkg. builder resolve deps., or make the dep. files spoke of above.
Sanity says make the files so it`s only done once for each Ubuntu distro. that`s supported.

Ideally all apps. would have relative paths, so links in AppPkg may be needed to the / FS.
But this solves any clean up problem as the mess is AppPkg self contained.
I had mixed luck modifying binary files, it`s sketchy at best not working on all of them.
Probably mainly the old apps., so as new replacements are written, maybe this will fade.

Most of the AppPkg setup file is linking the 4 paths and setting the PATHs.
But it runs quickly even on my piece of old crap Celeron PC, so it seems acceptable.

I think a good idea would be to run ldd in it when running the app`s. exec. fails.
Then an error popup showing the missing libs. ( if the AppPkg wasn`t built complete ).
Adds code to the setup file, but no big deal as the code only runs on a fatal error.
Could include the Ubu. download code ( once it`s done ), so it`d auto. fix the AppPkg.!


Thanks for conversing with me amigo, it helps with the global picture as well as details.

# If there`s any flaws with using Ubuntu packages this way... Yell.!!!

And any ideas you may have about creating dep. files is appreciated.
Script tools to parse Ubu. web pages seems like the best way to do it.
Complete dep. files with the sub URLs... I think that`s the ticket.!

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#6 Post by amigo »

No, handling of 'junction-links' would be separate from filesystem drivers -otherwise the code would only work for a particular filesystem. There's nothing hard about handling a PATH or junction-link with several possibilities -it's just that it is extra work which does not belong in the core parts of the kernel. The code for unionfs has not been mainlined exactly because of its complexity and size.

Lets back up a step though. The conclusion that using a union of APP_DIR/etc over /etc does *not* necessarily mean that all access attempts to /etc would be re-directed to APP_DIR/etc.

I have code in my src2pkg which uses a union+mount --bind+ chroot to do much the same as what you are wanting. It is not simple, but it is clean, flexible and doesn't require any links on your system. I have nothing against links and they are simple in principal. However, for your purposes, I I can't see the point of having a non-installed-but-available application which still needs to create/destroy links in the main filesystem. If you were running a 'sane' system, where the user wouldn't normally have write privileges it wouldn't work anyway. Maybe later I will post some code pointers which I was working out yesterday -it is an interesting project.

As for dpkg needing configuration, any program with *any* degree of flexibility will need some sort of configuration -either through a conf file or command-line options. dpkg does exactyl what you want as far as downloading and installing any extra depends. You do need to tell it which repos to use or ignore and you also need to blacklist quite a few packages -stuff which Puppy already supplies and which upgrades(from ubuntu) would break.

Creating a deps file is not terribly complicated, per se. It's just that it has to be done for each and every package on a system -you can only tell what package a file belongs to if you have a list of the files contained in each package on the system -at the time you build the new package.

My package manager (tpkg) uses files like this:

Code: Select all

# acl_2.2.50-i586-1
# Requires:                              |Supplied by:                           
# File: libattr.so.1.1.0                 |: attr_2.4.45-i586-1                   
# File: libc-2.13.so                     |: glibc_2.13-i586-4                    
attr_2.4.45-i586-1
glibc_2.13-i586-4
They can only be done accurately when each package in the dependency chain which preceeds this one has been processed similarly. That means things have to be compiled or upgraded in the correct order. My scheme does not support using version numbers of <, > or =. The info given tells you the exact version which the current package was compiled against. Providing </>/= info would require human input each time a 'range' was used. It is impossible to programaticaly determine if a package will work with an earlier or later version of the same-named dependency.
As I said before, the only way to accurately determine depends is at the time of package creation and the only way to reasonably use and distribute the info is to have it be contained within the package. A one-time or universal/central database will simply not work reliably.

In my case, src2pkg generates the dep files for me and they are then kept on the installed system for any possible use.

As for modifying binary files, I assume you mean opening a binary file using a hex editor, or such. Any modifictaions you make must not change the size of the binary or it will not run. You can't change some hard-coded path except with another of the exacty same link. To do so you must use other tools which modify the ELF header so that the correct size and internal offsets are used.

Have you actually tried using unionfs for your goals?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#7 Post by sunburnt »

No I haven`t tried using a union as I didn`t see how it could work.
However... I realized tonight that there`s a nifty way to do it.

For Puppy and other Linuxes with a union, just use a "sub save" layer.
A writable layer just under Puppy`s save layer, and write directly to it.
So the global view sees both save layers, and writes to the top one.
AppPkg apps. see the global view also, but write to the second layer only.
Put the sub save layer in a small ram disk as it holds only links anyway.
So then at each shutdown the sub save layer is wiped clean automatically.

As nice as this is, it negates AppPkg being portable, slaved to a union.
I gave up on SqApp as it required too much infrastructure ( not portable ).
AppPkg carries all of it`s infrastructure inside itself, assuring portability.

I`ve looked at dpkg and never understood most of it`s workings.
Need a clear picture of making it get lib. packages minus installed libs.

Correct. A dep. file would be: the app., the libs., the packages., and URLs.
It`d be nice to have a pool of bare lib. files making it easier to find them.

With Ubuntu handling the dep. chain, I don`t see why a solid set of dep.
files for that Ubuntu release ( ie: Precise ) would not work always.
Being a stable release it should never change... Right?
Again, no compiling anything, the target O.S. must be Ubuntu compatible.
### This is critical... Tell me if and how this view is flawed.!

Lots of the Ubuntu repository has no version release associated with it.
Many of the apps. and deps. ( libs.) work for the few releases around it.
Many Lucid apps. work in Precise, but newer apps. and libs. not in older.
Libs. seem to be backward compatible for a few numbers, sometimes.

This replaces all of a binary`s /usr paths with ././ ( relative path ):

Code: Select all

find . -type f -exec sed -i -e "s|/usr|././|g" {} \;
There are also utilities made to modify binary files ( bbe, patchelf, etc.).

### So where is the link code at then? Just in case I need to know... :wink:

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#8 Post by amigo »

Nah, you don't need to and shouldn't mess with the save layers of your existing aufs(right?) setup. Running the AppDir should set up a separate mount using aufs, unionfs, mhddfs or unionfs-fuse. The writes really belong in the users $HOME dir. The AppDir itself could be anywhere.

A separate mount point is used to mount the union on and, once layered you chroot into the mountpoint and run your program normally. Anything the program contains which is newer than the installed system take precedence, as they should. The mountpoint and the writes should really be in $HOME/Choices/$APP_DIR.

"slaved to a union" Shouldn't you really mean: I'm glad I have this flexible, just-works technique as an option?

Even if you use aufs, the whole thing is separate from any unions created by Puppy, but I find unionfs-fuse to work fine. grab the sources, compile and install -or wait, ubuntu has mhddfs in their repos (maybe unionfs-fuse, too), so you can simply install from dpkg, right?

Here's a very quick scenario that will demo it for you:

Code: Select all

#!/bin/sh
mkdir -p write union prog
# prog is where you have DESTDIR-installed program content
# write is where you want changes to be kept -should be ~/Choices/NAME_OF_THIS_APP_DIR
# union is where the union is mounted and where you'd chroot to run your program

# put some fake content in prog
mkdir -p prog/bin prog/etc
echo testcode > prog/bin/myproggie
# use a known-name to see how we override /
echo testconf > prog/etc/resolv.conf

# for use with unionfs patch and fuse
# mount -t unionfs -o dirs=write=rw:/=ro unionfs union
# mount -t aufs ... similar to above
# fusermount, pmount or other would be needed for non-privileged users

# this is the unionfs-fuse executable which uses only fuse
unionfs -o nonempty -o cow write=RW:/=RO:prog=RO union
# the write is RW, then root(/) is blended in, then our proggy itself which overrides /
View the file in union/etc/resolv.conf. Notice that your system file /etc/resolv.conf is not changed or showing the file union/etc/resolv.conf or write/etc/resolv.conf
Then, open the file union/etc/resolv.conf. Edit and save it.
Now the file will be saved at write/etc/resolv.conf.

To unmount it run: umount union

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#9 Post by sunburnt »

Thanks much amigo; I was wondering where you are that you`re so cold.

Much the same as my suggestion except using a separate union and chroot
so the app. thinks it`s normally installed. No links or paths, simple, clever...
# So prog, write, and union are all inside of AppPkg, so configs. go with it.

There`s a lot to like about it. But I`m designing a no union O.S. with no
apps. except VT, editor, file man., network, etc. Everything else is AppPkg.
A union, and the type of union can`t be relied on for the target O.S.
It`s necessary for AppPkg to carry it`s own solution inside of itself.
So UnionFS and aufs are out of the Q as they`re embedded in the kernel.

### You said kernel space apps. only work for privileged ( root? ) users?
# So only user space "union type" apps. will work run from inside AppPkg?

When I found mhddfs I thought it was my answer, not a J-Link but like it.
Then it was pointed out that it like all FUSE stuff is user space and slow.

### In light of all of this, a J-Link avoids all of the problems and is fast.

Need to rethink this whole setup, having AppPkg appear in / is a good idea.
It simplifies the deployment methods, and also the AppPkg builder as well.

### Time for more reading. Thanks again mi amigo... Terry B.

# P.S. ... Made your setup work with mhddfs, but not with fusermount.

# P.P.S. ... So where would the code for link behavior be located.?
.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#10 Post by amigo »

I'm in Germany, but not German.
There are no kernel-space 'apps'. kernel space is the kernel itself and naturally everything there is run as root. userspace is everything outside the kernel.
What was the syntax for mhddfs? I've used it before and seem to remember it was similar to unionfs-fuse -maybe more limited.
You'd have to ask on lkml or search to find exactly where links are handled in the kernel. But, your idea of junction-links is really no different than any of the already-implemented techniques. The only difference is in the location where it is handled.

unionfs and aufs handle it at the point precedding the files system drivers. Neither is a real file system -notice that they will work with any filesystem you like. They insert theirselves just between the VFS and the filesystem. They register themselves with the VFS as being a filesystem and *interpret* and handled read/write requests before handing them on to the filesystem drivers.

unionfs- and mhddfs use fuse, which is a sepcial access point to the VFS -designed exactly to allow any sort of wierd and/or ono-privileged access to the VFS -allowing normal users to use mount, creating compressed filesystems, etc.

COW solutions operate at the block level and interface with the kernel in the block-layer code.

I am unsure of the exacty point where links are resolved -but they resolve to inodes in the end. I believe the VFS speaks only inode-talk. Anyway, you have an order something like:
VFS - inodes - links - block-layer - mounts - filesystem driver

As I said, your idea only differs in the chosen point of interaction. The idea will never fly with kernel devs for the same reason that none of the others do. If you see an advantage because you only want a limited capability, then any solution besides aufs would be the best. aufs is the most capable of all the solutions. Hence, the largest and most complex. mhddfs or one of the old cow solutions are much simpler. Your implementation of j-links would also have to be done in the kernel -just like unionfs or aufs -unless you create such an implementation using fuse, which is outside the kernel but allows non-privileged use.

"designing a no union O.S." Do you mean you are building from the ground up with no puppyisms at all -like a normally installed distro?

"FUSE stuff is user space and slow." Yes, fuse implementations are slower than a builtin kernel feature. But the usage you suggest is not intensive. unionfs-fuse is fast enough that disk read/write speeds are the bottle-neck -just like with real kernel features.

"no apps. except VT, editor, file man., network, etc. Everything else is AppPkg" How/where do you draw the line? Isn't a mixed system more confusing and complex than using the concept (for making software available) throughout. If the AppPkg idea is so good, why can't it be applied and used for everything? (Go back to the top of the thread and read it all again to see why that is not the case). Any system is shot full of libs and programs which do not lend themselves to the AppPkg concept. While many larger applications can be fit into the scheme, their library requirements mostly do not. And lots of basic items also do not fit at all. The whole thing evolves into the issue of resolving dependencies. Even your AppPkgs have to deal with that unless you truly create each with *everything it needs (even glibc) included* in one deliverable item. This will always mean duplication and larger deliverables(read package, SFS, AppPkg, app, AppDir, etc).

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#11 Post by sunburnt »

Please forgive my quaint references ( kernel apps.), "kernel code and modules".
From AppPkg setup file ( mhddfs uses commas instead of colons ):

Code: Select all

# Mounting:
mhddfs $Pkg/$Pkg.rw,$Pkg/$Pkg.app,/ $Pkg/$Pkg.u

# Unmounting is: 
fusermount -u $Pkg/$Pkg.u
AppPkg runs faster, but then the setup file is half the size now, just I said it would be.
### If you`d like to see my files and code, I`d appreciate critique.

mhddfs works well, at 30 KB size it`s more capable than is needed. Only uses libfuse.
I tried to get fusermount to do it, but I think it doesn`t do unioning. Only uses libc.!
I got unionfs-fuse to try it, but bigger than mhddfs at 39 KB in size. Only uses libfuse.

### AppPkg portability... How to get AppPkg to load the fuse kernel module.?
This would make it possible for AppPkg to work on any compatible O.S.


Base target O.S. would be all Ubuntu ( or could be Slack ) with just needed utilities.
Much like Tiny Core, little installed apps. and infrastructure, unlike Puppy`s app. suite.
There`s new apps. all the time, it should be a snap to replace them ( not permanent ).

The line is just what`s needed to do O.S. maintenance, and maybe a little more.
This so you can do work with the base booted O.S., same reason TC`s like this.


With Ubuntu handling the tool chain, AppPkg could dnld. needed libs. to a shared dir.
Shouldn`t be any lib. conflicts, but if there are put the lib(s). in AppPkg to override.
So most AppPkg have no libs. or only the app. specific ones. Almost a normal setup.

I understand what you`re saying, Linux is not orderly... We try to provide it.
I think dep. files made from a given Ubuntu release`s web pages should be correct.
Otherwise Ubuntu has serious problems to address ( Nothings perfect in this life...).

### Again... If this method has flawed, how best to fix or address it.?
You`re saying dep. resolving is the problem, Ubuntu can get it right I hope.
Maybe I`m hoping for too much basing AppPkg and my O.S. on Ubuntu`s structure.?

# Sorry about the long posts, many subjects discussed here. Very interesting it is.

I thought about the difference between one Sq. file and links pointing into it,
or mounting a Sq. file on each r-o dir.: /bin, /lib, /sbin, /usr/bin, /usr/lib, /usr/sbin.
There`s more mounts, but there`s no link to resolve, slightly faster I`d think.
But like link vs mounts, I think I`m obsessing too much trying to optimize the O.S.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#12 Post by amigo »

"but if there are put the lib(s). in AppPkg to override" The problem with using a general location for any updated libs is the same as with the standard dirs. what to do when one of your apps needs a still different version?

My conjecture seems to be pretty close -that you want to build something based on ubuntu, using their kernel and everything -which is why you don't want unionfs or aufs. Actually that is good thinking. If you base your project on someone else's work, then the less alterations you make the less trouble you have.

You don't really have to worrry aboue FUSE then, ubuntu and all other distros will have it available. loading the kernel module takes no more: modprobe fuse
But, you may not even have to do that -if the kernel has auto-loading of modules enabled. If not, simple enough to add the command to whatever script you are writing which starts an app which needs it.

No need to maintain any deps file -for ubuntu it would really be huge. dpkg itself will handle fetching the packages for you and excluding anything already-installed -providing your system has the local dpkg database intact. Anything you don't want updated/overwritten can be blacklisted.

That said, you might do well to use Slackware or maybe archlinux instead of Eeeww-buntu LOL. Slackware is the ideal place for learning how things really work. arch is nearly as good there, but has a dependency-resolving pkg mgr. Of course, neither of them has the 'depth' of packages like ubuntu. At the very worst/best, use debian instead of ubuntu for more normality -use the testing branch if you want really up-to-date stuff. Or, use the stable branch for the best-tested combinations(and whole) anywhere.

"Sq. file" Are you taking about squashfs file-system images SFS files? Note that you can use that technique within any appdirs, as well. Instead of unioning an open dir structure, you can union an FS image also.

As far as the wisdom of trying to use appdirs everywhere, they really become lots of work when you try to maintain a few of them and are really inadequate for libraries or required software. At some point, it is just easier to create packages which use whatever dep-resolution which is available. And, at runtime, you could simply have an empty dir, mounted on tmpfs where you 'install' a series of packages temporarily for your chroot or whatever.

I might have a look at your code if you make it available -but no promises. If it's very long or hard-to-read I lose patience...

My system KISS-linux, is similar to Slackware, but you can do a truly-minimal install with a small number of packages(~30) and expand it without limits. That's just a bootable system with CLI login, no networking or non-local hardware. Minimal really means minimal. The list of packages is easy to understand and contains no cruft. Reading through the init scripts is a breeze and lets you understand exactly what is needed (and not) to bring up a basic system -I've out-slacked Slackware, as it were. So far, I do not provide actual resolution of dependencies -all the info is there as it is built right into every package. But I have been hesitant to implement resolution -I am an old Slacker, after all.

mhddfs vs. unionfs-fuse:
mhddfs is not well maintained, while unionfs-fuse is. Do you know how small 9K is? How many places are you willing to sacrifice usuability against a few K's? takes a lot of them to make 1MB...
There are other tools which might be helpful, but unionfs-fuse is probably the best candidate. Hey, I use it in src2pkg, so it must be good... Actually, I even got an email from the author once alerting me to a needed update -I had never written him, but he had noticed me using his stuff and wanted to be helpful.

Out of curiosity, why do you pursue the AppPkg line of thinking? Is it about run-time modularity, or about providing apps which are not available direct from ubuntu? Can you define more clearly the goals?

I don't mind batting things around with you -as long as it stays interesting! LOL

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#13 Post by sunburnt »

I`ll attempt to address each subject.

My experience is: links are reliable, mounts are less so, sockets and pipes are worse yet. Viva le link.!

Common libs. ( widely used ) go into a common dir. like /opt/lib. The dirs. /lib and /usr/lib are R-O in a Sq. file.
App. unique libs. go in the app`s. Sq. file or with the app`s. loose files in the mount dir.: (pkg. name + ver.).app
Optionally inside the AppPkg is a /lib dir., libs. in it are shared among any and all of the apps. inside the AppPkg.
So there`s 3 levels of libs., system wide shared level, AppPkg shared level, and app. unique non-shared level.

I spent a bit of time at the Debian site, I take it you put Debian above Ubuntu... That`s undoubtedly good advice.
A big package base makes it worth the effort to develop AppPkg and it`s app. downloader and the AppPkg builder.

I`m glad that it`ll be easy to load the needed kernel module, and union-fuse does seem more made for the job.
My learning curve will be making dpkg work for the AppPkg builder. I`ve used it and never got it to work properly.

Sq., a generic squash file, not an SFS that`s made to be unioned. Although now AppPkg is using unioned Sq. files,
which makes me realize that Puppy SFS files can be dropped straight into AppPkgs for Puppy without modification.
AppPkg is an AppDir or RoxApp. They don`t have a set internal structure. AppPkg sets a standard layout.

The AppPkg Sq. file is now essentially an SFS file, the rest of the AppPkg is 2 static scripts and 1 unique script.
There`s also the union-fuse files, an AppPkg popup menu exec. file and it`s unique menu.lst file for multi. apps.

AppPkgs are for apps., and maybe some services, daemons, etc. It`s not intended for installing libs. to the O.S.
If I get AppPkg to auto. download libs. to /opt/lib, it`s only in support of AppPkg, not for system lib. upgrading.

AppPkg is an enhanced AppDir, it does the same thing for the same reasons, modular, portable, no-install apps.
AppPkg improves on them with multiple apps., local libs., and a popup menu. They can be made do the same too.

An interesting idea, dump all the apps. into one dir. so only one union and chroot. It has fewer layers than Puppy.
But then it starts to have Puppy`s same conflict problems with layering so many apps. and libs. over each other.
Your idea of 1 union per app. runs the apps. in a sandbox effectively isolating them from other apps. and the O.S.


Ultimately a tiny Debian-Ubu. distro. with X, WM, GTK, Xorg, network, and utility apps.. About a 40 MB download.
Download only the AppPkgs you want, single apps., or same app. type "try them" bundles, or groups of like apps.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#14 Post by sunburnt »

Here`s the 3 scripts that make your union-chroot setup work.
The first script is unique for Chrome, the other 2 are generic:

Code: Select all

======= Script: chrome
#!/bin/sh
cd ${0%/*}/.AppPkg
export Pkg="chrome-24_i386"
./setup google-chrome --user-data-dir=$HOME/$Pkg/profile &

======= Script: setup
#!/bin/sh
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>	Mnt. Sq. Files & Union, Run App.
if [ "$1" ];then
	echo -e "\n###  Run:  $Pkg"
	App="$Pkg/$Pkg.app"									### Mount App. Sq. File
	Sq="$Pkg/$Pkg.sq"
	[ ! -d $App ]&& echo -e "\n###  ERROR:  NO App. Dir.:  $Pkg.app\n" && exit
	[ ! "`ls $App`" ]&& if [ -f "$Sq" ];then				# IF app dir. empty
		mount -t squashfs -o loop $Sq $App					# IF Sq. file, mnt.
		[ $? -gt 0 ]&&
			echo -e "\n###  ERROR:  Fail Mount App. File:  $Pkg.sq\n" && exit
		echo -e "\n#  Mount App. Sq. File:  $Pkg.sq"
	fi
	Lib="$Pkg/lib"
	Sq="$Pkg/lib.sq"									### Mount Lib. Sq. File
	[ -d $Lib ]&& if [ ! "`ls $Lib`" ];then					# IF lib dir. empty
		[ -f "$Sq" ]||										# IF no lib. Sq.
			echo -e "\n###  ERROR:  NO Lib. Sq. File:  lib.sq\n" && setup &
		mount -t squashfs -o loop $Sq $Lib					# IF lib. Sq., mnt.
		[ $? -gt 0 ]&&
			echo -e "\n###  ERROR:  Fail Mount Lib. File:  lib.sq\n" && setup &
		echo -e "\n#  Mount Lib. Sq. File:  lib.sq"
		fi
	[ "`ls $Lib 2>/dev/null`" ]&&							### Set lib. path
		LD_LIBRARY_PATH=`realpath ./`/$Pkg/lib:$LD_LIBRARY_PATH
	mhddfs $Pkg/$Pkg.rw,$Pkg/$Pkg.app,/ $Pkg/$Pkg.u			### Mount union fs
	[ $? -gt 0 ]&&
		echo -e "\n###  ERROR:  Fail Mount Union:  $Pkg.u\n" && setup &
	echo -e "\n#  Mount Union:  $Pkg.u"
	./run $@ &												### Run hook file

else #>>>>>>>>>>>>>>>>>>>>>>>>>>>>	Unmount Union & Sq. Files.
	echo -e "\n###  End:  $Pkg"
	fusermount -u $Pkg/$Pkg.u				    			# Unmount union fs
	[ $? -eq 0 ]&& echo -e "\n#  UnMount Union:  $Pkg.u" ||
		echo -e "\n###  ERROR:  Fail UnMount Union:  $Pkg.u\n"
	Mnt=`mount`
	App=`echo "$Mnt" |grep $Pkg.app`
	if [ "$App" ];then umount -d $Pkg/$Pkg.app				# Unmount App. file
		[ $? -eq 0 ]&& echo -e "\n#  UnMount App. Sq. File:  $Pkg.sq\n" ||
			echo -e "\n###  ERROR:  Fail UnMount App. File:  $Pkg.sq\n"
	fi
	Lib=`echo "$Mnt" |grep lib.sq`
	if [ "$Lib" ];then umount -d $Pkg/lib					# Unmount Lib. file
		[ $? -eq 0 ]&& echo -e "\n#  UnMount Lib. Sq. File:  lib.sq\n" ||
			echo -e "\n###  ERROR:  Fail UnMount Lib. File:  lib.sq\n"
	fi
fi

======= Script: run
chroot $Pkg/$Pkg.u $@
./setup &
I`m making a few AppPkgs with your setup as proof of concept tests.
It works with xMahjongg, but Chrome doesn`t like something about it.
Chrome`s startup script gets the wrong path, or no path, I`m not sure.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#15 Post by amigo »

Do you mean to have the OS as a LiveCD or set of a few files which can be manually copied to a USB stick? Or do mean a fully installable?

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#16 Post by sunburnt »

The O.S. on HD or USB, but could do CD-DVD boot too of course.
A CD and DVD are storage devices, but can boot an O.S. into ram well.

Base O.S. is 2 files like Murga`s Puppy with Sq. inside the image.gz file.
The kernel extracts a cpio file to initrdfs, and an image file to a ramdisk.
A ramdisk is simple, initrdfs is more complex and definitely overkill.
Root FS is dirs., links, and Sq. file(s), so a fixed size ramdisk of ~ 40 MB.
AppPkgs. can be loaded/unloaded in ram or not to manage the ram usage.

/bin, /lib, /sbin, and most of /usr are in a Sq. file or files. No union FS.
/dev, /mnt, /proc, /sys, /tmp are in image.gz and extracted to the ramdisk.
/etc, /home, /opt, /root, /var on partition, /usr/etc + /usr/local/etc => /etc.
/usr/share could be on partition, but AppPkg using a union it`s in a Sq. file.

O.S. has simple desktop and utilities, other WMs and desktops in AppPkgs.


# I`m no M$ fan, but they saw value in having a junction link in Windows...
.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#17 Post by amigo »

What do you mean by junction link in Windows? Windows doesn't even have symbolic links. A Windows 'shortcut' is a real file which contains 'directions' to the item short-cutted item.

Wouldn't it be simpler to create you setup so that anything the user wants can simply be installed including dependencies, using dpkg and that file would then all reside on the USB stick or HD? The maximum implementation of modularity is achieved with normal packages. I only create AppDirs where the nature of the application indictates the utility of it.

Under Puppy, most AppDirs wind up acting simply as shortcits to start a program. I mean the special edvantages of using an AppDir are not implemented or used at all. I use an AppDir when I want nice drag-n-drop functionality or a switchable default behaviour -right click to change options. Most Puppy stuff uses a complex right-click or default MIME-type script to offer such options. For instance, under Puppy, when you click on an html file, you might get a pop-up menu which lets you choose between opening the file with a browser or with an editor.

I find it nicer to simply have an AppDir for each -if I want to edit something I drop it on the editor AppDir. If I want to view it with a browser I drop it on the browser AppDir. These AppDirs don't necessarily *contain* the application. Instead, the AppDir provides a versatile, configurable way to access the application. This actually follows the original RISC/ROX concept -where the user is able to focus on *his files* and not the programs he uses to manipulate them. What I mean is this: the user always knows where his files are -right where he saved them. He can find them much easier than first finding an app to manipulate them, opening that app, then using the apps 'Open' feature to (again) browse to where the file is that he wants to open.

Another example. I created an AppDir which consists of just scripting for working with archives. If you drop a file or a directory on it, then it offers option windows for creation of an archive. If you drop an archive(or compressed file) on it, then it offers options for decompressing the archive or file.

Do you see the way the interaction with the user is turned around? It keeps a normal user from having to navigate through the bin dirs for apps. It also avoids navigating through menus -once the AppDir is on the desktop it is always easily available. Some folks like a GUI without any taskbar or menu at all.

Hope you don't think I'm trying to provoke you in a mean way or even convince you of a certain view of things. I'm just trying to tickle your brain as go along...

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#18 Post by sunburnt »

No no, it`s all "food for thought". MU made a number of desktop DnD apps.
And this is an excellent use of the AppDir concept, though a very simple one.
It`s very nice having "things" in dirs. and Sq., can even drop AppDirs on AppDirs!

Puppy doesn`t really need AppDir or RoxApp, SFS files do well in Precise and Slacko.
Puppy has a lot of utilities to deal with SFS files, booting, load/unload, upgrade, etc.

It`s good using dpkg to build AppPkgs, but for installing apps. the target O.S. must have it.
AppPkg doesn`t assume the target O.S. has any installed infrastructure, as none is needed.
To go the "installed" infrastructure route, then I`d put it in an AppPkg ( so it`s not installed ).
So then all the AppPkg apps. rely on an AppPkg dep. to work ( not really what I had in mind ).

2 intentions for AppPkg, app. portability ( for compatible O.S.s ), and for my AppPkg only O.S.
With AppPkg handling all of it`s functioning itself, the O.S. needs few installed utilities ( lean ).

Working from the file point of view is definitely much better than access through menus.
Puppy`s ROX right-click could be improved a lot though. Whatever menu type, sub menus suck.

Some like desktop icons, and some want a clean DT. I like the idea of many slide-out panels.
I posted a concept of labeled colored bars anywhere along the screen sides. hover to slide-out.
On a single left side panel: desktop buttons, drive/partition buttons, task-bar buttons, and tray.
Task-bar and drive/partition buttons are stacked vertically so lots of them fit in a small space.
Another left side panel bar labeled menu is a slide-out apps. menu that can also be DnD on.
Desktop icons get covered up, but the bars and panels are on-top, so drag-hover-select-drop.
Desktop icons are messy, slide-out panels organize items and the desktop`s clean and pristine.


### M$ junction is not a shortcut: http://en.wikipedia.org/wiki/NTFS_junction_point


### P.S.: Can`t seem to get Chrome to work, it should work.
I had no trouble getting it to work the old way with links. :?
Error and line 17 of google-chrome wrapper script:

Code: Select all

/usr/bin/google-chrome: line 17: /dev/null: Permission denied
Failed to open /dev/null

if ! which xdg-settings &> /dev/null; then
It`s looking for installed xdg utilities:
# We include some xdg utilities next to the binary, and we want to prefer them
# over the system versions when we know the system versions are very old. We
# detect whether the system xdg utilities are sufficiently new to be likely to
# work for us by looking for xdg-settings. If we find it, we leave $PATH alone,
# so that the system xdg utilities (including any distro patches) will be used.
I tried running a modified command in rxvt ( it works okay...):

Code: Select all

sh-4.1# if ! which xdg-settings &> /dev/null; then echo GOOD ;fi
GOOD
I checked the union dir. and everything seems to be there.
I don`t see how the union or chroot would make /dev/null inaccessible.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#19 Post by amigo »

The NTFS junction seem to be simple links -actually less-feature than linux symlinks in every way and do not provide a multi-target capability. They do introduce a term -reparse junction which is useful for the discussion, since any such multi-link-target capability will require parsing the optioanl points and deciding according to priorities which file should be linked to -must not forget handling the possibility that none of the possible links are valid/present.

About /dev/null, that is strange. Is chrome running as root or non-privileged user? Is the execution being done by a wrapper script? If so, llok for /dev/null in there to see more of what is going on.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#20 Post by sunburnt »

If none of the paths in a j-Link have the target dir. or file, it`s the same as any "not found" error.

Yes, the wrapper script errors at line 17 ( shown in post above ).
I thought about if unionfs-fuse was messing with the allowed users.
So I tried running it with the option:

Code: Select all

-o allow_other         allow access to other users
It made no difference.

gparted is always been a real pickle to get working also, "Buss error".
I didn`t think gparted is complex, uses lots of libs. But should be simple.

I`m guessing mPlayer and Handbrake will be lots of fun too.
Had good luck with music players, simpler? Not screwed up intentionally?

# If this is how the complex apps. go, the AppPkg builder becomes a tool.
Making build scripts for apps. is probably pointless, they keep changing.

### So the Linux O.S. is not the Linux problem, the damn apps. are.!!!
.

Post Reply