Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 30 Sep 2014, 10:01
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Bind mounts do the same job as links. Links are better.!
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 6 [84 Posts]   Goto page: 1, 2, 3, 4, 5, 6 Next
Author Message
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Mon 14 Jan 2013, 22:58    Post subject:  Bind mounts do the same job as links. Links are better.!  

Many Linux type O.S.s do some remapping of the root file system.
Links are usually used, I prefer them to bind mounts ( no mounting to do ).

I don`t see anything that a bind mount does that a link can`t do also.
In fact bind mount acts just like mount does, multiple and stacked mounts.
Both will mount a fs and image file, but bind mount will mount a dir.
Both will mount an item in many places, and stack mount many items in one.

Now if a bind mount could act like mhddfs and merge two dirs. into one.
mhddfs works very well, but is a user space program and so is rather slow.

### Q: Is there anything inherently better about bind mounts than links?
.

Last edited by sunburnt on Mon 21 Jan 2013, 23:38; edited 1 time in total
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Wed 16 Jan 2013, 05:13    Post subject:  

Of course. bind mounts are directories which are made visible in two places. The binding is able to make dirs visible even where links would not work correctly -like in a chroot. I still don't understand why you dislike mounting things -especially union mounts which do what you want. If you union-mount /etc/ and $APP_DIR/etc you get exactly what you want. The method you are using still forces you to still create/destroy links in the real rootfs and even have to backup/restore config files which you want to override. The union mount *is* the simple solution you seek. A 'multi-link' which you dream of, would not really provide a solution -you'd still need a way to tell/query the link the priorities to use. A union mount does this by letting you order the mounts any way you like and lets you use as many layers as you like.
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Wed 16 Jan 2013, 17:26    Post subject:  

amigo; Why did I know it would be you to answer this post? Very Happy

A junction link is just the current link that already holds a path in it now.
It just needs to be recognized by the kernel, Bash, or FS ( not sure which ).
Try this: ln -s /bin:/sbin:/usr/bin:/usr/sbin /root/000
Now look at it by hovering the mouse over it in the ROX-Filer.

A path is a priority or hierarchy. So the junction would work properly.
Read the path, first to last priority, and write to the first path only.
If the first path was read-only or not enough space then a write error.
This is it`s simplest form, more complex write behavior could be added.

Links surely use less cpu and ram resources than anything else.

### BUT... Back to reality as we know it now. Mad

Using aufs means a union layer each for: /etc, /var, /opt , /usr/share.
These are the dirs. that I`m linking in now. No others seem to be needed.

This would be nice for /opt and /usr/share as it would require no clean up.
But the links in /opt and /usr/share are easy to create and remove.

The config. files in /etc and messages in /var are kept in the AppPkg.
Being as the files can be in sub dirs., managing the paths is a bit complex.
Here a union ( or junction ) would make life a lot easier.

If I understand, mount AppPkg/etc on top of /etc, but isn`t it a conflict?
Any other process writing to /etc actually writes to AppPkg/etc ( right? ).
Doesn`t any union effect all processes writing to the unioned FS?
This is a real problem I think ( or am I not perceiving this properly? ).

But setting "$PATHs" only effect the current shell and die with it ( nice...).

Unions will most likely never be accepted into the kernel.
However a junction as I`ve described it could be... Just an enhanced link.
Here`s a good article about unions: http://lwn.net/Articles/324291/
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Thu 17 Jan 2013, 05:20    Post subject:  

"Why did I know it would be you to answer this post?" Because you are bright -and because no one else showed up to debate or provide feedback when you posted about this before...

"Any other process writing to /etc actually writes to AppPkg/etc ( right? ).
Doesn`t any union effect all processes writing to the unioned FS?"
You are getting brighter! Your appraisal is correct -and I'll admit that I hadn't really thought about that when touting a union. The above is so because of what you might call 'the point of intervention'. As mentioned in the article you link to, a union *mount* is not the same thing as a *union file system*, because the intervention point is different. The kernel has its own internal file system, the VFS, which is a *complete view* of what is 'available'. Each file system has to approach the VFS using a universal interface. The concepts of mounts and file systems lie *outside* the VFS -these things must be processed before talking to the VFS.

symbolic links also have their own special point of interpretation -at the inode level -which means they are below the file system/mount level -but before the VFS. The VFS canot deal with any multi-references -doing so would add complexity to the VFS and hence the resistance of the kernel devs to implement such a thing. For the same reason, they don't want unioning code in the VFS. All these multi-reference/layering/(over)mounting things should be handled in user space -otherwise the VFS code would become slower and less maintainable.

A symbolic link should point to a specific inode. Think about it this way: you are driving and come to a sign which says 'this way', but instead of a single arrow pointing some direction, there are two arrows pointing in two directions. Is the sign really helpful to you? No, you must drive first in one direction and if that is not the 'right' place, then you must go back to where you started and try the other one. And what happens if the second also doesn't bring you to your destination. In that case you must backup to the *last* such junction you met... So, the VFS would have lots to do -at least some of this would have to be done for every access to the VFS. Each section of a path would have to be buffered -think about what would happen when a 'multi-link' pointed to another 'multi-link', which, in turn, pointed to another...

So, implementing a multi-link would amount to the same complexity as a multi-layered mount or PATH -definitely work which should be done in userspace -not in the kernel. Think about it this way: the kernel only deals with 'reality' -all sorts of 'possibilities' which require lots of extra work, should be delegated to userspace.

Now, again, I'll try to be helpful with practical solutions. What has happened is that, by seeking to implement several/many of these 'single-dir' applications, you have run into some of the limitations. These limitations are exactly why the concept has not taken off everywhere. You'll say: "But what about Android apps?". They only work because they all depend on just one thing, java. And because they are written from scratch to work within the Android environment -if they need to over-ride config files from the main system, then the path to the new files is explicitly coded to do so.

As you know, I've created lots of ROX-Filer APPDIRs, and even a framework for compiling them from scratch -which works lots of times *but not always*. You are this same point where you see that the same solution does not work for all programs.

If the sources use autotools (autoconf & Co.), then you can easily set prefix=/path/to/this/app so that sysconfdir, sharedstatedir and others are all set automatically *at compile time*. An app compiled in such a manner will work properly when installed to the location which matches what you specified as 'prefix'. You can also, but less easily, tell sources which use cmake to use such a prefix. With both autotools and cmake, you can also individually set sysconfdir, and the others.

The real problems start when trying to build sources which use other configuration methods or those which use none at all(just a Makefile). In these last cases, you must usually intervene at the source level by patching the sources so that any reference to conf files, state files, etc are hard-coded into the binary. Another possibility is using relative paths. *Some* sources will already work this way. For such sources, an APPDIR can be created using 'prefix=.'. Some other sources can be modified to allow this, as well.

Do you see now why this is so hard to implement on any typical distro? Do you understand why Android makes this pretty easy? I'll point out that BSD systems also make this somewhat easier, because BSD is distributed as a complete OS. There the devs have complete control of what is included and where it is. This is not the case for any Linux distro. The best you can do is make plenty of assumptions about the use case. You can only reliably create such an app for a specific distro and version of it. *Good practice dictates that you do this for any and every program or library package/non-package you create anyway*.

What the above paragraph boils down to is this: If your app needs extra libs which are not available on the base system you are building for, then they should be built into your program (statically-compiled), if it is not likely that other apps would need them. If other apps are going to need them and they are largish, then they should be installed as a separate package -in the normal places and using the normal package format. If you try to put them into another non-installed single-dir thingy, then ate compile time you will have to juggle lots of things so that the compiler finds them -and then have to juggle LD_LIBRARY_PATH at runtime so that the linker finds them. LD_LIBRARY_PATH and PATH are easy enough to manipulate. As you have seen, it's these other paths/etc, /var..) which are harder to work with at runtime.

The more assumptions you make about the base system, the easier your job. I really loved the idea of a self-compiling APP_DIR, but it quickly reaches limits when implemented in any sort of universal way.

Being a program maintainer can be a very hard job, indeed. It is only possible for a single person to keep up with a limited number of packages. The more you must modify the sources or manipulate the build, the less of them you will be able to handle -period. This is why my super-duper package-creation software still does not create APP_DIRs automatically -too many variables in their creation. I am capable of doing a few of them, but doing it for many, many apps is just more than is possible. Instead, I use normally-packaged-and-located programs and libs along with proper dependency management. And dependency-management can only be done at the *package level* and *at compile-time*. What I mean is that dependency information must be generated at compile-time for later use by the package manager.

Gawd, I haven't written so much in at least a year! But do feel free to extend this if I haven't(still?) covered your questions -both asked and unasked. You have potential or I wouldn't take the time...
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Thu 17 Jan 2013, 19:53    Post subject:  

Thanks amigo, high praise from one of the few I`d consider as an authority.
Quote:
Albert Einstein: I have always despised any form of authority.
To punish me for my contempt,
the universe has seen fit to make me myself an authority.


True, I usually ask system level Qs that are rather base or cutting edge.

So I think you`re saying that a junction ( Karl Godt offered the term ) link
would be integrated into the file system drivers?

The JLink would make a generic path for anything, in this case /etc, /var, /opt, /usr/share.
If Linux had PATHs for configs. and such this wouldn`t be needed.
It would do what`s needed here, it couldn`t replace a union of course ( too simple ).

# A link target is a dir./file, just like an exec. that`s searched for through the PATH.

Are you sure a PATH is as complex to manage as a same level layered FS?
A union is very cpu and ram intensive, a path seems relatively simple.
The Q is mount vs link complexity, a tough Q for sure. I like a links simplicity.
Follow the PATH`s paths in precedence until the dir. or file is found.
A union has a precedence ( path ) of it`s mounted layers to follow.

# Android is another beast all together... Now if we could "clean" Linux...

Yes, an AppPkg builder will work for "good apps.", odd apps. ( many ) are still manual labor.
The app. folks seem to love to obfuscate their packages. Anarchists.!

I chose to build with Ubuntu binaries, they do the work to assure a good tool chain.
So no compiling`s needed, just a good method of utilization that works most of the time.
Should be no lib. conflicts, so completely library-less AppPkgs should be possible.

Dependency info... What if Ubuntu just had complete dep. files with the URL minus the mirror?
I know this just makes too much sense, so it`ll never happen because of them.
But it would eliminate much of this dependency hell mainly due to package management.!!!
It`s the "Uncle Bill" syndrome of: "Do it my way or not at all!!!" Planned incompatibility.

# Q to myself is; Have the pkg. builder resolve deps., or make the dep. files spoke of above.
Sanity says make the files so it`s only done once for each Ubuntu distro. that`s supported.

Ideally all apps. would have relative paths, so links in AppPkg may be needed to the / FS.
But this solves any clean up problem as the mess is AppPkg self contained.
I had mixed luck modifying binary files, it`s sketchy at best not working on all of them.
Probably mainly the old apps., so as new replacements are written, maybe this will fade.

Most of the AppPkg setup file is linking the 4 paths and setting the PATHs.
But it runs quickly even on my piece of old crap Celeron PC, so it seems acceptable.

I think a good idea would be to run ldd in it when running the app`s. exec. fails.
Then an error popup showing the missing libs. ( if the AppPkg wasn`t built complete ).
Adds code to the setup file, but no big deal as the code only runs on a fatal error.
Could include the Ubu. download code ( once it`s done ), so it`d auto. fix the AppPkg.!


Thanks for conversing with me amigo, it helps with the global picture as well as details.

# If there`s any flaws with using Ubuntu packages this way... Yell.!!!

And any ideas you may have about creating dep. files is appreciated.
Script tools to parse Ubu. web pages seems like the best way to do it.
Complete dep. files with the sub URLs... I think that`s the ticket.!
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Fri 18 Jan 2013, 03:39    Post subject:  

No, handling of 'junction-links' would be separate from filesystem drivers -otherwise the code would only work for a particular filesystem. There's nothing hard about handling a PATH or junction-link with several possibilities -it's just that it is extra work which does not belong in the core parts of the kernel. The code for unionfs has not been mainlined exactly because of its complexity and size.

Lets back up a step though. The conclusion that using a union of APP_DIR/etc over /etc does *not* necessarily mean that all access attempts to /etc would be re-directed to APP_DIR/etc.

I have code in my src2pkg which uses a union+mount --bind+ chroot to do much the same as what you are wanting. It is not simple, but it is clean, flexible and doesn't require any links on your system. I have nothing against links and they are simple in principal. However, for your purposes, I I can't see the point of having a non-installed-but-available application which still needs to create/destroy links in the main filesystem. If you were running a 'sane' system, where the user wouldn't normally have write privileges it wouldn't work anyway. Maybe later I will post some code pointers which I was working out yesterday -it is an interesting project.

As for dpkg needing configuration, any program with *any* degree of flexibility will need some sort of configuration -either through a conf file or command-line options. dpkg does exactyl what you want as far as downloading and installing any extra depends. You do need to tell it which repos to use or ignore and you also need to blacklist quite a few packages -stuff which Puppy already supplies and which upgrades(from ubuntu) would break.

Creating a deps file is not terribly complicated, per se. It's just that it has to be done for each and every package on a system -you can only tell what package a file belongs to if you have a list of the files contained in each package on the system -at the time you build the new package.

My package manager (tpkg) uses files like this:
Code:
# acl_2.2.50-i586-1
# Requires:                              |Supplied by:                           
# File: libattr.so.1.1.0                 |: attr_2.4.45-i586-1                   
# File: libc-2.13.so                     |: glibc_2.13-i586-4                   
attr_2.4.45-i586-1
glibc_2.13-i586-4


They can only be done accurately when each package in the dependency chain which preceeds this one has been processed similarly. That means things have to be compiled or upgraded in the correct order. My scheme does not support using version numbers of <, > or =. The info given tells you the exact version which the current package was compiled against. Providing </>/= info would require human input each time a 'range' was used. It is impossible to programaticaly determine if a package will work with an earlier or later version of the same-named dependency.
As I said before, the only way to accurately determine depends is at the time of package creation and the only way to reasonably use and distribute the info is to have it be contained within the package. A one-time or universal/central database will simply not work reliably.

In my case, src2pkg generates the dep files for me and they are then kept on the installed system for any possible use.

As for modifying binary files, I assume you mean opening a binary file using a hex editor, or such. Any modifictaions you make must not change the size of the binary or it will not run. You can't change some hard-coded path except with another of the exacty same link. To do so you must use other tools which modify the ELF header so that the correct size and internal offsets are used.

Have you actually tried using unionfs for your goals?
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Fri 18 Jan 2013, 04:58    Post subject:  

No I haven`t tried using a union as I didn`t see how it could work.
However... I realized tonight that there`s a nifty way to do it.

For Puppy and other Linuxes with a union, just use a "sub save" layer.
A writable layer just under Puppy`s save layer, and write directly to it.
So the global view sees both save layers, and writes to the top one.
AppPkg apps. see the global view also, but write to the second layer only.
Put the sub save layer in a small ram disk as it holds only links anyway.
So then at each shutdown the sub save layer is wiped clean automatically.

As nice as this is, it negates AppPkg being portable, slaved to a union.
I gave up on SqApp as it required too much infrastructure ( not portable ).
AppPkg carries all of it`s infrastructure inside itself, assuring portability.

I`ve looked at dpkg and never understood most of it`s workings.
Need a clear picture of making it get lib. packages minus installed libs.

Correct. A dep. file would be: the app., the libs., the packages., and URLs.
It`d be nice to have a pool of bare lib. files making it easier to find them.

With Ubuntu handling the dep. chain, I don`t see why a solid set of dep.
files for that Ubuntu release ( ie: Precise ) would not work always.
Being a stable release it should never change... Right?
Again, no compiling anything, the target O.S. must be Ubuntu compatible.
### This is critical... Tell me if and how this view is flawed.!

Lots of the Ubuntu repository has no version release associated with it.
Many of the apps. and deps. ( libs.) work for the few releases around it.
Many Lucid apps. work in Precise, but newer apps. and libs. not in older.
Libs. seem to be backward compatible for a few numbers, sometimes.

This replaces all of a binary`s /usr paths with ././ ( relative path ):
Code:
find . -type f -exec sed -i -e "s|/usr|././|g" {} \;

There are also utilities made to modify binary files ( bbe, patchelf, etc.).

### So where is the link code at then? Just in case I need to know... Wink
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Fri 18 Jan 2013, 09:56    Post subject:  

Nah, you don't need to and shouldn't mess with the save layers of your existing aufs(right?) setup. Running the AppDir should set up a separate mount using aufs, unionfs, mhddfs or unionfs-fuse. The writes really belong in the users $HOME dir. The AppDir itself could be anywhere.

A separate mount point is used to mount the union on and, once layered you chroot into the mountpoint and run your program normally. Anything the program contains which is newer than the installed system take precedence, as they should. The mountpoint and the writes should really be in $HOME/Choices/$APP_DIR.

"slaved to a union" Shouldn't you really mean: I'm glad I have this flexible, just-works technique as an option?

Even if you use aufs, the whole thing is separate from any unions created by Puppy, but I find unionfs-fuse to work fine. grab the sources, compile and install -or wait, ubuntu has mhddfs in their repos (maybe unionfs-fuse, too), so you can simply install from dpkg, right?

Here's a very quick scenario that will demo it for you:
Code:
#!/bin/sh
mkdir -p write union prog
# prog is where you have DESTDIR-installed program content
# write is where you want changes to be kept -should be ~/Choices/NAME_OF_THIS_APP_DIR
# union is where the union is mounted and where you'd chroot to run your program

# put some fake content in prog
mkdir -p prog/bin prog/etc
echo testcode > prog/bin/myproggie
# use a known-name to see how we override /
echo testconf > prog/etc/resolv.conf

# for use with unionfs patch and fuse
# mount -t unionfs -o dirs=write=rw:/=ro unionfs union
# mount -t aufs ... similar to above
# fusermount, pmount or other would be needed for non-privileged users

# this is the unionfs-fuse executable which uses only fuse
unionfs -o nonempty -o cow write=RW:/=RO:prog=RO union
# the write is RW, then root(/) is blended in, then our proggy itself which overrides /

View the file in union/etc/resolv.conf. Notice that your system file /etc/resolv.conf is not changed or showing the file union/etc/resolv.conf or write/etc/resolv.conf
Then, open the file union/etc/resolv.conf. Edit and save it.
Now the file will be saved at write/etc/resolv.conf.

To unmount it run: umount union
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Fri 18 Jan 2013, 17:33    Post subject:  

Thanks much amigo; I was wondering where you are that you`re so cold.

Much the same as my suggestion except using a separate union and chroot
so the app. thinks it`s normally installed. No links or paths, simple, clever...
# So prog, write, and union are all inside of AppPkg, so configs. go with it.

There`s a lot to like about it. But I`m designing a no union O.S. with no
apps. except VT, editor, file man., network, etc. Everything else is AppPkg.
A union, and the type of union can`t be relied on for the target O.S.
It`s necessary for AppPkg to carry it`s own solution inside of itself.
So UnionFS and aufs are out of the Q as they`re embedded in the kernel.

### You said kernel space apps. only work for privileged ( root? ) users?
# So only user space "union type" apps. will work run from inside AppPkg?

When I found mhddfs I thought it was my answer, not a J-Link but like it.
Then it was pointed out that it like all FUSE stuff is user space and slow.

### In light of all of this, a J-Link avoids all of the problems and is fast.

Need to rethink this whole setup, having AppPkg appear in / is a good idea.
It simplifies the deployment methods, and also the AppPkg builder as well.

### Time for more reading. Thanks again mi amigo... Terry B.

# P.S. ... Made your setup work with mhddfs, but not with fusermount.

# P.P.S. ... So where would the code for link behavior be located.?
.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Sat 19 Jan 2013, 05:57    Post subject:  

I'm in Germany, but not German.
There are no kernel-space 'apps'. kernel space is the kernel itself and naturally everything there is run as root. userspace is everything outside the kernel.
What was the syntax for mhddfs? I've used it before and seem to remember it was similar to unionfs-fuse -maybe more limited.
You'd have to ask on lkml or search to find exactly where links are handled in the kernel. But, your idea of junction-links is really no different than any of the already-implemented techniques. The only difference is in the location where it is handled.

unionfs and aufs handle it at the point precedding the files system drivers. Neither is a real file system -notice that they will work with any filesystem you like. They insert theirselves just between the VFS and the filesystem. They register themselves with the VFS as being a filesystem and *interpret* and handled read/write requests before handing them on to the filesystem drivers.

unionfs- and mhddfs use fuse, which is a sepcial access point to the VFS -designed exactly to allow any sort of wierd and/or ono-privileged access to the VFS -allowing normal users to use mount, creating compressed filesystems, etc.

COW solutions operate at the block level and interface with the kernel in the block-layer code.

I am unsure of the exacty point where links are resolved -but they resolve to inodes in the end. I believe the VFS speaks only inode-talk. Anyway, you have an order something like:
VFS - inodes - links - block-layer - mounts - filesystem driver

As I said, your idea only differs in the chosen point of interaction. The idea will never fly with kernel devs for the same reason that none of the others do. If you see an advantage because you only want a limited capability, then any solution besides aufs would be the best. aufs is the most capable of all the solutions. Hence, the largest and most complex. mhddfs or one of the old cow solutions are much simpler. Your implementation of j-links would also have to be done in the kernel -just like unionfs or aufs -unless you create such an implementation using fuse, which is outside the kernel but allows non-privileged use.

"designing a no union O.S." Do you mean you are building from the ground up with no puppyisms at all -like a normally installed distro?

"FUSE stuff is user space and slow." Yes, fuse implementations are slower than a builtin kernel feature. But the usage you suggest is not intensive. unionfs-fuse is fast enough that disk read/write speeds are the bottle-neck -just like with real kernel features.

"no apps. except VT, editor, file man., network, etc. Everything else is AppPkg" How/where do you draw the line? Isn't a mixed system more confusing and complex than using the concept (for making software available) throughout. If the AppPkg idea is so good, why can't it be applied and used for everything? (Go back to the top of the thread and read it all again to see why that is not the case). Any system is shot full of libs and programs which do not lend themselves to the AppPkg concept. While many larger applications can be fit into the scheme, their library requirements mostly do not. And lots of basic items also do not fit at all. The whole thing evolves into the issue of resolving dependencies. Even your AppPkgs have to deal with that unless you truly create each with *everything it needs (even glibc) included* in one deliverable item. This will always mean duplication and larger deliverables(read package, SFS, AppPkg, app, AppDir, etc).
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Sat 19 Jan 2013, 16:13    Post subject:  

Please forgive my quaint references ( kernel apps.), "kernel code and modules".
From AppPkg setup file ( mhddfs uses commas instead of colons ):
Code:
# Mounting:
mhddfs $Pkg/$Pkg.rw,$Pkg/$Pkg.app,/ $Pkg/$Pkg.u

# Unmounting is:
fusermount -u $Pkg/$Pkg.u

AppPkg runs faster, but then the setup file is half the size now, just I said it would be.
### If you`d like to see my files and code, I`d appreciate critique.

mhddfs works well, at 30 KB size it`s more capable than is needed. Only uses libfuse.
I tried to get fusermount to do it, but I think it doesn`t do unioning. Only uses libc.!
I got unionfs-fuse to try it, but bigger than mhddfs at 39 KB in size. Only uses libfuse.

### AppPkg portability... How to get AppPkg to load the fuse kernel module.?
This would make it possible for AppPkg to work on any compatible O.S.


Base target O.S. would be all Ubuntu ( or could be Slack ) with just needed utilities.
Much like Tiny Core, little installed apps. and infrastructure, unlike Puppy`s app. suite.
There`s new apps. all the time, it should be a snap to replace them ( not permanent ).

The line is just what`s needed to do O.S. maintenance, and maybe a little more.
This so you can do work with the base booted O.S., same reason TC`s like this.


With Ubuntu handling the tool chain, AppPkg could dnld. needed libs. to a shared dir.
Shouldn`t be any lib. conflicts, but if there are put the lib(s). in AppPkg to override.
So most AppPkg have no libs. or only the app. specific ones. Almost a normal setup.

I understand what you`re saying, Linux is not orderly... We try to provide it.
I think dep. files made from a given Ubuntu release`s web pages should be correct.
Otherwise Ubuntu has serious problems to address ( Nothings perfect in this life...).

### Again... If this method has flawed, how best to fix or address it.?
You`re saying dep. resolving is the problem, Ubuntu can get it right I hope.
Maybe I`m hoping for too much basing AppPkg and my O.S. on Ubuntu`s structure.?

# Sorry about the long posts, many subjects discussed here. Very interesting it is.

I thought about the difference between one Sq. file and links pointing into it,
or mounting a Sq. file on each r-o dir.: /bin, /lib, /sbin, /usr/bin, /usr/lib, /usr/sbin.
There`s more mounts, but there`s no link to resolve, slightly faster I`d think.
But like link vs mounts, I think I`m obsessing too much trying to optimize the O.S.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Sat 19 Jan 2013, 17:03    Post subject:  

"but if there are put the lib(s). in AppPkg to override" The problem with using a general location for any updated libs is the same as with the standard dirs. what to do when one of your apps needs a still different version?

My conjecture seems to be pretty close -that you want to build something based on ubuntu, using their kernel and everything -which is why you don't want unionfs or aufs. Actually that is good thinking. If you base your project on someone else's work, then the less alterations you make the less trouble you have.

You don't really have to worrry aboue FUSE then, ubuntu and all other distros will have it available. loading the kernel module takes no more: modprobe fuse
But, you may not even have to do that -if the kernel has auto-loading of modules enabled. If not, simple enough to add the command to whatever script you are writing which starts an app which needs it.

No need to maintain any deps file -for ubuntu it would really be huge. dpkg itself will handle fetching the packages for you and excluding anything already-installed -providing your system has the local dpkg database intact. Anything you don't want updated/overwritten can be blacklisted.

That said, you might do well to use Slackware or maybe archlinux instead of Eeeww-buntu LOL. Slackware is the ideal place for learning how things really work. arch is nearly as good there, but has a dependency-resolving pkg mgr. Of course, neither of them has the 'depth' of packages like ubuntu. At the very worst/best, use debian instead of ubuntu for more normality -use the testing branch if you want really up-to-date stuff. Or, use the stable branch for the best-tested combinations(and whole) anywhere.

"Sq. file" Are you taking about squashfs file-system images SFS files? Note that you can use that technique within any appdirs, as well. Instead of unioning an open dir structure, you can union an FS image also.

As far as the wisdom of trying to use appdirs everywhere, they really become lots of work when you try to maintain a few of them and are really inadequate for libraries or required software. At some point, it is just easier to create packages which use whatever dep-resolution which is available. And, at runtime, you could simply have an empty dir, mounted on tmpfs where you 'install' a series of packages temporarily for your chroot or whatever.

I might have a look at your code if you make it available -but no promises. If it's very long or hard-to-read I lose patience...

My system KISS-linux, is similar to Slackware, but you can do a truly-minimal install with a small number of packages(~30) and expand it without limits. That's just a bootable system with CLI login, no networking or non-local hardware. Minimal really means minimal. The list of packages is easy to understand and contains no cruft. Reading through the init scripts is a breeze and lets you understand exactly what is needed (and not) to bring up a basic system -I've out-slacked Slackware, as it were. So far, I do not provide actual resolution of dependencies -all the info is there as it is built right into every package. But I have been hesitant to implement resolution -I am an old Slacker, after all.

mhddfs vs. unionfs-fuse:
mhddfs is not well maintained, while unionfs-fuse is. Do you know how small 9K is? How many places are you willing to sacrifice usuability against a few K's? takes a lot of them to make 1MB...
There are other tools which might be helpful, but unionfs-fuse is probably the best candidate. Hey, I use it in src2pkg, so it must be good... Actually, I even got an email from the author once alerting me to a needed update -I had never written him, but he had noticed me using his stuff and wanted to be helpful.

Out of curiosity, why do you pursue the AppPkg line of thinking? Is it about run-time modularity, or about providing apps which are not available direct from ubuntu? Can you define more clearly the goals?

I don't mind batting things around with you -as long as it stays interesting! LOL
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Sun 20 Jan 2013, 12:03    Post subject:  

I`ll attempt to address each subject.

My experience is: links are reliable, mounts are less so, sockets and pipes are worse yet. Viva le link.!

Common libs. ( widely used ) go into a common dir. like /opt/lib. The dirs. /lib and /usr/lib are R-O in a Sq. file.
App. unique libs. go in the app`s. Sq. file or with the app`s. loose files in the mount dir.: (pkg. name + ver.).app
Optionally inside the AppPkg is a /lib dir., libs. in it are shared among any and all of the apps. inside the AppPkg.
So there`s 3 levels of libs., system wide shared level, AppPkg shared level, and app. unique non-shared level.

I spent a bit of time at the Debian site, I take it you put Debian above Ubuntu... That`s undoubtedly good advice.
A big package base makes it worth the effort to develop AppPkg and it`s app. downloader and the AppPkg builder.

I`m glad that it`ll be easy to load the needed kernel module, and union-fuse does seem more made for the job.
My learning curve will be making dpkg work for the AppPkg builder. I`ve used it and never got it to work properly.

Sq., a generic squash file, not an SFS that`s made to be unioned. Although now AppPkg is using unioned Sq. files,
which makes me realize that Puppy SFS files can be dropped straight into AppPkgs for Puppy without modification.
AppPkg is an AppDir or RoxApp. They don`t have a set internal structure. AppPkg sets a standard layout.

The AppPkg Sq. file is now essentially an SFS file, the rest of the AppPkg is 2 static scripts and 1 unique script.
There`s also the union-fuse files, an AppPkg popup menu exec. file and it`s unique menu.lst file for multi. apps.

AppPkgs are for apps., and maybe some services, daemons, etc. It`s not intended for installing libs. to the O.S.
If I get AppPkg to auto. download libs. to /opt/lib, it`s only in support of AppPkg, not for system lib. upgrading.

AppPkg is an enhanced AppDir, it does the same thing for the same reasons, modular, portable, no-install apps.
AppPkg improves on them with multiple apps., local libs., and a popup menu. They can be made do the same too.

An interesting idea, dump all the apps. into one dir. so only one union and chroot. It has fewer layers than Puppy.
But then it starts to have Puppy`s same conflict problems with layering so many apps. and libs. over each other.
Your idea of 1 union per app. runs the apps. in a sandbox effectively isolating them from other apps. and the O.S.


Ultimately a tiny Debian-Ubu. distro. with X, WM, GTK, Xorg, network, and utility apps.. About a 40 MB download.
Download only the AppPkgs you want, single apps., or same app. type "try them" bundles, or groups of like apps.
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5032
Location: Arizona, U.S.A.

PostPosted: Sun 20 Jan 2013, 12:23    Post subject:  

Here`s the 3 scripts that make your union-chroot setup work.
The first script is unique for Chrome, the other 2 are generic:
Code:
======= Script: chrome
#!/bin/sh
cd ${0%/*}/.AppPkg
export Pkg="chrome-24_i386"
./setup google-chrome --user-data-dir=$HOME/$Pkg/profile &

======= Script: setup
#!/bin/sh
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   Mnt. Sq. Files & Union, Run App.
if [ "$1" ];then
   echo -e "\n###  Run:  $Pkg"
   App="$Pkg/$Pkg.app"                           ### Mount App. Sq. File
   Sq="$Pkg/$Pkg.sq"
   [ ! -d $App ]&& echo -e "\n###  ERROR:  NO App. Dir.:  $Pkg.app\n" && exit
   [ ! "`ls $App`" ]&& if [ -f "$Sq" ];then            # IF app dir. empty
      mount -t squashfs -o loop $Sq $App               # IF Sq. file, mnt.
      [ $? -gt 0 ]&&
         echo -e "\n###  ERROR:  Fail Mount App. File:  $Pkg.sq\n" && exit
      echo -e "\n#  Mount App. Sq. File:  $Pkg.sq"
   fi
   Lib="$Pkg/lib"
   Sq="$Pkg/lib.sq"                           ### Mount Lib. Sq. File
   [ -d $Lib ]&& if [ ! "`ls $Lib`" ];then               # IF lib dir. empty
      [ -f "$Sq" ]||                              # IF no lib. Sq.
         echo -e "\n###  ERROR:  NO Lib. Sq. File:  lib.sq\n" && setup &
      mount -t squashfs -o loop $Sq $Lib               # IF lib. Sq., mnt.
      [ $? -gt 0 ]&&
         echo -e "\n###  ERROR:  Fail Mount Lib. File:  lib.sq\n" && setup &
      echo -e "\n#  Mount Lib. Sq. File:  lib.sq"
      fi
   [ "`ls $Lib 2>/dev/null`" ]&&                     ### Set lib. path
      LD_LIBRARY_PATH=`realpath ./`/$Pkg/lib:$LD_LIBRARY_PATH
   mhddfs $Pkg/$Pkg.rw,$Pkg/$Pkg.app,/ $Pkg/$Pkg.u         ### Mount union fs
   [ $? -gt 0 ]&&
      echo -e "\n###  ERROR:  Fail Mount Union:  $Pkg.u\n" && setup &
   echo -e "\n#  Mount Union:  $Pkg.u"
   ./run $@ &                                    ### Run hook file

else #>>>>>>>>>>>>>>>>>>>>>>>>>>>>   Unmount Union & Sq. Files.
   echo -e "\n###  End:  $Pkg"
   fusermount -u $Pkg/$Pkg.u                         # Unmount union fs
   [ $? -eq 0 ]&& echo -e "\n#  UnMount Union:  $Pkg.u" ||
      echo -e "\n###  ERROR:  Fail UnMount Union:  $Pkg.u\n"
   Mnt=`mount`
   App=`echo "$Mnt" |grep $Pkg.app`
   if [ "$App" ];then umount -d $Pkg/$Pkg.app            # Unmount App. file
      [ $? -eq 0 ]&& echo -e "\n#  UnMount App. Sq. File:  $Pkg.sq\n" ||
         echo -e "\n###  ERROR:  Fail UnMount App. File:  $Pkg.sq\n"
   fi
   Lib=`echo "$Mnt" |grep lib.sq`
   if [ "$Lib" ];then umount -d $Pkg/lib               # Unmount Lib. file
      [ $? -eq 0 ]&& echo -e "\n#  UnMount Lib. Sq. File:  lib.sq\n" ||
         echo -e "\n###  ERROR:  Fail UnMount Lib. File:  lib.sq\n"
   fi
fi

======= Script: run
chroot $Pkg/$Pkg.u $@
./setup &

I`m making a few AppPkgs with your setup as proof of concept tests.
It works with xMahjongg, but Chrome doesn`t like something about it.
Chrome`s startup script gets the wrong path, or no path, I`m not sure.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2251

PostPosted: Sun 20 Jan 2013, 12:25    Post subject:  

Do you mean to have the OS as a LiveCD or set of a few files which can be manually copied to a USB stick? Or do mean a fully installable?
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 6 [84 Posts]   Goto page: 1, 2, 3, 4, 5, 6 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1902s ][ Queries: 13 (0.0108s) ][ GZIP on ]