Static programs - isolating library $PATH [SOLVED}

Using applications, configuring, problems
Message
Author
User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

Static programs - isolating library $PATH [SOLVED}

#1 Post by greengeek »

Imagine that I am using a stripped down puppy and want to add a program which needs certain libraries. Those libs are expected to be in, say, usr/lib. However, some other static program may already be using usr/lib as storage for its own libs, and those libs may share the same name as the libs for the new program.

Is it possible to somehow "isolate" the location of certain libs so that a static program is "sandboxed" and unable to be contaminated by an incorrect lib from the host core, or some other program?
Last edited by greengeek on Wed 02 Jan 2013, 08:11, edited 1 time in total.

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#2 Post by RSH »

Hi.

i think "sunburnt" is working on something like this. Called: AppPkgs.

http://murga-linux.com/puppy/viewtopic.php?t=81578

RSH
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#3 Post by greengeek »

Thanks RSH, that looks very interesting. I'd like to see some fork of Puppy use this sort of approach and maybe build it's own repository. Doesn't matter if it is compatible with nothing else. I think it would allow a small core and lots of add-ons that were basically static packages, safe to run without risking conflicts with other packages.

Almost like lots of mini-sfs clustered around a nucleus.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#4 Post by amigo »

A statically-linked program means that the libraries are built right into the program, so libs aren't even looked for anywhere else.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#5 Post by greengeek »

Ok, I was misunderstanding this. I thought a static program simply loaded all of its necessary libs into the filesystem when it ran for the first time. My thinking was that this would potentially cause conflicts with other statics and I was visualising some way that each static could keep its necessary libs completely separate from the filesystem. (and just output data to a "safe" file or location/partition or display window)

Almost like the system core would be capable of running several VMs - each containing some static program (rather than an OS).

This is sort of how I visualise an sfs to be operating (like a VM), except that I think an sfs becomes part of a unified file system - which seems to carry the burden of ensuring that each sfs is compatible with all other programs running in the overall "unified filesystem".

I feel I'm not explaining this too well.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#6 Post by greengeek »

amigo wrote:A statically-linked program means that the libraries are built right into the program, so libs aren't even looked for anywhere else.
I'm definitely confused by this. I've been looking at several "static" programs and none of the ones I looked at carry a full load of libs - they may have some of their own libs within their specific directory, but the programs also seem to want to go looking for other libs that are either part of puppy itself, or else have been loaded by other pets or by me manually.

What I was hoping for was a mechanism to have a program running completely standalone (as much as possible anyway...) so that it did not need to access any libs that have been loaded by other programs or pets.

Maybe the "statics" I have been looking at are incorrectly assembled? Pseudo-statics maybe?

User avatar
Dingo
Posts: 1437
Joined: Tue 11 Dec 2007, 17:48
Location: somewhere at the end of rainbow...
Contact:

pdftk 1.41 static

#7 Post by Dingo »

as far I know, true STATIC apps, look like this pdftk version (14.1) I built following these tips

http://dokupuppylinux.info/programs:pdf ... lly_linked

link to non-upxed version

http://www.divshare.com/download/22405017-3bd

executing ldd on this pdftk static version, returns:

Code: Select all

 ldd ./pdftk
        not a dynamic executable
replace .co.cc with .info to get access to stuff I posted in forum
dropbox 2GB free
OpenOffice for Puppy Linux

npierce
Posts: 858
Joined: Tue 29 Dec 2009, 01:40

#8 Post by npierce »

greengeek wrote:I feel I'm not explaining this too well.
Well, I think I know what you are saying, although using the term "static programs" is confusing, since (as amigo has already pointed-out), it makes one think of statically-linked programs, which create no such problems with searching for libraries, since the necessary code is already in the executable binary. But you seem to be looking for something else.

I think you are actually referring to programs that use shared objects (e.g., shared libraries) and have one or more of their own libraries segregated from the general collections of libraries used by other programs. Right?
greengeek wrote:Those libs are expected to be in, say, usr/lib. However, some other static program may already be using usr/lib as storage for its own libs, and those libs may share the same name as the libs for the new program.
So if I understand you correctly, you are talking about a situation where a program uses a library that has the same name as a library that already exists in the /usr/lib/ directory, but is totally different.

If that is the case, then, yes, there are ways of handling that situation.

The ideal, of course, would be for the developer of the library to come up with a new name. But in a world with thousands of people developing libraries, such name conflicts are inevitable, and convincing one developer that her library should be the one that is renamed, and not the other, may not be easy.

So you get to work around the conflict.

As you have already guessed, you will create a new directory in which to put the library. Let's call it "/usr/local/lib/example_app/". And let's call the library "libexample.so.0", and the program "example_app".

So, in this example, you would have an existing library at /usr/lib/libexample.so.0 and a different library for use by example_app at /usr/local/lib/example_app/libexample.so.0 .

There are various methods that can be used to tell the loader to look in the latter directory at run time.

1. RUNPATH

If you have the source code, you can set the RUNPATH attribute at build time. This attribute is used to define a library search path and lives in the executable file, where the loader will find it at run time. Here is a simple example (which is all one long line):

Code: Select all

gcc -o example_app example_app.c -L/usr/local/lib/example_app -lexample -Wl,-rpath=/usr/local/lib/example_app,--enable-new-dtags
Unfortunately, although this is perhaps the best way to do what you want, this probably won't work for you in Puppy because many (perhaps all?) Puppies set the LD_LIBRARY_PATH environmental variable, which overrides RUNPATH. (I do not know why Puppy does this. The usual way to define the general library search path is to use the /etc/ld.so.conf file and ldconfig. LD_LIBRARY_PATH is usually just used temporarily for stuff like testing new libraries and debugging.)


2. RPATH

This is an older method of defining a library search path in the executable, using the RPATH attribute. This method is deprecated, since it cannot be overridden by LD_LIBRARY_PATH, which caused many headaches in the past because sometimes it is useful to be able to override the path. So it was replaced by RUNPATH, which is overridable. (Actually, the -rpath combined with the --enable-new-dtags options sets both RUNPATH and RPATH, but if RUNPATH is set, modern loaders ignores RPATH.)

RPATH is set similarly to the way RUNPATH was set, but without the --enable-new-dtags option:

Code: Select all

gcc -o example_app example_app.c -L/usr/local/lib/example_app -lexample -Wl,-rpath=/usr/local/lib/example_app
The above code is simplified to illustrate the options that need to be passed to the linker. But in the real world you are unlikely to be calling gcc directly to build a source package. You will probably need to either pass them to a configuration script, or set them manually in a Makefile (for instance, using LDFLAGS). Of course, the exact details of doing so will depend on the source package.

Because it is deprecated, I would not recommend that anyone use this method on a large-scale basis, but to solve an occasional problem like your library name conflict on Puppy, it could be useful.

By the way, the following command will show you if RPATH or RUNPATH are set in a program:

Code: Select all

readelf --dynamic example_app | grep -E "RPATH|RUNPATH"

3. LD_LIBRARY_PATH

This might be what you need if you have only the executable binary, not the source code, and therefore cannot set the RUNPATH or RPATH variables at build time.

As previously mentioned, LD_LIBRARY_PATH is usually just used temporarily for stuff like testing new libraries and debugging, but if the RUNPATH and RPATH methods are not available to you, this could do the job. For example, on the command line:

Code: Select all

LD_LIBRARY_PATH=/usr/local/lib/example_app:$LD_LIBRARY_PATH example_app
or in a wrapper script:

Code: Select all

#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/lib/example_app:$LD_LIBRARY_PATH
exec example_app
Note that the the new LD_LIBRARY_PATH also includes the old LD_LIBRARY_PATH, but the new directory is first in the path so that it is searched first. Of course, if all the needed libraries are in the new directory, or in /lib/ or /usr/lib/, or in /etc/ld.so.cache, you don't need the old value of LD_LIBRARY_PATH, and can simply use "LD_LIBRARY_PATH=/usr/local/lib/example_app".

Note also that the above examples cause LD_LIBRARY_PATH to be set to its new value only for running example_app. You would not want to set it permanently for a couple of reasons: first, because doing so would cause the loader to search the new directory first for libraries when running any program, which would be inefficient, and second, because the other library with the same name at /usr/lib/libexample would never be found when it was needed, since the one in the new directory would always be found first.

Be aware that this method will not work with programs that have their set-uid or set-gid permission bits set (unless, respectively, the user ID matches the file's owner or the user is a member of the file's group). (Allowing users to cause their own libraries to be loaded in such programs would, obviously, be a very large security hole.)


4. LD_PRELOAD

If you are just dealing with one or a few libraries, you could use this method to list those specific libraries so that the loader will load them first. It works similarly to LD_LIBRARY_PATH except that you list the absolute name of each library, not just the directory. (Normally this is used for stuff like testing a new library.)

For example, on the command line:

Code: Select all

LD_PRELOAD=/usr/local/lib/example_app/libexample.so.0 example_app
or in a wrapper script:

Code: Select all

#!/bin/bash
export LD_PRELOAD=/usr/local/lib/example_app/libexample.so.0
exec example_app
As was the case with LD_LIBRARY_PATH, this LD_PRELOAD method won't work with programs that have their set-uid or set-gid permission bits set (unless, respectively, the user ID matches the file's owner or the user is a member of the file's group).


Further reading - - -

I've tried to give a brief overview of some of your options, but I've probably not explained it well enough to make it really clear. Here are a few suggestions that may help to clear things up:

Program Library HOWTO Shared Libraries by David A. Wheeler

ld.so( 8 ) - Linux manual page

RPATH, RUNPATH, and dynamic linking by W. Trevor King

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#9 Post by amigo »

Let's get the definitions here. There is no such thing as a 'static program'. The correct term is statically-linked, although sometimes statcially-compiled is used.
It is possible to statically link only parts of the libs which any program needs. A completely statically-linked program will contain *all* the libs it uses whatsoever -meaning that the linker will not look for any libs outside the program. Obviously, if you used only statically-linked programs, there would be much duplication of code in your programs -each of them would be larger than the same program dynamically-linked (which means the linker looks for and uses *shared* libs found on your system (*.so libs).
LD_PRELOAD and LD_LIBRARY_PATH are used by the linker for searching for *shared* libs.
The 'linker' is the file /lib/ld-linux.so.2. The kernel does not directly execute programs -instead it calls the linker to do that. The linker looks at the first few bytes of the 'program' (or library) and figures out whether it needs any shared libs or not. If so, then it loads these external shared libs (if it finds them) before loading the actual program.

It's helpful to think of static-linking as 'including' -anything so linked will be included right into the executable.

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#10 Post by Flash »

I've heard the terms ROX-app and application-in-a-directory thrown around. ( file:///usr/local/apps/ROX-Filer/Help/Manual.html#appdirs ) I believe these are applications that contain all the necessary stuff in one directory so they don't depend on shared libraries and are easy to use and delete without affecting anything else.

Sharing libraries sounds like a good idea until you run into applications that require different versions of a library. I didn't even know there were different libraries that have the same name. :(

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#11 Post by greengeek »

Thank you all for the very helpful and indepth info. There is a lot for me to digest and research further.
npierce wrote:Well, I think I know what you are saying, although using the term "static programs" is confusing, since (as amigo has already pointed-out), it makes one think of statically-linked programs, which create no such problems with searching for libraries, since the necessary code is already in the executable binary. But you seem to be looking for something else.
I think you are actually referring to programs that use shared objects (e.g., shared libraries) and have one or more of their own libraries segregated from the general collections of libraries used by other programs. Right?
I think what I was confused about is there seem to be plenty of “statically-linked

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#12 Post by amigo »

In fact, even two libraries of the same version can be incompatible -according to the compile-time options chosen. One version may include a capability (implemented as extra code or inclusion of other externals) which the second lib does not have. If the program you want to run was compiled against lib #1, it will be expecting that specific function or functions. lib #2, failing that extra function will not always satisfy the programs real needs.

This is why picking and choosing libraries from just anywhere you find them will not always work and is a Bad Thing (TM). Properly upgrading a system means re-building everything which depends on something else -each time the something else gets upgraded (or re-compiled with different options which affect compatibility.) Since there are hundreds of programs which all depend on at least glibc, upgrading to a newer glibc means recompiling all those programs -whether or not upgrading those programs at the same time.

Also, about RoxApp's and other bundle/app/single-dir "packages". They commonly will depend on some exterior shared libraries being available on the system -including, most commonly, glibc.

The problem with having every program being completely independent from external libs, is that many, many programs will include a copy of some libs. If every program that needs glibc included the static library of those functions, then any system would be hugely bigger. If every GUI app included all of glibc, xlibs&Co. and GTK-plus-hangers, you'd have lots of apps being really huge. On the other hand, statically-compiled programs usually start faster -but use lots more RAM than shared-lib counterparts.

Anyway, apps usually *do* depend on some expected libs/progs -possibly with a specific version or range of versions.

OP, your best way to package nice, (mostly) self-contained apps, while avoiding duplication, would be to package any needed external libs which you do *not* normally expect to be installed as a separate library AppDir. This would also allow you to install and use alternate versions of already-installed libraries which you want to to 'over-ride', without over-writing. The AppRun script for the program which needs these alternate libs would simply set the LD_LIBRARY_PATH to point to the alternate location.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#13 Post by sunburnt »

Rule number 1; ### Always listen to amigo... :wink:

Puppy`s devx file is a good example of add-on libraries and apps.
There could be a few more for specific purposes, like media, etc.

But as amigo points out, all apps. that use them need to be compiled with them.
You can`t just install a programming language package to Puppy.
Expecting it to work as is,... Maybe yes, and maybe no.
If it uses libs. in the devx file or Puppy, they may not play well together.

That`s why I`ve had very good luck with Lucid Puppy.
It`s based on Ubuntu Lucid and using Ubuntu Lucid apps. works great!
I hope Precise Puppy gets it`s bugs worked out so there`s an upgrade path.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#14 Post by greengeek »

amigo wrote:On the other hand, statically-compiled programs usually start faster -but use lots more RAM than shared-lib counterparts.
Why would a statically compiled program use more RAM? I can easily see why you would need more RAM if you had a number of statically compiled programs all running at once (they would take up less ram if they shared common ground...) but wouldn't a single program need pretty much the same amount of ram to accomplish it's functions, regardless of static or no?

What I'm aiming for is a puppy with a small core (eg pupngo) and very little else - in fact I'd be happy with a word processor as the only other add on, if it allowed me to make use of the older PII PCs I'm trying to restore. If a statically compiled word processor is going to require significantly more RAM I'm probably barking (get it?) up the wrong tree.
amigo wrote:Bad Thing (TM)
<chuckle> so many of the things I've stumbled through with Puppy fit into the "BadThing" category. Much trial and error and lots of swearing.
Last edited by greengeek on Tue 01 Jan 2013, 01:06, edited 1 time in total.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#15 Post by greengeek »

Quote from one of the links I posted:
jpeps wrote:Have you checked out CDE? Automatic packaging of Code,Data,Environment. I used it a while ago to send exactimage to Dingo, and he was able to run it his own distro.
http://www.pgbovine.net/cde.html
Sounds quite interesting. Almost like the App-VM idea I was trying to convey.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#16 Post by amigo »

A statically-linked program will require more RAM simply because it is larger. The size difference can be dramatic, depending on just how many libraries it needs. Of course, using a couple may be alright. But some of the things you might most want to have as statically-linked applications may be very large indeed -FF, Seamonkey, Thunderbird, Chrome, etc.

Still, the best way to keep things small is to know exactly what each thing needs. Only very good dependency information can provide this -and dependency info can only be accurately determined *at compile time*. Since even compile-time options can change the requirements of a lib/program, let alone inter-versional differences, the real requirements can only be determined on the build system, at build-time.

npierce
Posts: 858
Joined: Tue 29 Dec 2009, 01:40

#17 Post by npierce »

[quote="greengeek"]Heres an example: Lets say I want to start with a stripped down Puppy (pupngo) and add only two other programs - a word processor and a browser: Imagine that both of those programs are labelled as being “statically linked

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#18 Post by sunburnt »

And that`s exactly what RoxApp or AppDir or AppPkg does.
They setup a run environment for the app. and then run it.
This is $PATH, $LD_LIBRARY_PATH, and links and dirs. needed.
No union[/n] is needed for these apps., they work with or without one.

If Linux and it`s apps. were setup properly, no need for the links and dirs.
But the original creators of Unix couldn`t have known of this type of use.

amigo has a RoxApp setup that compiles the app. for the distro ( very nice ).
This is the best arrangement for installing odd apps. into odd Linux distros.

As I`ve said, Ubuntu Lucid-Precise are very compatible with the Puppy versions.
I`m sure there`s a few apps. that would have problems, but not many.
So AppPkg or RoxApps made from these should work very well.

AppPkg is compatible with RoxApps, but it allows multiple apps. in one.
It allows app. specific local libs., and AppPkg shared local libs., and it
uses the Linux O.S. shared libs too of course. So no lib. conflicts...

There are other things that keep an app. from working, but libs. are a big one.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#19 Post by greengeek »

npierce wrote:OK, so you are basically trying to prevent the age-old problem of newly installed packages overwriting shared libraries with versions that are incompatible with previously installed programs. You want each program to be able to find a library that is compatible with it without stepping on the toes of other programs' libraries.
Yes you've summarised it nicely.
Still, while the implementation of library versioning is imperfect, it might be worth reviewing here how things are SUPPOSED to work.
Yes, thank you for the indepth info. It is very helpful. It seems to me there will be times when I want to try and do the right thing, and other times when I will want to cut corners, and just get something going the quick and dirty way - but it helps a lot to understand the risks and potential solutions.
Perhaps it was helpful to the understanding of why the problem exists, or perhaps not.
Very much so. Thanks.
I realize that we no longer live in a world where RAM is measured in kilobytes and disk drives are measured in megabytes -- if you've got the space, use it as you like.
Somehow buying a PC that has more than 1 GB of RAM seems morally wrong. And I have heaps of older machines that would benefit from a lean O.S. Apart from that I feel very insecure using software that is bloated and inefficient. Rightly or wrongly it makes me feel that it is potentially full of spyware and other nasties. (ok, it's probably not...)
Either way, I would agree with amigo that your best bet would be to place any needed libraries that do not already exist on the distro, and any that are incompatible with the existing libraries, in a separate directory, and use a wrapper script to set LD_LIBRARY_PATH before launching the program.
Excellent summary. Thanks. On that note I shall mark the topic solved, although I am still very keen to hear of other methods of "sandboxing" apps to avoid conflicts.
sunburnt wrote: And that`s exactly what RoxApp or AppDir or AppPkg does. They setup a run environment for the app. and then run it. This is $PATH, $LD_LIBRARY_PATH, and links and dirs. needed. No union[/n] is needed for these apps., they work with or without one.
Thanks. I will look into this a bit more. Howdo these apps compare with CDE for usability or practicality? Totally different?

npierce
Posts: 858
Joined: Tue 29 Dec 2009, 01:40

#20 Post by npierce »

You're welcome. I'm glad to hear it was helpful. Good luck with your project.

Post Reply