lib loader lib search issue

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
User avatar
Q5sys
Posts: 1105
Joined: Thu 11 Dec 2008, 19:49
Contact:

lib loader lib search issue

#1 Post by Q5sys »

First off before anyone wants to say this is a Slacko bash ITS NOT! I'm just using slacko as my example for this since I happen to have it running in a VM and it was easy to do this on. Also because that system is a default image that I haven't tampered with.

I'm asking this question because I not only because Im looking for the answer, but because dont undertstand the reason for this, as well as what possible results this may have.

When trying to debug things in the past I've routinely used strace. But one thing has always bothered me, and I guess its about time I just come out and ask it.

Below are the strace output segments from taking a fresh install of slacko-5.5 and loading rox filer. Entire file can be found here -> http://paste2.org/p/3452409

In the 1st example, the system has to search for the libncurses.so.5 lib 16 times before it finds it and can load it.
In the 2nd example, the system has to search for the libgtk-x11-2.0.so.0 lib 24 times before it finds it and can load it.

My first question is why is this happening? My second question is how much lag is this creating in our systems? If the system has to search for every lib a dozen times... that's alot of time that our system is searching and in my mind, wasting time and resources. Maybe it's not a big deal... but if this is happening at every lib load... it's got to add up. Since we focus on older hardware this slight resource expense makes a larger difference than on new fast hardware.
My third question is what can be done to resolve this? I'm sure there's a very complicated and complex solution, but is there a simple solution which we could implement?

Code: Select all

execve("/usr/local/bin/rox", ["rox"], [/* 55 vars */]) = 0
brk(0)                                  = 0x810e000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77a7000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/i686/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/i686/sse2", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/i686/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/i686", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/sse2", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/i686/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/i686/sse2", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/i686/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/i686", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/sse2", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0", {st_mode=S_IFDIR|0755, st_size=905, ...}) = 0
open("/lib/tls/i686/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xbfc065c0) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", 0xbfc065c0)     = -1 ENOENT (No such file or directory)
open("/lib/tls/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xbfc065c0)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0xbfc065c0)          = -1 ENOENT (No such file or directory)
open("/lib/i686/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xbfc065c0)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", 0xbfc065c0)         = -1 ENOENT (No such file or directory)
open("/lib/sse2/libncurses.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xbfc065c0)         = -1 ENOENT (No such file or directory)
open("/lib/libncurses.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\243\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=268236, ...}) = 0
and

Code: Select all

open("/usr/lib/firefox-19.0/tls/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/i686/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/i686", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/tls/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/tls", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/i686/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/i686", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/firefox-19.0/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/firefox-19.0", {st_mode=S_IFDIR|0755, st_size=905, ...}) = 0
open("/lib/tls/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", 0xbfd0e7a0)     = -1 ENOENT (No such file or directory)
open("/lib/tls/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xbfd0e7a0)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0xbfd0e7a0)          = -1 ENOENT (No such file or directory)
open("/lib/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xbfd0e7a0)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", 0xbfd0e7a0)         = -1 ENOENT (No such file or directory)
open("/lib/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xbfd0e7a0)         = -1 ENOENT (No such file or directory)
open("/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib", {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0
open("/usr/lib/tls/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0xbfd0e7a0)      = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686/sse2", 0xbfd0e7a0) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686", 0xbfd0e7a0)     = -1 ENOENT (No such file or directory)
open("/usr/lib/sse2/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/sse2", 0xbfd0e7a0)     = -1 ENOENT (No such file or directory)
open("/usr/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20C\5\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=4586808, ...}) = 0

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#2 Post by amigo »

We discussed just this awhile back in another thread(pupngo). It boils down to this: the search path is determined by the options used to configure glibc (and hence the loader) when compiling it. So, if you want to exclude those 'weird' search paths you need a different glibc -which means you need another distro or to build your won toolchain and everything else which follows. Since Puppy is built on other distros toolchain and binaries you don't have this choice.

Still, it's not as bad as it looks -you'd be hard-put to tell the difference in load-time from a seat-of-the-pants perspective. Once the libraries have been loaded, the long path search doesn't happen again.

User avatar
Q5sys
Posts: 1105
Joined: Thu 11 Dec 2008, 19:49
Contact:

#3 Post by Q5sys »

amigo wrote:We discussed just this awhile back in another thread(pupngo). It boils down to this: the search path is determined by the options used to configure glibc (and hence the loader) when compiling it. So, if you want to exclude those 'weird' search paths you need a different glibc -which means you need another distro or to build your won toolchain and everything else which follows. Since Puppy is built on other distros toolchain and binaries you don't have this choice.

Still, it's not as bad as it looks -you'd be hard-put to tell the difference in load-time from a seat-of-the-pants perspective. Once the libraries have been loaded, the long path search doesn't happen again.
Thanks for the info... I'll search for that pupngo post and read up on it. However I do have a follow up question if you wouldn't mind another (perhaps) dumb question.

I get what you're saying about it being determined by the options when glibc is configured... but if thats the case, why on earth is ld.so searching in the firefox directories first for the ncurses lib? Isnt ncurses a core included library in slackware? If memory serves the last time I installed it... the installer is a ncurses based installer. why would the glibc be configured to check in the firefox path? Firefox is an optional program, and might not be installed at all... like when someone is using slackware for a server.
It just seems idiotic (to me at least), that glibc would be configured to check a path like firefox before it'd check the common /lib/ path.
Or is there something else aside from the glibc configuration that comes into play?

Or is there something else I'm missing?

Also you mention that it's not really noticeable at all. The reason I ask is because I'm planning on installing puppy (havent decided which) on my libretto 50ct with a 75mhz pentium processor. So I thought maybe on something that slow it might be more of an impact.

Also a side question thats related, since we use a layered file system... does ld.so see the file system as layers or as a single functioning layer? (did I word that so you could understand me?)

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#4 Post by 01micko »

Code: Select all

# echo $LD_LIBRARY_PATH
/usr/lib/firefox-19.0.2:/lib:/usr/lib:/usr/X11R7/lib:/root/my-applications/lib:/usr/local/lib:/opt/lib:/usr/lib/seamonkey
Therein lies your clue. :wink:
Puppy Linux Blog - contact me for access

User avatar
Q5sys
Posts: 1105
Joined: Thu 11 Dec 2008, 19:49
Contact:

#5 Post by Q5sys »

01micko wrote:

Code: Select all

# echo $LD_LIBRARY_PATH
/usr/lib/firefox-19.0.2:/lib:/usr/lib:/usr/X11R7/lib:/root/my-applications/lib:/usr/local/lib:/opt/lib:/usr/lib/seamonkey
Therein lies your clue. :wink:
I know that $LD_LIBRARY_PATH can be altered by editing /etc/ld.so.conf. And I'm aware that if one program has something peculiar that you need, you can write a wrapper for a single program to change it for that binary only...

But Amigo is saying it's dependent on configuration of the glibc.

So is it both? Neither? One or the other? Half one half the other?

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#6 Post by amigo »

The search path is comprised of:
1. LD_LIBRARY_PATH
2. 'standard' locations (/lib and /usr/lib)
3. extra locations found in the file /etc/ld.so.conf
4. extra possibe locations determined by config options at glibc compile time

The weird ones you are seeing with 'tls' 'sse' 'i686', etc are all from number 4 above.

And, there may actually *be* a reason to bash slack -I think those particular lines with firefox-19 in them come from a 'sloppy' toolchain compile in slackware. Building gcc and/or glibc with extra stuff installed means these 'false' locations get included in the toolchain's
internals. OTOH, other distros glibc will turn up even more possibilities.

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#7 Post by 01micko »

amigo wrote: And, there may actually *be* a reason to bash slack -I think those particular lines with firefox-19 in them come from a 'sloppy' toolchain compile in slackware. Building gcc and/or glibc with extra stuff installed means these 'false' locations get included in the toolchain's
internals. OTOH, other distros glibc will turn up even more possibilities.
NO! That's puppy stuff I added to /etc/profile, so I'll take the heat for that. Barry has /usr/lib/seamonkey in there anyway and I add whatever firefox on the condition it is installed so flashplayer is happy and other stuff (so they can find nss, nspr etc). If this is a serious performance hit I can change it.
Puppy Linux Blog - contact me for access

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#8 Post by Ibidem »

The directories with names like sse, sse2, i686, tls, etc. are added at runtime based on the cpu capabilities ("flags" in /proc/cpuinfo)
touch /etc/ld.so.nohwcap to disable this, which should cut the attempts to a sixth of what they are.

LD_LIBRARY_PATH is the environment-based approach, ld.so.conf is the other approach; you shouldn't have a directory in both. Ideally, frequently used libs will be towards the start of the search path (if you start firefox rarely and don't care how long it takes to start, put it at the end of ld.so.conf so more frequently used libs get found sooner)

But this is why full-static binaries can start significantly faster...and part of why musl starts faster (ld-musl-*.so.1 is also libc.so, so there isn't the "search path for libc" spew, and there's no looking at cpu capabilities to determine lib search paths). For the average user, it isn't noticeable when starting one program; it is if you start a whole bunch, though (./configure runs MUCH faster).

A layered ("union") file system is not readily distinguishable from standard filesystems from userspace; the FS driver is all that sees the layers. Of course, it has to search each layer itself, starting at the top.
So for each of those calls that ld.so makes, the aufs module goes searching from the top layer down. The more SFS modules you have mounted as layers, the slower it will be...by a microscopic amount.

If you want an idea of how much effect it has, try (assuming /bin/true is a dynamically linked binary with this source:

Code: Select all

int main(){return 0;}
)
time /bin/true
touch /etc/ld.so.nohwcap
LD_LIBRARY_PATH=/lib time /bin/true

Post Reply