Need examples of static compiling of various apps.

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

Need examples of static compiling of various apps.

#1 Post by sunburnt »

I`ve only had marginal luck with compiling apps.
I need to create self contained ( relatively ) squash packages.
Criteria for static compiling the dependencies is common usage and size.

Some deps. like the gstreamer libs. are used by a number of media apps.
It probably is better to use the gstreamer libs. as shared.
But if dependencies are small, who cares if they`re widely used?

amigo`s src2pkg looks very good, a better idea of it`s usage may be the fix.
technosaurus also has a gui that could be modified to do static compiles.

goingnuts
Posts: 932
Joined: Sun 07 Dec 2008, 13:33
Contact:

#2 Post by goingnuts »

Not quite sure what your target apps are but attached a build script example that will

* create a build directory structure
* download the source
* compile it (using gcc/diet gcc/ulibc-gcc)/(static or dynamic)
* create package with the binaries

Also a root directory are populated with the content of the packages. This can be used to create sfs-packs with all the packages content.

Current script can make hotplug2stdout, cddetect, puppyinputdetect,
guess_fstype, disktype, setserial and elspci.

Some figures on the size of the different builds:
#gcc dynamic linked total size 489K
#gcc statatic linked total size 3002K
#uclibc static linked total size 242K
#diet libc static linked total size 173K

Update: the build script for puppyinputdetect is buggy - uses gcc all the time. Below should fix it - replace in script:

Code: Select all

puppyinputdetect () {
PUPPYINPUTDETECT="puppyinputdetect-1.1i"	
cd $SRC_DIR
rm -rf $PUPPYINPUTDETECT
#download	
if [ ! -f $PUPPYINPUTDETECT.tar.gz ]; then
  $FTP http://www.goingnuts.dk/userfiles/$PUPPYINPUTDETECT.tar.gz
fi
#build
$TARX $GTAR -f $PUPPYINPUTDETECT.tar.gz
cd $PUPPYINPUTDETECT
make clean
CONF='-DMOUSE_DETECT_SERIAL'
if [ "${CC}" = "gcc" ]; then
	make puppyinputdetectFULL CC="${CC}" CFLAGS="${CFLAGS} ${CONF}" LDFLAGS="${LDFLAGS}"
fi
if [ "${CC}" = "/usr/i386-linux-uclibc/bin/i386-uclibc-gcc" ]; then
	${CC} ${CFLAGS} ${CONF} ${LDFLAGS} -o puppyinputdetectFULL  main.c serial.c proc.c
fi
if [ "${CC}" = "diet gcc" ]; then
	${CC} ${CFLAGS} ${CONF} ${LDFLAGS} -o puppyinputdetectFULL  main.c serial.c proc.c -lcompat
fi
# prepare package layout
mkdir -p $TMP_DIR/sbin
cp -af puppyinputdetectFULL $TMP_DIR/sbin/puppyinputdetect
# create package
cprm $PUPPYINPUTDETECT
}

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#3 Post by sunburnt »

Thanks goingnuts; I appreciate the help.

The apps. will be common ones: browsers, media, office, games, utility, etc.

Pkgs. need to be configured with: --prefix /tmp/SqApp/mnt/( App.Pkg.Name )

I need a script to do most of the work, and getting the deps. is the hard part.
Is there a good place to get most app. pkgs.? And how to tell pkg. deps.?
Debian I noticed has an odd way of setting up pkgs. so they`re hard to use.
Different build scripts for each app., and the apps. are constantly changing.

### How about this for "Naaa, it just makes too much sense..."
A web site with a page of links listing app. source pkgs. and their sizes.
Each app. pkg. is in one dir with all it`s deps. No more looking for deps.
Each app`s. page lists the dir. contense and sizes, and a "Download All" link.

goingnuts
Posts: 932
Joined: Sun 07 Dec 2008, 13:33
Contact:

#4 Post by goingnuts »

If you compile the apps statically you have no dependency on additional libraries...So it would make more sense to have a list of applications and a link to a package containing all needed for the application to run. Some applications, although compiled statically, depends on other files to run (configuration files, other binaries ect.). Also the menu-entry, the manual pages and icons could/should be included in the package...

A lot of command line only applications will compile static by just using diet libc or uclibc. GUI application are more complicated if you want statically compiled binaries and you will have to build additional libraries and toolkits as well. This might include some modification of the sources to make things happen as most code seems to be written for dynamic linking.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#5 Post by sunburnt »

Thanks again goingnuts; This is what amigo was saying a while back.
Static compiling is a twisted route to follow, but the pkgs. run on more O.S.s.

Exactly my experience, CLI apps. were no problem, GUI apps. a bit tricky.
But if records are kept then future builds are much easier to accomplish.

I had good luck with the Dead Meat music player and xMajhongg.
I need to spend lots of time trying to build other apps. and track the steps.

The SqApp pkgs. will be like Tiny Core Linux`s SCM files, but widely usable.
The methods of use and locations are different between the two pkg. types.

SqApp pkgs. have exec. script and icon in root and are copied with configs.
Links for the apps. config. files and a /config dir. with the real config files.

Diet libc and uclibc... How different from gcc? Will one app. run on another?
I think libs. can be added as files or included in the apps. code, right?

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#6 Post by Ibidem »

sunburnt wrote:Diet libc and uclibc... How different from gcc? Will one app. run on another?
I think libs. can be added as files or included in the apps. code, right?
gcc is a compiler, which all linux libc versions are compatible with--gcc compatibility is pretty much mandatory.
glibc is the typical libc.
dietlibc, uclibc, and musl are not binary compatible with glibc; they have varying degrees of source compatibility. This means that you cannot mix/match libraries built against musl, uclibc, and glibc.
In order to cross-compile (use a compiler built with one libc to target another), you must specify several options to keep this from accidentally happening.
You can run apps built against any libc on the same system, without problems (provided that the libc isn't broken).

It's *possible* to #include "something.c" -but this makes most programmers cringe. You can also ship libraries with the source.
Usually, you just link libraries in at build time.
If you're linking statically, the "library" is just an archive of object files (*.o), plus an index; at link time, the linker will look through the index and use the object files with relevant symbols. Of course, a well designed modular library won't pull in extra functions by accident (this is one of the reasons for using musl).
So if you see

Code: Select all

gcc -o something -static -lc file1.o file2.o
, gcc is using crt1.o, crti.o, file1.o, file2.o, time.o, write.o, & crtn.o to construct an executable. The crt*.o files are automatically added startup/finish code, and write.o/time.o come from libc.a.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#7 Post by Karl Godt »

You might have a look for the hdparm source at sorceforge.net for a simple example .

The source has no ./configure script but a simple Makefile
which reads

LDFLAGS = -s
#LDFLAGS = -s -static

and if you shift the hash sign, it builds statically.


http://gcc.gnu.org/onlinedocs/gcc-4.4.7/gcc/Link-Options.html#Link-Options
-s
Remove all symbol table and relocation information from the executable.
-static
On systems that support dynamic linking, this prevents linking with the shared libraries. On other systems, this option has no effect.
-shared
Produce a shared object which can then be linked with other objects to form an executable. Not all systems support this option. For predictable results, you must also specify the same set of options that were used to generate code (-fpic, -fPIC, or model suboptions) when you specify this option.1

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#8 Post by sunburnt »

Thanks Ibidem; Each bit of info clears up the picture. I know gcc is a compiler.
I didn`t know how the C libs. related to it and each other ( they don`t play nice ).

Thinking about it, merging code seems better than files looking for other files.
Solves not being able to find dependencies for all the many reasons.
But if apps are written and setup to work a certain way, then it`s best.

Someone brought up an obvious point I`d thought of, how much to include?
This is where a statistical analysis of lib. usage would be so very much help.
My simple analysis showed only a small portion of libs. were commonly used.
So have the base distro contain the really common and bigger libs. as shared.
All other libs. would be compiled statically with the apps., so no lib. conflicts.

How to list an apps. dependencies before it`s compiled? Tree files are nice.
Debian lists the first level of deps., but it becomes a recursive nightmare.
And where to find and how to get the correct libs.? Automation?
This I`ve found to be the biggest effort and headache of building packages.


Thanks Karl; I`ll look at your link. Good... More info to assimilate.
I`m not sure about the -s description and what it means to my plans.
The -static I do understand, but I don`t see how to control what`s included.
Unless libs. in the build dir. are used and shared duplicates are ignored.
But perhaps your link will provide some of these answers.

Thanks to all for the help. Terry B.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#9 Post by jamesbond »

sunburnt wrote:How to list an apps. dependencies before it`s compiled? Tree files are nice.
Look inside the Makefile (warning: not easy read), look for anything that says *_LIBS= or LDFLAGS_*=. Anything that starts with -l (small L), e.g. -lz -lssl -lcurl -ldvdplay --- those are the shared library dependency (for this example, it means they depend on libz.so, libssl.so, libcurl.so, libdvdplay.so). If the app uses some kind of build automation tool (autoconf, cmake, autosetup etc) run these tool (configure, cmake, autosetup etc) first to generate or populate the Makefile.
And where to find and how to get the correct libs.? Automation?
Black magic :D Seriously, you need to hunt for them. And ensure you have the correct version. Fortunately though, Linux libraries are versioned so different versions of the library can work together.
This I`ve found to be the biggest effort and headache of building packages.
That makes two of us :)
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#10 Post by Ibidem »

jamesbond wrote:
sunburnt wrote:How to list an apps. dependencies before it`s compiled? Tree files are nice.
Look inside the Makefile (warning: not easy read), look for anything that says *_LIBS= or LDFLAGS_*=. Anything that starts with -l (small L), e.g. -lz -lssl -lcurl -ldvdplay --- those are the shared library dependency (for this example, it means they depend on libz.so, libssl.so, libcurl.so, libdvdplay.so). If the app uses some kind of build automation tool (autoconf, cmake, autosetup etc) run these tool (configure, cmake, autosetup etc) first to generate or populate the Makefile.
Also read the README/INSTALL/... or look up the program on packages.debian.org, then follow the link to "source".
Note, though, that (for example) -lssl does NOT mean you need libssl.so!
For static compiling, you instead need libssl.a (the static library)
If you have libssl.so only and build with -static, it will probably end up missing. If you link dynamically but only have libssl.a, gcc sees that libssl.so is missing, and uses libssl.a instead. If you have an uncommon library, moving the lib*.so out of the way should let you compile with that library as static.
Another trick (last resort!) is

Code: Select all

grep -hr 'include.*<' ./|sed -ne 's/^.*<\(.*\.h\)>.*/\1/g'|sort -u 
This will list all headers directly included by the source, and you can look those up...
jamesbond wrote:
And where to find and how to get the correct libs.? Automation?
Black magic :D Seriously, you need to hunt for them.
I'd try packages.debian.org/file:libabc.so

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#11 Post by sunburnt »

Hi jamesbond; Thanks for the info, I`m staring to get a little bit of a picture.

For the deps. I`m beginning to think that the best way is to use Debian or Ubuntu
as they have one of the largest repositories, and they auto. fetch and place deps.
I don`t want to run Debian, but it`s probably the most painless way to build pkgs.
Pussy is Debian compatible, but I couldn`t get it to run on my old PC.
I got Debian live, but it`s a loose file system, not like Puppy`s 3 files setup.
Looked at SlackWare but didn`t see that it had a lot of app. packages.

A small distro., preferably squash file based, lots of apps., that handles building pkgs.

Thanks again Ibidem; I know about the README files.

So if I understand correctly... "*.a" is a libs. static form and it`s code is included?.
Or is the lib. compiled and added as a separate "*.so" file anyway?

"*.so is the dynamic lib., and if it`s not in the build dir. but is pointed to or installed,
then it will be included in the pkg. as the "lib.so" file? ( seems so...)

Using Puppy`s installed libs. may not be a good idea. ( I don`t know...)
It seems best to use all downloaded libs. so the compile is completely new.
This way Puppy is just the O.S. running gcc but it`s not adding anything to the pkg.

Does this seem to make good sense? Any thoughts on distros. to use?

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#12 Post by technosaurus »

It helps to know how your linker works... by default, when it is passed -lmylib it will look in LD_LIBRARY_PATH for libmylib.so and if it isn't found, it will look for libmylib.a. Most of the time all you need to do is rm the .so symlinks for the libs you want built in statically and the linker will do the rest.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#13 Post by sunburnt »

Hi technosaurus; Did you ever rework your gui compiler app.?

So need a script to read a list of added libs. and rename any existing system links.
Run it again and it renames the links back to their original state.

This providing you have the correct libs. "*.a" in the build dir.

Q; Ibidem said:
If you have an uncommon library, moving the lib*.so out of the way should let you compile with that library as static.
So where do I put the "*.so" libs. to be added to the pkg.?
Or is Ibidem saying the same thing as technosaurus?
Make the lib. "*.so" inaccessable while having the "*.a" lib. available?

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#14 Post by technosaurus »

you can just do all files in LD_LIBRARY_PATH that match *.so (not *.so*) (use a white/black list if necessary) and you can use readlink to get store the file it links to before removing the link (its usually something like *.so.[0-9].[0-9])

some pseudo-ish code...

Code: Select all

cd to each directory in $LD_LIBRARY_PATH
for x in *.so; do
[ -f "${x%.so}.a" ] && readlink $x >>somefile && rm -f $x
done
btw, my projects that are over a year old, went away with my ntfs partition when it got fried
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#15 Post by Ibidem »

Yes, I'm saying the same thing as technosaurus. If you make the *.so files inaccessible but leave the *.a files, it will link statically.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#16 Post by sunburnt »

This is starting to make some good sense now...

Thanks for the code technosaurus; I think I`ll rename the links for restoring.
Linus Torvalds: Show me the code!
NTFS seems to have the M$ mark of death by design. Ext3 & 4 drivers for Win.

Thanks Ibidem; Clarification is everything to understanding.

# So a complete clarification as to lib. utilization. 5 Qs...

1: Both the *.a and *.so files can be used in the build dir. in their proper paths?
2: *.a files are built with everything else, *.so files are just added to the package?

3: *.a files must have any matching *.so system files hidden for them to be used.
4: But if a *.so file file is in the build dir., also hide matching *.so system files?

5: So if a system *.so file is found, it`s always used before anything in the build?

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#17 Post by Ibidem »

sunburnt wrote:Thanks Ibidem; Clarification is everything to understanding.
As my grandfather used to say, whelks!
# So a complete clarification as to lib. utilization. 5 Qs...
1: Both the *.a and *.so files can be used in the build dir. in their proper paths?
Don't understand this question.
2: *.a files are built with everything else, *.so files are just added to the package?
*.a files are compiled into the binary, *.so becomes a dependency (must be added to package unless you expect them to hunt em down)

(I'm not using your terminology because it means something different from what you think: literally, it would mean that the *.a libraries are produced by the compilation process along with the binaries)
3: *.a files must have any matching *.so system files hidden for them to be used.
Or you can use -static for a fully static binary, and there might be another option I don't know about, but usually, yes.
4: But if a *.so file file is in the build dir., also hide matching *.so system files?
Only if -L. is in the compiler options
5: But if system *.so file is found, it`s always used before anything in the build?
AFAIK, first paths specified by -L are searched, then system paths.
Current directory is not searched for libraries, only object files (*.o)

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#18 Post by jpeps »


User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#19 Post by sunburnt »

Thanks jpeps; From your link I got this static lib. compile:

Code: Select all

Compile: cc -Wall -c ctest1.c ctest2.c
Compiler options:

    -Wall: include warnings. See man page for warnings specified.

Create library "libctest.a": ar -cvq libctest.a ctest1.o ctest2.o
List files in library: ar -t libctest.a
Linking with the library:

    cc -o executable-name prog.c libctest.a
    cc -o executable-name prog.c -L/path/to/library-directory -lctest
And this dynamic lib. compile:

Code: Select all

 gcc -Wall -fPIC -c *.c
    gcc -shared -Wl,-soname,libctest.so.1 -o libctest.so.1.0   *.o
    mv libctest.so.1.0 /opt/lib
    ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so
    ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so.1
### Ibidem;
Both the *.a and *.so files can be used in the build dir. in their proper paths?
Meaning both the *.a and *.so files can be compiled / added into the pkg.
*.a files are not shared, *.so files could be shared but not likely if in the pkg.

# Again... These Squash files will mount but not be unioned ( not sfs files ).

I made a xMahjongg pkg. configured with --prefix (Mount Pt.), and it works
without adding the lib. paths to /etc/ld.so.conf or $LD_LIBRARY_PATH .
All I`ve read says not to use $LD_LIBRARY_PATH , but Puppy does.

rpath is the apps. internal "run path", two ways to set rpath:
> Set the environment variable $LD_RUN_PATH ( Puppy doesn`t have it...)
> Add -R /usr/local/lib (or whatever path) to the link command line.
Web page about paths and rpath: http://www.eyrie.org/~eagle/notes/rpath.html
# Best is: $LD_RUN_PATH ( set in Bash session before start of compile? ).
But if -R is used in the link command line then $LD_RUN_PATH is ignored.
# If Autoconf is used, the easiest way is to set LDFLAGS to "-R /usr/local/lib"
before running configure, which will then stick it into the Makefile.
Failing that, do something like: make CC="gcc -R usr/local/lib ( or path...) ".

# Now I`m not sure what --prefix is doing really, and what -static does also.
# Shouldn`t -static make both the *.a and *.so files "findable" in the pkg.?

Thanks again for all the help guys. Terry

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#20 Post by jpeps »

sunburnt wrote: # Shouldn`t -static make both the *.a and *.so files "findable" in the pkg.?

Thanks again for all the help guys. Terry
I'll take a novice guess, and say that -static builds the *.a libs into the binary, and that you need to indicate the path to the *.so files.

Post Reply