Compiling with musl (an alternate libc)

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Message
Author
Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

Compiling with musl (an alternate libc)

#1 Post by Ibidem »

Apologies if this is too on-topic...

What is it, and why would I be interested?
musl is an alternate libc (standard C library), with:
-a stable ABI
-strict full X/Open 2008 (SUS4), POSIX (latest version), and ISO C99 conformance.
-substantial _GNU_SOURCE compatability
-C++ compatability
-for a week or two, -fstack-protector compatability (I haven't tested)
-an MIT license (changed yesterday or so from LGPL 2.1)
-size competetive with uclibc (slightly larger than dietlibc)
-modular design that makes for small static binaries
-TCB shadow passwords
-supports i386, x86_64, and arm (mips is planned)
-fairly easy to port
-responsive developer

You might be interested if you...:
-Want to build static binaries for linux on supported architectures
-want a small libc that doesn't break the abi regularly (vs uclibc)
-want full standard conformance, rather than "we pick how much of the standard we follow"

Building it:
There are two options: as a cross-compiler, or migrating the system.
If you migrate, you'll want to read the documentation included. I can supply some details, but that's for a later post.

1. Compile and install musl, preferably from git head (a release is a tag, not a branch, so all fixes are in git).

Code: Select all

 git clone git://git.etalabs.net/musl
-Copy config.mak from dist/ and adjust as you want.
You may want to disable the shared library.
(PREFIX defaults to /usr/local/musl, and should not be set to /, /usr, or /usr/local without following the documentation on migrating)

2. Get the linux kernel headers (linux-libc-dev, not the ones for building modules) and put them somewhere (I use /opt/musl/lin)
There should be a linux/ subdirectory in the directory you install the headers in.

3. export CC=musl-gcc; export CFLAGS="-Os -static -fno-stack-protector -I/opt/musl/lin -D_GNU_SOURCE"
Don't forget the -D_GNU_SOURCE - most stuff won't build without that, though some will. -I/... should point to the linux headers.
(Note-the linux headers are needed for OSS and a lot besides)

4. Now run configure and make.


If you want to discuss anything related to musl, feel free to use this thread.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#2 Post by jamesbond »

Bookmarked for further reference.
This should be under "Cutting Edge" - interesting stuff here :D
I haven't tried musl, but I suppose it will compile busybox just fine?
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#3 Post by Ibidem »

jamesbond wrote:Bookmarked for further reference.
This should be under "Cutting Edge" - interesting stuff here :D
I haven't tried musl, but I suppose it will compile busybox just fine?
Much of Busybox.
You must install the kernel headers and point it at them (-I/opt/musl/lin)
Also, you need to disable NFS, unless you've built tirpc (RPC support is mandatory for NFS). I remember some other stuff I turned off, too...

Busybox won't build without -D_GNU_SOURCE, IIRC. You might also need -Wl,-z,muldefs -- but probably won't with any recent musl release.

Long story short: I got a static busybox binary that does anything most people need, in ~700 kilobytes. That includes vi, ash, init, and mdev, so you can boot to a shell with no other binaries.

I've also built gcc 3.4 (there were some breakages in gcc 4.x), make 3.81, ncurses 5.9, vim 7.3, m4 from netbsd 4.0, zlib, wireless-tools 30pre9 (which is harder to do...), and a few other packages.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#4 Post by jamesbond »

Impressive :D

This is my attempt at building dnsmasq as 32-bit executable under 64-bit environment.

1. I installed musl (0.9.0) - I added "-m32" to config.mak as directed. Everything went well. The build is *much much* faster than building uclibc using buildroot.
2. Unpacked dnsmasq, and did the export as indicated in the first post - fail. Looking at the output, I see it called gcc instead musl-gcc - so I passed CC and CFLAGS through make instead:

Code: Select all

make CC=musl-gcc  CFLAGS"=-Os -static -fno-stack-protector -D_GNU_SOURCE -m32 -I /mnt/sda2/tmp/linux-3.2.2/usr/include/"
3. Failed - missing netinet/in_systm.h. I checked, /opt/musl/include/netinet doesn't have in_systm.h. So I concocted my own from glibc, suitably modified:

Code: Select all

#ifndef _NETINET_IN_SYSTM_H
#define _NETINET_IN_SYSTM_H 1
typedef uint16_t n_short;      /* short as received from the net */
typedef uint32_t n_long;       /* long as received from the net  */
typedef uint32_t n_time;       /* ms since 00:00 GMT, byte rev   */
#endif /* netinet/in_systm.h */
4. Went smoothly until it tried to compile log.c, then failed - cannot find _PATH_LOG. This is defined in bits/syslog-path.h, but /opt/musl/include/bits doesn't have this. I just add _PATH_LOG to dnsmasq's config.h and compilation continued and finished.
5. But it failed at link step - saying that it skipped libc.a when searching -lc. This would not happened if I compiled for 64-bits, so I added LDFLAGS=-m32 to the make file, which now looks like this:

Code: Select all

make CC=musl-gcc  CFLAGS"=-Os -static -fno-stack-protector -D_GNU_SOURCE -m32 -I /mnt/sda2/tmp/linux-3.2.2/usr/include/" LDFLAGS=-m32
6. All went well - except that ld cannot find capset/capget.

Code: Select all

dnsmasq.c:(.text+0x1172): undefined reference to `capget'
dnsmasq.c:(.text+0x14d4): undefined reference to `capset'
dnsmasq.c:(.text+0x156a): undefined reference to `capset'
This is where I stopped. grep -Ril capset /opt/musl/* told me there is a definition of capset in bits/syscall.h, but there is none in the libs.

EDIT:
7. I edited dnsmasq.c to remove all references to capset and capget, and the compile ended successfully. I ended up with 172k stripped dnsmasq static 32-bit binary that runs successfully within my 64-bit system - but now I need to test it that my removal of capset/capget doesn't upset it too much :). The size compares favorably, my stripped *dynamic* 64-bit dnsmasq is 176k :shock:

EDIT: It works.

This capset/capget thing seems to be linux specific, and the POSIX version of the same stuff is implemented in libcap, however it can also be found in glibc (at least it seems to, I may be wrong).

Questions:
1. I'm sure the above is not the only one, so the general question is: as I understand it musl strives to be POSIX-compliant (not Linux compliant), however it also strives to be a glibc replacement. musl already implements _GNU_SOURCE, what about compatibilities at this level? Unfortunately, compatibility support == bloat :?

2. Assuming:
- not trying to cross-compile (not as above)
- not trying to migrate the whole system
- building a static binary
How does libs compiled with musl co-operate with libs compiled with glibc? Let's say I already have a libxxxx compiled with glibc (let say openssl's libssl). Will this work with musl - i.e. if I use musl-gcc and include does -lssl (pointing to the appropriate -L) - will this work? Probably not, I suppose.
If not, then I need to compile openssl again and install it on, say, /opt/musl/openssl, and make sure that all my compiles use that version of libgtk+, no?
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#5 Post by Ibidem »

jamesbond wrote:...
2. Unpacked dnsmasq, and did the export as indicated in the first post - fail. Looking at the output, I see it called gcc instead musl-gcc - so I passed CC and CFLAGS through make instead:

Code: Select all

make CC=musl-gcc  CFLAGS"=-Os -static -fno-stack-protector -D_GNU_SOURCE -m32 -I /mnt/sda2/tmp/linux-3.2.2/usr/include/"
Are those the cleaned headers corresponding to linux-libc-dev, or the linux kernel headers?
3. Failed - missing netinet/in_systm.h. I checked, /opt/musl/include/netinet doesn't have in_systm.h. So I concocted my own from glibc, suitably modified:

Code: Select all

#ifndef _NETINET_IN_SYSTM_H
#define _NETINET_IN_SYSTM_H 1
typedef uint16_t n_short;      /* short as received from the net */
typedef uint32_t n_long;       /* long as received from the net  */
typedef uint32_t n_time;       /* ms since 00:00 GMT, byte rev   */
#endif /* netinet/in_systm.h */
4. Went smoothly until it tried to compile log.c, then failed - cannot find _PATH_LOG. This is defined in bits/syslog-path.h, but /opt/musl/include/bits doesn't have this. I just add _PATH_LOG to dnsmasq's config.h and compilation continued and finished.
...
6. All went well - except that ld cannot find capset/capget.

Code: Select all

dnsmasq.c:(.text+0x1172): undefined reference to `capget'
dnsmasq.c:(.text+0x14d4): undefined reference to `capset'
dnsmasq.c:(.text+0x156a): undefined reference to `capset'
This is where I stopped. grep -Ril capset /opt/musl/* told me there is a definition of capset in bits/syscall.h, but there is none in the libs.

EDIT:
7. I edited dnsmasq.c to remove all references to capset and capget, and the compile ended successfully. I ended up with 172k stripped dnsmasq static 32-bit binary that runs successfully within my 64-bit system - but now I need to test it that my removal of capset/capget doesn't upset it too much :).
Questions:
1. musl already implements _GNU_SOURCE, what about compatibilities at this level? Unfortunately, compatibility support == bloat :?

2. Assuming:
- not trying to cross-compile (not as above)
- not trying to migrate the whole system
- building a static binary
How does libs compiled with musl co-operate with libs compiled with glibc? Let's say I already have a libxxxx compiled with glibc (let say openssl's libssl). Will this work with musl - i.e. if I use musl-gcc and include does -lssl (pointing to the appropriate -L) - will this work? Probably not, I suppose.
If not, then I need to compile openssl again and install it on, say, /opt/musl/openssl, and make sure that all my compiles use that version of libgtk+, no?
Removing capset/capget is most likely to be a security issue--they allow checking and dropping permissions in a more grnular way than setuid/seteuid.
Don't use grep -Ril --you will be very thoroughly fooled, until you actually look at the file. Here it's only SYS_capset and _NR_capset that are defined.
Currently musl is NOT ABI compatible with glibc; you will have to recompile everything. Substantial chunks of glibc headers are nonstandard, and aren't defined in musl yet.

If you report a missing function that breaks some program to the musl mailing list (refer to http://www.etalabs.net/musl/community.html), it may well be added, occasionally even the same day. This is what I mean by "responsive". I'd use a subject like
"Compatability: cap{set,get} needed for dnsmasq"
(identify that the issue is compatability, name functions, and name the program you're trying to build.)
The goal seems to be "enough glibc compatability for most software to work".

The "appropriate" -L is likely to be -L/usr/include, which will massively break everything.
I presume the "libgtk+" was a typo?
You can install in any prefix where there aren't glibc-based libraries.
In other words, if the musl prefix is /opt/musl, you can use that as the prefix for all the libraries, and musl-gcc will find them all.

HTH

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#6 Post by Ibidem »

I've confirmed this, sent an email to the musl mail-list with patches for the first two issues (missing header, add _PATH_LOG to syslog.h when _(GNU|BSD)_SOURCE is defined)
And the patches are committed in the bsd branch of my repo (github.com/idunham/musl).

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#7 Post by jamesbond »

Ibidem wrote:Are those the cleaned headers corresponding to linux-libc-dev, or the linux kernel headers?
cleaned headers corresponding to linux-libc-dev. I did try with raw kernel headers - didn't work, of course :oops:

Removing capset/capget is most likely to be a security issue--they allow checking and dropping permissions in a more grnular way than setuid/seteuid.
Yes.
Don't use grep -Ril --you will be very thoroughly fooled, until you actually look at the file. Here it's only SYS_capset and _NR_capset that are defined.
That's just a quick way to check stuff; if grep -Ril returns negative the the symbol is definitely not there. But you are right; I did check the file afterward.
Currently musl is NOT ABI compatible with glibc; you will have to recompile everything. Substantial chunks of glibc headers are nonstandard, and aren't defined in musl yet.
Yes, I must not be thinking when I asked this question. When libs are compiled they include constants and symbols and pointer offsets etc that come from the libc; and these will be different between glibc and musl. So everything must be re-compiled (and stored elsewhere) even if all I want to do is making a static binary out of it.
If you report a missing function that breaks some program to the musl mailing list (refer to http://www.etalabs.net/musl/community.html), it may well be added, occasionally even the same day. This is what I mean by "responsive". I'd use a subject like
"Compatability: cap{set,get} needed for dnsmasq"
(identify that the issue is compatability, name functions, and name the program you're trying to build.)
The goal seems to be "enough glibc compatability for most software to work".
Superb ! :D
The "appropriate" -L is likely to be -L/usr/include, which will massively break everything.
Which I tried and as you said - breaks everything (didn't even compile).
I presume the "libgtk+" was a typo?
Yes :oops: - was using libgtk+ for example, then changed my mind and change to libssl, but obviously I didn't replace all the references. :oops:
You can install in any prefix where there aren't glibc-based libraries.
In other words, if the musl prefix is /opt/musl, you can use that as the prefix for all the libraries, and musl-gcc will find them all.
Thank you, this is what I need to know. I want to avoid appending extra -L and -I unless it's really necessary. I suppose this will also work with packages using pkg-config? I probably have to append /opt/musl to PKG_CONFIG_PATH?

I will try to compile a few others and see how far I get.

EDIT: removed comment about in_systm.h, I see you have raised that on the mailing list too.

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#8 Post by Ibidem »

pkg-config seems designed to break cross-compiles.
if you use it, any packages missing for the cross-compiler and present for the host will get picked up...again, instant breakage.

I guess you need PKG_CONFIG_LIBDIR; set it to the directory containing .pc files in /opt/musl (probably /opt/musl/lib/pkgconfig, but some packages use oddball directories...)

capset & capget apparently belong in libcap; it seems glibc supports the symbols for backwards compatibility, and hasn't had it in headers since 1999 or so.. Rich doesn't think much of the "capabilities" approach, says it's mostly a false sense of security.

_PATH_LOG is used for some insane workaround for broken syslog daemons; looks like there should be an easy patch, though (change the SUN/ANDROID check to add !defined(_PATH_LOG).

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#9 Post by Ibidem »

I'm considering building a musl snapshot "SDK" of sorts.
Basically, it would have whatever's needed to start building against musl, installed in /opt/musl-09 .
Right now I think these would be what I'd start with:

musl
linux-libc-dev
ncurses (pdcurses is also an alternative, but I don't see a reason)
zlib
openssl
libcurl
libbz2
liblzma
libcap
libtirpc
libnl3

perl would be provided by the host, as would gcc and binutils (though I could just use pcc...); all parts except the Linux headers (which Torvalds appears to consider PD once sanitized) and libnl3 (LGPL) are PD, MIT, or BSD licensed, so strictly speaking don't require source (though I'd have that available).
But, OTOH, I'd be publishing the build-scripts and any patches...

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#10 Post by Ibidem »

Ibidem wrote: musl
linux-libc-dev
ncurses
zlib
openssl
libcurl
libbz2
liblzma
libcap
libtirpc
libnl3
libssh2 builds OOB with openssl and zlib installed
libnl3, zlib, ncurses, and openssl build with minimal patching
libcap needs pam
pam needs hacking, and still doesn't quite build
pam wants RPC, which means libtirpc...(dependency can be hacked out)
libtirpc is probably a porter's worst nightmare...

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#11 Post by jamesbond »

Thanks Ibidem, this is cool! I could get the SDK from your git repo I suppose?
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#12 Post by Ibidem »

I presume you mean github.com/idunham/musl ?
Probably not that one--I'm thinking of something more like /usr/src is on certain systems.
*If* I do an SDK, it would have a repo for the build environment, and then I'd have a binary download also available from that repo.
There's no good reason to build a musl libc binary package, without an environment--it takes maybe 2 minutes to build on an Atom, you need updates and bugfixes, and so on.
But if I distribute libnl3/pam, I need to distribute sources anyhow...

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#13 Post by jamesbond »

I was thinking along you'd post a build script on github.com/idunham/musl that will download source packages (the ones you've listed) and relevant patches to make it compile with musl; and compile it to some location (/opt/musl-09 is good) :D
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#14 Post by Flash »

You guys feel free to invite the developers to join our forum. Explain to them how useful Puppy could be to a developer. :)

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#15 Post by Ibidem »

Flash wrote:You guys feel free to invite the developers to join our forum. Explain to them how useful Puppy could be to a developer. :)
Most of them already have their own projects/distros ;)...
I'm aware of about 20 people using it, though there may be more.
I also know of at least 3 small distros that use it (jhuntwork's LightOS, Sabotage Linux which has as many git repos as users, bootstrap-linux which may not be commonly used, but can recompile itself with only 6 packages)
Openwall GNU/*/Linux is going to switch to musl at some point.

musl has an IRC channel (#musl on freenode) and a mailing list.
Also, I'm not sure the Puppy way/philosophy is acceptable to many of them...

Somehow, I doubt that the invitation would go over too well.

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#16 Post by Ibidem »

jamesbond wrote:I was thinking along you'd post a build script on github.com/idunham/musl that will download source packages (the ones you've listed) and relevant patches to make it compile with musl; and compile it to some location (/opt/musl-09 is good)
look at github.com/idunham/src-musl
Currently handles linux headers, musl, zlib, ncurses, and libnl.
Need to add openssl, libssh2, libcurl, maybe c-ares or such...

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#17 Post by jamesbond »

Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#18 Post by Ibidem »

jamesbond wrote:Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.
Thanks for the news!
Mentioned this on IRC, and Rich commented "if you bet that any given piece of software had a security vulnerability when linked to diet, you'd probably come out ahead."

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#19 Post by technosaurus »

so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewto ... 399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#20 Post by Ibidem »

technosaurus wrote:so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewto ... 399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?
Yes PLEASE! :D
There aren't many of the musl distros that have X11 really working, and the build approach for Sabotage (the main distro) are fairly similar.

On an unrelated note:
musl 0.9.1 has just been released.
At this point it's anticipated that glibc binaries not using pthreads and from source conforming to POSIX should mostly work with musl (ie, partial LSB ABI support).
There are several regex fixes, which among other things will allow native builds of ncurses with busybox sed...
The ARM port has several stability fixes.
In other words: It's a big improvement over 0.9.0, and is pretty much mandatory if you're using musl on ARM.

Post Reply