My long term someday goal: a Cross Linux From Scratch Pup

[Sit Heel Speak]

The purpose of this thread is to fully document an attempt to build a Puppy *not* using t2, not using woof, and not by back-engineering or adding-onto an existing Puppy.

Instead, I am going to attempt to build my very own Pup from scratch, by following the instructions of Cross Linux From Scratch, though I will vary from the prescribed procedure as I see fit.

This will be built on a desktop Athlon64 x2 6000+ with 2 G of ddr2-1066. So, I will be using a newer kernel, and this will not be a Puppy for older machines and netbooks.

However, once we have a reliable start-to-finish build procedure in hand, it should not be difficult to adapt it to older kernels, older machines, and specialty machines such as netbooks. Also to non-Intel, e.g. arm architectures.

Screen capture, showing the arrangement of the disks on my build machine. Sources are kept on sdb2 and the actual build will be done on sdc1.
-

[Sit Heel Speak]

Milestones will be here.
-

[Sit Heel Speak]

The book I will be following is:
http://cross-lfs.org/files/BOOK/CLFS-TR ... KS.tar.bz2

with a peek often into
http://www.linuxfromscratch.org/lfs/dow ... OK-6.7.pdf
and
http://www.linuxfromscratch.org/blfs/vi ... FS-6.3.pdf

The distro I am building from, is a September 5, 2010 local woofbuild of Quirky t2-8.0rc, much like Quirky-130. I will refer to it hereafter as "SHS-t2-Quirky". It has kernel 2.6.34.1.

OK, here we go:

page iv: Host System Requirements

The system should have these minimum versions:

Bash-2.05a
Binutils-2.12 (Versions greater than 2.20.1 are not recommended as they have not been tested)
Bison-1.875
Bzip2-1.0.2
Coreutils-5.0 (or Sh-Utils-2.0, Textutils-2.0, and Fileutils-4.1)
Diffutils-2.8
Findutils-4.1.20
Gawk-3.0
Gcc-2.95.3 (Versions greater than 4.5.1 are not recommended as they have not been tested)
Glibc-2.2.5 (Versions greater than 2.12 are not recommended as they have not been tested)
Grep-2.5
Gzip-1.2.4
Linux 2.6.22
Make-3.79.1
Ncurses-5.3
Patch-2.5.4
Sed-3.0.2
Tar-1.14
Texinfo-4.4

Using the modified version-check.sh, attached, we see:

Code: Select all

# ./version-check.sh
bash, version 3.00.16(1)-release
Binutils: (Linux/GNU Binutils) 2.20.51.0.2.20091009
bison (GNU Bison) 2.4.1
bzip2,  Version 1.0.5, 10-Dec-2007.
Coreutils:  2.20.51.0.2.20091009
diff (GNU diffutils) 2.8.1
find (GNU findutils) 4.4.2
GNU Awk 3.1.6
gcc (GCC) 4.3.4
GNU C Library stable release version 2.10.1,
GNU grep 2.5.4
gzip 1.3.11
Linux 2.6.34.1
GNU Make 3.81
ncurses 5.7.20081102
patch 2.5.9
GNU sed version 4.2.1
tar (GNU tar) 1.22
makeinfo (GNU texinfo) 4.13
#

So, SHS-t2-Quirky's binutils version is "not recommended as it has not been tested," however I am going to presume it is good, as I have seen no problems with these binutils in daily use for more than two months.

The above readout made me believe that coreutils would need to be brought up to the clfs minimum requirement first, before I began; and I did so, documented in the next message; however, by running version-check.sh before and after installing the newest coreutils, I discovered that it was unnecessary to upgrade coreutils, as the version 2.20.51.0.2.20091009 is in fact coreutils-8.7.

[Sit Heel Speak]

Gnu Coreutils home page:
http://www.gnu.org/software/coreutils/
Download page:
http://ftp.gnu.org/gnu/coreutils/

Looking at [url]file://localhost/mnt/sdb2/CLFS-BOOK-TRUNK-SVN-20100828/x86/index.html[/url]
we see that the version cited in the clfs instructions is Coreutils-8.5, however the newest version on offer is 8.7, and so I downloaded this:

http://ftp.gnu.org/gnu/coreutils/coreutils-8.7.tar.xz

into /mnt/sdb2/sources/input-coreutils

unpack it by left-click in Rox (XArchiver),

shift into the unpack directory coreutils-8.7

The standard Puppy configure parameters, given at http://bkhome.org/blog/?viewDetailed=00703, are:

Code: Select all

# ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --build=i486-pc-linux-gnu

However, looking first at the clfs instructions
[url]file://localhost/mnt/sdb2/CLFS-BOOK-TRUNK-SVN-20100828/x86/temp-system/coreutils.html[/url]
we see that, it is necessary to perform the following modifications on the coreutils source first, so that the coreutils Makefile does not attempt to regenerate the man files, and so that the df (report filesystem disk space usage) program will be built:

Code: Select all

cd /mnt/sdb2/sources/input-coreutils/coreutils-8.7
touch man/uname.1 man/hostname.1
cat > config.cache << EOF
fu_cv_sys_stat_statfs2_bsize=yes
gl_cv_func_working_mkstemp=yes
EOF

The above six commands were executed, and then the compile was run, per this amalgam of the Puppy standard configure and the clfs prescription plus the optional parameters listed by './configure --help':

Code: Select all

CFLAGS='-o2 -march=i486 -mtune=i686'
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --build=i486-pc-linux-gnu --cache-file=config.cache --enable-threads=posix --with-gnu-ld
make
new2dir make install

In new2dir, I opted to build just a single package; because, I foresee that it will be easier to, after the distro is completely built, separate the html, doc, and man files out by using a variant of my Clipper packaging script. This way, all documentation for the base distro will be available as a single .sfs or .pet file.
-

[Sit Heel Speak]

2.3. Creating a File System on the Partition

I am ignoring the warnings here and will go ahead and create the build on a pre-existing, empty ext3 partition (sdc1).

On my machine, sda1 is a 2 G swap partition. It is already turned on automatically by Puppy (i.e. by SHS-t2-Quirky), and so there is no need to format or initialize it.
-

[Sit Heel Speak]

Since I will be building on just a single partition, it is fairly straightforward to run the equivalent of this page's instructions, like this:

1. Place the

Code: Select all

export CLFS='/mnt/clfs'

command in SHS-t2-Quirky's (Puppy's) configuration file /etc/profile, just after the

Code: Select all

export QT_XFT=true

line.

2. Append these lines to Puppy's startup user-initialization file /etc/rc.d/rc.local:

Code: Select all

# added by SHS for building clfs pup: mount sdb2 so you have the sources up right away when booted:
mount -t auto /dev/sdb2 /mnt/sdb2
# added by SHS for building clfs pup: have the build partition up already right away when booted:
mount -t auto /dev/sdc1 /mnt/sdc1
# added by SHS for building clfs pup: remove symlink if it exists:
rm -f /mnt/clfs
# added by SHS for building clfs pup: re-create the symlink fresh at each boot:
ln -s -T /mnt/sdc1 /mnt/clfs

and reboot.

Looks like this now:
http://i54.tinypic.com/zwjapz.png
-

[l0wt3ch]

Hmm!

[Sit Heel Speak]

I am not storing my sources to ${CLFS}/sources (in other words, /mnt/sdc1/sources), rather am keeping them on a separate partition (sdb2) from the partition on which the build occurs (sdc1).

The "download list" on this page is non-existent, so, attached is a bash script, download1, which will download the packages and scripts from http://cross-lfs.org/files/packages/svn/.

First, create the sourcecode directory to hold them in:

Code: Select all

mkdir /mnt/sdb2/CLFS-PACKAGES_AND_SCRIPTS-SVN_NOVEMBER_02_2010
chmod -v a+wt /mnt/sdb2/CLFS-PACKAGES_AND_SCRIPTS-SVN_NOVEMBER_02_2010
cd /mnt/sdb2/CLFS-PACKAGES_AND_SCRIPTS-SVN_NOVEMBER_02_2010

Download and unpack the attached script, download1, into
/mnt/sdb2/CLFS-PACKAGES_AND_SCRIPTS-SVN_NOVEMBER_02_2010

Run the script to download them:

Code: Select all

./download1

Total size of the download is about 299 MB.

[Sit Heel Speak]

The list of files on page 3.2 is what has been downloaded (plus a few more) by the script attached to the above message. Total size is 299 MB according to ls -lah.

The file listed on page 3.3, grub-1.98.tar.gz, has also been downloaded by the above script.

[Sit Heel Speak]

The attached bash script, download-patches, will download all the patches listed on page 3.4 and 3.5 (in fact, everything in http://patches.cross-lfs.org/dev/ as of November 26, 2010). To use it:

Code: Select all

mkdir /mnt/sdb2/CLFS-PATCHES-SVN_NOVEMBER_02_2010
chmod -v a+wt /mnt/sdb2/CLFS-PATCHES-SVN_NOVEMBER_02_2010
cd /mnt/sdb2/CLFS-PATCHES-SVN_NOVEMBER_02_2010

Download and unpack download-patches into /mnt/sdb2/CLFS-PATCHES-SVN_NOVEMBER_02_2010. Run it:

Code: Select all

./download-patches

[Sit Heel Speak]

page 4.1, is already taken care of by what we did for page 2.4.

To create the subdirectory into which the toolchain will be built, the commands on page 4.2 were run exactly as given:

Code: Select all

install -dv ${CLFS}/tools
ln -sv ${CLFS}/tools /

This creates /mnt/sdc1/tools and a symlink to it, /tools. It is also accessible as /mnt/clfs/tools, of course.

[Sit Heel Speak]

To create a subdirectory into which a cross-toolchain can be built, the commands on page 4.3 are run exactly as given:

Code: Select all

install -dv ${CLFS}/cross-tools
ln -sv ${CLFS}/cross-tools /

Open a Rox window on /mnt/clfs and you'll see them.

[Sit Heel Speak]

Since SHS-t2-Quirky (i.e. Quirky Puppy) runs by default as root and does not carry the groupadd and useradd commands to create a new user, I will go ahead and build clfs-pup as root.

[Sit Heel Speak]

(this and the next few messages will be somewhat messy at first; I'm running out of real estate on my own screen, and need to think out loud)

One problem with the clfs-prescribed procedure of page 4.5, is that, overwriting /root/.bashrc with the suggested version will cause all instances of rxvt to observe the changed settings, in other words to see only the limited set of environment variables intended for the clfs build.

But I want to be able meanwhile to do other work on this machine while building the clfs-pup.

For example, in previous attempts to build individual packages under Puppy according to lfs or blfs or clfs instructions, it has often been my experience that I need to first compile-and-add a dependency into *not* the lfs/clfs build, but rather into the Puppy host distro (in this case SHS-t2-Quirky) itself. In such cases, where I need to compile-and-add something into SHS-t2-quirky itself as a preliminary requirement, I want to be able to do the compiling in the full SHS-t2-Quirky environment.

Also, I need to meanwhile compose these messages, and I want a bash command, being run for illustrative purposes of this thread, to observe the full SHS-t2-Quirky environment (e.g. PATH and LD_LIBRARY_PATH).

So, in sum, I want to be able to enter either environment, my regular one or the clfs-build one, at will.

Provisionally, my solution (still being developed, see next few messages) will be this:

1. Add to SHS-t2-Quirky, a second terminal emulator.
2. Start this second terminal emulator using a bash script which limits it to the prescribed clfs-build environment.
3. And, place this environment-limiting bash script somewhere where it does not affect regular rxvt.

As this second terminal emulator, the clfs-build-terminal, I choose Sakura, mainly because it looks very different from rxvt. And so, there is little-or-no chance that I will inadvertently start a program from Sakura forgetting that, when in Sakura, I am in the limited clfs-build environment.

aragon posted a .pet of Sakura 2.3.8 here, but its dependency vte-0.22.5 fails to install on SHS-t2-Quirky.

So, the plan now is to install Sakura from scratch, preparatory to proceeding.

[Sit Heel Speak]

Here is a section I have added to /etc/profile, so that I don't have to re-type the compiler and linker flags every time. Also so that I can just type

./configure $PUPPYCOMPILE

instead of

./configure --prefix=/usr --build=i486-pc-linux-gnu

et cetera:

Screen capture, showing the section of /etc/profile to which I have added CFLAGS, CPPFLAGS, CXXFLAGS, LDFLAGS, and PUPPYCOMPILE

I'm sticking with -O2 as the compiler optimization. I know that -Os produces smaller code, but -O2 is in my experience thoroughly reliable. My first goal is to produce a *working* distro. Then I'll go back and fine-tune it...

Code: Select all

# next line was added by SHS for purpose of CLFS:
export CLFS='/mnt/clfs'

# compiler and linker flags added by SHS:

# CFLAGS, CXXFLAGS, CPPFLAGS per technosaurus, see http://www.murga-linux.com/puppy/viewtopic.php?p=350313 --except I'm keeping -mtune=i686 -march=i486 -w, -O2, and -Wall):
CFLAGS="-mtune=i686 -march=i486 -O2 -pipe -ffunction-sections -fdata-sections -momit-leaf-frame-pointer -fomit-frame-pointer -mpreferred-stack-boundary=2 -fmerge-constants -w -Wall"
CXXFLAGS="-mtune=i686 -march=i486 -O2 -ffunction-sections -fdata-sections -momit-leaf-frame-pointer -fomit-frame-pointer -mpreferred-stack-boundary=2 -fmerge-constants -w -Wall"
CPPFLAGS="-mtune=i686 -march=i486 -O2 -ffunction-sections -fdata-sections -momit-leaf-frame-pointer -fomit-frame-pointer -mpreferred-stack-boundary=2 -fmerge-constants -w -Wall"
# LDFLAGS is per http://www.gentoo.org/proj/en/qa/asneeded.xml except
#  leaving out '--as-needed' because many programs are reported to break with it
#  --see http://hwoarang.silverarrow.org/2010/08/13/why-should-a-package-respect-ldflags-anyway/
#  --also see http://lwn.net/Articles/192624/
#  --also see http://www.gentoo-wiki.info/HOWTO_Hashstyle, LDFLAGS must be modified
#    --when emerging binutils and glibc, it is recommended to eliminate -Wl and --hash-style
LDFLAGS="-Wl,--hash-style=both,-O1"
# PUPPYCOMPILE is after gcc --ver from Puppy Quirky 130
#  plus the build parameter from http://bkhome.org/blog/?viewDetailed=00703:
PUPPYCOMPILE="--prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libdir=/usr/lib --datadir=/usr/share --includedir=/usr/include --infodir=/usr/info --mandir=/usr/man --sysconfdir=/etc --localstatedir=/var --build=i486-pc-linux-gnu "

# end of compiler and linker flags added by SHS

[Sit Heel Speak]

reserved

[Sit Heel Speak]

reserved

[Sit Heel Speak]

reserved

[amigo]

I hate to rain on your efforts as you may be having what seems to be a good start. I think you need to work with a later version of the LFS instructions. the problem is with these versions:

- Binutils-2.12 (Versions greater than 2.20.1 are not recommended as they have not been tested)
- Gcc-2.95.3 (Versions greater than 4.5.1 are not recommended as they have not been tested)
- Glibc-2.2.5 (Versions greater than 2.12 are not recommended as they have not been tested)

gcc-2.95 is ancient (from ~2003) as is glibc-2.2.5. They would be fine if you wanted to build a system which could run on a real i386 -in fact they'd be the last versions you could use. If you wanted to build a system which could easily use either linux-2.4.xx or linux-2.6.xx, then gcc-3.4.6 and glibc-2.3.6 would be the way to go(plus binutils-2.15.92.0.2)

- Linux 2.6.22
The final kernel used for running the system is much less relevant than the kernel version whose headers are used to compile glibc with.

Using the LFS recipe, you will be unable to compile a glibc version earlier than the one used on the host system. That means that to produce a system like you've outlined, you'll need to be running something like slackware-8.1 (which matches those specs exactly -HMMM).

And you probably will have a great deal of trouble even compiling gcc-2.95.3 for your now-running system -you'd need to first compile gcc-4.1 and then use that compile gcc-3.4.6, then use that to compile gcc-2.95.3.

This is also very outdated, but not critical:
Bash-2.05a
I'd use the last 3.2.xx (117??) available (or 4.whatever)

My guess is that you'll find my post unwelcome, but I really am trying to be helpful. I know you've been working towards all this for a couple of years now. The first thing you have to realize is how critical the choice of the toolchain componenets is. The versions for the kernel-headers, glibc, binutils and gcc must all be matched together using a known-working combination. The 'target' used becomes your 'brand' (if any). Puppy has never once used its' own branded toolchain like i486-puppy-linux. It has always used someone else's toolchain or a generic brand like what you are trying to use: i486-pc-linux. If you read enough about cross-compiling, you'll find that it is not recommended to use any generic target when creating your first-pass cross-compiler. Instead use some dummy name(like i486-shs-linux). Then when you have your cross-compiler working, you use it to create the final native compiler with your brand, or with any of the generic target names.

Once you have a native toolchain to work with, compile *everything* else using that toolchain -without doing any updates of any toolchain versions (glibc, gcc, binutils, kernel-headers) -or do any updates to the toolchain first and switch to the new versions before compiling everything else.

One thing to realize is that changing the 'target' *is* a cross-compile. But, if you are doing the work on the same system, you can avoid most of the hassle of cross-compiling because while you are producing the toolchains and basic stuff needed for a bootable system with devel environment, you can continue to usee all of the non-toolchain binaries which are on your running system. Only the toolchain has to be handled with kid-gloves. You can make that part much easier by producing statically compiled versions of binutils and gcc with the same target as your running system. then replace the installed gcc and binutils with the static versions and you are all set to rpoduce the native binaries for the new target. Once you have the new native binutils and gcc, use them to compile glibc with the new target. Install that and then re-compile gcc, binutils, gcc and glibc, install them and now you have a fully native toolchain. then you can re-compile everything else.

To emphasize, you need to re-compile gcc and binutils three times and glibc twice before you truly have a new native system.

I have produced a couple of systems using this method which I call 'cannabalization'. It avoids the use of any chroot at all. If you don't create static versions of gcc and binutils at the start, you need to compile versions with configure options for multiple targets(old target and new target). Compiling binutils this way is a snap, butt gcc is trickier. Note that when you specify a target, a directory under /usr gets created with the name of the target -this is where all the c++ libs and compiler libs and headers go. A mismatch between the installed target and compiler/bintuils target means you get a binutils/gcc combination which will not work, so make sure you always have proper packages of the original versions that you can easily re-install until you get it all right.

Don't even think about doing any of the work without using scripts until you really, really, really know all about it. As you've seen it's very complex and easy to forget what you were doing when take much-needed rests during the process.

I do wish you luck! You're trying to work out what even BK has not done...

Have a look at the DIY Linux site for a more complete explanation of cross-compiling. And look for 'jaguar linux', which is based on DIY Linux reference builds, but includes the use of (slackware) package management.

[Sit Heel Speak]

My guess is that you'll find my post unwelcome,...I hate to rain on your efforts...

Advice from one who has the experience and can communicate it well is, to the contrary, most welcome. Just because I haven't answered your posts in other threads immediately, doesn't mean I don't appreciate them! To the contrary, I am most grateful. I'm just not at a point yet where I know what I'm doing well enough to come up with a useful reply or relevant question...

The sort of iterative compiling necessary to arrive at an untainted toolchain is, in fact, what I'm doing right now, toward the compile of gtk+ so I can use Sakura. There is a circular dependency between glib and PyGobject, the solution of which requires incrementally stepping-up through the versions of both. It takes a few days for the disgust to wear off, so I can compose report messages that don't sound ...vindictive...

More later, when I arrive at the places you indicate.

Question: do you happen to know...is, in fact, Slackware an LFS/BLFS/CLFS/HLFS build? I know that Debian and Ubuntu and Fedora are not.
_____________________________
Creme tangerine and Montélimar
A ginger sling with a pineapple heart
A coffee dessert--yes you know its good news
But you'll have to have them all pulled out
After the Savoy truffle.
Yes, you'll have to have them all pulled out
After the Savoy truffle!