Unnamed Remastering Tool

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

Unnamed Remastering Tool

#1 Post by Iguleder »

Unnamed Remastering tool

Lately, I've been seriously involved in the development of automated package building infrastructures. I wrote a set of scripts which built, trimmed and packaged pretty much anything. I even made a magical script which takes a Puppy ISO and the matching devx module and uses them to build packages, so you can build packages for a Puppy version different from the one you use.

Some time ago I had a brilliant idea: if we have remastering tools and the means for automated package building on any Puppy for any Puppy, why don't we combine the three to form the ultimate remastering tool?

I decide to implement this idea in a tool, which takes a Puppy, builds your favorite packages inside it (so they're 100% compatible with it), optimizes it (strips all binary files, optimizes all images, re-compresses all archives, etc') and produces an optimized puplet with lower memory requirements and small size.

It is in alpha quality; use with care. Any damage done to your system is under your responsibility; if you don't agree with this, don't use it.

The tool is not meant for novice users; use it only if you have a certain degree of experience with Puppy, package building, package creation and remastering.

How It Does Whatever It Does

It's simple. In the UNIX family of operating systems (which includes Puppy), there is a cool utility called "chroot", which runs a command and changes the file system root it sees. For example, if you run this:

Code: Select all

chroot /path/to/directory /path/to/executable a b c
your executable will run with the given command-line arguments and see the specified directory as its file system root (/), while your system is unaffected.

Now, imagine a Puppy is extracted to a large partition with the devx module loaded on top of it, as in a running Puppy. If you use chroot with such setup, you can use this environment to build packages; the only thing you need is an automated way to build packages; that's where Builder fits in.

Builder is a simple tool I wrote for building packages. It receives one command-line argument, which is the path to a script which defines how to build a certain package.

Here's an example for a script skeleton:

Code: Select all

#!/bin/sh

PKG_NAME="mplayer"
PKG_VER="svn$(date +%d%m%Y)"
PKG_REV="1"
PKG_DESC="Multimedia player"
PKG_CAT="Multimedia"
PKG_DEPS="+yad"

download() {
    ...
}

build() {
    ...
}

package() {
    ...
}
The script defines the package name, version, description, category and dependencies. Then, it is split into three parts, which contain the instructions for downloading the package, building it and turning it into a Puppy PET package.

Builder prepares a build directory and builds each package according to the magic recipe contained in its build script. Once it's done, Builder trims the package, splits it and compresses it. It is 100% automatic.

In short, the remastering tool extracts the Puppy you give it, then uses chroot with Builder to build your desired packages inside it. Then, it trims and optimizes it; unneeded documentation and some useless files are removed to reduce its size.

Finally, the puplet is created from the extracted Puppy.

System Requirements

- squashfs-tools 4.1 or above, with support for at least one compression method. Make sure it supports the "-comp" option; if it doesn't, install a newer version. This one should be just fine.
- Wary, Slacko, dpup or any Puppy built with a recent Woof.
- Some build scripts require xz-utils, Git and Subversion.

Limitations

In its current implementation, the remastering tool has several limitations:
- It doesn't deal with dependencies; you have to build the packages in the right order if they depend on each other.
- It doesn't have a name :p

Included Build Scripts
- Develpment: AdvanceCOMP, OptiPNG.
- Fun: DOSBox, ScummVM and OpenTTD.
- Utilities: Galculator, gHasher, TrayFreq, Xarchiver, rxvt-unicode, emelFM2.
- Documents: Leafpad, Zathura.
- System tools: LXTask.
- Multimedia: Aumix, DeaDBeeF, FFmpeg, MPlayer, TiMidity++, freepats.
- Misc. stuff: file, Squashfs, syslinux, dhcpcd, Yad.
- Graphics: mtPaint, XSane, Viewnior.
- Desktop: LXAppearance.
- Fonts: Liberation, Ubuntu, Libertine, DejaVu, font-bh-ttf, FreeFont.
- Internet: Seamonkey, youtube-dl.

Desktop stuff:
- Window Maker: Window Maker, AlsaMixer.app, asmon, wmclock, wmsystemtray.

The included build scripts allow you to transform a Puppy into something very similar to Next Puppy, a minimalistic dpup puplet with Window Maker.

Notes

- The trimming script assumes you have Builder and all its dependencies available on the host distribution used to run the remastering tool. If you don't, simply copy the contents of the Builder directory to / and install both AdvanceCOMP and OptiPNG if you want to optimize your puplet. Bear in mind that puplets created using the remastering tool are "self hosting" and should have all those built-in, so they can be used to create the next versions of your puplet without any additional effort.
- If you read the list of included build scripts, the tool is able to build DOSBox and ScummVM, which are great if you're into retro DOS gaming. I also wrote build scripts for TiMidity++ and freepats (which contains sound samples), which provide MIDI emulation through software. It is useful since most modern sound cards lack the MIDI support that was popular back at the DOS era; most DOS use MIDI for music and sound effects.
- The MPlayer package built using the script can be built statically, without an external FFmpeg. This saves lots of space and some RAM.
- The MPlayer build script also installs a script called "mplayer_shell", which includes a file chooser that makes it easier to use the "mplayer" binary when GNOME MPlayer or any other GUI frontend is missing. It depends on Yad, so make sure you have it, too.
- The Seamonkey package is built from the static, official binary packages provided by Mozilla and not compiled, since it takes ages and because it updates so often it makes no sense to compile it on your own. Also, when you build it yourself, the updater is missing. Moreover - the official binaries are optimized for size, so there's very little (if any) difference in the result package size.
- The font packages are huge, especially the DejaVu fonts; use with care. For the best balance between small size and looks, go for the DejaVu and Liberation fonts.
- At the moment, the tool does not support selection of the default wallpaper, GTK theme, window manager theme, etc'.
- At the moment, the tool deletes all wallpapers in /usr/share/backgrounds, because they waste lots of space. I can change this behavior to make it keep only the default wallpaper.

How to Use It

Using the tool is very simple. Start by editing the configuration file.

Code: Select all

defaulttexteditor config
Make sure you give it the right paths to the Puppy ISO image and the matching devx module. Also, make sure the kernel your puplet uses supports the Squashfs compression you chose.

Once the configuration file is ready, run the extraction script to fully extract the Puppy you are remastering.

Code: Select all

./0extract
If you ever want to remove up the extracted files, run it with the "clean" option:

Code: Select all

./0extract clean
If the extraction succeeded, you'll see no errors. Proceed to the build environment preparation script:

Code: Select all

./1prepare
Bear in mind that this script mounts file systems and does some very dangerous stuff. Use it with care; never call 0extract with the "clean" option before you make sure the file systems mounted by 1prepare are unmounted.

As with 0extract, you can always clean up the directories created by 1prepare using the "clean" option:

Code: Select all

./1prepare clean
This will also unmount all file systems mounted by the script and make sure it's safe to call 0extract with the "clean" option. I'll say it again: never ever call 0extract with the "clean" option without doing this first.

Once the build environment is ready, you can proceed to the package building stage:

Code: Select all

./2build package
This script will build and install a single package. The PET packages built through it will be available in the "packages-pet" directory and the sources will be available under "sources". This is the heaviest part of the remastering tool.

Regarding the packages you can build, you can find all included build scripts under builder/var/builder. Of course, you can add your own build scripts to that directory. Just take a look at the existing ones to see how to write one. It's very easy.

In order to produce smaller and faster packages, build AdvanceCOMP and OptiPNG and make sure you build the former twice. AdvanceCOMP is used for optimizing packages (or more percisely, to compress them better), so you'll need to build it again to make it optimize its own package.

Of course, you can always clean up the working directory used for building packages using the "clean" option:

Code: Select all

./2build clean
Once you built all your packages, run 1prepare with the "clean" option to unmount all mounted file systems:

Code: Select all

./1prepare clean
This ensures you don't run into any synchronization or permission problems because of mounted file systems when you proceed to the next step.

Then, you can call the trimming script:

Code: Select all

./3strip
This script will optimize your puplet. It takes ages to run in some cases, but it's definitely worth it in my opinion. If you remaster a truly barebones puplet it won't take long.

Finally, call the puplet building script:

Code: Select all

./4builddistro
This script will take a long time, since it compresses lots of files. Once the script is done, the build output (your puplet's ISO image and a matching devx module) will be available in the "dist" directory.

You can easily clean the output of 4builddistro using the "clean" option:

Code: Select all

./4builddistro clean
FAQ

Does it work with Lucid Puppy?

I don't know, since Lucid Puppy was built using an older Woof. I think it won't work. The only thing I can say for sure is that Lucid Puppy supports only gzip compression.

Regarding Puppy 4.3.1 and older - I'm confident it won't work.

Which compression should I use?

Traditionally, Puppy uses gzip compression, which provides a good balance between performance and the puplet size. It is the sane default you should use if unsure.

You can also use XZ compression, which provides much better compression at the cost of time and speed. It requires Linux 2.6.38 and above; it is good for USB installations, since the bottleneck in many cases is the slow reading speed - if the puplet is smaller, less time is spent on reading it from the slow flash drive.

Also, there's LZO compression, my personal favorite - it is supported on Linux 2.6.36 and above and provides insanely fast decompression. It makes your puplet way faster, but much bigger. I think it's nice to have a LZO-compressed flavor for a puplet, for those who use it for hard drive installations and don't care much about size.

If you're still unsure after this explanation - try all. You'll find out that LZO is very fast and comfortable to work with, XZ produces smaller puplets at the cost of very long compression times and some loss of performance, while gzip is well-balanced.

Can I use it to optimize my Puppy?

Yes, you can use this tool just to optimize a Puppy, without building any packages.

Which packages should I choose?

As mentioned earlier: I suggest you to always include AdvanceCOMP and OptiPNG and build the former twice.

Regarding the rest, the answer is simple: listen to your heart. :lol:

How long does it take to build a puplet?

It depends on how fast your machine is, the compression you use, whether you run the 3strip script and your choice of packages.

Does it remaster the devx?

Yes, it does.

Can I use it with a Puppy that has a zdrv module?

Maybe; this feature exists but I haven't tested it.
Attachments
remasterer-001.tar.bz2
(17.22 KiB) Downloaded 317 times
screeny.png
Screenshot of dpup remaster with Window Maker
(133.68 KiB) Downloaded 432 times
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

User avatar
puppyluvr
Posts: 3470
Joined: Sun 06 Jan 2008, 23:14
Location: Chickasha Oklahoma
Contact:

#2 Post by puppyluvr »

:D Hello,
Downloaded, but not yet checked out...
Brilliant concept...
The talent pool around here is getting really deep...
I love being a small fish in an ever expanding pond...
So much learning to do.. 8)

I`ll get back after a bit of studying, and then a bit more...
Close the Windows, and open your eyes, to a whole new world
I am Lead Dog of the
Puppy Linux Users Group on Facebook
Join us!

Puppy since 2.15CE...

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#3 Post by technosaurus »

I have one that I did a couple years ago that handled any number of sfs files (including the main one) and would then do installed pets and any number of additional pets/sfs...never really felt like "marketing" it though. It's pre-woof so it may be useless now, but I can send an older version if it will be helpful.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

ldolse
Posts: 367
Joined: Fri 23 Oct 2009, 16:33

#4 Post by ldolse »

I've been working quite a bit with the remastering tool, and although the latest version is in the Next thread I figure this is probably a more appropriate place to talk about it in general.

Firstly - unti I read this thread I didn't really comprehend how awesome this tool is, or what it's doing under the hood - I'm still building all the packages for my first real remaster using it so I haven't stressed everything yet, but what I've used is looking really good.

One issue that I'm seeing is that the tool seems to assume that strippkg is in the sfs or the devx that's being targeted for remastering. They weren't there, and until I understood the deal with chroot I couldn't figure out why I kept getting file not found errors. Once I stuck them in the sfs the issue went away. Not sure if splitpkg and buildpkg are similarly affected.

I've got a couple feature requests - I suspect these might actually be quite simple based on the way aufs works:
  • Target packages for installation into the devx - e.g. I didn't realize I need cmake for a package, and I need to install it to the chrooted environment in order to use it, but I don't want it on the final sfs. Targeting to the devx would be perfect. Otherwise I need to figure out some manual install to the devx folder (and unfortunately installwatch doesn't support cmake), or worst case re-build the iso/devx from woof.
  • Target the packages for a third sfs folder - this would allow you to create pets/sfs files for projects that are typically difficult because installwatch doesn't support the installation methods (e.g. perl/python/cmake installers). As an example I'd like to build a Squeezebox sfs file as that project is huge, has lots of packages, and is mostly perl based. Targeting a different sfs would be perfect for a scenario like that.
I think because these would wind up in a different folder in the UnionFS it would be trivial to control which folder the files finally ended up when everything gets cleaned up. Let me know if I'm misunderstanding how it works.


I'm also seeing one issue with a little package - this package doesn't have a proper configure script, just a Makefile that's pre-made as it's a simple package. When you run make install it's executing these lines:

Code: Select all

install: all
	@echo Installing...
	$(Q)install -m 644 $(SND_PCM_BIN) ${DESTDIR}/usr/lib/alsa-lib/
	$(Q)install -m 644 $(SND_CTL_BIN) ${DESTDIR}/usr/lib/alsa-lib/
${DESTDIR} is mapping to the actual build directory when called from ./1build, and since usr/lib/alsa-lib/ doesn't reside inside the build directory the install fails. Do you think I just need to remove ${DESTDIR}? Working on an alternative install, but apparently Next is missing the alsa-lib development headers (a common issue on several puppy variants). I figured while I was fiddling with that I'd ask if there was a simple way to fix the Makefile.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#5 Post by amigo »

DESTDIR is null unless you give it a value. To use it you need to run:
make DESTDIR=/some/dir install
or sometime before executing make install:
export DESTDIR=/some/dir
...
make install

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#6 Post by Iguleder »

The first is something I wanted to implement in the long term, I think I'll get around to it soon, it's pretty easy.

Regarding the second feature request - it's something very hard to implement, since there's no generic way to do this. I think I won't implement that.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

ldolse
Posts: 367
Joined: Fri 23 Oct 2009, 16:33

#7 Post by ldolse »

Iguleder wrote:Regarding the second feature request - it's something very hard to implement, since there's no generic way to do this. I think I won't implement that.
If it's hard to implement then no big deal, but I guess I'm not understanding part of how it works. I thought it was working something like this:
  • Set up a temp layered file system with 3 layers, devx, sfs, and new content layer
  • Chroot to this new filesystem, build packages
  • Move the files from the 3rd layer for new content to the sfs layer
If I'm understanding it correctly couldn't you then choose to move the new content to either the devx, sfs, or a third custom folder? I guess if you're doing this with multiple packages and dependencies you need to copy to both the sfs and the third custom folder for each new package, but that wouldn't be a big deal.

If I'm misunderstanding and it's just two layers then I see why it would be difficult.

ldolse
Posts: 367
Joined: Fri 23 Oct 2009, 16:33

#8 Post by ldolse »

amigo wrote:DESTDIR is null unless you give it a value. To use it you need to run:
make DESTDIR=/some/dir install
or sometime before executing make install:
export DESTDIR=/some/dir
...
make install
Somehow it's getting a value from the build scripts, and the value is the root of the source folder - maybe it's just a variable collision. It sounds like I can delete it from the Makefile and I should be fine then.

Edit - I see where I can tweak the build script to get rid of it now, thanks for the tip!

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#9 Post by noryb009 »

It would be great if you make the variable and function names interchangeable with another build system's (like arch's PKGBUILD) variable and function names. It would at least give users an optional starting point to making Builder scripts.

Post Reply