A compressed Puppy SAVE file?

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

A compressed Puppy SAVE file?

#1 Post by sunburnt »

There`s many advantages to Puppy`s Save file being compressed.
It would take less space on the media, and in ram ( usb install ).
It also allows faster file access, and faster save/restore ( usb install ).

Btrfs is the Ext4 FS replacement, and will become the new Linux FS.

e2compr is light and fast, and maybe better suited for a Save file.

Both do on-the-fly transparent compression of the FS.

Both require compiling a new kernel and some of the FS tools.


I`ve never succeeded at compiling a kernel, and it`s far too critical.


# If anyone with experience is willing, it seems to be a worthy task...
.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#2 Post by Karl Godt »

Linux/i386 3.4.0 Kernel Configuration :

Symbol: BTRFS_FS [=n]
Type : tristate
Prompt: Btrfs filesystem (EXPERIMENTAL) Unstable disk format
Depends on: BLOCK [=y] && EXPERIMENTAL [=y]
Selects: LIBCRC32C [=n] && ZLIB_INFLATE [=y] && ZLIB_DEFLATE [=n] && LZO_COMPRESS [=y] && LZO_DECOMPRESS [=y]

Symbol: BTRFS_FS_CHECK_INTEGRITY [=n]
Prompt: Btrfs with integrity check tool compiled in (DANGEROUS)

I the beginning i did not choose btrfs, guess because of these statements .

Worth to read about kernel-2.6.30 btrfs version compatibility threshold :
http://lwn.net/Articles/342892/

https://btrfs.wiki.kernel.org/index.php/Main_Page

And about compression since kernel-2.6.38 :
https://btrfs.wiki.kernel.org/index.php/Mount_options

Some tables :
https://btrfs.wiki.kernel.org/index.php/Getting_started

I am thinking of compiling a new kernel since the last had been in
2.6.39-KRG-i486-cp-DELL running .. HURRAY
(4 posts) (3 voices)

Started 8 months ago by Karl Godt
http://puppylinux.info/topic/2639-krg-i ... ing-hurray

And since i have a Core2Duo as -iXeon .

Bruce B

#3 Post by Bruce B »

You have the kernel software available and the config file.

Maybe you only need to make one or two additions.

The hard part is getting a good set of instructions and following them.

There are some instructionals on the Internet and Barry has one, I don't know if it is current, you can check.

I say compile the kernel. If we never hear from you again, we can figure things didn't go well.

~

User avatar
pemasu
Posts: 5474
Joined: Wed 08 Jul 2009, 12:26
Location: Finland

#4 Post by pemasu »

http://www.murga-linux.com/puppy/viewtopic.php?t=80274

Kernel compiling has never before been this easy. Read the instructions. Edit build.conf file. Launch ./build.sh script. Choose ready DOTconfig as template. There are several to choose from. Use terminal gui editor with tickable choices to add that btrfs and possible dependent choices as Karl Godt cut showed. Called make menuconfig. It is ncurses based terminal gui. Just choose number one while executing the ./build.sh script.

Let the script do the job. You get kernel in one folder, dir2pet it to the kernel pet. An other folder called kernel_headers. Dir2pet it. And kernel source sfs has been created already. Done.

Then you need to switch the kernel of course. I use woof2 for that. I create new puppy with that new kernel. Has been easiest for me.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#5 Post by Karl Godt »

I say compile the kernel. If we never hear from you again, we can figure things didn't go well.
make ARCH=x64_64 menuconfig makes now the kernel while i just wanted to create a second dotConfig :lol:

My reboot problems with recent kernels seems to be related to the fact that the menuconfig autoselects "CONFIG_X86_REBOOTFIXUPS=y" if "ARCH=x86 (_32)" .

bash-3.2# grep -i 'not set' .config >not_set
bash-3.2# wc -l not_set
618 not_set
bash-3.2# wc -l .config
6850 .config

Guess 2hrs from now will make finish . Will try to create a simple install script that i buried sometime somewhere . AUFS is missing . Also a 3.4.X-version patch .

Xz compression for kernel , Server and 1000 Hz . Also enabled nearly everything related to boot into kernel even flash nand devices, scsi, ide, tape, raid so i don't have to add drivers to the initrd.gz .
Let 's look if the kernel becomes less then 5MB .
Image

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#6 Post by Karl Godt »

Nope, seems problems with at least one driver :

So 9. Sep 14:37:07 GMT+1 2012
/root/Downloads/KERNEL/linux-3.4:/root/Downloads/KERNEL/linux-3.4
/bin/sh: as68k: Kommando nicht gefunden.
make.bin[3]: *** [drivers/net/wan/wanxlfw.inc] Error 127

driver as command ? looks like some makefile syntax problem .

Also my make wrapper script to doublequote everything beyond = let make complain :

So 9. Sep 13:48:27 GMT+1 2012 ARCH=x86_64 menuconfig
/root/Downloads/KERNEL/linux-3.4:/root/Downloads/KERNEL/linux-3.4
Makefile:484: /root/Downloads/KERNEL/linux-3.4/arch/"x86_64"/Makefile: No such file or directory
make.bin: *** No rule to make target `/root/Downloads/KERNEL/linux-3.4/arch/"x86_64"/Makefile'. Stop.


and for _64 bit i need recent binutils it seems :

/root/Downloads/KERNEL/linux-3.4/arch/x86/Makefile:96: CONFIG_X86_X32 enabled but no binutils support

You will need a recent binutils (2.22 or later) with
elf32_x86_64 support enabled to compile a kernel with
CONFIG_X86_X32

bash-3.2# objcopy --version
GNU objcopy (Linux/GNU Binutils) 2.18.50.0.2.20071001

8)

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#7 Post by Karl Godt »

make finished with '0' ; modpost WARNING: modpost: Found 20 section mismatch(es) : sometimes i had only one and sometimes more than 50 for a kernel compile . Average . The driver error related to missing assembler binary as68k which seems to compile if binutils are compiled for Motorola target . First thought as68 would be a typo cos i have as86 installed :

CONFIG_WANXL_BUILD_FIRMWARE:
Allows you to rebuild firmware run by the QUICC processor.
It requires as68k, ld68k and hexdump programs.
You should never need this option, say N.

Have also applied the .9 version patch now .

And at least the Makefile did not like busybox xz cos too few -options :
xz: unrecognized option `--check=crc32'
BusyBox v1.18.3 (2011-05-01 19:45:13 CEST) multi-call binary.

Usage: xz -d [-cf] [FILE]...

Decompress FILE (or stdin)

Options:
-d Decompress
-c Write to stdout
-f Force

arch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
gcc: arch/x86/boot/compressed/piggy.S: No such file or directory
gcc: no input files
make.bin[2]: *** [arch/x86/boot/compressed/piggy.o] Error 1
make.bin[1]: *** [arch/x86/boot/compressed/vmlinux] Error 2
make.bin: *** [bzImage] Error 2

So 9. Sep 18:00:42 GMT+1 2012
/root/Downloads/KERNEL/linux-3.4:/root/Downloads/KERNEL/linux-3.4
Setup is 15168 bytes (padded to 15360 bytes).
System is 8244 kB
CRC f3be5482
WARNING: modpost: Found 20 section mismatch(es).
To see full details build your kernel with:
'make CONFIG_DEBUG_SECTION_MISMATCH=y'
Did not set
CONFIG_CC_OPTIMIZE_FOR_SIZE:
Enabling this option will pass "-Os" instead of "-O2" to gcc
resulting in a smaller kernel.
Symbol: CC_OPTIMIZE_FOR_SIZE [=n]

Next time with aufs added will set it, aufs driver is not so big :
76 /lib/modules/2.6.30.9-i586-dpup005-Celeron2G/kernel/fs/aufs/aufs.ko.gz

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#8 Post by sunburnt »

I knew this would get a response, I`m glad there`s interest.

Karl; The LWN article was 3 years old, the wiki page was up to date..
Web pages should have to be dated by law.

Looking at your posts, it`s like one of my worst jobs piecing an app. together.
Endless chasing of details and finding parts that might actually work.
Most of the dialog is unfamiliar to me, I wouldn`t know where to start.


Btrfs will become the new Linux FS, so using it`s forward looking.
But may not be stable enough, and it may be overkill for a Save file.
e2compr might be in keeping with Puppy`s motto of fast and small.

I figure there`s folks that are already setup for building a Puppy kernel.
Folks that know the tools and the build scripts from repeated use.


BruceB; I like your optimism... :lol:
But I know it`s hard for an experienced person to get it right (ie: Karl).
I`d be starting from scratch for the files and tools, and knowledge.
To integrate it into Puppy as an option, then it has to be done right.

pemasu; I assume a different kernel is needed for each Puppy type.
I hadn`t even thought about building a new Puppy around the kernel.
I figured to use one Puppy variant as a test bed, "Perfect" is good.
I got the kit you posted, now to begin reading and reading.

User avatar
pemasu
Posts: 5474
Joined: Wed 08 Jul 2009, 12:26
Location: Finland

#9 Post by pemasu »

You need to compile kernel using devx sfs of that Puppy. Used gcc version needs to be same for compiling kernel and when using the puppy. Otherwise extra third party kernel module drivers compile might fail.
Besides that. Those included DOTconfigs are all tested to work. Using Dotconfig from Puppy version you are going to use is safest pet. But all of the included ones have been used for compiling working Puppy kernel.

I know that there are guidelines in this forum to switch the kernel by remastering or by unsquashing, I dont know which way..... I havent tested that. James C frequently has posted about his experiments with kernel switching.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#10 Post by Karl Godt »

More news : Grub says 'Staring up ...' and the kernel says it is ' edd ' ing , also vga=ask works and than the kernel waits forever for something .

Recompile with a working dotConfig now with USB into kernel, more drivers, less debug : same symptoms .

There are actually several new features while make install regarding lilo+boot which had been always some sort of warning but now things error :

* Fatal: Cannot open: /etc/lilo.conf

* Warning: LBA32 addressing assumed
* Warning: Partition 3 on /dev/sda is not marked Active.
* Fatal: No images have been defined.

* Warning: LBA32 addressing assumed
* Fatal: Can't put the boot sector on logical partition 0xC0010300

which i went around by make -i install

*

Few things to remember :

* BLK_DEV_LOOP_MIN_COUNT seems to be newish and is set to 8 by default .

* CONFIG_BLK_DEV_RAM_SIZE is currently set to some 16384 by BK (Macpup-529/Precise-5.2.60) though i had read on his block that it would be
Posted on 17 Jul 2012, 7:57 by BarryK
Re advanced /sbin/init
K Godt,
My init script now checks the size of the ramdisk and requires >= 20000 KB.

I like your extra logic to look at filesystem check interval.
http://bkhome.org/blog/?viewDetailed=02907

*

In the whiletime i have compiled binutils-2.22 as

Code: Select all

./configure --build=x86_64-pc-linux-gnu
and this gives me a objcopy with :
supported targets: elf64-x86-64 elf32-i386 elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big elf32-little elf32-big srec symbolsrec verilog tekhex binary ihex

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#11 Post by Karl Godt »

bash-3.2# uname -a
Linux puppypc 3.4.9-KRG-iCore2-64gb #1 SMP Mon Sep 10 19:19:24 GMT+1 2012 i686 GNU/Linux
Finally up and running .
8)

Then dl http://packages.debian.org/source/sid/btrfs-tools source . Did not apply the debian patch , nor looked into it .
Compiled fine, simply make && make install . Installs by default all bins into /usr/local/bin . No bins into some */sbin .

bash-3.2# mkfs.btrfs /dev/ram5

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

SMALL VOLUME: forcing mixed metadata/data groups
Created a data/metadata chunk of size 3342336
fs created label (null) on /dev/ram5
nodesize 4096 leafsize 4096 sectorsize 4096 size 32.00MB
Btrfs Btrfs v0.19
bash-3.2# mkdir /mnt/ram5
bash-3.2# lsmod | grep btrfs
bash-3.2# modinfo btrfs
filename: /lib/modules/3.4.9-KRG-iCore2-64gb/kernel/fs/btrfs/btrfs.ko
license: GPL
alias: devname:btrfs-control
alias: char-major-10-234
depends: libcrc32c
intree: Y
vermagic: 3.4.9-KRG-iCore2-64gb SMP mod_unload CORE2
bash-3.2# mount -t btrfs /dev/ram5 /mnt/ram5
bash-3.2# lsmod | grep btrfs
btrfs 567896 1
libcrc32c 758 1 btrfs
bash-3.2# mount | grep btrfs
"/dev/ram5" on "/mnt/ram5" type "btrfs" (rw,relatime,space_cache)
bash-3.2# echo "HiAll RAM" >/mnt/ram5/Welcomest.text
bash-3.2# cat !$
cat /mnt/ram5/Welcomest.text
HiAll RAM
bash-3.2# df /mnt/ram5
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/ram5 32768 28 28656 1% /mnt/ram5

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#12 Post by Karl Godt »

And one thing had delayed everything while compiling the kernel :

* CONFIG_SOUND_PSS=m
* CONFIG_PSS_MIXER=y
* # CONFIG_PSS_HAVE_BOOT is not set

:
In the " OSS sound modules " section there is the menu choice :

* [ ] Have DSPxxx.LD firmware file

and the Help shows

* If you have the DSPxxx.LD file or SYNTH.LD file for you card, say Y
* to include this file. Without this file the synth device (OPL) may
* not work.

When i checked it, a sub-menu is shown :

* (/etc/sound/dsp001.ld) Full pathname of DSPxxx.LD firmware file (NEW)

I did not have this file but choose the default .

That led to compile error :
* make.bin[2]: *** No rule to make target `/etc/sound/dsp001.ld', needed by `sound/oss/pss_boot.h'. Stop.

The file sound/oss/pss_boot.h gets temporarily created from a Makefile and deleted again . Took me some time to figure things :
* /root/Downloads/KERNEL/linux-3.4/Documentation/sound/oss/PSS
says

* The PSS cards and other ECHO based cards provide an onboard DSP with
* downloadable programs and also has an AD1848 "Microsoft Sound System"
* device. The PSS driver enables MSS and MPU401 modes of the card. SB
* is not enabled since it doesn't work concurrently with MSS.

* CONFIG_SOUND_MSS=m
linux-3.4/sound/oss/Kconfig wrote: config SOUND_MSS
tristate "Microsoft Sound System support"
---help---
Again think carefully before answering Y to this question. It's
safe to answer Y if you have the original Windows Sound System card
made by Microsoft or Aztech SG 16 Pro (or NX16 Pro). Also you may
say Y in case your card is NOT among these:

ATI Stereo F/X, AdLib, Audio Excell DSP16, Cardinal DSP16,
Ensoniq SoundScape (and compatibles made by Reveal and Spea),
Gravis Ultrasound, Gravis Ultrasound ACE, Gravis Ultrasound Max,
Gravis Ultrasound with 16 bit option, Logitech Sound Man 16,
Logitech SoundMan Games, Logitech SoundMan Wave, MAD16 Pro (OPTi
82C929), Media Vision Jazz16, MediaTriX AudioTriX Pro, Microsoft
Windows Sound System (MSS/WSS), Mozart (OAK OTI-601), Orchid
SW32, Personal Sound System (PSS), Pro Audio Spectrum 16, Pro
Audio Studio 16, Pro Sonic 16, Roland MPU-401 MIDI interface,
Sound Blaster 1.0, Sound Blaster 16, Sound Blaster 16ASP, Sound
Blaster 2.0, Sound Blaster AWE32, Sound Blaster Pro, TI TM4000M
notebook, ThunderBoard, Turtle Beach Tropez, Yamaha FM
synthesizers (OPL2, OPL3 and OPL4), 6850 UART MIDI Interface.

For cards having native support in VoxWare, consult the card
specific instructions in <file:Documentation/sound/oss/README.OSS>.
Some drivers have their own MSS support and saying Y to this option
will cause a conflict.

If you compile the driver into the kernel, you have to add
"ad1848=<io>,<irq>,<dma>,<dma2>[,<type>]" to the kernel command
line.
I don't have any Windows installation disks with drivers to use for '/etc/sound/dsp001.ld' firmware file so it would not build if enabled .

But i could enable everything other OSS related as module with the intergrated =y .

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#13 Post by Karl Godt »

And short : technosaurus new guess_fstype snapshot works for me on btrfs :
http://murga-linux.com/puppy/viewtopic. ... 646#651646

Post Reply