Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 21 Oct 2014, 19:58
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Boot loaders, Gujin.
Moderators: Flash, Ian, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Author Message
Etienne
Guest


PostPosted: Tue 03 Jan 2006, 13:40    Post_subject:  Boot loaders, Gujin.  

Following:
http://www.murga.org/%7Epuppy/viewtopic.php?p=30958#30958
Several topic concerning the Gujin bootloader moved here.
http://gujin.org

kethd wrote:
I think that 1.2 truncates a CONFIG.SYS line shorter than 0.9 does, under FreeDOS under the same conditions. So this would have to be the fault of Gujin? I suggest that Gujin always by default echo the actual commandline it is going to fully process.


That is the first parameter passed on the command line which is the
name of the kernel file with its path (to know where is the file and so
deduce the root filesystem). You should still have 232 chars including
this filename with version 1.2 - IMHO it should be enough.
Gujin cannot know if something else is truncating the line (like SHELL=)
and so cannot display a warning.
There is a compilation switch to echo the command line, it is:
PRINT_COMMAND_LINE - but you have to regenerate tiny.exe
from source.
It is not enabled because some Linux version were already displaying
it - and usually still do it if you add the "debug" parameter to the
command line (and maybe remove "quiet"), I can send you a
modified version of tiny.exe tomorrow if you just want that, but
just try "debug" first.

kethd wrote:
linld can accept up to about 256 characters (or eight parameters) to pass to the kernel/init, in a file. This is extremely useful and helpful in Puppy, for potentially troubleshooting rc.sysinit. (Because rc.sysinit is extremely difficult to edit, it is desireable to make it highly flexible based on parameters passed by the bootloader.) Please add this feature, in a way compatible with linld.


OK, I'll have a look at it for v1.3, along with all the other stuff.
Something like "tiny.exe vmlinux initrd.img.gz @cmdfile" ?

kethd wrote:
I only partially understand the issues with decompressed image.gz, but I am quite sure other DOS bootloaders are over-all more efficient, regardless of who is doing the job when. More to the point, I know for sure that this computer is capable of decompressing image.gz about five times faster than Gujin is actually doing it; I'd be happy to try to work with you to speed it up. (Esp since so far in my experiments only Gujin can bootload a standard Puppy in 32MB.) It seems like maybe the simplest quick fix would be to allow a switch to Gujin to bypass the uncompression that you do and have Gujin just pass the uncompressed file like other bootloaders do? (To test this, be sure to use a computer that is only a Pentium-I.)


I can have a look at a switch to not decompress the initrd in the setup
screen, but I do not like it because people will complain that I do not
load correctly their initrd (for instance from their CompactFlash), and
that has to be my fault because they have formated their CompactFlash
using Windows and Windows did not find any erroneous sector, and
their CompactFlash is completely new and was extremely expensive.
Those still contain erroneous sectors.

The time Gujin spend is the time to read the file and then decompress
it - it has to be compared to a kernel doing the two steps and should
not be that different - but maybe to calculate the CRC32 (optimised
for size and not for speed).

kethd wrote:
Thank you for writing so much detail; I will try to study further.

(We should move this all onto a Gujin thread in Puppy forum, but I am not sure how to do that -- you could start one, and just link to it from here?)


Done.
Back to top
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Fri 06 Jan 2006, 10:52    Post_subject:  

Etienne,
Thank you very much for being willing to help us Puppies with our boot loader problems.

I need your help understand these three error warnings that I get only with Gujin, not with Linld or Grub.

You wrote to me:
Quote:
The problem of checking which data is given in the initrd memory
is quite important because most USB device *do not* report read
or write errors from the medium, and so you can get this kind of
errors after formating a CompactFlash using an USB adapter,
and the standard mke2fs:

> user.crit kernel: EXT2-fs error (device ramdisk(1,0)): ext2_check_page: bad entry in directory #1742: unaligned directory entry
> user.crit kernel: EXT2-fs error (device ramdisk(1,0)): ext2_check_page: bad entry in directory #1747: unaligned directory entry
> user.crit kernel: EXT2-fs error (device ramdisk(1,0)): ext2_check_page: bad entry in directory #1751: unaligned directory entry

This is simply few bytes which cannot be written in one sector.
If you use an IDE->CompactFlash adapter you will see those error
even running a standard Linux kernel, as I did.


But I do not understand what you are saying! I am using CF-IDE, nothing to do with USB at all at this time (although the CF was probably in some USB adapter long ago). The Puppy files were not written in USB adapter, and nothing is being written to the files during the kernel/init process? When I boot, it always works. But only with Gujin do I get these messages. I need to understand what they mean, if they are a serious problem, and what I can do to make them go away!

I have the complete /var/log/messages files from booting with each alternative boot loader under exactly the same conditions. (The Gujin log seems surprisingly longer than the other logs - 16948 vs 15173 bytes.) The parts that seem most relevant here are included below.

(Why do you pass these extra parameters that I do not set?
>NumLock=on LANG=en
How would I supress or change them?)

Thanks again for your help!

Code:
Gujin 1.2

Jan  5 19:21:16 (none) syslog.info syslogd started: BusyBox v1.01 (2005.12.05-21:34+0000)
Jan  5 19:21:16 (none) user.notice kernel: klogd started: BusyBox v1.01 (2005.12.05-21:34+0000)
Jan  5 19:21:16 (none) user.warn kernel: Linux version 2.4.29 (root@vector.linux.vnet) (gcc version 3.3.4) #1 Sat Jul 2 21:53:18 WST 2005
Jan  5 19:21:16 (none) user.info kernel: BIOS-provided physical RAM map:
Jan  5 19:21:16 (none) user.warn kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)

Jan  5 19:21:16 (none) user.info kernel: ACPI: DSDT (v001    SiS      735 0x00000100 MSFT 0x0100000d) @ 0x00000000
Jan  5 19:21:16 (none) user.warn kernel: Kernel command line: NumLock=on LANG=en   root=/dev/ram0 PHOME=hda1 PFILE=pup001-none-7000
Jan  5 19:21:16 (none) user.info kernel: Initializing CPU#0
Jan  5 19:21:16 (none) user.warn kernel: Detected 995.786 MHz processor.

Jan  5 19:21:16 (none) user.info kernel: Freeing initrd memory: 10752k freed
Jan  5 19:21:16 (none) user.warn kernel: VFS: Mounted root (ext2 filesystem) readonly.
Jan  5 19:21:16 (none) user.info kernel: Freeing unused kernel memory: 132k freed
Jan  5 19:21:16 (none) user.info kernel:  hda: hda1
Jan  5 19:21:16 (none) user.info kernel:  hda: hda1
Jan  5 19:21:16 (none) user.crit kernel: EXT2-fs error (device ramdisk(1,0)): ext2_check_page: bad entry in directory #1742: unaligned directory entry - offset=0, inode=3753478387, rec_len=6901, name_len=64
Jan  5 19:21:16 (none) user.crit kernel: _check_page: bad entry in directory #1747: unaligned directory entry - offset=0, inode=3753478387, rec_len=6901, name_len=64
Jan  5 19:21:16 (none) user.crit kernel: EXT2-fs error (device ramdisk(1,0)): ext2_check_page: bad entry in directory #1751: unaligned directory entry - offset=0, inode=3753478387, rec_len=6901, name_len=64
Jan  5 19:21:16 (none) user.info kernel: SCSI subsystem driver Revision: 1.00
Jan  5 19:21:16 (none) user.info kernel: Intel 810 + AC97 Audio, version 1.01, 10:55:54 Jun 17 2005

Code:
Linld 0.97

Jan  5 19:33:45 (none) user.warn kernel: Kernel command line: root=/dev/ram0 PHOME=hda1 PFILE=pup001-none-7000

Jan  5 19:33:45 (none) user.info kernel: Freeing initrd memory: 6619k freed
Jan  5 19:33:45 (none) user.warn kernel: VFS: Mounted root (ext2 filesystem) readonly.
Jan  5 19:33:45 (none) user.info kernel: Freeing unused kernel memory: 132k freed
Jan  5 19:33:45 (none) user.info kernel:  hda: hda1
Jan  5 19:33:45 (none) user.info kernel:  hda: hda1
Jan  5 19:33:45 (none) user.info kernel: SCSI subsystem driver Revision: 1.00
Jan  5 19:33:45 (none) user.info kernel: sis900.c: v1.08.07 11/02/2003
Jan  5 19:33:45 (none) user.info kernel: eth0: Realtek RTL8201 PHY transceiver found at address 1.
Jan  5 19:33:45 (none) user.info kernel: eth0: Using transceiver found at address 1 as default
Jan  5 19:33:45 (none) user.info kernel: eth0: SiS 900 PCI Fast Ethernet at 0xd400, IRQ 11, 00:0a:e6:0f:b5:96.
Jan  5 19:33:45 (none) user.info kernel: Intel 810 + AC97 Audio, version 1.01, 10:55:54 Jun 17 2005

Code:
Grub 0.4.1

Jan  5 19:38:23 (none) user.warn kernel: Kernel command line: root=/dev/ram0 PHOME=hda1 PFILE=pup001-none-7000

Jan  5 19:38:23 (none) user.info kernel: Freeing initrd memory: 6619k freed
Jan  5 19:38:23 (none) user.warn kernel: VFS: Mounted root (ext2 filesystem) readonly.
Jan  5 19:38:23 (none) user.info kernel: Freeing unused kernel memory: 132k freed
Jan  5 19:38:23 (none) user.info kernel:  hda: hda1
Jan  5 19:38:23 (none) user.info kernel:  hda: hda1
Jan  5 19:38:23 (none) user.info kernel: SCSI subsystem driver Revision: 1.00
Jan  5 19:38:23 (none) user.info kernel: sis900.c: v1.08.07 11/02/2003
Jan  5 19:38:23 (none) user.info kernel: eth0: Realtek RTL8201 PHY transceiver found at address 1.
Jan  5 19:38:23 (none) user.info kernel: eth0: Using transceiver found at address 1 as default
Jan  5 19:38:23 (none) user.info kernel: eth0: SiS 900 PCI Fast Ethernet at 0xd400, IRQ 11, 00:0a:e6:0f:b5:96.
Jan  5 19:38:23 (none) user.info kernel: Intel 810 + AC97 Audio, version 1.01, 10:55:54 Jun 17 2005
Back to top
View user's profile Send_private_message 
etienne
Guest


PostPosted: Fri 06 Jan 2006, 12:04    Post_subject:  

kethd wrote:
Etienne,
Thank you very much for being willing to help us Puppies with our boot loader problems.

I need your help understand these three error warnings that I get only with Gujin, not with Linld or Grub.
....
ext2_check_page: bad entry in directory #1742: unaligned directory entry
....
But I do not understand what you are saying! I am using CF-IDE, nothing to do with USB at all at this time (although the CF was probably in some USB adapter long ago). The Puppy files were not written in USB adapter, and nothing is being written to the files during the kernel/init process? When I boot, it always works. But only with Gujin do I get these messages. I need to understand what they mean, if they are a serious problem, and what I can do to make them go away!


Having used USB adapter before is not a problem.

Put your CompactFlash in the IDE adapter and boot Linux (those adapter are not usually hot plug-able). Then run "badblocks" on this ide device (man 8 badblocks) - something like "badblocks /dev/hdc"
You will see read and write errors of the medium - like bad floppy sectors. It will take quite a long time.
If you are using a FAT filesystem, use mbadblocks or if it is E2FS rebuild the filesystem with the "-c" option of mke2fs. Then, your filesystem has taken account of the un-accessible sectors and you will no more have those warnings. Those warnings are important, because there is probably file corruptions depending on the number and size of files on the CompactFlash.
But I do not understand something: the error seems to be on the RAMDISK, are you copying an uncompressed filesystem image from the CompactFlash to the RAMDISK? Else I would say your memory compoment are not working - check them with memtest
http://www.memtest86.com/
or
http://www.memtest.org/

kethd wrote:
I have the complete /var/log/messages files from booting with each alternative boot loader under exactly the same conditions. (The Gujin log seems surprisingly longer than the other logs - 16948 vs 15173 bytes.) The parts that seem most relevant here are included below.


Sorry, do not know why it is longer - maybe because you added "debug"?

kethd wrote:
(Why do you pass these extra parameters that I do not set?
>NumLock=on LANG=en
How would I supress or change them?)


I forgot about them, and sorry no way to remove them. NumLock would be automatically set to OFF depending on the keyboard status (notebooks usually have it OFF, desktop PC usually ON). When Gujin displays messages in French, LANG=fr (compile option).

Etienne.
Back to top
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Sat 07 Jan 2006, 18:37    Post_subject:  

Etienne,

Thank you very much for your tips about those critical kernel filesystem errors. I will study badblocks and mbadblocks, which I did not know about.

However, as you perhaps have begun to suspect, scanning my CF is not really relevant here. Gujin 1.2 is uncompressing Puppy image.gz from CF to ram. Then the kernel starts. Then the Puppy init script starts. At the end of that, it tries to copy the ram filesystem contents to new_root. This is where the 3 filesystem errors occur, trying to read the /lib files from ram -- that were built by Gujin. /lib should be 7.4MB, but is truncated to 5.7MB due to these errors. I have seen these specific errors so repeatedly, in so many places/situations, that I don't think they have anything to do with the CF-IDE. I think they are some kind of intrinsic problem with Gujin 1.2 and Puppy.

If you have a chance, try making a DOS 6.22 bootable hard drive with Gujin 1.2 and the Puppy 1.0.7(final) LiveCD files, and see if you get the same error. (You have to look carefully to see the messages flash by, or go back through the messages log file.) I will try to recreate the problem from scratch with new media when I get a chance.
Back to top
View user's profile Send_private_message 
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Sat 07 Jan 2006, 22:50    Post_subject:  

I have re-created the problem, from scratch, on different hardware, with a real hard drive, 128MB 1GHz Duron.

Gujin 1.2 is not compatible with Puppy 1.0.7. Gujin 0.9 is OK, and Linld is OK. (But they cannot boot in less than 64MB with no swap space, due to a bug in allocating enough tmpfs space.)
Back to top
View user's profile Send_private_message 
etienne
Guest


PostPosted: Tue 10 Jan 2006, 06:51    Post_subject: Booting with Gujin/tiny.exe  

kethd wrote:
I have re-created the problem, from scratch, on different hardware, with a real hard drive, 128MB 1GHz Duron.

Gujin 1.2 is not compatible with Puppy 1.0.7. Gujin 0.9 is OK, and Linld is OK. (But they cannot boot in less than 64MB with no swap space, due to a bug in allocating enough tmpfs space.)


Arrgg... I can recreate the problem too, but I do not understand.
Usually, the initrd.img file (that you call image.gz) is a GZIP compressed
file of an image of a filesystem.

To recreate, I downloaded puppy-1.0.7-mozilla.iso , write it on CD,
mount the CD, copy vmlinuz to the root of a FAT32 partition I have
(renaming it to vmlinuz-puppy), copy image.gz to the same place
(renaming it to initrd-puppy) and reboot the PC with a Win98 bootable
floppy to run "dbgload.exe" that you can find in install-1.2.tar.gz of Gujin
at sourceforge.
When running that dbgload and selecting to boot vmlinuz-puppy and
initrd-puppy, I get a log file named DBG which contains:

--------------------------
kernel parameter address: set it to 0x80000.
[HIMEM: present, using XMS] menu_load_system: using HIMEM.SYS freemem: 523200 Kb, BIOS totalmem: 523264 Kb
initrd_file: get_min_nb_initrd() = 0.
initrd_file: '/vmlinuz-puppy' vm_ptr: -puppy compared to "-puppy" exact match, stop searching =52, => best match /initrd-puppy, len 52, found.
menu_load_system: loading kernel '/vmlinuz-puppy' at 0x0, max size 535756800, [setup_sects=10, minor_root=0x1, major_root=0x3, kernel_version: 2.4.29 (root@vector.linux.vnet) #1 Sat Jul 2 21:53:18 WST 2005]
Loading kernel: 2.4.29 (root@vector.linux.vnet) #1 Sat Jul 2 21:53:18 WST 2005, Linux param address stay 0x80000.
[treat_start_bytes = -5633] [treat_start_bytes = -1537] [gzip header in 15496 bytes] [gzip signature: 0x8B1F] _XMS_alloc_memory(1280 Kb) OK, using handle 13080
_XMS_realloc_memory(1536 Kb) OK
_XMS_realloc_memory(1792 Kb) OK
_XMS_realloc_memory(2048 Kb) OK
_XMS_realloc_memory(2304 Kb) OK
_XMS_realloc_memory(2560 Kb) OK
_XMS_realloc_memory(2816 Kb) OK
_XMS_realloc_memory(3072 Kb) OK
_XMS_realloc_memory(3328 Kb) OK
inflate: uncompress finished successfully.
[inflate returns Z_STREAM_END, gzlib->avail_in = 937] [unsigned short after compressed data: 0x0] result: 0x0 (size: 1048576 compressed, 2178416 uncompressed bytes).

Locked handle 13080 at 0x110000, error 0x0
Will accept possible uncompressed initrd of size 6778196
menu_load_system: loading initrd '/initrd-puppy' at 0x0, max size 533397504, _XMS_alloc_memory(256 Kb) OK, using handle 13090
_XMS_realloc_memory(512 Kb) OK
_XMS_realloc_memory(768 Kb) OK
_XMS_realloc_memory(1024 Kb) OK
.....
_XMS_realloc_memory(9984 Kb) OK
_XMS_realloc_memory(10240 Kb) OK
_XMS_realloc_memory(10496 Kb) OK
_XMS_realloc_memory(10752 Kb) OK

Finnished loading an uncompressed initrd of 6778196 bytes [inflate returns Z_STREAM_END, gzlib->avail_in = 204] [unsigned short after compressed data: 0xBDEC] result: 0x0 (size: 6782976 compressed, 11010048 uncompressed bytes).
File is bigger than a single GZIP, contains 6778196 - 5316404 = 1461792 bytes extra
Stop accepting uncompressed file of size 6778196 to load

Locked handle 13090 at 0x450000, error 0x0
--------------------------

That tells me that the file /initrd-puppy is a correct GZIP file, but
with more stuff at end, exactly 1461792 bytes. Gujin ignores those
bytes - I do not know what to do with them.

This can be caused by a problem when generating this initial RAM disk,
not typing "sync" in between generating the filesystem image and trying
to compress this filesystem image - or that can be a special use that
I completely do not handle.
What I can tell is that I cannot reproduce the same file by uncompressing
and recompressing this image.gz, even the filesize are different after
a gunzip followed by gzip...

If I remember well, tiny.exe in Gujin-0.9 was a special case where
I had a quick patch not decompressing the initrd file (it is not that
important to check its integrity because it is read by DOS and not
directly with hardware direct access).

Could someone please tell me the intended behaviour? What are
those extra bytes at end of the GZIP file image.gz? Did you really
have sync'ed the filesystem image before compression?

Thanks a lot,
Etienne.
Back to top
GuestToo
Puppy Master

Joined: 04 May 2005
Posts: 4078

PostPosted: Tue 10 Jan 2006, 07:39    Post_subject:  

image000.gz = the original 1.0.7 image.gz file
image.gz = the file gunzipped then gzipped again (gzip -9)

6778196 image000.gz
6778202 image.gz

there seems to be no significant difference in size

there is of course a slight difference in size, but that is because the date and time are always gzipped in the file, and it will gzip slightly differently depending on the exact time (for example, 11:11:11 will gzip to a slightly smaller size than 12:34:56) ... every time you gzip a file, it may be a slightly different size and will probably have a different md5sum because the time that is gzipped into the file will be different

i am not an expert in how the kernel handles caching in ram, but i would think that if it worked properly, it should not be necessary to sync unless a drive is unmounted ... i would think that if a file or part of a file is in the ram cache, then the kernel driver should retrieve it from the ram cache, if it is on a drive, it should retrieve it from a drive ... that is, whether gzip reads the file from a ram buffer or from a hard drive, it should read the exact same bytes of the exact same file ... if it does not do that, i would think that the kernel's cache system is buggy

http://www.linuxinfor.com/english/man8/sync.html
"The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance"

in any case, i seem to remember reading that ext file systems automatically sync every 5 seconds anyway, to help preserve the data in case of a crash
Back to top
View user's profile Send_private_message 
GuestToo
Puppy Master

Joined: 04 May 2005
Posts: 4078

PostPosted: Tue 10 Jan 2006, 08:00    Post_subject:  

on second thought, i think it's a tar that has the date and time in the file ... so if a tar file is gzipped, it is always slightly different because of the date and time ... but image.gz would not change because of the time, because it is only gzipped, not tar and gzipped

i think that's the way it works ... a tar file will always be slightly different because of the date and time ... a tar.gz file will always be slightly different because it is a tar file ... but a gzipped file like image.gz that is not a tar file, does not have the date and time in it, so it will always compress the same way each time
Back to top
View user's profile Send_private_message 
Etienne
Guest


PostPosted: Tue 10 Jan 2006, 09:34    Post_subject: Booting with Gujin/tiny.exe  

GuestToo wrote:
image000.gz = the original 1.0.7 image.gz file
image.gz = the file gunzipped then gzipped again (gzip -9)

6778196 image000.gz
6778202 image.gz


I also get a small difference in size:
# mkdir /mnt/cdrom
# mount -t iso9660 /dev/hdc /mnt/cdrom
mount: block device /dev/hdc is write-protected, mounting read-only
# ls /mnt/cdrom
boot.cat boot.msg image.gz isolinux.bin isolinux.cfg usr_cram.fs vmlinuz
# ls -l /mnt/cdrom/image.gz
-rw-r--r-- 1 root root 6778196 Dec 29 01:00 /mnt/cdrom/image.gz
# cp /mnt/cdrom/image.gz /win98/initrd.gz
# ls -l /win98/initrd.gz
-rwxr-xr-x 1 root root 6778196 Jan 10 08:40 /win98/initrd.gz
# gunzip /win98/initrd.gz
# gzip -9 /win98/initrd
# ls -l /win98/initrd.gz
-rwxr-xr-x 1 root root 6778203 Jan 10 08:40 /win98/initrd.gz
# mv /win98/initrd.gz /win98/initrd-puppy

Which does not seem right. Moreover, I am not sure to be able to simply mount this decompressed file in loopback mode.

GuestToo wrote:

i am not an expert in how the kernel handles caching in ram, but i would think that if it worked properly, it should not be necessary to sync unless a drive is unmounted ... i would think that if a file or part of a file is in the ram cache, then the kernel driver should retrieve it from the ram cache, if it is on a drive, it should retrieve it from a drive ... that is, whether gzip reads the file from a ram buffer or from a hard drive, it should read the exact same bytes of the exact same file ... if it does not do that, i would think that the kernel's cache system is buggy


IHMO, the thing with a loopback is that it is a user task writing the data and it is also a user task reading it for GZIP, but that is independant tasks (not the same file descriptor). A bit like when you do on a xterm "make vmlinux > log" and on another xterm "vi log": you get a snapshot.
If you have a file open, you do not ask for the data you have written to be imediately updated on disk - they can and are kept in local buffer until the file is closed. The file we are talking here is in fact the complete image of a filesystem, but it is still a file from the kernel point of view, from the writing task. It has to be sync-ed before being read by another application.
Note that I am not sure "sync" is enough - because the filesystem may need to be completely unmounted (so there is a clean unmount written and no ext2fsck at next reboot).

GuestToo wrote:
in any case, i seem to remember reading that ext file systems automatically sync every 5 seconds anyway, to help preserve the data in case of a crash

I bet you do not have a wait 5 second in you scripts, and I have a correct beginning of a filesystem (It is a complete GZIP and I checked its CRC32 on the totality of the data), just something has been appended after the CRC32...

GuestToo wrote:
i think that's the way it works ... a tar file will always be slightly different because of the date and time ... a tar.gz file will always be slightly different because it is a tar file ... but a gzipped file like image.gz that is not a tar file, does not have the date and time in it, so it will always compress the same way each time


It would seem reasonable.

Thanks for the quick answer,
Etienne.
Back to top
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Tue 10 Jan 2006, 10:46    Post_subject:  

Because I am testing on slower computers, I *know* that Gujin 0.9 (and linld+kernel) *are* uncompressing image.gz, with no apparent problems. (I time the uncompressions with a stopwatch.)

And I am quite sure that Gujin 1.2 has the same problems with other image.gz files that I have made, by hand, no scripts involved, no sync issue in my opinion....

(Some differences in sizes of gzipped files depend on which level of compression you choose.)

I don't understand most of this technical discussion, and will try to study and learn. I am especially interested in learning about good boot debugging approaches -- what I went through was very painful!

But in this case, the facts seem quite simple. You can take the image.gz file from a Puppy LiveCD (or extract same from within the iso file) and unzip that file on any Linux. I predict it will unzip fine -- I've never had a problem with it. Then you can mount that filesystem and work with it. So it seems obvious (even though nothing is ever certain) that the problem is that Gujin 1.2 is not properly processing the gz file, even though it is actually OK.

I think, you would need to study the tech details of exactly why G1.2 thinks there is something wrong with this gz file, that all other tools think is OK. There might indeed be something peculiar or odd about it, but somehow the other tools seem to be able to handle it! (This all leads into the other important subject, of why Gujin is so much slower at uncompressing gz files, compared to other tools, but that is of course less important than for the uncompression to work.)

One approach would be to study how your uncompression algorithm changed from .9 to 1.2 -- I have noticed that 1.2 is slightly faster.
Back to top
View user's profile Send_private_message 
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Tue 10 Jan 2006, 14:35    Post_subject:  

Another related thought:
The whole purpose of your non-standard philosophy with Gujin, that you uncompress things yourself instead of passing the compressed initrd to the kernel like other boot loaders do, is that you want to be extra-safe-and-sure -- you want to check that everything is OK.

But here is a case where the opposite seems to be happening. The standard tiny.exe for G12 uncompresses an image that it thinks is malformed, but does not say anything to alert the user to the potential problem. (The only error msgs I am aware of are issued later, during the init sequence, when Linux tries to actually use the files that were made. For example, walking the tree with du -a.)
Back to top
View user's profile Send_private_message 
etienne
Guest


PostPosted: Tue 10 Jan 2006, 20:30    Post_subject:  

kethd wrote:
Another related thought:
The whole purpose of your non-standard philosophy with Gujin, that you uncompress things yourself instead of passing the compressed initrd to the kernel like other boot loaders do, is that you want to be extra-safe-and-sure -- you want to check that everything is OK.

But here is a case where the opposite seems to be happening. The standard tiny.exe for G12 uncompresses an image that it thinks is malformed, but does not say anything to alert the user to the potential problem. (The only error msgs I am aware of are issued later, during the init sequence, when Linux tries to actually use the files that were made. For example, walking the tree with du -a.)


I am currently debugging, so will find something. What I can say is that I am sure the CRC32 is checked and valid, and I checked that my function to decompress, compiled for Linux in protected mode, gives the same result byte to byte than Gunzip.
The problem I seem to have is after that, sending the uncompressed data to extended memory: I just calculate the CRC32 on the "real mode" window memory and then send the complete 32Kbytes window to upper memory - I do not try to recalculate the CRC32 at the end of decompression (in case the XMS memcopy do not completely work - maybe I should).

Note that it does not seem to be related to decompression, because if you just call tiny.exe with a pre-decompress initrd, I just send the data in upper memory without processing and get the exact same error on the filesystem. I also get the same error using DOS with or without HIMEM and using the BIOS memcopy - probably only the tiny_img will work differently.

Sorry for bothering you,
Etienne.
Back to top
Etienne
Guest


PostPosted: Tue 17 Jan 2006, 10:30    Post_subject: Booting with Gujin/tiny.exe  

etienne wrote:
I am currently debugging, so will find something


I found the bug that is bothering you, I successfully booted Puppy yesterday.
While decompressing the initrd, a part of the data is stored because it cannot be compressed (as GZIP specs) and if this part is big enough that triggers a "dirty patch" I have done to support non-compressed initrd.
Corrected by a patch like this:
Code:

diff -b -ur gujin-1.2/gzlib.c gujin/gzlib.c
--- gujin-1.2/gzlib.c   2005-07-12 20:41:10.000000000 +0100
+++ gujin/gzlib.c       2006-01-17 08:41:32.000000000 +0000
@@ -916,6 +1025,7 @@
                  if (LOADER.accept_uncompressed_initrd_filesize) {
                      ZDBG (("\r\nWill load an uncompressed initrd of %u bytes ", LOADER.accept_uncompressed_initrd_filesize));
                      z->sub.stored.left = LOADER.accept_uncompressed_initrd_filesize;
+                     z->sub.stored.load_uncompressed = 1;
                      z->mode = STORED_COPY;
                      z->avail_in += 2;
                      z->next_in -= 2;
@@ -1264,10 +1374,11 @@
              continue;
          case STORED_LENS:
              z->sub.stored.left = utmp;
+             z->sub.stored.load_uncompressed = 0;
              z->mode++;        /* = STORED_NOTLENS */
              continue;
          case STORED_NOTLENS:
              if (z->sub.stored.left != (utmp ^ 0xFFFF)) {
                  ZDBG (("invalid stored block lengths"));
                  z->mode = BAD;
                  continue;
@@ -1309,7 +1420,7 @@

 #ifdef INITRDNAME
              /* Dirty patch to load an uncompressed initrd */
-             if (z->sub.stored.left == 0 && LOADER.accept_uncompressed_initrd_filesize) {
+             if (z->sub.stored.left == 0 && z->sub.stored.load_uncompressed) {
                  ZDBG (("\r\nFinnished loading an uncompressed initrd of %u bytes ", LOADER.accept_uncompressed_initrd_filesize));
                  z->mode = ALLDONE;
 #ifdef GZLIB_USE_INTERMEDIATE_WINDOWdiff -b -ur gujin-1.2/gzlib.h gujin/gzlib.h
--- gujin-1.2/gzlib.h   2005-07-12 20:41:10.000000000 +0100
+++ gujin/gzlib.h       2006-01-17 08:41:32.000000000 +0000
@@ -156,6 +156,7 @@
        struct {
            /* patch 16->32 bits to load an uncompressed initrd */
            unsigned /*short*/  left;
+           unsigned            load_uncompressed;
            } stored;
        struct {
            unsigned short      len;


But I will release Gujin v1.3 really soon now so you will not have to recompile. Gujin 1.3 adds the display of the command line for few seconds if you press and keep pressed the Control key alone while loading the files - and re-check the CRC32 of the kernel/initrd to check for bad RAM.
You are also saying Gujin is very slow, could you tell me where you find it slow, is it before the "weel turning", during or after?

Thanks,
Etienne.
Back to top
kethd

Joined: 20 Oct 2005
Posts: 451
Location: Boston MA USA

PostPosted: Tue 17 Jan 2006, 17:16    Post_subject:  

Etienne,

Thank you for working on the Gujin 1.2 bug, and I look forward to trying 1.3.

I don't understand your explanation, however. Are you saying that there is actually something wrong with the Puppy image.gz?

Please consider always displaying the command line that you will be passing, immediately, by default. The reason this is needed is so that people who are having problems with the line being truncated will have some chance to realize what is wrong. There is no need to provide a special way for people to check for this problem. People who want to check for the problem already have ways to check. But most people who have this problem have no idea that it might be the cause of their boot problems, and so will not check for it.

The problem with Gujin unzipping time is while the wheel is spinning, to uncompress image.gz. Gujin takes about five times as long to do this on a Pentium 233MMX, compared to any other unzipping tool. And the 20 second additional delay during each boot does seem significant when you are booting over and over to do debugging.

And I would like to urge you again to allow an include file, using the same format as linld. It is very important to be able to allow the passing of max info on the command line -- at least the 8 variables / 256 characters that the Puppy kernel allows. The include file allows for a more sensible format for passing large amounts of command line info. It is very important for a Linux like Puppy to be able to pass giant hunks of parameters because the boot code itself is extremely hard to change for most Puppy users. Being able to pass lots of parameters from the command line means that very fancy Puppy boot scripts can be invented by a very few people, and then all the other Puppy users can learn how to pass special parameters to do all kinds of special things -- like find out why their Puppy is not booting, and be able to tell it how to boot the way that they want!
Back to top
View user's profile Send_private_message 
Etienne
Guest


PostPosted: Wed 18 Jan 2006, 07:38    Post_subject: Boot loaders, Gujin.  

kethd wrote:
I don't understand your explanation, however. Are you saying that there is actually something wrong with the Puppy image.gz?


There isn't anything wrong with Puppy image.gz, just an obvious bug in Gujin that I did never detect - and nobody else reported before you.
I would just say that this file is misnamed, name is usually initrd* and then it is autodetected by other forms of Gujin than tiny.exe as the initial RAM disk to load with the kernel.

kethd wrote:
Please consider always displaying the command line that you will be passing, immediately, by default. The reason this is needed is so that people who are having problems with the line being truncated will have some chance to realize what is wrong. There is no need to provide a special way for people to check for this problem. People who want to check for the problem already have ways to check. But most people who have this problem have no idea that it might be the cause of their boot problems, and so will not check for it.


Thinking twice, for tiny.exe only, displaying the command line before loading the files will give something to read to users during the loading process - I'll do it.

kethd wrote:
The problem with Gujin unzipping time is while the wheel is spinning, to uncompress image.gz. Gujin takes about five times as long to do this on a Pentium 233MMX, compared to any other unzipping tool. And the 20 second additional delay during each boot does seem significant when you are booting over and over to do debugging.


So it seems to be the CRC32 calculation - I can replace that with a simple recompilation - but I lack some code space for the other forms of Gujin bootloader than tiny.exe. This needs to be done because now I re-check the complete CRC32 later on. I need one or two days more.
By the way, I tried yesterday to boot on a 486/32 Mb (takes a long time but I knew that) and it seems that some utilities are compiled for 586 at least...

kethd wrote:
And I would like to urge you again to allow an include file, using the same format as linld. It is very important to be able to allow the passing of max info on the command line -- at least the 8 variables / 256 characters that the Puppy kernel allows. The include file allows for a more sensible format for passing large amounts of command line info. It is very important for a Linux like Puppy to be able to pass giant hunks of parameters because the boot code itself is extremely hard to change for most Puppy users. Being able to pass lots of parameters from the command line means that very fancy Puppy boot scripts can be invented by a very few people, and then all the other Puppy users can learn how to pass special parameters to do all kinds of special things -- like find out why their Puppy is not booting, and be able to tell it how to boot the way that they want!


That is what I done yesterday evening.
Basically you can now type:
- simple:
tiny vmlinuz initrd root=/dev/ram0 ro param1 param2
- second param contains '=' so no initrd there:
tiny vmlinuz root=/dev/ram0 ro param1 param2
- insert one file:
tiny vmlinuz initrd root=/dev/ram0 ro param1 @param.txt param2
- insert multiple files (but param3=42@53 is not expanded):
tiny vmlinuz initrd root=/dev/ram0 ro param1 @param_a.txt param2 @param_b.txt param3=42@53
- file @param_b.txt can be empty (i.e. contains only space, tabs, linefeeds and return) and the correct empty field will be inserted.
- filename in @param has to be smaller than 80 chars, content of those file is truncated to 255 chars.
- file in @file has to exists, else an error is printed and loading is stopped.
- linefeeds / newlines in @file are converted to space, and the ending linefeeds / newlines is trimed to fit a command line.

Etienne.
Back to top
Display_posts:   Sort by:   
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1970s ][ Queries: 11 (0.0205s) ][ GZIP on ]