reviving pupray project

[nosystemdthanks]

the "changes between versions" part doesnt just refer to official versions.

what is my attraction to puppy after all this time? puppy is probably the most remixed gnu/linux distro there is (except debian perhaps, either way puppy takes a more casual approach to the whole thing.)

pupray (it wasnt called pupray) is the same project that led to mkfigos and any other tools i use for remixing puppy (or other distros.)

but originally, the goal was just to produce an elaborate html table of puppy features per distro.

and i do mean elaborate. with a field for just about anything you could want.

the recent event that made me think of rebooting it, is that one particular user wanted to know why pidof X didnt work. pidof Xorg did, but it took me about 5-10 minutes to figure that out.

pupray can answer all sorts of questions like this and more-- and it can do it across distros.

one of the things that slowed down pupray before was the storage requirements.

ive figured out a way to make pupray far more modest in its requirements.

if youre still curious how it works, run pupsysinfo.

now imagine pupsysinfo as an html table instead of a gui.

now imagine instead of information about the computer, it focuses on information about the puppy cd:

1. kernel image
2. window manager
3. included software (highlights)
4. information about libraries
5. other included features (ones i think of, or ones you request)

ally has an extensive collection of old and new puppies:

https://ia801708.us.archive.org/isoview ... -puppy.iso

BOOT.CAT 2003-07-09 02:36:49 2048
IMAGE.GZ 2003-07-09 02:24:40 19526010
ISOLINUX.BIN 2003-07-08 16:53:04 9396
ISOLINUX.CFG 2003-06-21 06:45:24 104
VMLINUZ 2003-06-10 09:15:40 1130790

thats from 2003. note the lack of initrd and sfs. but we can still automate looking through that.

dont need anybodys help with this, though id be very interested in feature requests.

thats not a promise to implement, though its a likelihood.

as for how to deal with size restrictions, thats going to be fun too. basically it will get file information on everything it can, then discard most of the actual files-- but not all of them.

then zip them up into an archive that can be copied/uploaded (and downloaded again) with more modest requirements. so if you want to keep icons and wallpaper or some config files from (MANY) old versions? it should make it easier to do that.

by the way, some newer versions of puppy (xenial for example) cannot get most of the files from that 2003 version of cd-puppy.

its a minix filesystem, xenial doesnt have a kernel module for that. grub does, refracta (from last year) does, void linux does not, older versions of puppy do.

i dont think barry stuck with minix filesystems for a very long time, but for those it is important that the program runs on a system that supports mounting it.

this is actually one of the things it can check for.

how to get the kernel version:

# file vmlinuz

how to check if the kernel supports minix (even if it is not found under /lib/modules)

https://stackoverflow.com/questions/120 ... or-bzimage

# dd if=vmlinuz skip=`grep -a -b -o -m 1 -P '\037\213\010\000' vmlinuz | cut -d: -f 1` bs=1 | zcat | grep 'minix_fs' | wc -l

newer puppies use sfs, which is more reliable across distros.

[oui]

your introduction is for non English speaking people as I very difficult and I did have to invite Google translations to help me as you mix words like barry and ally without respect on the right writing rules with low characters but Google, our friend, did help and all is good now...

This subdivision of our forum seems not to be the best place for it (I would prefer «Cutting Edge»)

this matter is very interesting and remember from times being very far in the past. I hold an old i486-computer being able to read an write my old CD-RW's from Puppy on CD-RW (I suppose Puppy 1.03 .. 1.04) and have yet old CD's really used with Puppy 0.4 and later.

but my reflexion on your proposal: 2003! that is very late! is a bit as use IBM-DOS. why?

[nosystemdthanks]

your introduction is for non English speaking people as I very difficult and I did have to invite Google translations to help me :idea: as you mix words like barry and ally

thats how it works in hindi too. barry is barry, ally is ally, and i have never asked someone speaking french or german to write their own language differently. but im glad that google translate is helping.

This subdivision of our forum seems not to be the best place for it (I would prefer «Cutting Edge»)

your preference is noted. this is a puppy project, its related to new and old distros, what is "cutting edge" about this project?

this matter is very interesting and remember from times being very far in the past. I hold an old i486-computer being able to read an write my old CD-RW's from Puppy on CD-RW (I suppose Puppy 1.03 .. 1.04) and have yet old CD's really used with Puppy 0.4 and later.

i used puppy 2, barry had already switched to sfs by then.

but my reflexion on your proposal: 2003! that is very late! is a bit as use IBM-DOS. why?

its the oldest version of puppy i can find. its also the first year that puppy existed.

this is designed to work on new puppy versions too, but i wanted to start with the earliest puppy available. you can find more here: https://archive.org/details/%40allytonx ... publicdate

if you have any questions i will be happy to answer them.

[nosystemdthanks]

so here is the initial toolkit and a better explanation of what pupray is.

the goal of pupray is to compare and document many versions of puppy. compare and document what? lots of things.

first lets talk about our input files-- about 1 or 2 terabytes of cd and dvd files. now lets talk about limitations-- i dont have terabyte storage for this.

now there are two sorts of contents we want to process when we get a cd or dvd image:

1. we want the file data. not just the basic file listing, but the sfs files too. oh, and really old puppies dont even use sfs. some of those can also be processed. but that depends on minix filesystem support. (available in some distros.)

2. we want actual files. but we dont want the isos to take up so much room, and many of the files we want to examine are smaller.

so what do we do?

.

stage 1: we open the file and process its filesystems, within reason. thats what pupray.fig does.

pupray creates a log of filepaths, sha256sums, and date and size info. it creates a string to zip everything into a new tar.gz file.

once we have the file info, we can delete the largest files. but we want to preserve wallpaper, icons and config files. the log is included with the tar.gz. then we can delete the dvd and the tar.gz keeps valuable data for the next stages.

.

stage 2: using a subset of puppy cds and dvds we figure out what to remove so we can redo this subset and apply the design tweaking to the full range of input. we use the existing logs for this, with a tool called pupraytally.

pupraytally is small:

Code: Select all

#### license: creative commons cc0 1.0 (public domain) 
#### http://creativecommons.org/publicdomain/zero/1.0/ 

proginf = "pupraytally 0.1, nov 2018 mn"

function getsize p
    f split p " " mid 1 1
    try
        now f int
    except
        f 0
        resume
    now return f
    fig

function gettype p
    fn split p "/" right 1 mid 1 1
    cdot instr fn "."
    iftrue cdot
        now split fn "." right 1 mid 1 1 reverse plus "." reverse return now
    else
        now return fn
        fig
    fig

tal arr
python
    tal = {}
    fig

function taldo p t
python
    global tal
    try: tal[p] = int(tal[p]) + t
    except: tal[p] = t
    fig
    fig

now taldo "a" 1

files arrstdin
forin p files
    tab 9 chr
    ctab instr p tab 
    csf  instr p "/mnt/pupray/sfs/"
    cf   instr p "/mnt/pupray/fs/"  plus csf
    iftrue ctab
        iftrue cf
            size getsize p 
            filetype gettype p   
            now taldo filetype size
            fig
        fig
    next

now tal len print

talfig arr mid 0 1
python 
    for p in tal.keys(): 
         if int(tal[p]) > 32767: talfig += [(int(tal[p]), p)]
    talfig.sort()
    for p in talfig: print p[0], p[1]
    print
    fig

here is a demo with 5 puppy versions:

Code: Select all

for p in xenialpup-7.5-uefi.iso "cd-puppy(092).iso" beaver64-8.7.iso JXS_5.7_version3.iso pizzapup-3.0.1.iso ; do cat "$p.log" | ./pupraytally.fig.py | awk -v p=$p '{print $_ " " p}' ; done | sort -n # public domain

what this does is tell us what types of files use the largest amount of dvd. heres the tail of the output:

Code: Select all

12414008 .100 xenialpup-7.5-uefi.iso
13347399 .ja xenialpup-7.5-uefi.iso
13529680 .05 JXS_5.7_version3.iso
13685784 .100 beaver64-8.7.iso
14687176 .0 beaver64-8.7.iso
15083256 .18 xenialpup-7.5-uefi.iso
15185543 Packages-ubuntu-bionic-universe beaver64-8.7.iso
16142368 .22 beaver64-8.7.iso
19007895 .ja beaver64-8.7.iso
26904264 .2 beaver64-8.7.iso
31379740 .so JXS_5.7_version3.iso
45387776 usr-cramfs cd-puppy(092).iso
53381204 .1 xenialpup-7.5-uefi.iso
59299316 .so xenialpup-7.5-uefi.iso
61080848 .1 beaver64-8.7.iso
79207904 .so beaver64-8.7.iso

.so files and .1 files use the most dvd of any files (not including vmlinuz and the iso or sfs files themselves.)

specifically, of the 5 dvds we processed, .so files in beaver64-8.7.iso (the log says sha256 eabbec4d991e7766a2b95b6ab3aa628dc302b043b9eeeca682ba2bd9f48f9464) use the most dvd.

.

stage 3: we have lots (maybe 1000) tar.gz files which contain wallpaper, icons, config files, and other files. these are much easier to process and much easier to manage. they dont include binary libraries, they dont include kernel modules. they hopefully include all the text and config files we could possibly want from the history of puppy linux.

and we can use those to create a more comprehensive version of pupray, which includes all sorts of data about whats in each puppy version-- and perhaps output that to a giant html table, automatically-generated website, or even a gui.

included are version 0.1 of pupray and pupraytally.