Yet another basic remaster script for frugal install

How to do things, solutions, recipes, tutorials
Message
Author
User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#16 Post by nic007 »

ndujoe1 wrote:Question: so if I remove all the default puppy applications I don't wish to use, the will remaster everything I do use, personal files and sfs's?
It will include all loaded sfs's, pets you have installed and personal files that are stored in the savefile/folder (not personal files that are stored elsewhere on your drive). As far as removal of default built-in applications goes: - this will depend how you remove it. If you use some application that only white-out (mark them that they can not be seen by the operating system) files, I don't think this basic script will actually delete them from the base sfs (but it should be masked so that there won't be a menu entry for them, etc.). If you physically remove them by editing the base sfs, it will definitely be gone.
Last edited by nic007 on Fri 02 Jun 2017, 07:01, edited 1 time in total.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

#17 Post by mikeslr »

Repeating my previous caution "coding is not my strong suit", as I understand nic007's command line:

cp -a /initrd/pup_ro2/* /initrd/pup_ro3/* /initrd/pup_ro4/* /initrd/pup_ro5/* /initrd/pup_ro6/* /initrd/pup_ro7/* /initrd/pup_ro8/* /initrd/pup_ro9/* /initrd/pup_rw/* /mnt/home/puppyfilesystem

whatever you have in /root will be copied. /root is where all application settings are stored.

The builtin remaster application will not include in your custom remastered version the applications you removed using Menu>Setup>Remove builtin packages. The way this works --when the builtin remaster works-- is that remove builtin doesn't actually remove the package, only "white's out" the link to it. So the builtin remaster, doesn't have a link to follow and, consequently, doesn't include the application. nic007's remaster simply copies what's there including both the applications and the "white outs".

Frankly, removing applications IMHO is a waste of time (unless, of course, you're involved in some grand project, like taking a Puppy which uses JWM as its Window Manager and substituting XFCE as Window Manager). Storage space is cheap. Recently I purchased a package of 3 8Gbs USB-Keys for $12. Hard-drives can be found for as little as $.04 per Gb*. Most Devs creating Puppies don't include large applications --such as LibreOffice-- in their creation; and a Frugally installed Puppy doesn't load an entire builtin** application into RAM, but rather creates links to it which are followed as needed. The only downside of builtin applications is that they appear on the Menu delaying your selection of those applications you actually want to use. Newer Puppies have builtin Menu>Setup>Menu Manager --edit the Menu, which can remove an application's appearance from the Menu. In older Puppies, you can edit their desktop files, /usr/share/applications/xxx.desktop, so that the category argument is one which will not produce a display, e.g Categories=Calculator to Categories=ZCalculator. Both techniques would, I believe, be carried over into your remaster using nic007's remaster script.

mikesLr

* If it takes me an hour to remove 25 Mbs of applications, conserving the space worth 4 cents per Gb, the value of my time is .1 cent per hour. Would you work for that?

** When you install a new version of an application or load an SFS, say Palemoon, into a Puppy where the Dev has builtin a version of Palemoon, both versions continue to exist. But the links Puppy uses will now point to the new version rather than the old.

nic007 can explain it better. I've used the term "links" which I, and I think most of us, understand. But nic007 can probably explain how inodes function.

User avatar
smokey01
Posts: 2813
Joined: Sat 30 Dec 2006, 23:15
Location: South Australia :-(
Contact:

#18 Post by smokey01 »

I always have the devx loaded as I do a lot of compiling. This method would allow me to include the devx in the main base so I could compile in a virtual environment like Vbox and Qemu. Nice.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#19 Post by nic007 »

mikeslr wrote:Repeating my previous caution "coding is not my strong suit", as I understand nic007's command line:

cp -a /initrd/pup_ro2/* /initrd/pup_ro3/* /initrd/pup_ro4/* /initrd/pup_ro5/* /initrd/pup_ro6/* /initrd/pup_ro7/* /initrd/pup_ro8/* /initrd/pup_ro9/* /initrd/pup_rw/* /mnt/home/puppyfilesystem

whatever you have in /root will be copied. /root is where all application settings are stored.

The builtin remaster application will not include in your custom remastered version the applications you removed using Menu>Setup>Remove builtin packages. The way this works --when the builtin remaster works-- is that remove builtin doesn't actually remove the package, only "white's out" the link to it. So the builtin remaster, doesn't have a link to follow and, consequently, doesn't include the application. nic007's remaster simply copies what's there including both the applications and the "white outs".

Frankly, removing applications IMHO is a waste of time (unless, of course, you're involved in some grand project, like taking a Puppy which uses JWM as its Window Manager and substituting XFCE as Window Manager). Storage space is cheap. Recently I purchased a package of 3 8Gbs USB-Keys for $12. Hard-drives can be found for as little as $.04 per Gb*. Most Devs creating Puppies don't include large applications --such as LibreOffice-- in their creation; and a Frugally installed Puppy doesn't load an entire builtin** application into RAM, but rather creates links to it which are followed as needed. The only downside of builtin applications is that they appear on the Menu delaying your selection of those applications you actually want to use. Newer Puppies have builtin Menu>Setup>Menu Manager --edit the Menu, which can remove an application's appearance from the Menu. In older Puppies, you can edit their desktop files, /usr/share/applications/xxx.desktop, so that the category argument is one which will not produce a display, e.g Categories=Calculator to Categories=ZCalculator. Both techniques would, I believe, be carried over into your remaster using nic007's remaster script.

mikesLr

* If it takes me an hour to remove 25 Mbs of applications, conserving the space worth 4 cents per Gb, the value of my time is .1 cent per hour. Would you work for that?

** When you install a new version of an application or load an SFS, say Palemoon, into a Puppy where the Dev has builtin a version of Palemoon, both versions continue to exist. But the links Puppy uses will now point to the new version rather than the old.

nic007 can explain it better. I've used the term "links" which I, and I think most of us, understand. But nic007 can probably explain how inodes function.
Unfortunately this does NOT seem to be the case. I "removed" Seamonkey using "Remove builtin Packages" and then ran the builtin remaster script. Only the menu entries were deleted, Seamonkey was still there in the remastered sfs. My script also only removes the menu entries. I remember pup412 had an application called pet-be-gone which worked but Remove builtin packages doesn't seem to.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

#20 Post by mikeslr »

Another reason not to actually remove "builtins" unless you really know what you are doing. With the "remove builtin" application so limited, it would seem that in order to actually remove an application and all of its files, you'd have to open its listing in /root/.packages in a text viewer, then manually delete the folders, and in some cases, the specific files shown on that list. But to safely do so, you have to be certain that every file on that list isn't also used by some application you still want to be functional.*

Perhaps the coding requirement of a program that would keep track of all files within each system and which applications make use of each file explains why "pet-be-gone" is no longer supported, and "remove builtins" has limited value.

mikesLr

* and if you possess that knowledge, its probably faster to woof your own Puppy version than to strip someone else's version of what you don't want.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#21 Post by nic007 »

Improved the script.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#22 Post by nic007 »

Revised version uploaded to first post.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#23 Post by nic007 »

Revision 2 uploaded, probably the final release.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#24 Post by musher0 »

Hi, nic007.

I tried your remaster script for my DPupStretch-7.0.0, and by the time the
script gets to squashing, file /lib/libgcc_s.so.1 (normally 114 Kb) had been
altered, so rxvt couldn't kick in to support mksquashfs.

(/lib/libgcc_s.so.1 is needed by rxvt and a number of other apps. Among
other difficulties its alteration creates,

Code: Select all

xwin [window manager]
won't work either.)

I copied /lib/libgcc_s.so.1 back into place and repeated the test, and this
bug still happened.

Probably this is a quirk only in the DPupStretch, but I thought I'd mention it.

IHTH. BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#25 Post by nic007 »

Hi, musher0. I assume you are saying that the script does not finish because rxvt is not envoked. I've tested the script on many official released puppys from 412 upwards and haven't had any of those sort of problems. An important note - this script is designed to run first thing after rebooting with savefile/folder (before running applications). This will make for a "clean" remaster because there are no temp stuff, altering of files,etc. to deal with at that stage (and this script does not deal with temp files, etc. I've specifically made it that way). So in short, if you are going to run the script later on in the session after working with applications, etc. the remaster will in all probability be a mess, ie. the running of the script itself and any resulting remaster (for one, I would be surprised if you don't have a problem with xwin).

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#26 Post by nic007 »

Rev 3 uploaded. Had a bit of rethink how the seperate use of the adrv, ydrv, zdrv, etc. could influence things (configurations, etc.) so decided to include everything in the base SFS in the correct sequence (hopefully).

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#27 Post by musher0 »

Hi nic007.

Please find attached some edits for the script version 2 you offered us
yesterday. (This post is NOT about the version 3 you just uploaded,
although it includes its "reverse" approach for copying the pup_ro?'s.)

My modifications are not meant as criticism, since you have provided
us with a solid base. They are only an "I" statement (of sorts), since they
involve only me and my work on the DPupStretch-7fr.

You may wish to read the remarks I incorporated.

With this edit, the serious problem I encountered and reported on
yesterday has gone away, as well as a few minor bugs I had. Also, further
testing is needed, but I believe this 2.1 version could be used at any time
during one's session.

Feedback and discussion are of course welcome!

IHTH. In co-operation,

musher0

~~~~~~~~~~~~~~

Code: Select all

#!/bin/sh
# A Script by nic007 to remaster the base sfs.  Published 2017.
# When happy with setup reboot with savefile/folder and do remaster immediately when at desktop
# (load any extra sfs's if necessary first).
# This script does not combine the adrv, ydrv and zdrv (if present) with the base SFS.
# Those files are loaded automatically at bootup, separately from the base SFS.
####
# Version 2.1, avec/with modifications de/by  musher0, 16-17 juin 2017.
#### set -xe
# The command above is useful when de-bugging.
# Below, I created strings for repeating commands, and the much-used target directory. musher0
FilSys="/mnt/home/puppyfilesystem"
RxvT="rxvt +tr -bg lightblue -fg black -title 'Progress' -g 80x6" # touch-up
MksQ="mksquashfs $FilSys /mnt/home/$PUPPYSFS -noappend -all-root" # Here, I added the -all-root
# parameter in case some new file we installed would not be "root". musher0

cd /mnt/home
if [ -d $FilSys ];then # Added this condition because I was getting an error when no directory to remove. musher0
	rm -r $FilSys;wait
fi
mkdir $FilSys
Xdialog -title "WELCOME!!!" -left -wrap -yesno "Thank you for using nic007's Quick Remaster Script.  This script will customise your Puppy's base sfs.

I need to know which storage media contains your savefile/folder.  Select 'Yes' or 'OK' if on harddrive otherwise 'No' or 'Cancel' if it's on something else like a usb pendrive/card.

Click 'Yes' or 'OK' if on harddrive.
Click 'No' or 'Cancel' if on usb pendrive/card.

After making your selection, I will start copying files.  This will take some time, please be patient." 0 80
# Modif., musher0
mount | awk '/_ro/ { print $3 }' > /mnt/home/_ro.lst
# We only need to copy the mounted systems, and the above line provides us with the list.
for i in `tac /mnt/home/_ro.lst`;do cp -a $i/* $FilSys;done # tac instead of cat, following your usage of
# copying in reverse that you introduced in your version 3. (tac reads from last line to first line.)
# If all extra sfs's are unloaded, we'll have only pup_ro2 in this list, but I think having it is useful.
if [ -d /initrd/pup_rw ];then
	cp -f -R -n --preserve=links /initrd/pup_rw/* $FilSys # I use some symlinks in /root for directories
# such as Download, my-documents and .packages, and cp was giving me the error: "Cannot copy a
# symlink over a non-sysmlink" or similar wording. So I changed the -a param. to -f -R -n --preserve=links.
# The -n param. prevents cp from clobbering a file or directory that would already be in the target dir.
else
	cp -f -R -n --preserve=links /initrd/pup_ro1/* $FilSys # Used here as well, in case of similar situation.
fi
#
cd $FilSys
rm -r -f ./initrd ./sys # musher0: File removal separated from dir. removal
rm -f ./etc/.XLOADED ./root/.XLOADED # End of modifs by musher0.

Xdialog -title "REMASTER OPTIONS" -left -wrap -yesno "/mnt/home/puppyfilesystem is now ready and fully customised for the same computer.  If, however, the remastered system is going to be used on different computers, you may want to keep some pristine settings.
If you select 'Yes' or 'OK' here, you can run Setup from Puppy's menu to change any settings after booting the remastered system on a different computer.

No user intervention is necessary irrespective of the option selected.  If, however, you wish to do some editing of /mnt/home/puppyfilesystem yourself (eg. adding or removing something), do so now before selecting the option.

Click 'Yes' or 'OK' to keep some pristine settings for different computers.
Click 'No' or 'Cancel' to keep the fully customised settings for the same computer." 0 80
if [ $? -eq 0 ];then
	rm -r $FilSys/etc
	cp -a /initrd/pup_ro2/etc $FilSys
fi
Xdialog -title "NEW PUPPY OR OLD PUP" -left -wrap -yesno "This remaster script should work on all official released versions and has been tested on Pup412, Pup431, Wary, Racy, Precise and Tahr605.

If you are remastering a series 5 or newer Puppy, there may be more compresssion options available and the remastered sfs should automatically be named correctly (if not, please rename it correctly).  These Puppys normally carry the 'puppy' prefix in the base sfs name.  If you are remastering one of these, select 'Yes' or 'OK' here.

If you are remastering an old Pup, there will be fewer compression options and you will have to correctly rename the remastered sfs.  These Pups normally carry the 'pup' prefix in the base sfs name.  If you are remastering one of these, select 'No' or 'Cancel' here.  This option should also work for all versions by default (also if the 'Yes' or 'OK' option fails).


Click 'Yes' or 'OK' for series 5 and newer Puppys.

Click 'No' or 'Cancel' for old Pups or as a fallback if the other option does not work ." 0 80
if [ $? -eq 0 ];then # Increased the readability (IMO!) of this section with tabs. musher0
	. /etc/DISTRO_SPECS
	PUPPYSFS="$DISTRO_PUPPYSFS"
	answer=`Xdialog --left --wrap --stdout --title "Compression Settings"  --combobox "/mnt/home/puppyfilesystem is now ready to be compressed to sfs file.
Higher compression levels produce smaller files but take longer to compress.
Gzip compression produce bigger files than xz compression but compress faster.
Older Puppys with pre-series-3 kernels must use gzip OLD compression.
Please select your choice from the menu, click 'OK' and wait.  This will take a while, please be patient." 0 80 "xz  DEFAULT" "xz  MEDIUM" "xz  HIGH" "xz  HIGHEST" "gzip  DEFAULT" "gzip  OLD" "NO COMPRESSION"`
	case "$answer" in "xz  DEFAULT")$RxvT -e $MksQ -comp xz ;;
        "xz  MEDIUM")$RxvT -e $MksQ -comp xz -Xbcj x86 ;;
        "xz  HIGH")$RxvT-e $MksQ -comp xz -b 1048576 -Xdict-size 100% ;;
        "xz  HIGHEST")$RxvT -e $MksQ -comp xz -b 1048576 -Xbcj x86 ;;
        "gzip  DEFAULT")$RxvT -e $MksQ -comp gzip ;;
        "gzip  OLD")$RxvT -e $MksQ ;;
        "NO COMPRESSION")$RxvT -e $MksQ -noI -noD -noF -noX ;;
	esac
else
	answer=`Xdialog --left --wrap --stdout --title "Compression Settings" --combobox "/mnt/home/puppyfilesystem is now ready to be compressed to sfs file.
Please select your choice from the menu, click 'OK' and wait.  This will take a while, please be patient." 0 80 "gzip  OLD" "NO COMPRESSION"`
	case "$answer" in "gzip  OLD")$RxvT -e mksquashfs $FilSys /mnt/home/pup_remastered.sfs -noappend -all-root ;;
        "NO COMPRESSION")$RxvT -e mksquashfs $FilSys /mnt/home/pup_remastered.sfs -noappend -all-root -noI -noD -noF ;;
	esac
fi
rm -r $FilSys;rm -f /mnt/home/_ro.lst # To remove the list of mounted systems we used. musher0
Xdialog -center -title "SUCCESS!!!"  --no-buttons -infobox "Remaster completed, exiting now" 0 80 5000
####THE END#### # set +xe
# (De-bugging command de-activated.) musher0
Attachments
Quick_Remaster_Rev2.1.zip
Same as above, zipped
(2.84 KiB) Downloaded 138 times
Last edited by musher0 on Sun 18 Jun 2017, 16:21, edited 1 time in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#28 Post by nic007 »

Hi, musher0. Thanks for your input. Few things:

rxvt -e compression line not working on my Puppys. Had to change $MksQ to mksquashfs $FilSys /mnt/home/$PUPPYSFS...

Content of save file is not copied.

I see you have only copied the ro's. I think the best way will be to copy all mounted stuff (including adrv, fdrv, ydrv, zdrv) all in one base sfs. But this will have to be in the correct sequence as people put all kinds of things in these drives. Not sure if your method of copying them all at once is going to work unless the listing sequence is absolutely correct.

The script does not deal with white-out files (record of "deleted" base files). The original files are showed in the remastered sfs. I would like these files to be deleted permanently. Can you assist with this?

In co-op nic007.

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

#29 Post by belham2 »

nic007 (and musher),

Just wanted to say again this little script is useful! The whiteout files thing is the only thing that is bothersome (but easily enough ignored). Though if you both could get the 'white-out files' issue resolved, this script would be perfect. :wink:

Thks again.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#30 Post by nic007 »

There are of course always alternatives too. Attached my reworked version of the original builtin puppyremaster script. A little bit more sophisticated compared to my humble effort here. No need for savefile, choose your working area, etc. :)

Edit: Uploaded to first post
Last edited by nic007 on Sun 18 Jun 2017, 13:18, edited 1 time in total.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#31 Post by nic007 »

belham2 wrote:nic007 (and musher),

Just wanted to say again this little script is useful! The whiteout files thing is the only thing that is bothersome (but easily enough ignored). Though if you both could get the 'white-out files' issue resolved, this script would be perfect. :wink:

Thks again.
Hi Belham2. I think I have good news for you. The remaster script posted below does indeed seem to get rid of files you have deleted in the base sfs. I've just checked with Wary Puppy. I deleted a few files in /usr/bin, did a remaster and bingo, gone in the remaster. :)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#32 Post by musher0 »

nic007 wrote:Hi, musher0. Thanks for your input. Few things:

rxvt -e compression line not working on my Puppys. Had to change $MksQ to mksquashfs $FilSys /mnt/home/$PUPPYSFS...

Content of save file is not copied.

I see you have only copied the ro's. I think the best way will be to copy all mounted stuff (including adrv, fdrv, ydrv, zdrv) all in one base sfs. But this will have to be in the correct sequence as people put all kinds of things in these drives. Not sure if your method of copying them all at once is going to work unless the listing sequence is absolutely correct.

The script does not deal with white-out files (record of "deleted" base files). The original files are showed in the remastered sfs. I would like these files to be deleted permanently. Can you assist with this?

In co-op nic007.
Hi nic007,

As I said, that was a suggested edit of your version 2. By the time I
had finished, you had released your version 3 !!!

adrv, fdrv, etc. were meant as separate sfs's and can stay separate, IMO.

In particular, the zdrv. That's where the kernel drivers and similar are. It's
very important to have it alongside the main puppy sfs, but it is not
necessary to blend the zdrv into the remaster.

As to the proper order of copy, it applies to the pup_rw. It should be
copied last. The code on line 37 of my edit should have copied the
pupsave file. That's strange, no problem here.

For the pup_ro? files, I believe my

Code: Select all

mount | awk '/_ro/'
and following lines above do a good job of it. They certainly save bytes
compared to the mile-long line you're using!? ;) The "tac" utility is also
respecting the reverse order in which you decided to do things in your
version 3.

I touched up the RxvT line in my script above, replacing English quotation
marks (") with German quotation marks ('). I had no problem on the
DPupStretch-7, but maybe that small change will bring success with your
version of rxvt.

I'll see what I can find on removing the "whiteouts".

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

mfb

#33 Post by mfb »

I have used the new version, Remasterpup(BaseSFS).zip, on Xenial-64-7.0.7 and tahr-64-6.0.5 both of which I run entirely in RAM.

Both tests resulted in total success and nic007's code, with its clear and helpful advice windows on available options at the decision stages made it a pleasure to use.

I only use 3 files - initrd.gz, vmlinuz and the main (remastered) sfs because I run entirely in RAM and do not need a save file, folder or dedicated save partition nor do I need to make any choices when booting.

I thoroughly recommend readers to try it on their frugal installs (after precautionary renaming of any current file(s) before using your remastered sfs - per nic007's opening post) and then report back with details of where they tried it and how they found it.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#34 Post by nic007 »

Added final versions of Remasterpup(ISO-CD) and Remasterpup(Base SFS) to opening post.

User avatar
nic007
Posts: 3408
Joined: Sun 13 Nov 2011, 12:31
Location: Cradle of Humankind

#35 Post by nic007 »

Users encouraged to use either version of Remasterpup instead. It seems to deal correctly with white-out files and builtin applications removed by the "Remove builtin packages" application seems to be removed accordingly.

Post Reply