Savefolder without 'mount -o bind' - works

[gyro]

The following are patches to "init" and "rc.shutdown" against TahrPup 5.8.3.rc1,
that implement 'savefolder' using symbolic links instead of 'mount -o bind'.

Patch 1) "init" script in "initrd.gz":

Code: Select all

--- init.orig	2014-10-04 01:58:54.960910335 +1000
+++ init	2014-10-04 02:26:50.898497667 +1000
@@ -1078,7 +1078,10 @@
      ;;
    esac
   fi
-  if [ "$CRYPTO" != "" ] ; then
+  if [ -d /mnt/dev_save$PUPSAVEFILE ]; then
+   rm -r -f $CREATEPUPSAVE2FS
+   ln -s /mnt/dev_save${PUPSAVEFILE} $CREATEPUPSAVE2FS
+  elif [ "$CRYPTO" != "" ] ; then
    echo "" >/dev/console
    echo "Mounting encrypted $PUPSAVEFILE..." > /dev/console
    while true; do
@@ -1124,8 +1127,6 @@
      losetup -d /dev/loop1
     fi
    done
-  elif [ -d /mnt/dev_save$PUPSAVEFILE ]; then
-   mount -o bind /mnt/dev_save${PUPSAVEFILE} $CREATEPUPSAVE2FS
   else #pupsave not encrypted.
    # if pupsave increase requested
    if [ -f /mnt/dev_save/pupsaveresizenew.txt ];then #131225 changed to function
@@ -1967,6 +1968,12 @@
 do
  mount -o move $ONEMNT /pup_new/initrd${ONEMNT}
 done
+if [ ${PUPSAVEFILE} -a -d /pup_new/initrd/mnt/dev_save${PUPSAVEFILE} ]; then
+ if [ ${CREATEPUPSAVE2FS} -a -L ${CREATEPUPSAVE2FS} ]; then
+  rm -r -f /pup_new/initrd${CREATEPUPSAVE2FS}
+  ln -s ./mnt/dev_save${PUPSAVEFILE} /pup_new/initrd${CREATEPUPSAVE2FS}
+ fi
+fi
 
 #v4.02 bring back, but allocate more space (/4 instead of /8)....
 #v3.97 a problem can run out of /tmp space, remove...

To provide access to the save directory during "init", "/pup_rw" is changed to be a symbolic link to the save directory.
To provide direct access to the save directory after the switch root, "/pup_new/initrd/pup_rw" is changed to be a symbolic link to the save directory.
Fortunately 'aufs' cooperates nicely, when the specified branch is a symbolic link, it follows the link and uses the real path to the directory.

Code: Select all

# cat /sys/fs/aufs/si_*/br0
/initrd/mnt/dev_save/tahr/tahrsave=rw

I've also changed the logic from:

Code: Select all

if savefile is encrypted #assume is a file
else if savefile is a directory #assume no encryption
else #is unencrypted file

to

Code: Select all

if savefile is a directory #could include encrypted option
else if savefile is encrypted #must be a file
else #is unencrypted file

Patch 2) "rc.shutdown":

Code: Select all

--- rc.shutdown.orig	2014-08-28 06:07:44.000000000 +1000
+++ rc.shutdown	2014-10-04 02:04:09.922537392 +1000
@@ -440,12 +440,11 @@
   [ "`echo -n "$SAVEFILE" | grep "2fs"`" != "" ] && FILEFS="ext2"
   [ "`echo -n "$SAVEFILE" | grep "4fs"`" != "" ] && FILEFS="ext4" #120427 01micko
 
-  if [ "$CRYPTO" = "" ];then
-   if [ -d $SMNTPT$SAVEFILE ]; then
-    mount -o bind $SMNTPT$SAVEFILE /tmp/save1stpup
-   else
-    mount -t $FILEFS -o noatime,rw,loop $SMNTPT$SAVEFILE /tmp/save1stpup
-   fi
+  if [ -d $SMNTPT$SAVEFILE ]; then
+   rm -r -f  /tmp/save1stpup
+   ln -s $SMNTPT$SAVEFILE /tmp/save1stpup
+  elif [ "$CRYPTO" = "" ];then
+   mount -t $FILEFS -o noatime,rw,loop $SMNTPT$SAVEFILE /tmp/save1stpup
   else
    #note: loop2 is kept available for scripts to use. but, do this to find free one...
    DEVLOOP="`losetup-FULL -f`"
@@ -488,7 +487,7 @@
    [ "$BASENAME" = "root" ] && cp -a $ONEDIR/.[0-9a-zA-Z]* /tmp/save1stpup/${BASENAME}/ #v2.16exp4
   done
   sync
-  umount /tmp/save1stpup
+  [ -L /tmp/save1stpup ] || umount /tmp/save1stpup
 
   #w481 if low-ram, save-file is on a fast media, and no swap file/partition, then create one...
   swapfile_func $SMNTPT $SAVEPART
@@ -675,8 +674,10 @@
    sync
   fi
  fi
- busybox mount -t $SAVEFS -o remount,ro $SAVEDEV /initrd${SAVE_LAYER} 2>/dev/null
- umount-FULL -i -n -l /initrd${SAVE_LAYER} 2>/dev/null #-l is lazy unmount.
+ if [ $SAVEDEV ]; then
+  busybox mount -t $SAVEFS -o remount,ro $SAVEDEV /initrd${SAVE_LAYER} 2>/dev/null
+  umount-FULL -i -n -l /initrd${SAVE_LAYER} 2>/dev/null #-l is lazy unmount.
+ fi
 fi
 
 #v2.16 try one more thing for ntfs... lazy unmount, suggested by GuestToo...

At first shutdown a symbolic link is created to the actual save directory so the copying code will work.
After prepending the dummy rw layer to the aufs stack, only do the mount commands for a savefile, there is no mount corresponding to a savefolder.

Note: If only we had implemented it this way in the first place, we might have avoided the headaches caused by 'mount -o bind'.

gyro

Edit: The code above is a fix to the 'savefolder' facility that already exists in TahrPup. I intend to provide an example for a puppy with no existing 'savefolder' facility.

Edit2: An "initrd.gz" for TahrPup 5.8.4.sfs can be downloaded from here http://www.fishprogs.software/puppy/tahr/initrd.gz
A ydrv for TahrPup containing "rc.shutdown" and "sfs_load" can be downloaded from here http://www.fishprogs.software/puppy/tah ... _5.8.4.sfs

[Flash]

Will this work with a multisession Puppy DVD?

[gyro]

Will this work with a multisession Puppy DVD?

No.
'savefolder' requires that saving is done on a Linux partition.

gyro

[mavrothal]

The code above is a fix to the 'savefolder' facility that already exists in TahrPup. I intend to provide an example for a puppy with no existing 'savefolder' facility.

Have a test ISO?

[starhawk]

I remember that there was an option to do a sort of hybrid install where the install itself was frugal but you could save to / and (theoretically) not have to bother with savefiles filling and all -- it would fill the remaining space in the root partition with savefile goodness automatically, or something like that.

Unfortunately it was abandoned before it became functional, and eventually it got removed. The few times I've tried it, as soon as I rebooted from the first boot, Puppy would make a mess on the carpet and generally not work. I don't recall the exact details but it was lots of red text having to do with one of the pup_ro bits (I want to say ro2?) followed by a kernel panic.

I would love to see that option actually made functional and brought back as such. Would make Puppy just that much more easy to use! (I wonder if it could be implemented simply as "put the savefile in / rather than a folder, and fill the entire partition with it"?)

[gcmartin]

Great. Seemingly plays into the hands of "remaster this very system".

[mikeb]

bind mount was used for the save folder to avoid altering the pup_rw folder arrangement to keep compatability. Also its puppies ill thought through scripts unable to deal with it that was the problem rather than something wrong with the technique itself.

With any addition/modification to save techniques the designer has to try and keep everything as-is as much as possible otherwise you would need to change the name to Porteus or something
I dynamically created additional folders for extra sfs but still keep pup_ro2 for exactly the same reasons. Certainly rearranging the initrd structure would make life a whole lot easier at times but compromise is the key word here.

mike

[gyro]

bind mount was used for the save folder to avoid altering the pup_rw folder arrangement to keep compatability. Also its puppies ill thought through scripts unable to deal with it that was the problem rather than something wrong with the technique itself.

Yes, at the time I couldn't work out how the single bind mount could be replaced with a single symbolic link.
I still can't, but in this solution, there are 2 symbolic links, one that works during 'init' and another one that works for the running system, after the switch root.

gyro

[gyro]

(I wonder if it could be implemented simply as "put the savefile in / rather than a folder, and fill the entire partition with it"?)

And why can't the contents of a directory "fill the entire partition"?

gyro

[gyro]

Have a test ISO?

Hmm, I seem to remember you asking me this question in a different place.
The answer is the same, No.
However I plan to make an initrd.gz and a ydrv available for download.

gyro

[starhawk]

(I wonder if it could be implemented simply as "put the savefile in / rather than a folder, and fill the entire partition with it"?)

And why can't the contents of a directory "fill the entire partition"?

gyro

They can. I'm just working with what I'm used to I rather liked the idea of that feature (frugal with save to root)... like I said, would make life with Puppy that much easier...

[gyro]

Url's for downloading an initrd.gz and ydrv for TahrPup 5.8.3.rc1 are now included at the end of the first post.
The ydrv contains sfs_load, to avoid partitions remaining mounted on first boot. (It's fixed in QuickPet, but that's too late for first boot.)

gyro

[mavrothal]

Note: If only we had implemented it this way in the first place, we might have avoided the headaches caused by 'mount -o bind'.

So is this working without all the other changes implemented so the current "bind" method would work properly? Would it work say, on Slacko 5.7 or PrecisePup properly without additional changes (other than the shutdown config)?

BTW, which would you say is the advantage of the symlink method over the bind method. Which current or future problem is it addressing better? (I would think device management by different file managers maybe. Does it?)

[gyro]

So is this working without all the other changes implemented so the current "bind" method would work properly? Would it work say, on Slacko 5.7 or PrecisePup properly without additional changes (other than the shutdown config)?

Yes, that is my contention.
I'm currently working on porting it to Dpup Wheezy 3.5.2.11, without any of the additional changes. I've got a patched 'init' and 'rc.shutdown' working, I just need to work out the best way to incorporate, an updated 'shutdownconfig' so a savefolder can be created at first shutdown. (I currently create a small savefile and then use my 'savefile2dir' utility to convert it to a savefolder.)
I will be posting the files for this port as an example, in this topic.

BTW, which would you say is the advantage of the symlink method over the bind method. Which current or future problem is it addressing better? (I would think device management by different file managers maybe. Does it?)

Any code that does not appropriately handle the existence of more that 1 mount entry for a single device, will not get broken by the symlink method.
A simple example is "df", using the symlink method, it works as expected in TahrPup.
Things like 'PcmanFM' should work without any modification.
The long term advantage is that we don't know if there is more software out there that would fail if it was run in an environment with more than 1 mount entry per device.

gyro

[gyro]

I have attached the diff files I used for Dpup Wheezy 3.5.2.11.
I have also attached my utility for converting an existing savefile to a savefolder.
The files are really gzipped, so download and move them to where you want them and then gunzip them.

I post these as an example of porting this savefolder facility to puppies that are not based on a current version of woof-ce, as TahrPup is, and hence do not contain any savefolder facility.

Warning, while this version of savefolder avoids the "more than 1 mount entry" problems that plagued the development of the bind mount version, it does not make any difference to other problems.
Example: when I previously ported savefolder to Dpup Squeeze Exprimo, the "rc.shutdown" would fail. I expect that this version could have the same problem.

Note: Once I have sorted out the easiest way to include an updated 'shutdownconfig' I intend to upload a patched 'initrd.gz' and a 'ydrv_wheezy_3.5.2.11.sfs' containing 'rc.shutdown' and 'shutdownconfig'.

Note2: I have included 'savefile2dir' because without an updated 'shutdownconfig' you can't create a savefolder. But this utility can convert an existing savefile to a savefolder. So testing of the savefolder facility can begin once you have a successfully patched 'initrd.gz' and 'rc.shutdown'.

gyro

[gyro]

@mikeb
I do appreciate that the 'mount -o bind' method is all correct and above board Linux, and the scripts and programs are wrong.

gyro

[mikeb]

No worries and I cannot think of any software that is affected other than puppy scripts. I once went around removing instances of rm -rf $VARIABLE from several places as such code can potentially wipe yer system if the generation of $VARIABLE gets upset. (lost a partition that way )

Just as an aside slax manages to keep mounts made by the initrd separate from the main system so when running you only see 'normal' mounts and the loops and multiple mounts can only be seen by cat /path/to/initrd/proc/mtab ... not sure what the subtle difference is but it may be that the initrd system is not destroyed once booted since its reused at shutdown to keep things clean so I assume the kernel does not transfer over such special files/devs/proc info like it does in puppy.

Well have fun with the shoehorn... guess you like the challenge like me

mike

[gyro]

An 'initrd.gz' for dpup Wheezy 3.5.2.11 can be downloaded from here http://www.fishprogs.software/puppy/wheezy/initrd.gz
A ydrv for Wheezy containing 'rc.shutdown' and 'shutdownconfig' can be downloaded from here http://www.fishprogs.software/puppy/whe ... 5.2.11.sfs
The 'shutdownconfig' has been lifted directly from TahrPup.

This combination works ok.
The only failure I have noticed is 'freememapplet' did not seem to start. But then I always disable 'freememapplet' in any puppy using savefolder, since I don't think it's providing critical information in this situation.

gyro

[gyro]

Ok, I've found the problem with this implementation of 'savefolder'.
It's programs that try to find out how much space is left in the savefile. And they expect to get this from a mount entry. And of course there is no mount entry for '/initrd/pup_rw' (or '/initrd/pup_ro1' in pupmode=13).

I just looked at 'snapmergepuppy' source, and it definitely has this problem. So I suspect that in pupmode13 doing a 'save' will result in an error message saying that you have 0 space, and 0 files were copied. I haven't worked out a patch yet, but I don't think it will be too difficult.

I suspect that 'freememapplet' could have a similar problem. But it's not a script. Besides, I can easily live without it when using a savefolder, in fact I do. Which might explain why I didn't spot this one sooner.

I can't think of any other candidates for the problem.

It's a little frustrating that once again a side-effect breaks some puppy code, because it nullifies an assumption that seemed reasonable at the time the code was written.

gyro

[gyro]

Here is a possible patch for 'snapmergepuppy' in TahrPup 5.8.3.rc1.

Code: Select all

--- snapmergepuppy.orig	2014-08-28 06:07:44.000000000 +1000
+++ snapmergepuppy	2014-10-06 18:00:13.693546346 +1000
@@ -66,9 +66,15 @@
  BASE="/initrd/pup_ro1"
 fi
 
+if [ -L $BASE ]; then #we have savefolder based on symbolic links
+ BASEMTP="/initrd${PUP_HOME}"
+else
+ BASEMTP="$BASE"
+fi
+
 # Precautions
 [ $PUPMODE -eq 3 -o $PUPMODE -eq 7 -o $PUPMODE -eq 13 ] || { echo "Wrong PUPMODE ($PUPMODE)!"; exit 1; }
-mountpoint -q "$BASE" || { echo "$BASE is not mounted!"; exit 1; }
+mountpoint -q "$BASEMTP" || { echo "$BASEMTP is not mounted!"; exit 1; }
 
 echo "Merging $SNAP onto $BASE..."
 
@@ -146,7 +152,7 @@
 
 #100222 a quick hack: BOOTCONFIG written to in init, before timezone set, can cause trouble...
 touch /etc/rc.d/BOOTCONFIG
-FREEBASE=`df -B 1 | grep -w "$BASE"| head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah #110222 shinobar # SFR: result in bytes (see 'find' below)
+FREEBASE=`df -B 1 | grep -w "$BASEMTP"| head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah #110222 shinobar # SFR: result in bytes (see 'find' below)
 
 rm -f /tmp/snapmergepuppy-nospace #110222 shinobar
 rm -f /tmp/snapmergepuppy-error   #140102 SFR
@@ -171,7 +177,7 @@
 
  #stop saving if not enough room left in ${DISTRO_FILE_PREFIX}save file...
  if [ $((NSIZE+204800)) -gt $FREEBASE ]; then	# 204800 = 200K slack space
-  FREEBASE=`df -B 1 | grep -w "$BASE" | head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah: this is very slow; try not to check every iteration #110222 shinobar: fix for pup_ro10 and more
+  FREEBASE=`df -B 1 | grep -w "$BASEMTP" | head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah: this is very slow; try not to check every iteration #110222 shinobar: fix for pup_ro10 and more
   if [ $((NSIZE+204800)) -gt $FREEBASE ]; then	#110212 Jemimah.
    touch /tmp/snapmergepuppy-nospace  #110222 shinobar
    break

This is untested, I don't run any pupmode=13.
I have also include the patched version of 'snapmergepuppy' in the ydrv for TahrPup here http://www.fishprogs.software/puppy/tah ... .3.rc1.sfs

Edit: I'm terribly sorry, there is a bug in the original code I published here. (Missing 'fi') I have fixed the diff here and the patched 'snapmergepuppy' in http://www.fishprogs.software/puppy/tah ... .3.rc1.sfs

gyro