Pupsave on USB when using separate data partition

Using applications, configuring, problems
Message
Author
User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

Pupsave on USB when using separate data partition

#1 Post by greengeek »

I am seeking clarification of my understanding of exactly what happens to the data when a pupsave is being used on a usb flash installation.

Example:
Imagine a usb stick which has a boot partition and a separate data partition. The pupsave is in a folder on the boot partition, but I keep my data in the second partition for safety.

It's my understanding that in a default usb flash installation the pupsave will update only occasionally in order to minimise write cycles to the flash memory on the stick (unlike an HDD install where the savefile is continuously updated). This means that there are periods of time where my "fresh" data is just sitting in RAM and not yet permanently saved into the pupsave filesystem - right?

So if I had a powercut at that moment my fresh data would be lost - right?

But what about this case: Imagine that I am using Abiword to type up a long project, and I occasionally save the file to my DATA partition. Does the Abiword save dialog ACTUALLY save immediately to the data partition (which is also a flash memory just like the boot partition) or does Abiword save into the RAM filesystem, and wait for the pupsave routine to reach it's appropriate timeout and then the data gets saved to the appropriate permanent locations.

I am hoping that any data I try to save direct to my data partition ACTUALLY goes there immediately. But does it?

This is of particular interest to me as I generally turn off my pupsave function and only do a pupsave at shutdown time (if I'm happy with the changes I've been making) - believing that all the data I saved to the other partition was immediately and permanently saved.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#2 Post by Karl Godt »

That's a good question .

If in PUPMODE=13 on USB the mountpoint "/initrd/pup_rw" is tmpfs filesystem i think.

The save-file.[2-4]fs should have it's own mountpoint in /initrd/ directory, in PUPMODE=13 it is "/initrd/pup_ro1" i think.

User avatar
Dave_G
Posts: 453
Joined: Thu 21 Jul 2011, 13:53

#3 Post by Dave_G »

That is a good question so I decided to try it out on Puppy4.3.1 frugal install
on USB flash drive.

Created a .abw (Abiword document) and saved it to /root.
Switched power off to computer.
When I restarted, the document was no longer there.

Interestingly, when I restarted the computer, Xwin did not complain about
an improper power down even though the file .XLOADED was in /etc before power down.
This can only mean that nothing was saved.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#4 Post by Karl Godt »

The relevant code for saving from temporary filesystem layer is in `which snapmergepuppy' :

Code: Select all

SNAP="/initrd/pup_rw"
cd $SNAP || exit 1

if [ "$SAVE_LAYER" ];then #defined in PUPSTATE
 BASE="/initrd${SAVE_LAYER}"
else
 BASE="/initrd/pup_ro1"
fi
Note : PUPSTATE is /etc/rc.d/PUPSTATE configuration file .

User avatar
Dave_G
Posts: 453
Joined: Thu 21 Jul 2011, 13:53

#5 Post by Dave_G »

So I guess the moral of the story is, if one needs stuff saved permanently,
enable the periodic saving feature or use a UPS.

Considering how cheap USB flash drives are these days, as long as one
backs up the save file say once a week to another medium it does not matter
if the USB flash drive is written to say every 30 minutes or so and only lasts
around 6 months.

On my main Puppy machine, I have my save interval set to every hour and I have
been using the same USB flash drive for the last 8 months and the machine is operational every day for around 8 to 10 hours a day.

User avatar
rcrsn51
Posts: 13096
Joined: Tue 05 Sep 2006, 13:50
Location: Stratford, Ontario

#6 Post by rcrsn51 »

Just to be clear:

1. You are running an application like Abiword.

2. You are working on a file that is located in a different partition, like sdb2.

3. You click File > Save.

4. You have a power failure.

The file might not be saved, but only because it was still in cached memory. If you opened a terminal window and did a "sync", I'm pretty sure that it would survive the power failure.

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#7 Post by RSH »

Hi.

I do save each and every document i do create with the LazY Puppy outside the running system.

I do use mostly a 8 gb usb stick. Putting all the LazY Puppy files to this usb stick gives me about 4 GB free space for data.

On my home workstation i do not use any save file to 100% - i did never lost any file (except the usb stick crashes).

On my usb stick i do have a directory, that contains several sub directories for different data types like office docs, pictures, scripts, music and some more. On the root partition of the usb stick i do have a file that contains the names of theses directories and sub directories - so, they are known by the system (no matter how they are named or re-named).

Code: Select all

# LazY Puppy 2.0.2 MMS Pro external directories
# 2012-07-24 RSH for LazY Puppy
#------------------------------------------------------------------------------

# main directory of files
mydirsdir="Dateien"

# sub directories of main directory
myguidesdir="Anleitungen"
myaqualungdir="Aqualung"
mybashscriptsdir="Bash"
myimagesdir="Bilder"
myhtmldir="HTML"
myiconsdir="Icons"
mylazypuppydir="LazYPuppy"
mymusicdir="Musik"
myofficedir="Office"
mytransferdir="Transfer"
myuploaddir="Upload"
myvideosdir="Videos"

# files in office directory
mynotes="Notizblock.txt"
mylpmmstodolist="LazYPuppy-Bugs-ToDo.txt"

# Script Directories
myscripts="Skripte"
mymousesettings="MouseSettings"

# Software and Installation Directories
mysoftware="Software"

#------------------------------------------------------------------------------
# End of LazY Puppy 2.0.2 MMS Pro external directories
#------------------------------------------------------------------------------
LazY Puppy (the downloadable version as well) contains a file in /usr/sbin/ceelab/lp2_sys_data named LP2_BootDir, which is been created automatically at every boot up. This file contains different paths of LazY Puppy.

Code: Select all

LazY_Puppy_Boot_Partition_Link="/mnt/sdd1"
LazY_Puppy_Boot_Partition="/mnt/sdd1"
LazY_Puppy_Boot_Dir_Link="/mnt/sdd1/LazY"
LazY_Puppy_Boot_Dir="/mnt/sdd1/LazY"
LP2BDL="/mnt/sdd1/LazY"
LP2BD="/mnt/sdd1/LazY"
LP2MNT="+mnt+sdd1+LazY"
LP2MNTDIR="+mnt+sdd1+LazY"
LP2MNTPRTLNK="+mnt+sdd1"
LP2MNTPRT="+mnt+sdd1"
Using these two files (i mean the data) i can build every needed path and use it from inside the system to run scripts from outside the system etc.pp.

I do use it f.e. to make symlinks for aqualung media library, config and playlists.

Code: Select all

#!/bin/sh
# Create needed symlinks to external files and directories in /mnt/home
#------------------------------------------------------------------------------

yaf-splash -deco "Links erzeugen" -icon gtk-info -fontsize large -bg orange -text "Erzeuge Links für: Aqualung Audioplayer" &
X3PID=$!

# make file that contains the boot directory
/usr/sbin/ceelab/tools/get_boot_dir

sleep .5

# include needed files to build the paths
. /usr/sbin/ceelab/lp2_sys_data/LP2_BootDir
. $LazY_Puppy_Boot_Partition_Link/lped

# make symlink for Aqualung Files Diretory
AQUALUNGDIR="$LazY_Puppy_Boot_Partition_Link/$mydirsdir/$myaqualungdir"
if [ -d /root/.aqualung ]; then
	echo "removing /root/.aqualung"
	rm -r /root/.aqualung
fi
echo "make symlink to $AQUALUNGDIR/.aqualung in /root"
ln -s $AQUALUNGDIR/.aqualung /root/.aqualung
if [ /root/.aqualung ]; then
	echo "symlink /root/.aqualung created"
fi
# end of symlink for Aqualung

sleep .5

kill $X3PID

exit 0
If i want to increase the functions of this file or to add some new features, i can do it without the need to remaster the LazY Puppy. I just go the the menu and do click on its entry. The script is always been found as long as the usb stick is mounted - even if i would change the directory.

RSH

EDIT:

rcrsn51 has posted while i did wrote my post.
rcrsn51 wrote:If you opened a terminal window and did a "sync", I'm pretty sure that it would survive the power failure.
LazY Puppy has a menu entry for that (menu LazY Puppy - LP2 Sync) :D :lol: 8)
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#8 Post by Karl Godt »

Dave_G wrote:So I guess the moral of the story is, if one needs stuff saved permanently,
enable the periodic saving feature or use a UPS.

Considering how cheap USB flash drives are these days, as long as one
backs up the save file say once a week to another medium it does not matter
if the USB flash drive is written to say every 30 minutes or so and only lasts
around 6 months.

On my main Puppy machine, I have my save interval set to every hour and I have
been using the same USB flash drive for the last 8 months and the machine is operational every day for around 8 to 10 hours a day.
I think you can use the "Save" icon on the desktop that should be created by /etc/rc.d/rc.sysinit i think .
Or run

Code: Select all

snapmergepuppy
manually via cli
Or create a PUPSnap.desktop file in /usr/share/applications/ that executes snapmergepuppy .
Or navigate with the abiword fileselector to "/mnt/home" or "/initrd/mnt/dev_save"(if in pfix=ram mode) and save the blank sheet first so you only need to hit the save button on abiword and it should bypass the tmpfs layer . In fact every writable mount point in the /initrd/* directory can be used to bypass the tempfs buffer.

(I run full installs and don't have all these problems) :wink:

User avatar
rcrsn51
Posts: 13096
Joined: Tue 05 Sep 2006, 13:50
Location: Stratford, Ontario

#9 Post by rcrsn51 »

Since talk is cheap :wink: , I decided to test this.

1. I made a flash drive with two partitions.

2. I started an Abiword document on sdb2.

3. If I did a Save, then immediately killed the power, the document was lost.

4. If I did a Save followed by a "sync", then killed the power, the document was saved correctly. The blinking lights on the flash drive are the clue.

User avatar
Dave_G
Posts: 453
Joined: Thu 21 Jul 2011, 13:53

#10 Post by Dave_G »

Puppy creates (amongst others) a mount point for sdb1 (on my system)
for the USB flash drive as read-write then loop1 also as rw for the save file
and others such as loop0 and loop2 depending if other sfs files have also
been loaded such as devx.

Since sdb1 is accessible as /mnt/home then if we save directly there it is permanent.
Tried it by simulating a power failure.

To make it more organised, do as RSH wrote and make several directories
such as docs ,pics, music and so on.

The big advantage as RSH has said is we no longer limited by the size of the
save file but only by the size of sdb1 itself.

The disadvantage is making back ups will involve multiple directories and files.

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#11 Post by RSH »

Hi, me again.

I also did make a openbox pipe menu to get easily access to the externally saved files via the openbox menu. I just do install this once after boot up - if i want to use it.

Example script /etc/xdg/openbox/scripts/images_extern.sh

Code: Select all

# Raster Image documents --> pipe menu for openbox
# RSH #

. /usr/sbin/ceelab/lp2_sys_data/LP2_BootDir
. $LazY_Puppy_Boot_Partition_Link/lped

IMGSEXTDIR="$LazY_Puppy_Boot_Partition_Link/$mydirsdir/$myimagesdir"

# Begin pipe menu for openbox
echo "<openbox_pipe_menu>"

for IMGSEXT in ` ls "$IMGSEXTDIR/" | egrep ".bmp|.jpg|.jpeg|.png|.tiff|.tif|.xpm|.BMP|.JPG|.JPEG|.PNG|.TIFF|.TIF|.XPM" `
do
	echo "<item label=\"$IMGSEXT\">
		<action name=\"Execute\">
			<execute>
				/etc/xdg/openbox/scripts/select_img_action \"$IMGSEXTDIR/$IMGSEXT\"
			</execute>
		</action>
	</item>"
done

echo "</openbox_pipe_menu>"

exit 0

# End
example script (what to do with the image) /etc/xdg/openbox/scripts/select_img_action

Code: Select all

#!/bin/sh
#RSH

bild=$1
bnf=`basename $bild`

Xdialog --title "Bilddatei" --icon "/usr/share/pixmaps/artbrush-b.xpm" --timeout 60 --ok-label "Ansehen" --cancel-label "Bearbeiten" --yesno "Wie möchten Sie mit der Bilddatei \n\n'$bnf' \n\nverfahren? Bitte wählen Sie..." 0 0

case $? in
	0) defaultimageviewer $bild ;;
	1) defaultimageeditor $bild ;;
	255) exit 255 ;;
esac

exit 0
and exapmle html code (insert in file) /root/.config/openbox/menu_top.xml

Code: Select all

<menu execute="/etc/xdg/openbox/scripts/images_extern.sh" id="allimgs-ext-menu" label="Pixelgrafik"/>
Please take my apologize (not appointment :lol: ) for the German Language parts in the scripts. I am just to lazy to translate them by now, because i did make this only for my private use. This is not available in the downloadable versions of LazY Puppy - but can be created though.

I hope this is interesting, somehow...

RSH
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#12 Post by Karl Godt »

Description


The sync() function shall cause all information in memory that updates file systems to be scheduled for writing out to all file systems.

The writing, although scheduled, is not necessarily complete upon return from sync().
Return Value

The sync() function shall not return a value.
from man 3 sync

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#13 Post by greengeek »

rcrsn51 wrote:The blinking lights on the flash drive are the clue.
Yes, good point. One of the reasons I started worrying about this issue is that I have recently been trialling puppy installs on micro-SD (Tflash) cards, so that I can carry several of them in my wallet. Unfortunately the adapter/controller they slide into does not have an LED so I feel like I'm flying blind with regard to data integrity.

It would be really nice if it was possible to have a function that brought up a gtk dialog describing the file save progress (especially when saving to a separate partition which is what I do for the data I really want to keep)
The writing, although scheduled, is not necessarily complete upon return from sync().
Hmmm, now I feel even less secure :-)
Please take my apologize (not appointment :D ) for the German Language parts in the scripts.
Thanks RSH. I do not yet understand what is the purpose of the openbox script - I need to look at this in more detail.
Or navigate with the abiword fileselector to "/mnt/home" or "/initrd/mnt/dev_save"(if in pfix=ram mode) and save the blank sheet first so you only need to hit the save button on abiword and it should bypass the tmpfs layer . In fact every writable mount point in the /initrd/* directory can be used to bypass the tempfs buffer.
Interesting. I will read up a bit more about the tempfs structure.

User avatar
rcrsn51
Posts: 13096
Joined: Tue 05 Sep 2006, 13:50
Location: Stratford, Ontario

#14 Post by rcrsn51 »

greengeek wrote:
The writing, although scheduled, is not necessarily complete upon return from sync().
Hmmm, now I feel even less secure :-)
This just means that there might be a short gap between when the sync command ends and the data is completely written.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#15 Post by greengeek »

rcrsn51 wrote:This just means that there might be a short gap between when the sync command ends and the data is completely written.
I'm finding that when I save a big file it sometimes takes way longer on some flash cards than others. There seems to be a huge variation in write speed between brands/models. Without an LED it's guesswork. Might have to stay away from micro-SDs. :-)

User avatar
rcrsn51
Posts: 13096
Joined: Tue 05 Sep 2006, 13:50
Location: Stratford, Ontario

#16 Post by rcrsn51 »

So how do you know when writing is fully completed on these SD cards?

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#17 Post by greengeek »

To be honest I don't know for sure when it is finished, I just try to gauge it from the cpu graph and from whether or not the system will respond to a shutdown message (if I am at that stage..). When I find a flash card that seems to take longer than others I give it a test with something like a 100-200mb iso file copy and see at what point it will allow an unmount action. Not very scientific I'm afraid. So much easier if there is an LED.

User avatar
rcrsn51
Posts: 13096
Joined: Tue 05 Sep 2006, 13:50
Location: Stratford, Ontario

#18 Post by rcrsn51 »

Are you doing this with a "sync"? It's been my experience that when the sync is done, the partition is immediately ready to unmount and shut down.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#19 Post by greengeek »

No, I haven't been using sync. If I have been wanting to eject a usb stick / flash card I normally use Pmount (or whatever icon method might be available) to unmount the stick, or else just wait for the shutdown to complete if the relevant partition is on my boot stick.

My main concern was not really having any true indication (other than interpreting LED activity) that data had actually finished being written at the time I was expecting. From the responses I have received here, it sounds as if data that I'm saving to my data partition IS in fact starting to be written there as soon as I use the save dialog within Abiword etc, but I still don't have any way of knowing when the save terminates. (and this effect is much more pronounced if it is a large file like an iso that is being copied across with the file manager - I assume that this type of copy begins as soon as I ask it to, but I have little confidence that it finishes as quickly as the dialog suggests).

I think I will do a bit of 'sync' experimenting with some of my slower flash cards and see what I can find out regarding transfer of large amounts of data.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#20 Post by Karl Godt »

I just greped sync in /usr/sbin/pmount and found one match. Also /usr/local/bin/drive_all has one match. Probably it could be implemented in /bin/umount again:

Code: Select all

#sync #paranoid precaution. v2.12 kernel call to modprobe, sync hangs.
because if the partition is unmounted by using rox and not by clicking the drive icon on desktop the bare /bin/umount script gets run and not the copied /usr/local/bin/drive_all as /root/.pup_event/DRIVE/AppRun which would sync .

FWIW : I use xosview to check if writes are done :

Post Reply