snap2 rotating snapshot backups for Puppy

Miscellaneous tools
Message
Author
Jim1911
Posts: 2460
Joined: Mon 19 May 2008, 20:39
Location: Texas, USA

#76 Post by Jim1911 »

Hi lstandish,

It's working well.

One issue, there still isn't a prominent caution ie., BEFORE PROCEEDING, MAKE SURE THAT THE SOURCE AND DESTINATIONS ARE MOUNTED, otherwise backups will be deleted. I tested this and sure enough, with a partition unmounted, a previous Mirror backup Is deleted very quickly. This is a necessary reminder for the first time user and a good reminder for forgetful old timers like myself.

You have really improved the content of the help file.

Thank you for this fine utility program,
Jim

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#77 Post by 8-bit »

Jim1911,
I have some thing for you to try.
Say your destination is set as /mnt/sda3.
Also let us say you forgot to mount /dev/sda3
After doing a backup/snapsot, and before mounting /dev/sda3
check /mnt/sda3 by opening that directory and see if the backup file is there.
That is something I was trying to express in a previous post.
That before a backup is done, the destination should be checked to see if it is mounted.
I have as a test copied files to an unmounted drive, NOT USING /dev/[drive/partition] but instead, using /mnt/[drive/partition].

The files copied fine to the unmounted /mnt/drive directory.
But they would disappear on mounting that drive.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#78 Post by lstandish »

Sylvander,
Part of the idea of a snapshot backup is to document the state of the files on a given date. If you delete the snapshot just because it is the same as a previous snapshot, you lose that documentation. The only reason to delete it would be, I think, to save disk space. Yet, the overhead (extra disk space) used by an identical snapshot is very small (only the size of the hard links).

In sum, I don't recommend deletion of identical snapshots, unless you are very low on backup storage space. snap2's built in support for deleting snapshots is intended mainly to be used when a backup was interrupted (and therefore incomplete).

I don't think there is any advantage to having snap2 try to rename snapshots to fill up 'holes' left by snapshot deletions.

Sorry, I have not yet had time to get to study what will be involved to include the date-time stamp in the snapshot directory name.

Jim1911,
Hmmm, I never considered the scenario of the *source* partition not being mounted! To fix that, snap2 would have to be able to tell when a source directory was really deleted, and when the source partition is only not mounted. I'll try to make snap2 detect the unmounted partition case, or if that is not practical, I'll add a warning about unmounted source partitions to the docs.

8-bit,
I see your problem: if /mnt/sda3 is specified as the backup storage location, the directory /mnt/sda3 will exist whether or not the partition is mounted. That means the snap2 setting to *not* allow creation of a missing backup storage directory will have no effect, and backup will proceed.

To prevent snap2 from proceeding with a backup when the destination partition is not mounted, specify a backup storage folder (on the BACKUP STORAGE tab) like this one: /mnt/sda3/mybackups Now, when sda3 is not mounted, /mnt/sda3/mybackups will not exist, and if the 'Advanced' setting to allow creation of missing backup storage folder is unchecked, then snap2 will not do the backup.

(You can move your backups in /mnt/sda3/ to /mnt/sda3/mybackups/ without having to actually move data, so it will be fast. If you do this remember to update the backup storage directory in snap2.)
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#79 Post by Sylvander »

1. "Part of the idea of a snapshot backup is to document the state of the files on a given date"
Can't think why I'd be interested in "the state of the files on a given date". :?
That's not something that's useful to me.
What I really want is a series of snapshots of [the contents of partitions? in] DIFFERENT STATES, so I can perhaps restore one of those to get back to that state...
Or perhaps to recover a lost file, or [less likely] one or more files in a particular state.

2. "The only reason to delete it would be, I think, to save disk space. Yet, the overhead (extra disk space) used by an identical snapshot is very small"
(a) Imagine making a great list of snapshots, that are all [or perhaps only nearly all] IDENTICAL.
But you don't know [unless you expend time and effort], which are identical to which.
For my purposes, those identical snapshots are only clutter getting in the way.
[Perhaps there could be a tick-box to not save a snapshot that is identical to the previous snapshot? = CHOICE]
It's little consolation to know that at least they don't use much "real" space.
Which raises another important point as follows below:

(b) If I'm looking for a suitable snapshot to restore the way [an operating system? on] a partition used to be, when it worked a certain way [rather well?]...
I need something...
Some information about the snapshot...
That tells me what I need to know.
Does snap2 allow a user to make notes and associate them with a snapshot?
[Can't check here; this OS doesn't have snap2 installed]

3. "I don't think there is any advantage to having snap2 try to rename snapshots to fill up 'holes' left by snapshot deletions"
I like to keep things orderly.
If the folder names were not = snap.1, snap.3, snap.7, but rather a series of dates which were in date order, then it wouldn't look "untidy".

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#80 Post by lstandish »

I guess we have different ideas about snapshot backups. I will rethink this, but for now I should should state my premises. I think my premises about snapshot backups represent "conventional wisdom" on this. (To a large extent, snap2 was developed based on these premises.)

In the first place, I think backups should be run on a regular basis, not sporadically. snap2 includes a backup scheduler to make automated, regular backups easy to do.

My cron-triggered automatic backup job is is run at 9:00 AM, and I don't mind that often only a few files are changed between one snapshot and the next. Sometimes no important files at all are changed, but I don't think that makes the backup unnecessary.

When I examine my snapshot backups, it is important to me to see that backups were run every day. If there is a day when no backup was run, I know that something "went wrong" that day. In my case, that usually happens when my computer is not on at 9:00 AM.

If I need to recover a file and I want the latest version, I look in yesterday's snapshot. The fact that yesterday's snapshot is there is my assurance that I am really getting the latest version of a file. (This is the "documentation" I spoke of.)

If I were to delete the last three days' snapshots because they did not contain important changes, then at file recovery time, how do I know that I'm getting the LATEST version? I would have to use a 3 or 4 day old snapshot and hope that it is really represents yesterday's state of my files, even though the snapshot date stamp is 3 or 4 days old.

For the same reason, I want to see that my "weekly" snapshots have date stamps about one week apart, etc.

The new log file viewer makes it very fast and easy to browse snapshots and see exactly what files were backed-up each time.

There is at the present time no way to add comments to a snapshot backup. I might add that, but as you can imagine it does not fit very well into the idea of regular, automated snapshot backups.

I will try to accommodate as far as possible and practical different ways to use snap2. I'm not saying that my premises on snapshot backups are the only "right" or useful ones. For example, snap2 was originally intended to back up data files, not an entire OS. I'll try to keep Operating System partition backup in mind.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Jim1911
Posts: 2460
Joined: Mon 19 May 2008, 20:39
Location: Texas, USA

#81 Post by Jim1911 »

Hi Istandish,
Hmmm, I never considered the scenario of the *source* partition not being mounted! To fix that, snap2 would have to be able to tell when a source directory was really deleted, and when the source partition is only not mounted. I'll try to make snap2 detect the unmounted partition case, or if that is not practical, I'll add a warning about unmounted source partitions to the docs.
Your idea of automatic detection is a much better choice than a warning. In my case, I have over 12 partitions and usually the only one mounted is the one that I'm working on, so it is very important that the partitions be mounted before running a backup. It would be great to have snap2 mount the necessary partitions, especially in the case of automated backups. I hope this won't be too difficult to achieve.
Cheers,
Jim

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#82 Post by lstandish »

I spent hours yesterday coding to detect when a directory is on a mounted partition, and I had to give up. I got it working for cases where the mount point is listed in fstab, but USB drives have mount points that just "appear" out of nowhere. The only way I can think of would be to allow the user to specify a device file and mount point for each source directory - or for each "backup set". However, I have not wanted to limit source directories for a backup set to a single partition.

(I don't think there is a problem with unmounted destination directory, if the "Allow creation of missing destination dir" option is unchecked.)

Jim1911
Posts: 2460
Joined: Mon 19 May 2008, 20:39
Location: Texas, USA

#83 Post by Jim1911 »

lstandish wrote:The only way I can think of would be to allow the user to specify a device file and mount point for each source directory - or for each "backup set". However, I have not wanted to limit source directories for a backup set to a single partition.
This would work well when setting up automated backups.

I've just tried snap2 on dpup484b4. It appears to install fine, however, it gives an error when trying a backup "Process exited abnormally with status 0. Press any key to close tab." Previously, I have been running it on Barry's 432 experimental version which has worked great.

Jim :)

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#84 Post by lstandish »

I have code written now to take an optional mount point and device file on the same line as the source directory, lke this:

# source directories
/mnt/sde1 DEVICE=/dev/sde1 MOUNTPT=/mnt/sde1

It will do the mount if necessary. After backup, a previously unmounted source partition is unmounted again. Does this look OK?
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Jim1911
Posts: 2460
Joined: Mon 19 May 2008, 20:39
Location: Texas, USA

#85 Post by Jim1911 »

lstandish wrote:It will do the mount if necessary. After backup, a previously unmounted source partition is unmounted again. Does this look OK?
Great! :D

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#86 Post by lstandish »

On the dpup484b4 error, make sure rsync is available (in $PATH), and that /tmp is writable. Does the snap2 GUI open OK, failing only upon "...Backup Now"?

Jim1911
Posts: 2460
Joined: Mon 19 May 2008, 20:39
Location: Texas, USA

#87 Post by Jim1911 »

lstandish wrote:On the dpup484b4 error, make sure rsync is available (in $PATH), and that /tmp is writable. Does the snap2 GUI open OK, failing only upon "...Backup Now"?
Yes, it opens fine and source and destinations set up. That was just FYIO, no need to pursue trying to get it to work on dpup since a new beta for dpup should be out soon. It works good on pups 431 and 432, I'll try it on Lighthouse which is based on 431.
Jim

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#88 Post by Sylvander »

@lstandish
Regarding your post above at 11:23 on Jan 13th...
1. Consider the following "thought experiment".

(a) Imagine I had configured snap2 to automatically make a snapshot every hour, on the hour.
24 snapshots a day, every day of every year.
That's a lot of snapshots, right?

(b) Now lets imagine that most of those snapshots happen to fall into groups of identical snapshots.
Perhaps there were differences only once a week, or once a month.
That seems to me to rather a lot of clutter.
Fairly pointlessly listing lots of identical snapshots.

(c) Now what if:
When a snapshot was being made...
If it was discovered to be identical to the previous snapshot...
Some alternative automatic action was taken by snap2. [But only if a tick-box was ticked]
Say the snapshot was aborted...
And perhaps some logfile was made that listed that the snapshot process had been completed, and found to be identical to such-and-such a snapshot.

(d) And all following snapshots that were found to be identical were similarly logged.

(e) Then when a snapshot was found to be no-longer-identical...
It would be completed and saved.

(f) So you end up with a series of snapshots that are DIFFERENT to each other.
Because they include at least ONE item that is not identical.
This would let the user instantly see WHICH snapshots were different to each other...
And also which snapshots were run, and found to be identical to those, so aborted.

(g) It would minimize the number of snapshots listed/saved.
Yet still allow the user to know which snapshot applied to a certain point in time.
And a user like myself, who wasn't interested in such things would also be happy.

2. You may consider this a crazy idea that would make snap2 much too complex. :(
But it would certainly meet my needs even better than now.
I'd be going "WOW, that's cool"! :D So FLEXIBLE! 8)

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#89 Post by 8-bit »

lstandish wrote:I spent hours yesterday coding to detect when a directory is on a mounted partition, and I had to give up. I got it working for cases where the mount point is listed in fstab, but USB drives have mount points that just "appear" out of nowhere. The only way I can think of would be to allow the user to specify a device file and mount point for each source directory - or for each "backup set". However, I have not wanted to limit source directories for a backup set to a single partition.

(I don't think there is a problem with unmounted destination directory, if the "Allow creation of missing destination dir" option is unchecked.)
Look at the code in "Hot Backup for frugal installs found Here.
It may shed some light on how to detect if source or destination is mounted.
The detection code works there.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#90 Post by lstandish »

Hi Sylvander,
Thank you for your thoughts on duplicate snapshots. What you suggest is possible, but after giving this some thought I realized that keeping identical snapshots around has some unexpected advantages, due to the discarding of snapshots that happens as snapshots are "rotated."

Snap2 does not keep every snapshot backup. It only keeps, for example, 'recent' snapshots up to the configured maximum. After that, it "thins them out". After the 'recent' snapshots reach the maximum, the oldest one is only 'promoted' to being a 'daily' snapshot if it is at least 24 hours newer than the newest daily snapshot. If not, it is discarded, no matter how much unique data it may have.

In a similar way, once you have reached the configured maximum of 'daily' backups, the oldest one is either 'promoted' to being a weekly backup (if it is 7 days newer than the newest weekly snapshot), or discarded. If you run snap2 once per day, only one out of 7 snapshots will survive to become a 'weekly' snapshot.

So, no matter how many times per day you run snap2, the limits you have set on recent, daily, weekly, and monthly backups will put a ceiling on the total number of snaphots that can exist. The out-of-the-box values for each of these add up to a total of 26 snapshot backups that can exist at one time. This is of course configurable, but I don't think many people would need to keep more than about 50 snapshots.

Now to explain the advantage of keeping duplicate snapshots. Suppose we have these snapshots:

daily.6 (2010-02-11)
daily.7 (2010-02-10)
daily.8 (2010-02-09)
weekly.1 (2010-02-04)

Daily.6 has no changes with respect to daily.7. But daily.7 has changes with respect to the daily.8.

Now, suppose we have snap2 configured to keep 8 daily snapshots. So the next time snap2 is run, daily.8 will be discarded, since it is too close in time to weekly.1. The next snapshot that is eligible to become a 'weekly' is daily.6

So, daily.8 and daily.7 are doomed to disappear in only 2 more days (assuming backups are done daily). But daily 6, which also holds the unique data in daily.8, will be promoted to 'weekly' and will survive for several more weeks.

Now, you would have deleted daily.6 because it had no unique data with respect to daily.7. By doing the manual deletion, you will have greatly shortened the lifespan of the data, due the way snapshot promotion/deletion works.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#91 Post by Sylvander »

@lstandish
Re your post of today, Sun 14th Feb at 13:23...
1.
(a) Thank you for listening and giving careful consideration to what I had to say.
On reading your post its quite difficult for me to fully understand exactly how all of that works.
I only roughly get the idea:
One of the daily backups being promoted to a weekly.
Then one of the weekly backups being promoted to a monthly.
I guess after some time you end up with a set of dailies, a set of weeklies, and a set of monthlies?
And new snapshots go in at the top, and old snapshots drop out at the bottom?

(b) Perhaps after using snap2 for some time it will begin to become clear to me.
Is there anything [especially visual] that would make it all seem simple?

2. New topic:
(a) Today I was trying something new.
Wanted to run snap2 within a Puppy loaded from a USB2 Flash Drive...
So that it wouldn't load anything from any partition on the internal HDD [no partitions auto-mounted].
This so the Puppy could be used for imaging [all of the internal HDD] as well as using snap2.
After various failures to get a good install of Twinkle & Puppy-4.31.1 I settled for using an existing install of Puppy-4.2.1 on a 1GB USB2 Flash Drive.
Installed snap2 successfully to that.
Had to do some of the installation of dirdiff manually [the PET failed], but it worked.
Also installed from the PPM:
tk-8.5.6-V1-DEV
tcl-8.5.6-V1-DEV
In the hope these might supply needed dependencies.
Both snap2 & dirdiff seemed to be working normally.

(b) Ran snap2 to make updated snapshots of:
sda1&sda5; sda3; sda6; sda7.
All of those took hours to complete, and seemed to be backing up ALL of the contents of the source partitions.
And yet, when I used dirdiff to look at sda7 & sda6...
And compare the latest snapshot with the previous snapshot...
The snapshots were identical for both sda7 & sda6.
Any idea what's going on there?
Problem with installing snap2 in Puppy-421?

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#92 Post by lstandish »

Sylvander,

(a) I'll try to explain snap2 backup rotation through a directory listing. (Following is from my website http://www.linuxbackups.org/node/29)

"Over time, the program rotates the backups to create a directory structure like this:

lloyd@debiandesk:/media/BACKUP8GIG/snapbackups$ ls -l
total 96
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-23 21:12 daily.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-22 08:35 daily.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-21 08:35 daily.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-18 08:35 daily.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-17 08:35 daily.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-16 08:35 daily.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-14 08:35 daily.7
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-13 08:35 daily.8
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-12 08:35 daily.9
drwxr-xr-x 3 lloyd lloyd 4096 2009-10-07 23:06 mirror
drwxr-xr-x 4 lloyd lloyd 4096 2009-05-25 08:35 monthly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-04-19 08:35 monthly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-03-19 08:35 monthly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-02-08 22:24 monthly.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-15 22:36 recent.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-13 20:59 recent.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-07 23:04 recent.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-09-07 13:51 recent.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-27 08:35 recent.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-06 08:35 weekly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-30 08:35 weekly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-22 08:35 weekly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-14 18:00 weekly.4"

My configuration specified maximum 6 'recent' snapshots, 9 'dailies', 4 'weeklies' and 4 'monthlies'. Most of the backups were done automatically, daily at 8:35 AM.

As you can see, snap2 deletes a whole lot of snapshots in order to get them 'spaced out' as 'weekly' and 'monthly' snapshots. For example, only one daily snapshot per week is 'promoted' to a weekly snapshot. All the other 'dailies' are discarded at rotation time.

Since many snapshots will be deleted, it is advantageous to allow duplicate shapshots to exist, so your data will live longer (unless the duplicates are 'recent' snapshots that were done the same day).

(b) Check the backup logfile for the most recent snapshots. At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was. Also, if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing. Only a few or no files should be listed if there were few or no changes between the 2 snapshots.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#93 Post by Sylvander »

1. " At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was"
There was indeed a "hard link reference" found as follows:
"Backup run on Sun Feb 14 19:56:28 GMT 2010
Using settings from /root/.snap2/sda7.set/settings
Using rsync compression
Snapshot hardlink reference: recent.1
"

2. "if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing"
(a) Yes, a great long list of files in the log, and yet that snapshot is found by dirdiff to be identical to the previous snapshot.
Any idea why the long backup list?

(b) I'll try making yet another snapshot of [for example] sda7, from the copy of snap2 within Twinkle, and see what happens.
Just did that; it completed within about 3 seconds, nothing listed. :D

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#94 Post by lstandish »

Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#95 Post by Sylvander »

1. "Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?"
(a) I believe there was not. [What's a mount point?]
e.g. dirdiff said that the latest snapshot [for sda7 for example] was identical to the previous snapshot.
And the path to the source was identical in each.
And the path to the destination was identical in each.

(b) Decided to by-pass that problem as follows:
I copied the pupsave->[plus companion files] for Twinkle...
To a different 1GB Flash Drive...
And booted the Twinkle live CD-RW, which sought and found and used the pupsave on sdb1.
Twinkle is using the pupsave on sdb1 [seen at /mnt/home].
That had the desired effect; no partitions on the internal HDD are mounted.
The backup destination partition sdc1 is on the external USB HDD.
I'm in the process of deleting the existing snapshots for all sources, and making new snapshots.

Post Reply