snap2 rotating snapshot backups for Puppy

[Jim1911]

Hi lstandish,

Just tried your new snap2-4 (version 4.0beta). Impressive, however, I got more files than I expected on the snapshot backup.

I mirrored a 26.54GB partition with 18.69GB used to a 31.25GB partition. The mirror operation worked fine. Then I made one file change of a few KB to the source partition and then selected 'Snapshot Backup Now' using the GUI. It created a recent.1 directory in which I expected to just find the changed information of only a few KB, however it created another entire partition backup in the recent.1 directory, therefore the entire 31.25GB drive partition was filled. I then deleted the recent.1 backup and the drive now shows 18.73GB used for the mirror which seems about right.

Am I misunderstanding what should take place with the snapshot?

Your dirdiff Graphical Directory Comparision utility is really nice too.

Thank you for all the hard work that you've put into this. Looking forward to further tests.
Jim

[Sylvander]

With reference to my previous post, here's an example:

[lstandish]

Jim1911,

Each snapshot backup is supposed to look like a full backup. It is a "snapshot" of the state of your files at a given date. However, the storage space used for each snapshot is essentially only that of changed and modified files, thanks to the use of hard links for all files that are unchanged between snapshot backups.

Many file size reporting utilities are fooled by hard links and will incorrectly report the space used as though each snapshot were a separate copy. I suspect this is what happened in your case.

If this happened, you will find that the log report of the second backup (recent.1 - look on the second tab of the snap2 GUI for log display) only shows that one file was backed up. (There may have been a few others too that the system automatically updated or created.)

Also, running 'du -cbs backuproot' in a terminal (where 'backuproot' is the top level directory for your backup storage) will show the correct amount of storage space used by your backups.

To illustrate how space reporting utilities are fooled, below is a screenshot of my Debian 'disk properties' for my 8 gig USB flash drive, which I used for months as snapshot backup storage. It appears to hold 30.2 gigs! However, look at the (correct) "Free disk space" line at the bottom.

Sylvander,
Yes, recent.1 must ALWAYS be newer than recent.2 (etc.). If you mess with the date stamps so that this is no longer true, backups can no longer be done.

I'll try to find a more descriptive button name for the logfile/reporting/backup deletion utilities. Thanks very much for your suggestions and feedback!

[Sylvander]

Don't know if I mentioned or not, but...

Prompted by me using your snap2 program to make snapshots...
It then occurred to me that I needed to make an image of the whole of the internal 80GB HDD.

That Pudd image was 43GB, and I had to delete some of the snapshots to make space.
So I bought a new Allcam SATA to USB external enclosure, plus a 1TB SATA HDD.

Everything is up and working now.
Made new snap2 snapshots of sda1&sda5, sda3, sda6, sda7.
All very easy/routine to do.

Now making images using the FREE "Seagate Disk Wizard - Bootable Media" [CD-RW].
This has USB drivers, so my USB mouse is functioning and there's a cursor, and it can see the destination partition on the external USB HDD.
It took ONLY TWO HOURS to image my 80GB internal HDD.
I'd prefer to use some program within Puppy to do the job, but...
Pudd took 20 hours to do the same job.
Plus there were difficulties due to mounted partitions [Linux swap & pupsave], so I had to delete the Linux swap, and use the...
puppy pfix=ram command.
Much easier just to use the Seagate disk.

My FREE "Acronis True Image 11" "Emergency Disk" wasn't able to see any USB devices, so is useless to me.

[lstandish]

Version 4.0 does not show deleted files correctly (it also shows changed files). This is fixed in version 4.1, which I will upload soon (and announce here).

[Jim1911]

Also, running 'du -cbs backuproot' in a terminal (where 'backuproot' is the top level directory for your backup storage) will show the correct amount of storage space used by your backups.

Sorry, took so long to reply, but as you can see from the output below, there is a duplicate mirror being created by the snapshot created by clicking on "Snapshot Backup Now". The only difference should be an OpenOffice3.2-sfs4_431.sfs which was downloaded and added to the mirror location. (your great little dirdiff utility confirms the difference) Bottom entry is after deletion of the recent.1 backup.

Code: Select all

# cd /mnt/sdc2/
# ls
kids  mirror  MU  recent.1  upup432
# du -cbs
61012170345     .
61012170345     total
# du -cbs
36961235181     .
36961235181     total
# 

I'll delete the contents and start over when your next release is posted. Thank you for all the hard work that you are devoting to this project.
Cheers,
Jim

[lstandish]

Hi Jim,

You configured and ran both a mirror-type backup AND a snapshot-type backup. These are 2 different backup types. They are configured separately and are run by clicking on separate buttons. If you make a mirror backup and also a snapshot backup you will have 2 separate copies taking up full disk space in each, as you observed.

Use a mirror-type backup when you only need a SINGLE copy of your files. Use snapshot backups when you want to be able to obtain past versions of files. Do not use both for the same source directories.

If you have made a mirror backup, make a few changes to the source, and run it again, only the changed/added files will be backed up - it will be very fast.

If you have made a snapshot backup (which on first run results in a directory called recent.1), make a few changes to the source, and then do another snapshot backup, you will get what appears to be a whole new copy of your files, but in fact only the changed/added files will actually take up disk space - it will ALSO be very fast.

After the second run you will have, then, a 'recent.1' and a 'recent.2'. recent.1 will be the most recent snapshot. recent.2 will be the previous one.

recent.1 is always an exact 'snapshot' of the source files at the time of the last backup. For that reason, it will be the same as a mirror-type backup specifying the same source directories. However, unlike a mirror backup, with snapshot backups you also have previous snapshots of your files, allowing access to files you have deleted from the source, and previous versions of files.

On dirdiff, sorry for the confusion: I am not the author of that. I just made the Puppy package.

The most recent version of snap2 (4.0) will also give information on files deleted from one snapshot to the next, but if you want to try that, you should wait for me to release version 4.1, since 4.0 is also reporting changed files.

[Jim1911]

Am I misunderstanding what should take place with the snapshot?

As stated in my first post, I was, thanks for clarifying. Suggest that you include a prominent "CAUTION: Before proceeding make sure that your source and destination partitions are mounted." on your opening screen. Thanks for packaging dirdiff, it's a very handy little utility.

Looking forward to your next release,
Jim

[lstandish]

I agree some sort of prominent warning is necessary. For the moment I have added the following to the "3 Easy Steps" info at the top of the GUI:

"On the DIRECTORIES TO BACK UP tab, add folders to EITHER 'Snapshot Backup Paths' OR 'Mirror Backup Paths' (Do not use both for the same source folders)"

I am open to suggestions on how to clear up the snapshot vs mirror backup confusion. Of course, I try to avoid overwhelming the user with wordy explanations.

Thanks very much for your feedback on this. All feedback helps a developer in the struggle for user-friendliness.

[8-bit]

What would it take to add some code to check if the destination partition is mounted?
I ask this because one can write to /mnt/[partition or drive] of an unmounted partition as /mnt/[partition or drive] is just a directory used when mounting a partition or drive.
In the case of the destination being unmounted, the backup will appear to work, but it will not go to the destination as expected.
Do I make any sense?

[Jim1911]

Hi lstandish,

I am open to suggestions on how to clear up the snapshot vs mirror backup confusion. Of course, I try to avoid overwhelming the user with wordy explanations.

Consider just adding some of the comments that you provided me above to your nice Help file, like the bold faced markups below:

BACKUP METHODS

This program does backups of sets of source directories using rsync. There are
2 types of backups:
1. mirror
2. snapshot
NOTE: Select only one type of backup for each source directory.

'MIRROR' TYPE BACKUPS:
A mirror backup is just a single copy of the files in the source directories. Selecting "Mirror Backup Now" the first time will create a mirror backup of the source directory. Subsequent mirror backups will create a new mirror backup that reflects only the current state of the source directory. . . . .

'SNAPSHOT' TYPE BACKUPS:
This method is used to keep a series of backups, to allow recovery of files to
any of several different points in time. However, unlike a mirror backup, with snapshot backups you also have previous snapshots of your files, allowing access to files you have deleted from the source, and previous versions of files. A snapshot backup on first run results in a directory called recent.1. Subsequent snapshots will result in a 'recent.1' and a 'recent.2'. Recent.1 will be the most recent snapshot. recent.2 will be the previous one. . . . .
Cheers,
Jim

[lstandish]

OK, I'll use most of your wording, but I think saying that "...subsequent mirror backups will create a new mirror backup..." might be confusing. There is in fact only one mirror backup which is updated each time the backup is done. snapshots, on the other hand, could each be said to be a "new backup."

Anyway, I'll try to fix up the docs and keep it simple and clear. Thanks for the suggestions!

[lstandish]

8-bit, you wrote,

What would it take to add some code to check if the destination partition is mounted?
I ask this because one can write to /mnt/[partition or drive] of an unmounted partition as /mnt/[partition or drive] is just a directory used when mounting a partition or drive.
In the case of the destination being unmounted, the backup will appear to work, but it will not go to the destination as expected.
Do I make any sense?

I used to have exactly the problem you describe when I backed up to my USB drive. That's why I created the option "Create missing backup storage dir" on the ADVANCED OPTIONS tab. Uncheck it to avoid the problem you described. (Hint: hover your mouse over the advanced options to see the intended use.)

[lstandish]

snap2 4.1beta has been released.
Release notes: http://www.linuxbackups.org/node/36

[Sylvander]

@lstandish

Trying snap2 4.1beta:
1. It seems to be working well.

2. When I make recent.2 and the report and log show no differences, and I check and confirm that using dirdiff...
I use "View backup logs->Delete snapshot" [I don't want to keep a snapshot with no differences to the previous snapshot], and when done, recent.2 isn't renamed back to its previous name of recent.1
Wouldn't it be best to have that done?
I can do it manually, but wouldn't it be best done automatically by snap2?

3. "Report Deleted Files" is working well.

4. Don't see the "year/month/date-hour:min" being added to the recent.x folder name.
You said you were going to do that.

[Jim1911]

Hi lstandish,

It's working well.

One issue, there still isn't a prominent caution ie., BEFORE PROCEEDING, MAKE SURE THAT THE SOURCE AND DESTINATIONS ARE MOUNTED, otherwise backups will be deleted. I tested this and sure enough, with a partition unmounted, a previous Mirror backup Is deleted very quickly. This is a necessary reminder for the first time user and a good reminder for forgetful old timers like myself.

You have really improved the content of the help file.

Thank you for this fine utility program,
Jim

[8-bit]

Jim1911,
I have some thing for you to try.
Say your destination is set as /mnt/sda3.
Also let us say you forgot to mount /dev/sda3
After doing a backup/snapsot, and before mounting /dev/sda3
check /mnt/sda3 by opening that directory and see if the backup file is there.
That is something I was trying to express in a previous post.
That before a backup is done, the destination should be checked to see if it is mounted.
I have as a test copied files to an unmounted drive, NOT USING /dev/[drive/partition] but instead, using /mnt/[drive/partition].

The files copied fine to the unmounted /mnt/drive directory.
But they would disappear on mounting that drive.

[lstandish]

Sylvander,
Part of the idea of a snapshot backup is to document the state of the files on a given date. If you delete the snapshot just because it is the same as a previous snapshot, you lose that documentation. The only reason to delete it would be, I think, to save disk space. Yet, the overhead (extra disk space) used by an identical snapshot is very small (only the size of the hard links).

In sum, I don't recommend deletion of identical snapshots, unless you are very low on backup storage space. snap2's built in support for deleting snapshots is intended mainly to be used when a backup was interrupted (and therefore incomplete).

I don't think there is any advantage to having snap2 try to rename snapshots to fill up 'holes' left by snapshot deletions.

Sorry, I have not yet had time to get to study what will be involved to include the date-time stamp in the snapshot directory name.

Jim1911,
Hmmm, I never considered the scenario of the *source* partition not being mounted! To fix that, snap2 would have to be able to tell when a source directory was really deleted, and when the source partition is only not mounted. I'll try to make snap2 detect the unmounted partition case, or if that is not practical, I'll add a warning about unmounted source partitions to the docs.

8-bit,
I see your problem: if /mnt/sda3 is specified as the backup storage location, the directory /mnt/sda3 will exist whether or not the partition is mounted. That means the snap2 setting to *not* allow creation of a missing backup storage directory will have no effect, and backup will proceed.

To prevent snap2 from proceeding with a backup when the destination partition is not mounted, specify a backup storage folder (on the BACKUP STORAGE tab) like this one: /mnt/sda3/mybackups Now, when sda3 is not mounted, /mnt/sda3/mybackups will not exist, and if the 'Advanced' setting to allow creation of missing backup storage folder is unchecked, then snap2 will not do the backup.

(You can move your backups in /mnt/sda3/ to /mnt/sda3/mybackups/ without having to actually move data, so it will be fast. If you do this remember to update the backup storage directory in snap2.)

[Sylvander]

1. "Part of the idea of a snapshot backup is to document the state of the files on a given date"
Can't think why I'd be interested in "the state of the files on a given date".
That's not something that's useful to me.
What I really want is a series of snapshots of [the contents of partitions? in] DIFFERENT STATES, so I can perhaps restore one of those to get back to that state...
Or perhaps to recover a lost file, or [less likely] one or more files in a particular state.

2. "The only reason to delete it would be, I think, to save disk space. Yet, the overhead (extra disk space) used by an identical snapshot is very small"
(a) Imagine making a great list of snapshots, that are all [or perhaps only nearly all] IDENTICAL.
But you don't know [unless you expend time and effort], which are identical to which.
For my purposes, those identical snapshots are only clutter getting in the way.
[Perhaps there could be a tick-box to not save a snapshot that is identical to the previous snapshot? = CHOICE]
It's little consolation to know that at least they don't use much "real" space.
Which raises another important point as follows below:

(b) If I'm looking for a suitable snapshot to restore the way [an operating system? on] a partition used to be, when it worked a certain way [rather well?]...
I need something...
Some information about the snapshot...
That tells me what I need to know.
Does snap2 allow a user to make notes and associate them with a snapshot?
[Can't check here; this OS doesn't have snap2 installed]

3. "I don't think there is any advantage to having snap2 try to rename snapshots to fill up 'holes' left by snapshot deletions"
I like to keep things orderly.
If the folder names were not = snap.1, snap.3, snap.7, but rather a series of dates which were in date order, then it wouldn't look "untidy".

[lstandish]

I guess we have different ideas about snapshot backups. I will rethink this, but for now I should should state my premises. I think my premises about snapshot backups represent "conventional wisdom" on this. (To a large extent, snap2 was developed based on these premises.)

In the first place, I think backups should be run on a regular basis, not sporadically. snap2 includes a backup scheduler to make automated, regular backups easy to do.

My cron-triggered automatic backup job is is run at 9:00 AM, and I don't mind that often only a few files are changed between one snapshot and the next. Sometimes no important files at all are changed, but I don't think that makes the backup unnecessary.

When I examine my snapshot backups, it is important to me to see that backups were run every day. If there is a day when no backup was run, I know that something "went wrong" that day. In my case, that usually happens when my computer is not on at 9:00 AM.

If I need to recover a file and I want the latest version, I look in yesterday's snapshot. The fact that yesterday's snapshot is there is my assurance that I am really getting the latest version of a file. (This is the "documentation" I spoke of.)

If I were to delete the last three days' snapshots because they did not contain important changes, then at file recovery time, how do I know that I'm getting the LATEST version? I would have to use a 3 or 4 day old snapshot and hope that it is really represents yesterday's state of my files, even though the snapshot date stamp is 3 or 4 days old.

For the same reason, I want to see that my "weekly" snapshots have date stamps about one week apart, etc.

The new log file viewer makes it very fast and easy to browse snapshots and see exactly what files were backed-up each time.

There is at the present time no way to add comments to a snapshot backup. I might add that, but as you can imagine it does not fit very well into the idea of regular, automated snapshot backups.

I will try to accommodate as far as possible and practical different ways to use snap2. I'm not saying that my premises on snapshot backups are the only "right" or useful ones. For example, snap2 was originally intended to back up data files, not an entire OS. I'll try to keep Operating System partition backup in mind.