snap2 rotating snapshot backups for Puppy

[Sylvander]

@lstandish
Re your post of today, Sun 14th Feb at 13:23...
1.
(a) Thank you for listening and giving careful consideration to what I had to say.
On reading your post its quite difficult for me to fully understand exactly how all of that works.
I only roughly get the idea:
One of the daily backups being promoted to a weekly.
Then one of the weekly backups being promoted to a monthly.
I guess after some time you end up with a set of dailies, a set of weeklies, and a set of monthlies?
And new snapshots go in at the top, and old snapshots drop out at the bottom?

(b) Perhaps after using snap2 for some time it will begin to become clear to me.
Is there anything [especially visual] that would make it all seem simple?

2. New topic:
(a) Today I was trying something new.
Wanted to run snap2 within a Puppy loaded from a USB2 Flash Drive...
So that it wouldn't load anything from any partition on the internal HDD [no partitions auto-mounted].
This so the Puppy could be used for imaging [all of the internal HDD] as well as using snap2.
After various failures to get a good install of Twinkle & Puppy-4.31.1 I settled for using an existing install of Puppy-4.2.1 on a 1GB USB2 Flash Drive.
Installed snap2 successfully to that.
Had to do some of the installation of dirdiff manually [the PET failed], but it worked.
Also installed from the PPM:
tk-8.5.6-V1-DEV
tcl-8.5.6-V1-DEV
In the hope these might supply needed dependencies.
Both snap2 & dirdiff seemed to be working normally.

(b) Ran snap2 to make updated snapshots of:
sda1&sda5; sda3; sda6; sda7.
All of those took hours to complete, and seemed to be backing up ALL of the contents of the source partitions.
And yet, when I used dirdiff to look at sda7 & sda6...
And compare the latest snapshot with the previous snapshot...
The snapshots were identical for both sda7 & sda6.
Any idea what's going on there?
Problem with installing snap2 in Puppy-421?

[lstandish]

Sylvander,

(a) I'll try to explain snap2 backup rotation through a directory listing. (Following is from my website http://www.linuxbackups.org/node/29)

"Over time, the program rotates the backups to create a directory structure like this:

lloyd@debiandesk:/media/BACKUP8GIG/snapbackups$ ls -l
total 96
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-23 21:12 daily.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-22 08:35 daily.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-21 08:35 daily.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-18 08:35 daily.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-17 08:35 daily.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-16 08:35 daily.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-14 08:35 daily.7
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-13 08:35 daily.8
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-12 08:35 daily.9
drwxr-xr-x 3 lloyd lloyd 4096 2009-10-07 23:06 mirror
drwxr-xr-x 4 lloyd lloyd 4096 2009-05-25 08:35 monthly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-04-19 08:35 monthly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-03-19 08:35 monthly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-02-08 22:24 monthly.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-15 22:36 recent.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-13 20:59 recent.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-07 23:04 recent.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-09-07 13:51 recent.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-27 08:35 recent.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-06 08:35 weekly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-30 08:35 weekly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-22 08:35 weekly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-14 18:00 weekly.4"

My configuration specified maximum 6 'recent' snapshots, 9 'dailies', 4 'weeklies' and 4 'monthlies'. Most of the backups were done automatically, daily at 8:35 AM.

As you can see, snap2 deletes a whole lot of snapshots in order to get them 'spaced out' as 'weekly' and 'monthly' snapshots. For example, only one daily snapshot per week is 'promoted' to a weekly snapshot. All the other 'dailies' are discarded at rotation time.

Since many snapshots will be deleted, it is advantageous to allow duplicate shapshots to exist, so your data will live longer (unless the duplicates are 'recent' snapshots that were done the same day).

(b) Check the backup logfile for the most recent snapshots. At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was. Also, if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing. Only a few or no files should be listed if there were few or no changes between the 2 snapshots.

[Sylvander]

1. " At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was"
There was indeed a "hard link reference" found as follows:
"Backup run on Sun Feb 14 19:56:28 GMT 2010
Using settings from /root/.snap2/sda7.set/settings
Using rsync compression
Snapshot hardlink reference: recent.1"

2. "if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing"
(a) Yes, a great long list of files in the log, and yet that snapshot is found by dirdiff to be identical to the previous snapshot.
Any idea why the long backup list?

(b) I'll try making yet another snapshot of [for example] sda7, from the copy of snap2 within Twinkle, and see what happens.
Just did that; it completed within about 3 seconds, nothing listed.

[lstandish]

Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?

[Sylvander]

1. "Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?"
(a) I believe there was not. [What's a mount point?]
e.g. dirdiff said that the latest snapshot [for sda7 for example] was identical to the previous snapshot.
And the path to the source was identical in each.
And the path to the destination was identical in each.

(b) Decided to by-pass that problem as follows:
I copied the pupsave->[plus companion files] for Twinkle...
To a different 1GB Flash Drive...
And booted the Twinkle live CD-RW, which sought and found and used the pupsave on sdb1.
Twinkle is using the pupsave on sdb1 [seen at /mnt/home].
That had the desired effect; no partitions on the internal HDD are mounted.
The backup destination partition sdc1 is on the external USB HDD.
I'm in the process of deleting the existing snapshots for all sources, and making new snapshots.

[Sylvander]

PROBLEM?

1. Making yet another manual snapshot of /mnt/sda7 to /mnt/sdc1/backups/snap2/sda7.
ALL partitions are mounted.
Using Twinkle-421 [built on Puppy-431] with pupsave+companion files recovered to a repartitioned & reformatted Flash Drive.
Twinkle & snap2 seem to be working well, except for this "problem".

2. Prior to commencement, there existed recent.1, recent.2, recent.3

3. The new snapshot [recent.0] was using as its reference point, recent.2 rather than recent.1

4. sda7 had probably not been changed, or possibly a very few files, and yet as I type it appears to be making a snapshot of ALL OF SDA7.
How come?
Now completed, and dirdiff shows recent.1 & recent.2 are identical.

5. To repeat: both source and destination partitions are mounted.
sdc1 has had its partition file system checked/fixed using GParted.
I'll do the same for sda7 ASAP.

[lstandish]

For the actual backup, snap2 calls rsync. I don't think rsync has any major bugs.

If snap2 used recent.2 rather than recent.1 for the hard link reference, it must not have found recent.1

If recent.2 is newer than recent.1, snap2 will detect that and will refuse to run. I don't think there is any problem in the snap2 code for finding the newest snapshot (for the hard link reference)

Remember that during the rsync run, the new snapshot is recent.0, but as soon as it is finished it will rename all the snapshots and recent.0 will become recent.1.

Using the information you sent, I can't help. You could try sending me a detailed listing of your destination dir ('ls -l') just before you run the backup.

Or, if you can send me a source and destination directory structure that reproduces the problem, then I could help. Of course, that could not be entire partitions.

I suggest you set up a simple test backup of a small directory and see if you can reproduce the trouble. Then you could zip the whole thing up and send it to me.

[Sylvander]

SUCCESS?!

1. Rebooted a couple of time into various Puppies.
Rebooted into Twinkle.

2. Ran snap2 yet again, and made more snapshots of...
sda1&sda5; sda3; sda6; sda7.
All behaved EXACTLY as they aught.
Said that there were no changes to sda1&sda5; sda6; sda7. [There should have been no changes]
Said there were changes to the contents of sda3, which is as expected.

3. So all seems to be OK.
Don't know why the previous problem; will keep a watch for a repeat, otherwise do nothing.

[Sylvander]

Problem?

1. Brought yet another Puppy into use [Lighthouse-4.43f built on Puppy-4.31.1 I believe].

2. Saved the pupsave+companions on an 8GB Flash Drive.
Decided to use this method so that no partitions on the internal HDD would be auto-mounted, and therefore the Puppy would be free to do anything/whatever with the internal HDD.

3. Lighthouse auto-mounts ALL partitions it can see, which is either a convenience or a pest.
Doesn't see my external USB HDD, unless I power on AFTER Lighthouse gets to the desktop, so I normally do that when needed, and then mount it [/mnt/sdc1].

4. Installed lots of programs including snap2 and dirdiff.

5. Ran snap2 to make yet another snapshot of sda1+sda5.
[After having made all the necessary configurations, and mounting sdc1]
snap2 began making a total backup of ALL of the contents of sda1, so I halted it by closing the window, then the program, then deleted recent.0
How about providing and "ABORT" button on the window?

6. Looks like snap2 does the above IF run from a different Puppy.
Is it possible to make it behave normally, even if run from different Puppies?
Otherwise I'll be forced to always make snap2 snapshots from the same Puppy, and what do I do when/if it fails?

7. Will now try running snap2 from Twinkle and see if it behaves as it aught, then report back.

[Sylvander]

The above problem is only happening when working with FAT32 source partitions.

When making a snapshot of the ext3 sda3 partition, all's well.

i.e. Everything works OK in Twinkle.
The previous snapshots were made in Twinkle, and the new snapshots of unchanged source partitions show as unchanged in Twinkle, but NOT in Lighthouse [EXCEPT for ext3 sda3].

[lstandish]

I thought snap2 would work OK (in snapshot mode) as long as the destination partition is a Linux partition. I need to check for myself what happens when the source partition is FAT32.

There should be no problem running snap2 from different Puppies to backup the same source folders to the same storage partition. The mount point of the destination folder should not make a difference, either. That is, if you mount your storage partition at /mnt/backups/ on one run and at /tmp/12345.temp/ the next time, that should not make any difference, even though that changes the overall path to the backups.

I am working on automount for both source and destination folders, on both local and remote storage media. However, I have not been able to test this enough to make a release.

[Sylvander]

EXPERIMENT-1...
On sda1+sda5:
[Windows partition + its data partition (both FAT32)]

1. Booted into Lighthouse, powered-on the ext HDD and mounted the partition [all other partitions already mounted].

2. Deleted all snapshots of sda1+sda5.

3. Made a new snapshot of the partition pair.

4. Waited overnight and [this morning] made a new snapshot of the pair.
This done from within Lighthouse.
snap2 recognized that the partition contents were unchanged, and didn't list any folders/files for backup.

5. Deleted the newly made and identical recent.1, and renamed recent.2 to recent.1

6. Re-booted into Twinkle and ran snap2 and made fresh snapshots of sda1+sda5.
snap2shell began listing files being copied.
Terminated the copy, and deleted recent.0

7. Conclusions anyone?

[Sylvander]

Continuing EXPERIMENT-1:

8. Booted back into Lighthouse.

9. Ran yet another snap2 snapshot on sda1+sda5.

10. As expected, it detected there had been no change to the contents of these partitions, so listed no changed files.

11. Deleted recent.1 and renamed recent.2 to recent.1

[lstandish]

Were the full source paths of the mounted partitions exactly the same on both Lighthouse and Twinkle?

[Sylvander]

Yes, and the destination too was good.
And I checked the configurations for those, and they were good also.
And the snapshot by Twinkle began to be saved as recent.0 right above recent.1

The only problem seems to be that if I change to a different Puppy->[snap2], it acts like it doesn't know of the files saved by the previous Puppy->[snap2].

Does snap2 save [somewhere in the Puppy filesystem in the pupsave file] a record of the files saved in the previous snapshot?
In which case, when I switch to a different Puppy, then its snap2 wouldn't know of the previous list of files in the snapshot made by the other Puppy->[snap2].

I'm using Lighthouse->[snap2] and gradually working through from sda1+sda5 to sda3 to sda6->[now completed]...
Deleted all the existing snapshots, and made a new snapshot.
Now, I bet when I run snap2 new snapshot...
All will be well...
But if I use Twinkle->[snap2], it will act like the previous snapshot doesn't exist, and backup ALL of the contents of the partitions all over again.

[keo01]

hello lstandish, your software seems to be what I was looking for...

I've one question, my idea is to "synchronize" files beetween desktop and laptop trough a backup media (USB, file server, etc).

The way I want to do this, is:
I make a change to a file (or a group of files) in PC1.
Backup it under request (it can be done with your program, right?) to a backup media (usb flash drive, NAS, etc).

Then when I want to make a change in the latest version of a file (either in PC1 or PC2 or elsewhere) , check the backup and get the latest version.
Save the changes and backup it again, so when I want work in the last version I can repeat the process.

Could this be possible? I don't have clear that I can backup the same file from two (or more) computers to only one snapshot(I mean, that don't be one snapshot for this file for PC1, and other snapshot for this file for PC2, etc).

Thanks.

[lstandish]

Sorry, I was dealing with some urgent sys-admin work, and haven't visited this forum for a while. For some reason I did not get (or see) the emailed notice of a new post here.

To Sylvander and keo01:
I think that both of you are needing to back up from more than one OS to the same backup media. In my previous posts, I said I thought this was possible, but now I think that you will not be able to do this, at least not with the current snap2 version. It may already work for mirror backups, which do not involve hard links, but I'm not sure.

For snapshot type backups, which rely on hard links, I think any attempt to back up from 2 different operating systems to the same backup media will fail due to the fact the destination mount point directory on each OS will have different attributes. I believe I know of a workaround for this, already built into rsync, but I am too busy right now to investigate.

I'll post here later when I have had time to study this and perhaps make a new snap2 release that can ignore the source/destination mount directories.

[Sylvander]

@lstandish

1. Whew!
Nice to know this phenomenon is real; not just my mind playing tricks.

Since last reporting, I've been making all snapshots from within Lighthouse-4.43f
Great Puppy, GREAT snap2.

The layout of snap2 seems a little strange at first encounter...
All those buttons and tabs together in one view...
But as I become accustomed to it, it's easy to use, and things are easy to get to because they're all available in that one view.

And now, of course, a snapshot typically takes only a minute or few to complete.

2. Only improvement I can think of is to be able to also do restores from within snap2.

At the moment I would use Xfe.
Delete the whole of the contents of a partition...
Then "COPY-to..." from the snapshot folder holding [the backup copy of] all of the contents of that partition...
Back to the partition.

That involves a LOT of un-necessary deletions and restores...
Of folders and files that are identical in both locations.

I imagine if snap2 were doing that job it would only copy over where there were differences.

[lstandish]

Glad snap2 is working out for you!

I have no plans to give snap2 file manager abilities, to use it for restores. For that, a 2-pane file manager works great. Just browse to the folders/files you want in one pane, and copy them to the destination in the other pane. I use mc (midnight commander) for this, available for Puppy (see attachment - this is a old package - not a pet - with an install script but it works). It has loads of cool features, and even allows you to have a remote server connection in one pane and the local filesystem in the other pane, to browse files, mark them, and restore from a remote server.

As I said, I hope to soon fix snap2 to optionally ignore mount points for both source and destination, to avoid trouble with using it between different OS's.

[lstandish]

It has been a relatively long time since the last release of snap2, which was 4.1. I have been extremely busy with my small business, which involves harvesting and processing a wild tropical forest fruit called "carao".

In my last post, I mentioned my intention to allow snap2 to auto-mount partitions that are not mounted at backup time, and I made extensive code changes to allow auto-mount of both source and target partitions, for backup to both local and remote backup media.

Unfortunately, I ran into some issues with this, and ran out of free time. I therefore decided to put the auto-mount development on "hold" (sorry, Jim1911).

Meanwhile, I have some tips on how to use snap2 when backing up from more than one machine to the same backup storage. I understand that this is what Sylvander has tried to do. The following explanation is based on my theoretical understanding of how rsync works. That is, it is untested.

Suppose you specify a source path of /mnt/sda1 with backup storage as /mnt/mybackups. If you try to back up /mnt/sda1 from a distinct Puppy installation compared to the one used for the original backup, rsync will probably see a changed time stamp on the /mnt and /mnt/sda1 directories (which are distinct directories for each Puppy filesystem), and will therefore create a new backup copy rather than linking identical files together with hard links.

To avoid this for this example, tell rsync to ignore the /mnt/sda1 portion of the source path. This is done by adding a /./ to the end of the path. The source path becomes /mnt/sda1/./

In this case, rsync should not pay attention to the timestamp of directory /mnt nor directory /mnt/sda1. Moreover, it will not recreate this portion of the directory structure on the backup storage drive. Therefore, if the files under /mnt/sda1 are identical to files at the time of the previous backup, they should be linked together with hard links, even if backups are done from different installations of Puppy.