snap2 rotating snapshot backups for Puppy

Miscellaneous tools
Message
Author
User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#31 Post by lstandish »

Sylvander,
The (optional) logfile stored with each snapshot backup indicates which files were backed up (the differences between that backup and the next oldest.) I do think it would be good to allow viewing the latest backup log (whether stored locally or on a remote machine) in a snap2 gtkdialog window.

For more in-depth information, I think the user should open the storage directory and browse through the log files in each snapshot directory.

I think these snapshot logfiles should normally give you what you want to know about a backup event. It's true they don't tell you which files were NOT backed up compared to the previous backup. As pointed out previously, to provide that information snap2 would have to create the hard links via cp rather than via rsync, and this would introduce the ownership/permissions bug (old snapshots can acquire permissions/ownerships of newer snapshots).

If you want more in-depth info about the differences between snapshots, you could run a directory comparison tool such as Krusader's "Synchronize Directories." That's non-Puppy, of course. It's really first-class.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#32 Post by lstandish »

Sylvander, I found a great graphical directory comparison utility for you (called dirdiff), and make a pet package out of it (attached). You can compare up to 5 directories at once. It needs the Puppy tk/tcl libraries. Since they are listed as dependencies, they should be installed automatically.

Here's the Debian package description:
"Dirdiff can handle up to 5 trees. It displays a main window with a
list of the files which are different between the trees, with colored
squares to indicate the relative ages of the versions. A menu allows you to display the differences between any two of the versions in another window. Another menu allows you to copy the file from one tree to another."

I think I'll put it in the snap2 download area. (For Puppy 4.3.1)
Attachments
dirdiff-2.1-4fixed2.pet
(34.14 KiB) Downloaded 1412 times
Last edited by lstandish on Mon 25 Jan 2010, 02:28, edited 1 time in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#33 Post by 2byte »

Istandish,

Can you add checking for mounted folders to snap2?
I just tried to backup an unmounted partition to another unmounted partition, it failed of course but there was no error reported.
.


Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#34 Post by Sylvander »

1. @lstandish
(a) I installed dirdiff using the pet you supplied.
At the possible end of the installation there wasn't the usual display of notifications; just saw a quick flash of more than 1 window.

(b) "Menu->Utility->dirdiff" is listed, but if I click on this nothing happens.

(c) Entered the command dirdiff in a terminal and got the response:
/usr/bin/dirdiff: line3: exec: wish: Not found

2. Any ideas how to get it working?

3. Ran "Menu->Setup->Check dependencies" on dirdiff-2.1-4, which reported:
/usr/lib/libfilecmp.so.0.0
Has missing library files:
libtcl8.5.so.0
And that Puppy had examined all user installed packages and found the following missing dependencies:
tcl tk

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#35 Post by lstandish »

Sylvander,
The problem with dirdiff is that tcl and tk need to be installed. When I built the pet I specified that tcl and tk are dependencies. Maybe I need to specify the full package name when I specify the dependency. Anyway, if you install the tcl and tk libraries, dirdiff ought to work. Search for "tcl" in the Puppy package installer program - you'll see a package with a name like tcl8.5nothread.pet (that's not exactly it - I don't have Puppy running now to check it.) The tcl package will I think require the tk package, plus a few other pet packages, totaling something over 1 meg in size. I'll try to fix the dirdif package.

2byte, I'll add a check for mounted partitions to a future version - thanks for that excellent suggestion!

Sylvander and 2byte, please make sure that you are using version 3.2-20. 3.2-19 had mirror backup broken and 3.2-18 also had SSL remote authentication broken.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#36 Post by Sylvander »

1. OK...
(a) Used PPM to install both:
tcl-8.5a4nothreads
And...
tk-8.5a4nothreads

(b) dirdiff-2.1-4 now runs and functions. :D
It's looking good so far; looks like it will provide useful info on "differences" between partition contents and backups, or between this backup and that.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#37 Post by lstandish »

Good! Notice that dirdiff also can show color-coded differences between one file and another, so you can see exactly what changed from one version of a file to another. Plus you can apply or unapply file differences.

I made a new pet package for dirdiff giving the fill name of the dependency packages. I'll test it soon. (I made it using dir2pet running on Debian.)

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#38 Post by Sylvander »

1. "you can apply or unapply file differences"
Woa...
Could you elaborate?
e.g. Exactly what is done, and How?

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#39 Post by lstandish »

I just discovered this program myself, looking for a lightweight directory difference analyzer for Puppy, so I don't know much about it. However, I just did some tests, comparing a directory containing a file from snap2 (snap2engine), with a directory containing a previous version of snap2engine.

It is really cool. When I first ran dirdiff and selected the 2 directories, I saw this:
Image

This shows that both directories have the file 'snap2engine' and that the right-hand file is older than the left-hand file. Also, it shows that the left-hand dir has 'snap2' and 'snap2shell', files that are not found in the right-hand directory.

Then I double-clicked the snap2engine line, and saw this:

Image

This window shows the differences between the 2 files. The green text is the section found in the newer file that is not found in the older file. Note that I have ticked the first 3 checkboxes at the start of the lines. You do this to indicate which changes you want to work with for a subsequent "merge" operation. I clicked 'merge' in the menu and chose to "update /home/lloyd/bin". That's the older, right-hand file (pink name).

Then a window appeared to show me the result of merging my ticked changes to the older file. I scrolled down, and saw this:

Image

At this point you can save the merged files whereever you wish. I'm impressed. I really like dirdiff.

I made a new dirdiff pet and replaced the original (above).
Last edited by lstandish on Mon 25 Jan 2010, 02:26, edited 2 times in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#40 Post by 8-bit »

Dirdiff fixed pet is not quite fixed.
You did not set the executable dirdiff as being executable
When I installed the Pet for it, and clicked on the menu entry, nothing happened.
So I tried running it from a terminal and got Permission Denied.
I then went to it's location of /usr/bin/dirdiff and set the executable properties to allow the scrip to run.
That solved it.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

snap2 version 3.2.21beta released

#41 Post by lstandish »

snap2 version 3.2-21beta is ready for download. The most important change is that rsync output is no longer buffered, so you can (again) see the progress of the backup as it occurs. Puppy pet download: http://files.lstandish.com/

Sorry, 8-bit, about the bad permissions in the dirdiff pet. Did it at least load the dependencies properly?

I'll try to go back and upload another fixed pet. (done: dirdiff-2.1-4fixed.pet)
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#42 Post by 8-bit »

The dependencies I already had covered.
And I took the program as far as the entry window for dirdiff.
But I thought you should know.
I also will again try your latest version of snap2.

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#43 Post by 8-bit »

I'm back with news.
The latest version of Snap2 failed to do a new snapshot when using the "Snapshot backup now" button in the main Snapshot-GUI program.
It seems like it worked before.
The Puppy menu selection of "Snap2 snapshot backup now!" still works.
Also when trying the new version, I uninstalled the old one before installing the new version.
In the uninstall, my snap2 settings were preserved.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#44 Post by lstandish »

8-bit,
Oh, damn, I know what it is. I left a line in there hard-coding 'konsole' (a KDE program) as the terminal emulator, rather than allowing it to detect the installed terminal emulator program. I'll fix that and upload a new version in a few minutes.

... OK, fixed in version 3.2-22beta

On the snap2 pet uninstall not wiping out your settings: That's true, but do we really want all the backup configuration files to be deleted every time you upgrade?
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#45 Post by Sylvander »

@lstandish
1. I'm trying to clarify my understanding of snap2.
(a) The snap2shell window displays ALL differences, right?
i.e.
(b) Newly created folders/files that didn't exist previously, and therefore were not included in any previous snapshot.
And/or...
(c) Changes to existing files.
And/or...
(d) Files that previously existed [are therefore possibly included in previous snapshots], but have been eliminated/deleted.

2. I've been using dirdiff to compare/contrast snapshots [say recent.2 & recent.3], and finding that there were NO DIFFERENCES between them, so I deleted the newest [recent.2], and renamed recent.3 to recent.2
Is it OK to do that do you think?

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#46 Post by lstandish »

Hi Sylvander,
You wrote:

>1. I'm trying to clarify my understanding of snap2.
>(a) The snap2shell window displays ALL differences, right?
>i.e.
>(b) Newly created folders/files that didn't exist previously, and >therefore were not included in any previous snapshot.

It shows files that were backed-up during the run, which are the changed or new files, but it does NOT show directory entries.

>(c) Changes to existing files.
Yes

>(d) Files that previously existed [are therefore possibly included
>in previous snapshots], but have been eliminated/deleted.

It does NOT show files that existed in the previous snapshot but do not exist in the current snapshot. In order to make it show those, I would have to make a change would allow the "permissions/ownership bug" to appear. This "bug" allows a new snapshot to affect the permissions/ownership of files in a previous snapshot, when the only change to a file is in permissions/ownership.

I guess you realize that the output of the snap2shell window is stored into the backup logfile, so you can refer to it later.

>2. I've been using dirdiff to compare/contrast snapshots [say >recent.2 & recent.3], and finding that there were NO >DIFFERENCES between them, so I deleted the newest [recent.2], >and renamed recent.3 to recent.2
>Is it OK to do that do you think?

The "overhead" (disk space use) for a snapshot backup snapshot is so low that I don't think deletion of duplicate snapshots is necessary. If you delete it, then you lose the record of how your files were on a given day. However, you can delete snapshot backups whenever you wish. There is no need to rename them, since snap2 will always find and use the earliest snapshot backup for the hardlink references, and it will always rotate the backups so as to fill up any "holes" created by missing snapshots.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#47 Post by Sylvander »

1. "It does NOT show files that existed in the previous snapshot but do not exist in the current snapshot"
(a) Ouch, that isn't so good.
I take it you are saying there is NO WAY to do this?
e.g. The SyncBack "differences" list [which is routinely displayed] includes deleted files, and this can warn that certain VITAL files have been deleted and urgent action needs to be taken to IMMEDIATELY recover those.

(b) At least it IS possible to use dirdiff to compare the latest snapshot with that previous, and make such a discovery.
The problem is that such a check is not a routine part of the snap2 snapshot backup sequence, so is likely to be missed.

2. "I guess you realize that the output of the snap2shell window is stored into the backup logfile, so you can refer to it later"
Yes, but the dirdiff display gives more [complete] information [e.g. missing files].

3. "The "overhead" (disk space use) for a snapshot backup snapshot is so low that I don't think deletion of duplicate snapshots is necessary. If you delete it, then you lose the record of how your files were on a given day"
(a) I wouldn't want a list of snapshots that were all identical, don't even want 2 snapshots that are identical; I want snapshots that are all different.
Hopefully one of those would be the ideal one to restore to achieve some desirable result.

(b) Can't see what use it would be to have 2 identical snapshots taken on different days, to prove they were identical on these days.

4. " There is no need to rename them, since snap2 will always find and use the earliest snapshot backup for the hardlink references"
Huh!?
I fail to comprehend what you are saying.
e.g.
(a) What are "hardlink references"?

5. "and it will always rotate the backups so as to fill up any "holes" created by missing snapshots"
Similar failure to comprehend.
What exactly will snap2 do [when next I make a snapshot backup], if I [for example] delete recent.2 from between recent.1 & recent.3?

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#48 Post by lstandish »

Responses to your questions/comments:
1. (a) It is possible to make snap2 tell which files were deleted (compared to the previous backup), but as I said that introduces the "ownership/permissions bug". That is, if you change only the ownership or permission of a file, and then back up, then the previous exact copies of that file in previous snapshots would all acquire the new ownership/permissions. That seems to me like an unacceptable side effect.

If you want to revert to the way your files were on a given day in the past, you can always just copy all the files over, and you will get back any files that were deleted after the date of your snapshot.

I'll post to the rsync mailing list to see if there are any solutions to this. Meanwhile, if you really must know which files you deleted from one backup to the next, you will have to do a directory comparison.

Or, you can use mirror type backup rather than snapshot type. mirror backup reports on files deleted during a run.

>(b) Can't see what use it would be to have 2 identical snapshots >taken on different days, to prove they were identical on these >days.

It IS useful. Suppose you suffer a total loss of your hard drive, and need to recover your machine to the way it was yesterday. But you don't have a snapshot for yesterday, you only have one from 3 days ago. Why? Because someone deleted the snapshots from the last 2 days because they were identical to the one from 3 days ago. But you may not know (or remember) that at the time you are restoring the files.

I back up about 1.3 gigs of data every day to my server in USA (I am in Costa Rica). Thousands of files and directories are involved. However, most of the time, less than 100 files are new/changed and actually need to be transmitted to the remote server. All the rest of the 1.3 gigs are duplicated by hard links, which take up almost no hard disk room, and did not require any bandwidth at all. The end result is that I can browse over these backups and see exactly the state of my files at a particular time in the past, and recover from a particular day.

You asked me to explain this:
>4. " There is no need to rename them, since snap2 will always >find and use the earliest snapshot backup for the hardlink >references"

The "hard link reference" is always the LATEST backup. snap2 compares this latest backup to the files being backed-up, and if identical, it creates a hard link to files in the hard link reference instead of copying data. In sum, there is no need to rename the snapshots, although you can as long as they remain in chronological order.

>What exactly will snap2 do [when next I make a snapshot >backup], if I [for example] delete recent.2 from between recent.1
>& recent.3?

snap2 will rename recent.1 to recent.2, and the new snapshot will be named recent.1. recent.3 will not be affected. As you can see, it "filled in the whole" in the snapshots.

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#49 Post by 2byte »

Sylvander

There is a perl script that may be helpful to you. http://www.sanitarium.net/unix_stuff/ba ... kup.pl.txt

#H# Diffs 2 rsync backup trees to show what files have changed.
#H# Usage:
#H# diff_backup.pl <dir1> <dir2>
#H#
#H# The output is sort of patterned after RedHat's rpm -y command which I feel has
#H# the perfect format for what it does and I was so happy to see that the Gentoo
#H# people copied it in the epm command.
#H# Output Legend:
#H# + File is new (no further checking is done)
#H# - File is no longer present (no further checking is done)
#H# I Inode number has changed
#H# P File permissions have changed
#H# O File UID has changed
#H# G File GID has changed
#H# S File size has changed
#H# M File mtime has changed
#H# 5 File md5sum has changed (assumed if Size is different)


User avatar
alienpup
Posts: 9
Joined: Sun 18 Oct 2009, 16:18
Location: Arizona, USA

Odd Snap2 backup directory structure

#50 Post by alienpup »

Lloyd,
I'm very new to Snap2 and find it fits my needs splendidly. Thanks for the
time, effort and talent!

Snap2 created it's first daily backup this morning. The results were as
advertised save for one surprise - my recent.4 backup had been copied down
into the daily.1 folder, as shown:

# tree -L 1 ./daily.1/
./daily.1/
|-- initrd
|-- mnt
|-- recent.4
`-- snapshot.log.gz

This surprised me as it was neither expected nor beneficial.

One other fact may have some bearing on this. At the time of this backup, I
had accumulated six recent backups. Just prior to running the backup (which
produced daily.1), I used the GUI to change the maximum number of recent
backups from seven to four. I assumed that Snap2 would delete recent.5 and
recent.6, "promote" recent.1 to daily.1 and add the latest changes. The actual
result was different as revealed by the parent directory after this last
backup:

drwxr-xr-x 5 root root 4096 Feb 1 09:56 daily.1
drwxr-xr-x 4 root root 4096 Feb 1 09:52 recent.1
drwxr-xr-x 4 root root 4096 Jan 31 19:26 recent.2
drwxr-xr-x 4 root root 4096 Jan 31 19:24 recent.3
drwxr-xr-x 4 root root 4096 Jan 30 17:39 recent.4
drwxr-xr-x 4 root root 4096 Jan 30 13:09 recent.6
drwxr-xr-x 4 root root 4096 Jan 30 13:07 recent.7

As you can see, recent.5 is missing while recent.6 and recent.7 remain. Odd?
I may simply be misunderstanding Snap2.

Attached is a tar of my Snap2 config directory.
Version: 3.2-22 beta.

Thanks
Attachments
alienpup_snap2_config.tar
(20 KiB) Downloaded 535 times

Post Reply