snap2 rotating snapshot backups for Puppy

Miscellaneous tools
Post Reply
Message
Author
Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#16 Post by Sylvander »

I've been experimenting with snap2 as follows:

1. Made a [39.2GiB] ext3 Linux partition on my [80GB] external USB HDD, to which to save the backups.
Had to use GParted to non-destructively re-arrange partitions to achieve this.

2. Configured snap2 to backup sda1, sda3, sda5, sda6, sda7.
Backup storage on /mnt/sdb1/snap2.
Unticked "Add Default Exclusions" and clicked "Save above settings".
Despite this it seems these default exclusions are being used! :(

3. Made a couple of snapshot backups yesterday, and another today.
38.89GiB used, and 327.86MiB free. [Whew, that was close!]

4. Deleted one of the snapshots made yesterday, with no discernible ill effect.
Snapshots have been auto-named recent.1 [newest] & recent.2 [next newest = oldest]

5. QUESTIONS
(a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?

(b) Would it be possible to display "differences" as in SyncBack?

(c) Is it OK to exclude the Windows pagefile.sys?
Attachments
00.jpg
(12.71 KiB) Downloaded 1154 times

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#17 Post by Sylvander »

1. Completed yet another snap2 snapshot backup.

(a) Added a custom exclusion for [highlighted]->sda1 = - pagefile.sys
The result was that this file wasn't present in new recent.1 mentioned in (c) below [the backup made immediately after].

(b) Clicked on "Snapshot Backup Now".
This took 5 min to complete and reduced the remaining space from 2.8GB to 2.6GB [0.2GB used].

(c) Having deleted the recent.2 folder prior to the backup...
During this new backup...
snap2 renamed recent.1 to recent.2, and made a new recent.1->[the latest backup].

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#18 Post by lstandish »

Sylvander,
> Unticked "Add Default Exclusions" and clicked "Save above
> settings". Despite this it seems these default exclusions are being used!

You're right, 'Add Default Excludes' only adds (or not) the default exclude patterns to any custom exclude patterns. If there are no custom exclude patterns, then unticking 'Add Default Excludes' is ignored, and the default patterns are used. (It never occurred to me that anyone would want no exclude patterns!)

I'll change that behavior to work as expected. Meanwhile, to get rid of all exclude patterns you can create an empty custom exclude list for each specified backup path by clicking the 'Exclusions for Selected Snapshot Entry' button, then click 'Save'. By doing that, and unticking the 'Add Default Excludes', you'll have what you want.

You may want to subscribe to the 'snap2 version history' post (http://www.linuxbackups.org/node/26) to be notified when I release the next snap2 version.

a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?
Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude? If that's what you mean, yes. See http://www.linuxbackups.org/node/31

(b) Would it be possible to display "differences" as in SyncBack?
I think that would be extremely complicated. I have no plans for doing that.

(c) Is it OK to exclude the Windows pagefile.sys?
That's the Windows virtual memory file, right? If so, I don't think you would want to back that up. I would certainly exclude it.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#19 Post by lstandish »

Sylvander,
> Unticked "Add Default Exclusions" and clicked "Save above
> settings". Despite this it seems these default exclusions are being used!

You're right, 'Add Default Excludes' only adds (or not) the default exclude patterns to any custom exclude patterns. If there are no custom exclude patterns, then unticking 'Add Default Excludes' is ignored, and the default patterns are used. (It never occurred to me that anyone would want no exclude patterns!)

I'll change that behavior to work as expected. Meanwhile, to get rid of all exclude patterns you can create an empty custom exclude list for each specified backup path by clicking the 'Exclusions for Selected Snapshot Entry' button, then click 'Save'. By doing that, and unticking the 'Add Default Excludes', you'll have what you want.

You may want to subscribe to the 'snap2 version history' post (http://www.linuxbackups.org/node/26) to be notified when I release the next snap2 version.

a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?
Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude? If that's what you mean, yes. See http://www.linuxbackups.org/node/31

(b) Would it be possible to display "differences" as in SyncBack?
I think that would be extremely complicated. I have no plans for doing that.

(c) Is it OK to exclude the Windows pagefile.sys?
That's the Windows virtual memory file, right? If so, I don't think you would want to back that up. I would certainly exclude it.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#20 Post by Sylvander »

1. "I'll change that behavior to work as expected"
Great! Many thanks. :D

2. "By doing that, and unticking the 'Add Default Excludes', you'll have what you want"
This will do quite nicely in the meantime.
I was concerned that perhaps something useful or vital might have been excluded.

3. "Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude?"
(a) No, what I mean is...
Within snap2, I specify a list of partitions whose contents I MAY want to backup at this time or that...
But I probably do NOT want to backup ALL at each and every backup session.
There may be a good reason to backup one [or more] particular partitions at this time or that.
It's nice to be able to choose certain partitions [for backup] from amongst those listed.
[Hold down "Ctrl" and highlight those to be backed -up [from among the list], then click the backup button?]
Instead, the program is backing-up each and every listed partition at each and every backup session.
NO choice permitted?

(b) I can do this right now using SyncBack, and I'm trying to decide whether it would be sensible/prudent/justified to switch from using SyncBack to using snap2 for this type of job [making folder/file copies of partition contents].
To supplement the making of image backups.
Snap2 is better able [than SyncBack] to make backups of BOTH Linux AND Windows partitions.
If I don't lose too much functionality I'll switch. :D
This would be an important change for me.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#21 Post by lstandish »

Snap2 started as a program to do automatic non-interactive backup (via cron), and it is still oriented toward running a predefined backup job. However, you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'.

You could create a new backup set to backup any group of directories that you want. The result is that you back up only part of the partitions or folders. If you do this, you should however specify a different storage folder.

You can also set up a menu entry to execute a backup (via execution of snap2shell), and you can of course specify which backup set to back up, as an argument to snap2shell.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#22 Post by Sylvander »

1. "you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'"
(a) Are there any instructions on how to do that?

(b) Seems I'd need to create various backup sets for every possible combination of partitions.
Not a very elegant solution.
I don't see myself doing that. :(

(c) Or else configuring the program anew [to specify the partitions to backup] each time I run it.
Wouldn't want to do that either. :(

2. "The result is that you back up only part of the partitions"
I don't think I'd ever want to backup only PARTS of chosen partitions.
I'd want to backup only certain chosen partitions, but backup ALL of the contents of those.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#23 Post by lstandish »

Yes, you'd need to create a backup set for every combination of partitions you would back up. However, that's easy: A new set is created on the ADVANCED SETTINGS page by just typing in a new set name and clicking 'Change'. Then you follow the 3 step instructions at the top to configure the new backup set.

As I said, this is designed for those who do regular backups of predefined sets of directories (partitions in your case). It is really not intended for sporadic use or one-time backup. Wouldn't you want to back up your partitions (the same ones) on a regular basis?

In order to assist and encourage regular backup, snap2 will set up cron tasks to execute a backup set on certain days at a certain time.

I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked.

Also, starting with the latest version (3.2-18), backup paths that have a custom exclusion pattern list are marked. That was a glaring deficiency, I think. 3.2-18 should be available a few minutes after I make this post.
Last edited by lstandish on Thu 21 Jan 2010, 17:47, edited 2 times in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#24 Post by lstandish »

duplicate post removed

User avatar
alienpup
Posts: 9
Joined: Sun 18 Oct 2009, 16:18
Location: Arizona, USA

#25 Post by alienpup »

lstandish wrote:Yes, you'd need to create a backup set for every combination of partitions you would back up.
I've run into a very significant issue when attempting to backup pup_save files using any rsync-based utility. The problem is that Puppy does not "touch" the pup_save while/after updating it. Because the file's time stamp does not change, rsync does not detect that the file has changed. The pup_save is backed one time and one time only.

I've solved the problem in a backup script of my own by explicitly touching the pup_save file before running rsync.

As this problem likely exists in all Puppies and could cause other problems, it probably ought to be brought to Barry's attention.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#26 Post by lstandish »

rsync decides to back up a file when it has changed in size or in last-modified time. If pupsave files don't get a fresh timestamp when Puppy updates them, I think that just about any backup utility (not just rsync) would have a problem detecting that it should be backed-up.

Your use of touch just before running a backup sounds like the best workaround until Barry fixes this.

By the way, since rsync transmits only the (compressed) differences in files when backing up, it is particularly suited to backing up big files such as pupsave files.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#27 Post by Sylvander »

1. "you'd need to create a backup set for every combination of partitions you would back up"
OK, beginning to get a feel for this. :D
(a) Within the "Advanced Settings" pane...
Made an "Active Backup Set" [new name] for [almost] each individual partition [sda1+sda5 (Windows+its-data-files), sda3 [BoxPup], sda6 [music], sda7 [photos]]...

(b) After giving each new name [e.g. sda1,sda5.set], clicked the "Change" button and then completed the "3 easy steps".
See the screenshot below of the storage folders I made/used.
One folder for each partition.
Might need to eliminate the default that makes a backup of ALL partitions to the "snap2" folder.
Would that result in a time-consuming repeat of first-time backing-up of all the individual partitions?

(c) So now I can choose to backup partitions individually, and when I like.
Might it be possible [upon running the program after configurations have been completed] to make that choice more prominent?
e.g. When the user runs the "Backup Now..." from the menu...
If there's more than 1 backup set configured...
To have a separate/first window that opens and asks the user to choose WHICH backup set to run?

(d) If I now make a snapshot of one of these individual partitions...
Would it start anew, or make use of a previous snapshot included in the default backup set?
e.g. /mnt/sdb1/snap2/recent.1/mnt/sda1

2. "Wouldn't you want to back up your partitions (the same ones) on a regular basis?"
(a) I don't backup to a fixed schedule.

(b) I backup quite regularly according to whether/when I think there's a need.
e.g. Perhaps I'd backup the Windows partition immediately prior to making some significant or risky change to the software. [Installing a program?]
But no need to make yet another backup of data partitions I know have not changed since the last backup.

3. "I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked"
Nice! :D 8)
I uninstalled the old and installed the new.
Noticed that the old configurations were present in the new!
Did uninstalling leave in place certain files?
Not keen on that, but it was quite helpful in these circumstances.
Would it be possible for the user to be asked about this at uninstall?

4. "starting with the latest version (3.2-18), backup paths that have a custom exclusion pattern list are marked"
That's good; very helpful to see the X against the backup sources listed that have custom exclusion entries.
No X's against those with [NEW] empty custom exclusion files.
[Those with old empty custom exclusion files had X's against them until I reset them]
Attachments
00.jpg
(20.33 KiB) Downloaded 987 times

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#28 Post by Sylvander »

1.
(a) What is it that the "snap2shell" window displays?
Differences?

(b) e.g. I just updated windows, and also the Avira Anti-virus program installed within Windows, and now here in Twinkle-421 [having completed a new snapshot] I can see at the bottom of the snap2shell window, what appears to be a list of [only = nothing but] the changed files in the Firefox profile held on sda5 .

2. Would it be possible to include a scroll-bar in snap2shell so that it would be possible to browse back up through the display of folders/files listed?
Pretty much the most obvious thing missing from snap2 is the ability to view "differences".
This ability is much to be desired, because the user can discover from viewing these whether there has been any interesting/undesirable/disastrous change in the folders/files.
e.g. Vital files deleted in error.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#29 Post by lstandish »

Sylvander,
I'm glad snap2 is working out for you.

Unfortunately, the code changes to handle default exclusion patterns differently, and (previously) to remove the xterm dependency broke the code in 2 different ways! I'm very sorry, but, well, it's still 'beta'.

This affected the (1) remote authentication process, and (2) mirror-type (not snapshot) backups. So if you did only snapshot-type local backups, it didn't affect you. The fixed version is 3.2-20, download: http://files.lstandish.com/

I'll consider adding a way to choose the backup set upon execution. Currently of course you can only execute backups for the currently active set. However, the feature set is frozen now, since I want to finish testing and create a stable/tested release.

Different backup sets will not share hard link references, unless they also share the same backup storage directory, which I don't think would work for you. So the default backup set is not seen by any other sets.

You should avoid having any partitions named in more than one set. To avoid a time-consuming repeat of what you've backed-up, you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over. If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast.

I too like the freedom of being able to run a backup at any time, not bound to a fixed time schedule. When I first wrote this years ago (the non-GUI shell script), that was one the gripes I had with existing snapshot backup systems. They would ONLY work when run at a certain time by cron.

snap2, on the other hand, allows the user to back up on a whim, several times per day if desired. Still the "daily", "weekly", and "monthly" backups will be automatically 'spaced out" in time, and rotated. So snap2 is a merger of the notions of sporadic backup and structured, time-based backup.

>(a) What is it that the "snap2shell" window displays?
Differences?
Yes, it lists ONLY the files that are backed-up on that run (the differences. It is recorded in the logfile in the top of each backup (for example, in directory recent.1). To see what was changed in a given backup, you would open its logfile. But I see that a button to display this file in a window would be nice. I'll add that later. That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups. Hmmm, maybe it would be better to use local storage for ALL logfiles... or do both? What do you think?

Unfortunately, since I modified the code to allow rsync to create the hardlinks (fixing the problem with permissions/ownerships of older snapshots being changed to those of the newest snapshot), the logfile no longer indicates when files are deleted, only when files are backed-up. I will try increasing the verbosity level of rsync to see if that helps.

Also, I will add the scrollbar to snap2shell's terminal window. That would certainly be useful. Thanks for the suggestions!
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#30 Post by Sylvander »

1. "You should avoid having any partitions named in more than one set"
Done. :D

2. "you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over"
Guessed that could be done from what you'd said previously.
Hence it's already done, and it appears to have worked well. :D

3. "If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast"
That's exactly the way it was. :D

4. " I see that a button to display this file in a window would be nice"
(a) So isn't it possible to have a scrollbar on the snap2shell window?
If not, then...

5. "That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups"
Whoa, a bit too technical for me.
Does that mean a logfile held on a remote server cannot be viewed on the local machine?
Unless copies are held on the local machine?

6. "maybe it would be better to use local storage"
Seems like a bad idea to separate the log files from the backups; I like that they are held very near the backups.

7. "or do both?"
Seems less than ideal to duplicate files.

8. "the logfile no longer indicates when files are deleted, only when files are backed-up"
(a) That makes sense to me; seems like a good way to do things.
[Not so, faulty thinking by me. I want to know ALL differences]
After all, this program only makes backups, which seems good...
[Again faulty thinking, see (b) below]
No need to make it restore, so no duplication of functionality.
Restoring any chosen number of folders/files is done using other programs; again, I like that.
e.g. Xfe can/would do that quite nicely.

(b) Oops, just realized an important difference.
One of the good features in SyncBack is that it not only will alter a backup to add the "differences", but it will do the same during a restore.
i.e. Restore only the differences to make the [recently changed?] source the same as the backup [of the arrangement prior to the change].

9. What about a program that could compare 2 existing snap2 backups [e.g. sda3 recent.1 versus recent.2 (or recent.3)], and list the differences?
In Xfe I can compare 2 file-panes side-by-side, and see the obvious differences, but not a definitive list of differences.
I suppose a file-compare program could compare 2 log-files, but I'm skeptical how easily/well it would do that.

10. "I will add the scrollbar to snap2shell's terminal window"
SUPERB! :D 8)

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#31 Post by lstandish »

Sylvander,
The (optional) logfile stored with each snapshot backup indicates which files were backed up (the differences between that backup and the next oldest.) I do think it would be good to allow viewing the latest backup log (whether stored locally or on a remote machine) in a snap2 gtkdialog window.

For more in-depth information, I think the user should open the storage directory and browse through the log files in each snapshot directory.

I think these snapshot logfiles should normally give you what you want to know about a backup event. It's true they don't tell you which files were NOT backed up compared to the previous backup. As pointed out previously, to provide that information snap2 would have to create the hard links via cp rather than via rsync, and this would introduce the ownership/permissions bug (old snapshots can acquire permissions/ownerships of newer snapshots).

If you want more in-depth info about the differences between snapshots, you could run a directory comparison tool such as Krusader's "Synchronize Directories." That's non-Puppy, of course. It's really first-class.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#32 Post by lstandish »

Sylvander, I found a great graphical directory comparison utility for you (called dirdiff), and make a pet package out of it (attached). You can compare up to 5 directories at once. It needs the Puppy tk/tcl libraries. Since they are listed as dependencies, they should be installed automatically.

Here's the Debian package description:
"Dirdiff can handle up to 5 trees. It displays a main window with a
list of the files which are different between the trees, with colored
squares to indicate the relative ages of the versions. A menu allows you to display the differences between any two of the versions in another window. Another menu allows you to copy the file from one tree to another."

I think I'll put it in the snap2 download area. (For Puppy 4.3.1)
Attachments
dirdiff-2.1-4fixed2.pet
(34.14 KiB) Downloaded 1412 times
Last edited by lstandish on Mon 25 Jan 2010, 02:28, edited 1 time in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#33 Post by 2byte »

Istandish,

Can you add checking for mounted folders to snap2?
I just tried to backup an unmounted partition to another unmounted partition, it failed of course but there was no error reported.
.


Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#34 Post by Sylvander »

1. @lstandish
(a) I installed dirdiff using the pet you supplied.
At the possible end of the installation there wasn't the usual display of notifications; just saw a quick flash of more than 1 window.

(b) "Menu->Utility->dirdiff" is listed, but if I click on this nothing happens.

(c) Entered the command dirdiff in a terminal and got the response:
/usr/bin/dirdiff: line3: exec: wish: Not found

2. Any ideas how to get it working?

3. Ran "Menu->Setup->Check dependencies" on dirdiff-2.1-4, which reported:
/usr/lib/libfilecmp.so.0.0
Has missing library files:
libtcl8.5.so.0
And that Puppy had examined all user installed packages and found the following missing dependencies:
tcl tk

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#35 Post by lstandish »

Sylvander,
The problem with dirdiff is that tcl and tk need to be installed. When I built the pet I specified that tcl and tk are dependencies. Maybe I need to specify the full package name when I specify the dependency. Anyway, if you install the tcl and tk libraries, dirdiff ought to work. Search for "tcl" in the Puppy package installer program - you'll see a package with a name like tcl8.5nothread.pet (that's not exactly it - I don't have Puppy running now to check it.) The tcl package will I think require the tk package, plus a few other pet packages, totaling something over 1 meg in size. I'll try to fix the dirdif package.

2byte, I'll add a check for mounted partitions to a future version - thanks for that excellent suggestion!

Sylvander and 2byte, please make sure that you are using version 3.2-20. 3.2-19 had mirror backup broken and 3.2-18 also had SSL remote authentication broken.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Post Reply