Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 21 Aug 2014, 10:05
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
snap2 rotating snapshot backups for Puppy
Post new topic   Reply to topic View previous topic :: View next topic
Page 4 of 11 [158 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, ..., 9, 10, 11 Next
Author Message
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 26 Jan 2010, 09:00    Post subject:  

Hi Sylvander,
You wrote:

>1. I'm trying to clarify my understanding of snap2.
>(a) The snap2shell window displays ALL differences, right?
>i.e.
>(b) Newly created folders/files that didn't exist previously, and >therefore were not included in any previous snapshot.

It shows files that were backed-up during the run, which are the changed or new files, but it does NOT show directory entries.

>(c) Changes to existing files.
Yes

>(d) Files that previously existed [are therefore possibly included
>in previous snapshots], but have been eliminated/deleted.

It does NOT show files that existed in the previous snapshot but do not exist in the current snapshot. In order to make it show those, I would have to make a change would allow the "permissions/ownership bug" to appear. This "bug" allows a new snapshot to affect the permissions/ownership of files in a previous snapshot, when the only change to a file is in permissions/ownership.

I guess you realize that the output of the snap2shell window is stored into the backup logfile, so you can refer to it later.

>2. I've been using dirdiff to compare/contrast snapshots [say >recent.2 & recent.3], and finding that there were NO >DIFFERENCES between them, so I deleted the newest [recent.2], >and renamed recent.3 to recent.2
>Is it OK to do that do you think?

The "overhead" (disk space use) for a snapshot backup snapshot is so low that I don't think deletion of duplicate snapshots is necessary. If you delete it, then you lose the record of how your files were on a given day. However, you can delete snapshot backups whenever you wish. There is no need to rename them, since snap2 will always find and use the earliest snapshot backup for the hardlink references, and it will always rotate the backups so as to fill up any "holes" created by missing snapshots.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3420
Location: West Lothian, Scotland, UK

PostPosted: Tue 26 Jan 2010, 10:05    Post subject:  

1. "It does NOT show files that existed in the previous snapshot but do not exist in the current snapshot"
(a) Ouch, that isn't so good.
I take it you are saying there is NO WAY to do this?
e.g. The SyncBack "differences" list [which is routinely displayed] includes deleted files, and this can warn that certain VITAL files have been deleted and urgent action needs to be taken to IMMEDIATELY recover those.

(b) At least it IS possible to use dirdiff to compare the latest snapshot with that previous, and make such a discovery.
The problem is that such a check is not a routine part of the snap2 snapshot backup sequence, so is likely to be missed.

2. "I guess you realize that the output of the snap2shell window is stored into the backup logfile, so you can refer to it later"
Yes, but the dirdiff display gives more [complete] information [e.g. missing files].

3. "The "overhead" (disk space use) for a snapshot backup snapshot is so low that I don't think deletion of duplicate snapshots is necessary. If you delete it, then you lose the record of how your files were on a given day"
(a) I wouldn't want a list of snapshots that were all identical, don't even want 2 snapshots that are identical; I want snapshots that are all different.
Hopefully one of those would be the ideal one to restore to achieve some desirable result.

(b) Can't see what use it would be to have 2 identical snapshots taken on different days, to prove they were identical on these days.

4. " There is no need to rename them, since snap2 will always find and use the earliest snapshot backup for the hardlink references"
Huh!?
I fail to comprehend what you are saying.
e.g.
(a) What are "hardlink references"?

5. "and it will always rotate the backups so as to fill up any "holes" created by missing snapshots"
Similar failure to comprehend.
What exactly will snap2 do [when next I make a snapshot backup], if I [for example] delete recent.2 from between recent.1 & recent.3?
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 26 Jan 2010, 11:52    Post subject:  

Responses to your questions/comments:
1. (a) It is possible to make snap2 tell which files were deleted (compared to the previous backup), but as I said that introduces the "ownership/permissions bug". That is, if you change only the ownership or permission of a file, and then back up, then the previous exact copies of that file in previous snapshots would all acquire the new ownership/permissions. That seems to me like an unacceptable side effect.

If you want to revert to the way your files were on a given day in the past, you can always just copy all the files over, and you will get back any files that were deleted after the date of your snapshot.

I'll post to the rsync mailing list to see if there are any solutions to this. Meanwhile, if you really must know which files you deleted from one backup to the next, you will have to do a directory comparison.

Or, you can use mirror type backup rather than snapshot type. mirror backup reports on files deleted during a run.

>(b) Can't see what use it would be to have 2 identical snapshots >taken on different days, to prove they were identical on these >days.

It IS useful. Suppose you suffer a total loss of your hard drive, and need to recover your machine to the way it was yesterday. But you don't have a snapshot for yesterday, you only have one from 3 days ago. Why? Because someone deleted the snapshots from the last 2 days because they were identical to the one from 3 days ago. But you may not know (or remember) that at the time you are restoring the files.

I back up about 1.3 gigs of data every day to my server in USA (I am in Costa Rica). Thousands of files and directories are involved. However, most of the time, less than 100 files are new/changed and actually need to be transmitted to the remote server. All the rest of the 1.3 gigs are duplicated by hard links, which take up almost no hard disk room, and did not require any bandwidth at all. The end result is that I can browse over these backups and see exactly the state of my files at a particular time in the past, and recover from a particular day.

You asked me to explain this:
>4. " There is no need to rename them, since snap2 will always >find and use the earliest snapshot backup for the hardlink >references"

The "hard link reference" is always the LATEST backup. snap2 compares this latest backup to the files being backed-up, and if identical, it creates a hard link to files in the hard link reference instead of copying data. In sum, there is no need to rename the snapshots, although you can as long as they remain in chronological order.

>What exactly will snap2 do [when next I make a snapshot >backup], if I [for example] delete recent.2 from between recent.1
>& recent.3?

snap2 will rename recent.1 to recent.2, and the new snapshot will be named recent.1. recent.3 will not be affected. As you can see, it "filled in the whole" in the snapshots.
Back to top
View user's profile Send private message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Sun 31 Jan 2010, 16:37    Post subject:  

Sylvander

There is a perl script that may be helpful to you. http://www.sanitarium.net/unix_stuff/backups/diff_backup.pl.txt

Quote:

#H# Diffs 2 rsync backup trees to show what files have changed.
#H# Usage:
#H# diff_backup.pl <dir1> <dir2>
#H#
#H# The output is sort of patterned after RedHat's rpm -y command which I feel has
#H# the perfect format for what it does and I was so happy to see that the Gentoo
#H# people copied it in the epm command.
#H# Output Legend:
#H# + File is new (no further checking is done)
#H# - File is no longer present (no further checking is done)
#H# I Inode number has changed
#H# P File permissions have changed
#H# O File UID has changed
#H# G File GID has changed
#H# S File size has changed
#H# M File mtime has changed
#H# 5 File md5sum has changed (assumed if Size is different)

_________________

Back to top
View user's profile Send private message 
alienpup


Joined: 18 Oct 2009
Posts: 9
Location: Arizona, USA

PostPosted: Mon 01 Feb 2010, 14:03    Post subject: Odd Snap2 backup directory structure  

Lloyd,
I'm very new to Snap2 and find it fits my needs splendidly. Thanks for the
time, effort and talent!

Snap2 created it's first daily backup this morning. The results were as
advertised save for one surprise - my recent.4 backup had been copied down
into the daily.1 folder, as shown:

# tree -L 1 ./daily.1/
./daily.1/
|-- initrd
|-- mnt
|-- recent.4
`-- snapshot.log.gz

This surprised me as it was neither expected nor beneficial.

One other fact may have some bearing on this. At the time of this backup, I
had accumulated six recent backups. Just prior to running the backup (which
produced daily.1), I used the GUI to change the maximum number of recent
backups from seven to four. I assumed that Snap2 would delete recent.5 and
recent.6, "promote" recent.1 to daily.1 and add the latest changes. The actual
result was different as revealed by the parent directory after this last
backup:

drwxr-xr-x 5 root root 4096 Feb 1 09:56 daily.1
drwxr-xr-x 4 root root 4096 Feb 1 09:52 recent.1
drwxr-xr-x 4 root root 4096 Jan 31 19:26 recent.2
drwxr-xr-x 4 root root 4096 Jan 31 19:24 recent.3
drwxr-xr-x 4 root root 4096 Jan 30 17:39 recent.4
drwxr-xr-x 4 root root 4096 Jan 30 13:09 recent.6
drwxr-xr-x 4 root root 4096 Jan 30 13:07 recent.7

As you can see, recent.5 is missing while recent.6 and recent.7 remain. Odd?
I may simply be misunderstanding Snap2.

Attached is a tar of my Snap2 config directory.
Version: 3.2-22 beta.

Thanks
alienpup_snap2_config.tar
Description 
tar

 Download 
Filename  alienpup_snap2_config.tar 
Filesize  20 KB 
Downloaded  380 Time(s) 
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 01 Feb 2010, 14:36    Post subject:  

alienpup,
Thanks for the encouragement on snap2!

When coding snap2, I did not anticipate the consequences of reducing the maximum number of recent backups.

recent.4 should have been renamed to daily.1, not put under daily.1. This is a bug. Sorry - I'll fix it for the next release. Meanwhile, you will have to fix the snapshot directories "by hand." To avoid further trouble, you should rename or remove recent.5 and recent.6

PS
To clear up what happened: snap2 tried to move recent.4 to recent.5, but since recent.5 existed (unexpected!), it made recent.4 into a subdirectory of recent.5. Then recent.5 was renamed to daily.1. That leaves us with daily.1 and daily.1/recent.4.

In sum, snap2 needs to able to rename the latest recent (and daily, weekly, and monthly) to max+1, which in this case was recent.5. As a fix, snap2 now first checks to see whether or not recent.5 exists. If it does, it renames it like this: recent.5~ Then it can proceed.

I don't think snap2 should delete automatically delete any snapshots, so I opted for renaming the directory. However, if recent.5~ already exists, then it deletes it.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3420
Location: West Lothian, Scotland, UK

PostPosted: Mon 01 Feb 2010, 16:41    Post subject:  

@2byte
1. "There is a perl script that may be helpful to you"
Hmm...
(a) I'm rather new to Puppy and Linux [used Windows since 1997], and I don't even know what a perl script is. Confused

(b) Would I need this?
I have dirdiff installed after all, and that seems to do a very good job.
Would this perl script do the job better?
Or what?

2. I have a new 1TB external HDD ordered from Amazon, and dispatched today... Very Happy
Should now have plenty room for all those snap2 backups and image backups.
Here's the enclosure...
And here's the HDD...
Back to top
View user's profile Send private message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Tue 02 Feb 2010, 10:39    Post subject:  

Istandish ,
Installed snap2_3.22-beta_all.zip to / as root on my server and it refuses to run. It will not go past a dialog that says "You have not specified the storage directory for all backups, on the 'BACKUP STORAGE' tab." Clicking OK only repeats the dialog message. Then I tried installing as a user and had the same results. Running from console as user or root shows an error: -- gtkdialog: Error in line 4, near token 'string' : syntax error --.
Any suggestions?

@ Sylvander ,
If you are happy with dirdiff then stick with it. I haven't tried the perl script yet, it just looks handy Smile

_________________

Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 02 Feb 2010, 17:04    Post subject:  

Hello 2byte,

Since you say you see a dialog and click 'OK' I assume you are running snap2 (the GUI), rather than snap2engine or snap2shell (command line utilities). Have you been able to configure it using the GUI (including the backup storage directory on the second tab)? Are you seeing this error when you click a "...Backup Now" button?

What Linux distro are you using? Are you running the snap2 (the GUI) via a ssh shell session to a *remote* server? I'm not sure whether or not that would work. However, you should be able to run snap2shell or snap2engine on a remote server IF the program is properly configured first.

Regarding the error - did you check to make sure that ~/.snap2/default.set/settings exists, and has an entry like this:
dst=/path/to/your/storage/directory?

Of course, this entry is supposed to be created by snap2 (GUI).

Finally, in snap2, on the backup storage tab, did you click 'Save Storage Settings'?

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Wed 03 Feb 2010, 10:49    Post subject:  

Hi Istandish,

This is exactly what I did.
Logged in as root, unzipped snap2_3.22-beta_all.zip to /.
Created a shortcut on the desktop to execute /usr/local/bin/snap2.
Rebooted.
Logged in as root.
Clicked the desktop shortcut and the first thing I get is the message dialog with "You have not specified the storage directory for all backups, on the 'BACKUP STORAGE' tab." That's all. The dialog has two butons on it: 'About snap2' and 'OK'. Clicking OK just reloads the same message dialog. I never get to the snap2 GUI.

/root/.snap2/default.set/settings contained a line 'dst=none'. I changed this to dst=/mnt/sda2/snap2-bkps with no trailing slash, verified the partition as mounted and ran snap2 from xterm.
Code:
 [root@localhost ~]# snap2
which: no defaulthtmlviewer in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:.:/usr/lib/qt3/bin)
which: no mozilla in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:.:/usr/lib/qt3/bin)
which: no opera in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:.:/usr/lib/qt3/bin)
which: no mrxvt in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:.:/usr/lib/qt3/bin)
which: no rxvt in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:.:/usr/lib/qt3/bin)

** ERROR **: gtkdialog: Error in line 4, near token 'string': syntax error

aborting...
/usr/local/bin/snap2: line 1203: 11667 Aborted                 (core dumped) $GTKDIALOG --program=MAIN_DIALOG

** ERROR **: gtkdialog: Error in line 4, near token 'string': syntax error

aborting...
/usr/local/bin/snap2: line 1203: 11731 Aborted                 (core dumped) $GTKDIALOG --program=MAIN_DIALOG

This repeats endlessly until killed.
The OS is BEL-SBK_1-5, based on pclos2007 with gtkdialog-0.7.20-7.
.

_________________

Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Wed 03 Feb 2010, 11:58    Post subject:  

The "which" warnings are of no consequence. I believe your system is finding konsole as the terminal emulator, and is choking on MAIN_DIALOG.

I do not have BEL-SBK_1-5 or anything similar, so I cannot reproduce the problem. I tried dumping MAIN_DIALOG on my Debian machine, but I see no clues there.

For the time being, I will have to suggest that you configure snap2 manually (sorry). You will not be able to run snap2, only snap2engine from command prompt. snap2engine will create missing skeleton configuration files. Then open ~/.snap2/default.set/settings and make any adjustments.

You will have to store your directories to back up into ~/.snap2/default.set/snapshotsourcepaths (snapshot type backups) and ~/.snap2/default.set/mirrorsourcepaths (mirror backups).

Backup exclusion patterns files go into ~/.snap2/default.set/exclude/ and are named (for example) "varmailuser" for a backup path /var/mail/user (snapshot backups) or "mirrorvarmailuser" for /var/mail/user (mirror backups).

For info on syntax of exclude file entries, see http://www.linuxbackups.org

You can also run snap2 on a system on which it works, such as Puppy, just to configure it. Afterwards, copy the generated configuration files to your server.

PS
I just figured out that the message "You have not specified the storage directory for all backups..." is due to the fact that the main snap2 GUI is not opening (the output you sent with reference to a syntax error on line 4 shows that). However, rather than quit, the snap2 script continues, and generates the message about the missing storage directory, which is only supposed to be seen when the user clicks on any snap2 GUI button without specifying the storage directory. Then, when you accept that dialog, it loops and tries again to show the snap2 GUI, fails again, and brings up the "missing storage directory" message again, etc.

At the very least, I can check for a failure to load the main snap2 GUI, and have the program quit in that case.

PPS I have sent a special debug version of snap2 to 2byte by PM which dumps the MAIN_DIALOG XML, and also prints the environment, saving both to a disk file, which I hope to examine.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Sun 07 Feb 2010, 17:26    Post subject: snap2 new release (new code branch)  

I have released snap2-4 (version 4.0beta) today for testing and comments. It adds the following backup management actions:

1. View logfile for any backup
2. Delete any backup
3. Report on files missing from a snapshot but present in the previous backup. This is essentially a report on files deleted between one backup and another.

These actions for on both local storage backups and remote server backups (via ssh).

The download directory for both version 3.x and 4.x is:
http://files.lstandish.com/

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3420
Location: West Lothian, Scotland, UK

PostPosted: Mon 08 Feb 2010, 09:50    Post subject:  

1. Uninstalled the old and installed the new, and started afresh with new snapshots on sda1+sda5, and sda3.
The new seems to be functioning as it aught.

2. Hope to see the new features working well.
(a) Tried "sda3->Backup Storage->View backup Logs->recent.1->Show Logfile" and IT WOKS! Very Happy

(b) I expect "Report Deleted Files" won't work until I have a 2nd later snapshot with deleted files [on the source partition] to show.

3. I want to put to you an idea I have:
Regarding the naming of snapshot folders = recent.1 etc.
Would it be possible to name the folders with the year/month/day/time?
e.g. I often name [backup?] folders, or files as:
090208 = [year =09; month=02; day=08]
You might add the time like this:
090209-09:23
Then...
090209-17:12
Folders or files so named would automatically be listed in date/time order.
Then the user would be able to immediately see WHEN each backup was made.
I find this to be a VITAL bit of info that needs to be displayed VERY PROMINENTLY.
Naming the folder this way does just that.
See example in screenshot below.
00.jpg
 Description   
 Filesize   25.75 KB
 Viewed   629 Time(s)

00.jpg

Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 08 Feb 2010, 10:36    Post subject:  

On snapshot naming with date/time, good idea! I could probably incorporate the name like this: recent.1-2010-02-10_9:23. (I can't discard the 'recent', 'daily' name, etc. completely.)

Note that the new GUI interface to view/delete/report deleted files shows the date prominently. However, I can see that making the date part of the name would be convenient for when you access the backup directories via a file manager.

Last night I worked late finally giving decent error checking to the remote authentication code. Now it detects when the process goes wrong at any stage. (This enhancement is not published yet.)

So far, nobody has given me any feedback about the use of the program for backup to a remote server...

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3420
Location: West Lothian, Scotland, UK

PostPosted: Mon 08 Feb 2010, 11:36    Post subject:  

1. "recent.1-2010-02-10_9:23"
That looks OK.
The 2010-02-10 is more explicit & obvious to the casual observer or new user.
It isn't quite so short-and-sweet, but that's OK.
Is recent.1 ALWAYS more recent than recent.2?
Therefore, will they ALWAYS be in date/time order?

2. "Note that the new GUI interface to view/delete/report deleted files shows the date prominently"
(a) That helps; any chance of also having that displayed at the top of the log?
I REALLY like the display of the log; easier than having to find it at the destination recent.x folder.

(b) I initially had trouble finding these new additional features.
They are buried 1 level down with no clue at the top level that they can be got to by clicking the "View Backup Logs" button.
Would it be possible to name the button something like:
Logs/Deleted-files/Delete-snapshots?
Or else put buttons for each of these at the top level?
Might involve a cluster of buttons, but then they will make the user curious to know what these do, and they will click them to see.

3. "I can see that making the date part of the name would be convenient for when you access the backup directories via a file manager"
I would normally have a list of backup folders [or image files] all [automatically put] in order by year-month-date; it makes it so obvious which-is-which.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 4 of 11 [158 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, ..., 9, 10, 11 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1306s ][ Queries: 12 (0.0049s) ][ GZIP on ]