Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 02 Oct 2014, 06:31
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
snap2 rotating snapshot backups for Puppy
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 11 Posts_count   Goto page: 1, 2, 3, ..., 9, 10, 11 Next
Author Message
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 22 Dec 2009, 20:03    Post_subject:  snap2 rotating snapshot backups for Puppy
Sub_title: simple yet powerful rsync-based backup program
 

Status: Stable/Tested.
Download latest pet for Puppy at http://files.crnatural.net/snap2
Website: http://standish.home3.org/snap2

May 26, 2012: Released version 4.24

Here are the changelog entries for the last 5 updates:

26-May-2012 (snap2 4.24)
    (snap2engine remoteutils2) Bugfix: Roll back to previous handling of snapshot promotion, to allow e.g. zero daily snapshots with weekly snapshots.

25-May-2012 (snap2 4.23)
    (snap2) Bugfix: If filesystem root ('/') is Backup Source Directory, backup exclusions could not be added.
    (snap2) Bigfix: Upon deleting a Backup Source Directory, additional directories could possibly be deleted.
    (snap2 snap2engine) Bugfix: Corrected command line messages when zero daily, weekly, or monthly backups are configured.
    (snap2engine): Add special handling of root source directory ('/') backup exclusions

24-Oct-2011 (snap2 4.22)
    (snap2) Improved user prompts about generation and use of SSH keys for remote server authentication. No functional changes.

24-Oct-2011 (snap2 4.21)
    (snap2, remoteutils2.sh) changed remote SSH authentication method to "manual," add "Generate SSH Keys" button to "Advanced" tab.

23-Oct-2011 (snap2 4.20)
    (snap2, remoteutils2.sh) Correct issues with multi-core systems: correct remote SSH authentication detection and remove automatic remote authentication setup.


If you would like to keep 'snapshot's of your files (or, usually, a subset of your files) as they were hours ago, days ago, weeks ago, etc., this program will do that, automatically "spacing out" the backups over time.

Thanks to the use of hard links, it can keep several gigabytes of files on a one-gig USB drive, without compression. Your backups are accessed as ordinary directories and files.

It can also do backup to a remote server. Thanks to the hard link trick, duplicate files (from one backup to another) are "copied" without using disk space. And thanks to rsync magic, only changed portions of files are transmitted.

You access and browse your backups just as you would ordinary directories and files, with any file manager. Despite the huge savings in disk space through the use of hard links, no compression is used. This allows you to easily inspect backup files and compare one version of a file with another.

I keep about 1.4 gigs of files from my home directory backed up to my VPS (remote server), over a regular broadband connection, and the whole process usually only takes a couple of minutes (since only new and changed files need to be sent.)

Here are some screenshots directly from my home directory backup configuration. There is a screenshot for each of the program's 3 tabs:
'Directories to Back Up' Tab:


'Backup Storage' Tab:


'Logs & Reports' Tab:


'Advanced' Tab:


'Automatic Backup Scheduler':


'Backup File Exclusions Editor':


Here's the link to the program website:
http://standish.home3.org/snap2

Look for the latest Puppy pet here:
http://files.crnatural.net/[/b]

Edited_times_total
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Thu 24 Dec 2009, 11:22    Post_subject:  

(There was a false alarm here, which I removed, about what I thought was a bug. )

version 3.2-3 will also have a revised (simpler-looking) GUI.

PS see screenshots of 3.2-4 in the next post.
Back to top
View user's profile Send_private_message 
8-bit


Joined: 03 Apr 2007
Posts: 3368
Location: Oregon

PostPosted: Mon 28 Dec 2009, 16:24    Post_subject:  

ON version 2.3.4, snap2, on doing a snapshot backup, creates a "snaplog.gz" and also seems to create a mirror backup in the same directory.
I thought it was supposed to create either a compressed backup file or a mirror backup, not both.
I did not set up the mirror backup. I only set up the Snapshot backup.
The above happens when I click on the "Snapshot Backup" button.
Also, it creates a new directory for each backup.
Should I have unchecked the create directory option to prevent this?
EDIT: The picture was with the create directory option unchecked.
Backup_directory.png
 Description   Result of second backup with new directory name of recent.2
 Filesize   22.25 KB
 Viewed   5146 Time(s)

Backup_directory.png

Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 28 Dec 2009, 22:34    Post_subject:  

Hi 8-bit,
snap2 does not create any compressed backup files. It does however, with the default settings, create a compressed log file of what was done during the backup. The 'snaplog.gz' file is that compressed log file. The next version of snap2 will include the option of not compressing the log file. The current version only has the option of not storing a log file (look on the ADVANCED SETTINGS tab).

A snapshot backup does indeed create a new directory for each backup (snapshot), inside the directory you specified on the BACKUP STORAGE tab. This will look like a brand new full copy of your files with full directory structure: a 'snapshot' of the files at the time the backup was run.

However, due to the use of hard links, each new backup only uses extra disk space for new and modified files (and of course for the hard links, which are really just filenames.) That's how you can get, say, 10 gigs of snapshot backups on a 2 gig USB drive. You will be able to retrieve past versions of files from hours, days, weeks, or months ago, files you have long since deleted from your hard drive.

snap2 automatically rotates and 'spaces out' the snapshot backups in time, according to the maximums set for 'recent', 'daily'. 'weekly', and 'monthly' backups. These settings are on the ADVANCED SETTINGS page, and the defaults are good for most people.

I consider the snapshot backup method to be the chief feature of snap2.

The 'Create missing destination top-level dir' (advanced option), is, I admit, not clearly explained by the tooltip text. If checked, it allows snap2 to create the directory you named in 'BACKUP STORAGE', if it does not yet exist. The idea to to avoid accidentally backing up to the wrong storage media. I added this because I used to back up to a USB drive, and I did not want to accidentally back up to the wrong USB drive. Now I back up my 1.5 gig home directory to a remote server.

For more explanation of the concept of snapshot backups, see my website at http://www.linuxbackups.org/snap2 Also, this page is good: http://www.mikerubel.org/computers/rsync_snapshots/

If you have no possible need of past versions of files, you can use the 'mirror' backup method, which will result in a single backup with all your files as they were at the time of the last backup. This one backup will be in a directory called 'mirror' in the backup storage directory. I use this method for my Opera mail backup.
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3447
Location: West Lothian, Scotland, UK

PostPosted: Mon 18 Jan 2010, 08:39    Post_subject:  

1. Is it possible to use this [Snap2] to backup/restore ALL of the folder/file contents of a partition holding an operating system?

2. Am I missing something?
(a) I installed Snap2 within Twinkle-421, ran the program, made the necessary configurations, specified the [mirror] backup path of the partition sda3 [source], and the backup storage location [destination].

(b) Clicked "Mirror Backup Now", and up came a window named "snap2shell mirror default.set".
What is this?
No sign of a backup being made.
Nothing appeared in the destination [/mnt/sda8/snap2-backup-of/sda3].
[Both the source (ext3) and the destination (ext3) were mounted]
Back to top
View user's profile Send_private_message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Mon 18 Jan 2010, 16:42    Post_subject:  

OK, this looks good for backups.
How do you restore from these snapshot backups?
Can individual files be restored from specific times?

Are all file ownerships and privileges preserved?
Are my symlinks included in the backups?
.

_________________

Back to top
View user's profile Send_private_message 
8-bit


Joined: 03 Apr 2007
Posts: 3368
Location: Oregon

PostPosted: Mon 18 Jan 2010, 17:06    Post_subject:  

Sylvander,

Did you install the xterm pet package?
It is currently needed for the program to make backups.
I have made a few and Snap2 works only if you install the xterm pet package.
If you do not see an xterm window pop up giving progress of the backup, you most likely did not install xterm.
Back to top
View user's profile Send_private_message 
abushcrafter


Joined: 30 Oct 2009
Posts: 1447
Location: England

PostPosted: Mon 18 Jan 2010, 18:24    Post_subject:  

This looks good....
Back to top
View user's profile Send_private_message Visit_website 
Sylvander

Joined: 15 Dec 2008
Posts: 3447
Location: West Lothian, Scotland, UK

PostPosted: Mon 18 Jan 2010, 18:59    Post_subject:  

@8-bit
1. Success! Very Happy
Used the PPM to install xterm-229-1, and when I ran the backup, instead of an empty do-nothing "snap2shell mirror default.set" window, a list of files and folders being backed-up flashed down the window.

2. Questions:
(a) Having completed a single "Mirror Backup" [dated as per the "modified" date and time of the "mirror" folder]...
I guess this specifies when I'm going back to if I restore this, right?

(b) If I now [some minutes or hours later] make a snapshot backup...
Is it best to make it in the same /mnt/sda8/snap2-backup-of/sda3 folder?
And will a snapshot make use of the previous mirror backup?
i.e. By only backing-up the "differences" [between the up-to-date state and the last (mirror) backup]?
I'll give this a try whilst awaiting a reply.
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3447
Location: West Lothian, Scotland, UK

PostPosted: Mon 18 Jan 2010, 19:26    Post_subject:  

1. OK, completed the snapshot backup...
And it has the appearance of comprising ALL of the folders and files on the partition, as though EVERYTHING on the partition has been backed-up...
But my understanding so far is that most of these items are only links to folders/files already copied to the mirror backup...
And that only the "differences" are actually included in the snapshot.
Is that right?

2. So does the original mirror use the most space, and the snapshot very little space?
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 18 Jan 2010, 23:13    Post_subject:  

Sylvander,
A mirror backup is an exact SINGLE copy of your files. If you delete files on your computer, and then run a mirror backup, the corresponding files in the mirror backup are also deleted. So you can only recover to a single point in time (the last time you ran the mirror backup).

A snapshot backup keeps a SERIES of 'snapshots' of your files, which look like full backups. Each snapshot directory has ALL your files as they were at the time the snapshot was made. However, due to the use of hard links, each new snapshot "full backup" only uses disk space to save new and modified files.

For both mirror and snapshot-type backups, after the first run, subsequent backups will only, of course, copy over only new and modified files. In fact, when updating an existing file, only the differences between files will be transmitted by rsync, which can give a huge savings in time and/or bandwidth.

As to where you should put your backups:
I use the same backup storage directory for both my snapshot and my mirror-type backups. They will not get mixed up. Mirror backups will be in a directory named 'mirror', and snapshot backups will be in a series of directories with names like recent.1, recent.2, daily.1, daily.2, etc. For an explanation of how this backup naming works and how snapshot backups are automatically 'rotated', see http://www.linuxbackups.org/node/29

Snapshot and mirror backups are completely different types of backups and do not interact. Snapshot backups will not make use of a previous mirror backup for the hard-link references.

Just remember that mirror backups are a subset of snapshot backups. Snapshot backups give you the same thing as a mirror backup, plus access to previous versions of files (previous snapshots), for only a relatively small amount of extra disk space. As I have pointed out, you can often fit 7 or 8 gigs of snapshot backups on a 1 gig USB drive.

For recovery, look at the date stamp of the 'mirror' directory (for mirror backups) or the various snapshot directories (for snapshot-type backups. The timestamp of the directory is the date of the backup. To recover your files, just copy them back. I highly recommend mc (Midnight Commander) for copying files around in Puppy. In mc, you can use the fish filesystem to copy files between a remote server and your computer, just like you were copying between 2 local disks. There is an mc pet for Puppy.

You could certainly use snap2 to back up an entire partition, but I don't think that is usually necessary. I recommend you identify the data files you need to recover your computer, and only back those up. snap2 uses 'exclusion patterns' to specify which groups of files NOT to back up. I have detailed information on this, including examples, on the snap2 website http://www.linuxbackups.org

2byte,
>OK, this looks good for backups.
>How do you restore from these snapshot backups?
Copy files/directories back using your favorite file manager. For recovery from a remote server I suggest mc + fish rather than FTP, but you can use FTP.

>Can individual files be restored from specific times?
Yes, if snapshot backups are used. Mirror backups, on the other hand, will only keep a single copy representing your files the last time you ran the backup.

>Are all file ownerships and privileges preserved?
Yes, although a non-privileged user will not be able to back up files for a different user and preserve file ownership. There may be a minor issue with changed directory ownerships under certain circumstances, but I don't think it affects Puppy, where this would presumably be run by root.

>Are my symlinks included in the backups?
Symlinks yes, hard links no.

I have updated the screenshots at the top of this post.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 19 Jan 2010, 00:29    Post_subject:  

PS
To clarify the "minor issue with changed directory ownerships under certain circumstances..."

In fact, this is an issue that has concerned me for some time. I delved into into it today, and confirmed that by using cp to create the hard links, permissions and ownerships in file in old snapshots can get the value of the file in the most recent snapshot, if the only change to a file is a change in ownership or permission.

Fortunately, rsync after version 2.5.6 has a special --link-dest switch designed to create hard links (or not) properly for snapshot backups.

I had tried this previously, but rejected it because it cluttered up the log files with directory entries. My workaround is to throw away all directory entries from the rsync log.

3.2-17 is the fixed version, and will be uploaded tomorrow.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3447
Location: West Lothian, Scotland, UK

PostPosted: Tue 19 Jan 2010, 05:52    Post_subject:  

1. OK, I think I now substantially understand how Snap2 works. Very Happy
Tell me if I'm right or wrong.

e.g.
2. "mirror backup is an exact SINGLE copy of your folders/files"
This keeps only ONE copy of the way things were the most recent time a mirror backup was made.
When you make yet another mirror backup of the source...
Only DIFFERENCES are copied over to alter/update the mirror copy.
The 1st time you make a mirror is probably the only time EVERYTHING is copied over, so it takes the most time and involves the greatest diminishment of available free space.

3. "A snapshot backup keeps a SERIES of 'snapshots' of your files, which look like full backups"
The first snapshot looks/behaves the same as the first mirror.
The big difference is in the making of subsequent snapshots.
Whereas a mirror alters the content to incorporate the "differences", and the latest mirror is now a perfect copy of the latest state of the source...
A snapshot looks as though it has made a whole new additional backup that includes all the "differences"since the previous snapshot.
In fact it really only comprises the differences...
But the new snapshot has been made to appear like a full new backup by adding links for all the unchanged folders/files, to the original copies in the 1st snapshot.
What if you then DELETE the 1st snapshot?

4. "only the differences between files will be transmitted...which can give a huge savings in time and/or bandwidth"
This is the main motivation for me switching to this from using Xfe to make and restore backup copies of the folder/file contents of Partitions.
At the moment I have multiple [4] Xfe backups of sda3 made between last November 11th and now [Jan15].
I could save about 4Gb there alone; and there will be a BIG saving in time. Very Happy Cool

5. "I use the same backup storage directory for both my snapshot and my mirror-type backups"
I see why this is the way to go; the program makes/names its own mirror & snapshot sub-folders within the chosen/specified backup folder.

6. "Snapshot and mirror backups are completely different types of backups and do not interact"
OK, I get it...
But isn't it possible to make Snap2 make use of a mirror [to subsitute for the 1st snapshot] if it's available?
Or is your reasoning: "why do that when you can make a snapshot?"

7. "mirror backups are a subset of snapshot backups"
I don't understand this statement; don't know what a SUBSET is. Confused

8. "To recover your files, just copy them back. I highly recommend mc (Midnight Commander)"
I normally use the SUPERB "X File Explorer" [Xfe] to do such work.
I can be easily configured to display twin panes [in addition to the leftmost pane displaying the folder hierarchy].
These can be used to "Copy to..." or "Move to..." from one pane to the other.
Is MC better than Xfe?

9. "You could certainly use snap2 to back up an entire partition, but I don't think that is usually necessary"
(a) I've been doing this [backing-up entire partitions] routinely for YEARS.
I always move [almost] ALL of the data files off the OS partition, so as to reduce the content to only the OS and program files.
e.g. I moved the Mozilla folder off sda3 to sda11 [with a Symlink back to the original location], and made the necessary configurations to have sda11 auto-mounted.
This prevents the Mozilla contents "jumping-back" when I restore the sda3 partition contents to an earlier time.
I normally then backup ALL of the contents of the [sda3? and others?] OS partition[s].
Backing up [and restoring] the contents of an OS partition at various point in time...
Gives the possibility of of eliminating [ALL] software problems.
e.g. If the OS has ceased functioning normally, you can restore a normally working system.
Or if you make some [complex?] change, then think better of it, you can easily take things back to the way they were.

(b) I backup the data partitions separately.
Certain types of data files don't need backed-up so often as the OS partition contents.
And certain data files [emails, address book, etc] you wouldn't want to revert to an earlier time.
Back to top
View user's profile Send_private_message 
2byte

Joined: 09 Oct 2006
Posts: 357

PostPosted: Tue 19 Jan 2010, 10:44    Post_subject:  

Istandish,

I have a specific purpose for snap2 in mind and would like to clarify it's usage.
I have a Linux LAN server (not puppy) that holds files for several groups and users. If I (root) perform the snapshot backups for all users and then restore them while being root, will the original file and directory ownerships and permissions that existed at the time of backup be restored?
.

_________________

Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Tue 19 Jan 2010, 15:16    Post_subject:  

2byte,
YES, you can do it as you say and all the permissions and ownerships will be preserved.

Sylvander,
#2 You got it!

#3: As you say, when a new snapshot backup is made, all files that are unchanged from the previous snapshot are duplicated using hard links rather than by making a separate copy. The hard links for a new snapshot (the newest is always in a directory called recent.1) are linked to files in the next-newest backup (usually recent.2). However, if most of the files in recent.2 are themselves hard links, where is the "real" file? Is it the first copy made?

The answer is "no". In a Linux filesystem, ALL files are hard links to a storage area on the filesystem. So the first copy of a file is really just a hard link itself, no different from any other hard link that might be created to the same file. There are no "original copies." When you think in terms of "original copy" and "link to the original copy", you are thinking of symbolic links, not hard links.

So, you can delete ANY hard link, and the actual file (on disk) will not be deleted until the LAST hard link is removed.

You can use the 'ls -l' command in a terminal to see how many hard links a file has.

#6 The latest snapshot backup is exactly the same as a mirror backup, so there is no point in having both. If you have made a mirror backup and want to convert to snapshot-type backups, and you want the mirror backup to be used as the hard link reference, you would first configure your snapshot backup with snap2. Then, before you run a backup, rename the 'mirror' directory to 'recent.1' (assuming they are in the same 'backup storage' directory).

#8 "Is MC better than Xfe?" No room for that discussion here! Smile

#9 snap2 should work fine to give you the multiple versions of OS backup that you want. Have fun!
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 1 of 11 Posts_count   Goto page: 1, 2, 3, ..., 9, 10, 11 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1213s ][ Queries: 13 (0.0060s) ][ GZIP on ]