Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 21 Apr 2014, 00:45
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
snap2 rotating snapshot backups for Puppy
Post new topic   Reply to topic View previous topic :: View next topic
Page 7 of 11 [158 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 Next
Author Message
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Sun 14 Feb 2010, 19:48    Post subject:  

@lstandish
Re your post of today, Sun 14th Feb at 13:23...
1.
(a) Thank you for listening and giving careful consideration to what I had to say.
On reading your post its quite difficult for me to fully understand exactly how all of that works.
I only roughly get the idea:
One of the daily backups being promoted to a weekly.
Then one of the weekly backups being promoted to a monthly.
I guess after some time you end up with a set of dailies, a set of weeklies, and a set of monthlies?
And new snapshots go in at the top, and old snapshots drop out at the bottom?

(b) Perhaps after using snap2 for some time it will begin to become clear to me.
Is there anything [especially visual] that would make it all seem simple?

2. New topic:
(a) Today I was trying something new.
Wanted to run snap2 within a Puppy loaded from a USB2 Flash Drive...
So that it wouldn't load anything from any partition on the internal HDD [no partitions auto-mounted].
This so the Puppy could be used for imaging [all of the internal HDD] as well as using snap2.
After various failures to get a good install of Twinkle & Puppy-4.31.1 I settled for using an existing install of Puppy-4.2.1 on a 1GB USB2 Flash Drive.
Installed snap2 successfully to that.
Had to do some of the installation of dirdiff manually [the PET failed], but it worked.
Also installed from the PPM:
tk-8.5.6-V1-DEV
tcl-8.5.6-V1-DEV
In the hope these might supply needed dependencies.
Both snap2 & dirdiff seemed to be working normally.

(b) Ran snap2 to make updated snapshots of:
sda1&sda5; sda3; sda6; sda7.
All of those took hours to complete, and seemed to be backing up ALL of the contents of the source partitions.
And yet, when I used dirdiff to look at sda7 & sda6...
And compare the latest snapshot with the previous snapshot...
The snapshots were identical for both sda7 & sda6.
Any idea what's going on there?
Problem with installing snap2 in Puppy-421?
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Sun 14 Feb 2010, 20:45    Post subject:  

Sylvander,

(a) I'll try to explain snap2 backup rotation through a directory listing. (Following is from my website http://www.linuxbackups.org/node/29)

"Over time, the program rotates the backups to create a directory structure like this:

lloyd@debiandesk:/media/BACKUP8GIG/snapbackups$ ls -l
total 96
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-23 21:12 daily.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-22 08:35 daily.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-21 08:35 daily.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-18 08:35 daily.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-17 08:35 daily.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-16 08:35 daily.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-14 08:35 daily.7
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-13 08:35 daily.8
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-12 08:35 daily.9
drwxr-xr-x 3 lloyd lloyd 4096 2009-10-07 23:06 mirror
drwxr-xr-x 4 lloyd lloyd 4096 2009-05-25 08:35 monthly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-04-19 08:35 monthly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-03-19 08:35 monthly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-02-08 22:24 monthly.4
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-15 22:36 recent.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-13 20:59 recent.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-10-07 23:04 recent.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-09-07 13:51 recent.5
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-27 08:35 recent.6
drwxr-xr-x 4 lloyd lloyd 4096 2009-08-06 08:35 weekly.1
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-30 08:35 weekly.2
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-22 08:35 weekly.3
drwxr-xr-x 4 lloyd lloyd 4096 2009-07-14 18:00 weekly.4"

My configuration specified maximum 6 'recent' snapshots, 9 'dailies', 4 'weeklies' and 4 'monthlies'. Most of the backups were done automatically, daily at 8:35 AM.

As you can see, snap2 deletes a whole lot of snapshots in order to get them 'spaced out' as 'weekly' and 'monthly' snapshots. For example, only one daily snapshot per week is 'promoted' to a weekly snapshot. All the other 'dailies' are discarded at rotation time.

Since many snapshots will be deleted, it is advantageous to allow duplicate shapshots to exist, so your data will live longer (unless the duplicates are 'recent' snapshots that were done the same day).

(b) Check the backup logfile for the most recent snapshots. At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was. Also, if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing. Only a few or no files should be listed if there were few or no changes between the 2 snapshots.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Mon 15 Feb 2010, 03:16    Post subject:  

1. " At the beginning it will indicate whether or not it found a "hard link reference", and if so, what it was"
There was indeed a "hard link reference" found as follows:
"Backup run on Sun Feb 14 19:56:28 GMT 2010
Using settings from /root/.snap2/sda7.set/settings
Using rsync compression
Snapshot hardlink reference: recent.1
"

2. "if a lot of files are reported as backed-up in that logfile, then you indeed got a new disk copy of the whole thing"
(a) Yes, a great long list of files in the log, and yet that snapshot is found by dirdiff to be identical to the previous snapshot.
Any idea why the long backup list?

(b) I'll try making yet another snapshot of [for example] sda7, from the copy of snap2 within Twinkle, and see what happens.
Just did that; it completed within about 3 seconds, nothing listed. Very Happy
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 15 Feb 2010, 08:22    Post subject:  

Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?
_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Mon 15 Feb 2010, 09:51    Post subject:  

1. "Was there any difference in the source paths (due to different mount point?) between the snapshot run within your regular Puppy and your USB Flash Drive Puppy?"
(a) I believe there was not. [What's a mount point?]
e.g. dirdiff said that the latest snapshot [for sda7 for example] was identical to the previous snapshot.
And the path to the source was identical in each.
And the path to the destination was identical in each.

(b) Decided to by-pass that problem as follows:
I copied the pupsave->[plus companion files] for Twinkle...
To a different 1GB Flash Drive...
And booted the Twinkle live CD-RW, which sought and found and used the pupsave on sdb1.
Twinkle is using the pupsave on sdb1 [seen at /mnt/home].
That had the desired effect; no partitions on the internal HDD are mounted.
The backup destination partition sdc1 is on the external USB HDD.
I'm in the process of deleting the existing snapshots for all sources, and making new snapshots.
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Wed 17 Feb 2010, 06:36    Post subject:  

PROBLEM?

1. Making yet another manual snapshot of /mnt/sda7 to /mnt/sdc1/backups/snap2/sda7.
ALL partitions are mounted.
Using Twinkle-421 [built on Puppy-431] with pupsave+companion files recovered to a repartitioned & reformatted Flash Drive.
Twinkle & snap2 seem to be working well, except for this "problem".

2. Prior to commencement, there existed recent.1, recent.2, recent.3

3. The new snapshot [recent.0] was using as its reference point, recent.2 rather than recent.1

4. sda7 had probably not been changed, or possibly a very few files, and yet as I type it appears to be making a snapshot of ALL OF SDA7.
How come?
Now completed, and dirdiff shows recent.1 & recent.2 are identical.

5. To repeat: both source and destination partitions are mounted.
sdc1 has had its partition file system checked/fixed using GParted.
I'll do the same for sda7 ASAP.
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Wed 17 Feb 2010, 10:13    Post subject:  

For the actual backup, snap2 calls rsync. I don't think rsync has any major bugs.

If snap2 used recent.2 rather than recent.1 for the hard link reference, it must not have found recent.1

If recent.2 is newer than recent.1, snap2 will detect that and will refuse to run. I don't think there is any problem in the snap2 code for finding the newest snapshot (for the hard link reference)

Remember that during the rsync run, the new snapshot is recent.0, but as soon as it is finished it will rename all the snapshots and recent.0 will become recent.1.

Using the information you sent, I can't help. You could try sending me a detailed listing of your destination dir ('ls -l') just before you run the backup.

Or, if you can send me a source and destination directory structure that reproduces the problem, then I could help. Of course, that could not be entire partitions.

I suggest you set up a simple test backup of a small directory and see if you can reproduce the trouble. Then you could zip the whole thing up and send it to me.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Wed 17 Feb 2010, 11:03    Post subject:  

SUCCESS?! Very Happy Confused

1. Rebooted a couple of time into various Puppies.
Rebooted into Twinkle.

2. Ran snap2 yet again, and made more snapshots of...
sda1&sda5; sda3; sda6; sda7.
All behaved EXACTLY as they aught. Very Happy
Said that there were no changes to sda1&sda5; sda6; sda7. [There should have been no changes]
Said there were changes to the contents of sda3, which is as expected.

3. So all seems to be OK. Very Happy Cool
Don't know why the previous problem; will keep a watch for a repeat, otherwise do nothing.
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Sat 20 Feb 2010, 09:48    Post subject:  

Problem? Sad

1. Brought yet another Puppy into use [Lighthouse-4.43f built on Puppy-4.31.1 I believe].

2. Saved the pupsave+companions on an 8GB Flash Drive.
Decided to use this method so that no partitions on the internal HDD would be auto-mounted, and therefore the Puppy would be free to do anything/whatever with the internal HDD.

3. Lighthouse auto-mounts ALL partitions it can see, which is either a convenience or a pest.
Doesn't see my external USB HDD, unless I power on AFTER Lighthouse gets to the desktop, so I normally do that when needed, and then mount it [/mnt/sdc1].

4. Installed lots of programs including snap2 and dirdiff.

5. Ran snap2 to make yet another snapshot of sda1+sda5.
[After having made all the necessary configurations, and mounting sdc1]
snap2 began making a total backup of ALL of the contents of sda1, so I halted it by closing the window, then the program, then deleted recent.0
How about providing and "ABORT" button on the window?

6. Looks like snap2 does the above IF run from a different Puppy.
Is it possible to make it behave normally, even if run from different Puppies?
Otherwise I'll be forced to always make snap2 snapshots from the same Puppy, and what do I do when/if it fails?

7. Will now try running snap2 from Twinkle and see if it behaves as it aught, then report back.
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Sat 20 Feb 2010, 12:54    Post subject:  

The above problem is only happening when working with FAT32 source partitions. Confused

When making a snapshot of the ext3 sda3 partition, all's well. Very Happy

i.e. Everything works OK in Twinkle.
The previous snapshots were made in Twinkle, and the new snapshots of unchanged source partitions show as unchanged in Twinkle, but NOT in Lighthouse [EXCEPT for ext3 sda3].
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Sun 21 Feb 2010, 17:16    Post subject:  

I thought snap2 would work OK (in snapshot mode) as long as the destination partition is a Linux partition. I need to check for myself what happens when the source partition is FAT32.

There should be no problem running snap2 from different Puppies to backup the same source folders to the same storage partition. The mount point of the destination folder should not make a difference, either. That is, if you mount your storage partition at /mnt/backups/ on one run and at /tmp/12345.temp/ the next time, that should not make any difference, even though that changes the overall path to the backups.

I am working on automount for both source and destination folders, on both local and remote storage media. However, I have not been able to test this enough to make a release.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Mon 22 Feb 2010, 04:45    Post subject:  

EXPERIMENT-1...
On sda1+sda5:
[Windows partition + its data partition (both FAT32)]

1. Booted into Lighthouse, powered-on the ext HDD and mounted the partition [all other partitions already mounted].

2. Deleted all snapshots of sda1+sda5.

3. Made a new snapshot of the partition pair.

4. Waited overnight and [this morning] made a new snapshot of the pair.
This done from within Lighthouse.
snap2 recognized that the partition contents were unchanged, and didn't list any folders/files for backup.

5. Deleted the newly made and identical recent.1, and renamed recent.2 to recent.1

6. Re-booted into Twinkle and ran snap2 and made fresh snapshots of sda1+sda5.
snap2shell began listing files being copied.
Terminated the copy, and deleted recent.0

7. Conclusions anyone?
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Mon 22 Feb 2010, 06:30    Post subject:  

Continuing EXPERIMENT-1:

8. Booted back into Lighthouse.

9. Ran yet another snap2 snapshot on sda1+sda5.

10. As expected, it detected there had been no change to the contents of these partitions, so listed no changed files.

11. Deleted recent.1 and renamed recent.2 to recent.1
Back to top
View user's profile Send private message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Mon 22 Feb 2010, 11:12    Post subject:  

Were the full source paths of the mounted partitions exactly the same on both Lighthouse and Twinkle?
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 3252
Location: West Lothian, Scotland, UK

PostPosted: Mon 22 Feb 2010, 13:29    Post subject:  

Yes, and the destination too was good.
And I checked the configurations for those, and they were good also.
And the snapshot by Twinkle began to be saved as recent.0 right above recent.1

The only problem seems to be that if I change to a different Puppy->[snap2], it acts like it doesn't know of the files saved by the previous Puppy->[snap2].

Does snap2 save [somewhere in the Puppy filesystem in the pupsave file] a record of the files saved in the previous snapshot?
In which case, when I switch to a different Puppy, then its snap2 wouldn't know of the previous list of files in the snapshot made by the other Puppy->[snap2].

I'm using Lighthouse->[snap2] and gradually working through from sda1+sda5 to sda3 to sda6->[now completed]...
Deleted all the existing snapshots, and made a new snapshot.
Now, I bet when I run snap2 new snapshot...
All will be well...
But if I use Twinkle->[snap2], it will act like the previous snapshot doesn't exist, and backup ALL of the contents of the partitions all over again.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 7 of 11 [158 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1070s ][ Queries: 12 (0.0155s) ][ GZIP on ]