rsync - Sep 2014 - Recycling and keeping backups - Tower of Hanoi management of backups using rsync

Thanks to Kevin and Paul for responses.


We use a modified Tower of Hanoi scheme (on top of rsync and --link-dest
and recycling) for deciding which backups to keep.

Here is a sample of our holdings for one area:

 	home.20111124.seq.0   set  0
 	home.20130512.seq.512 set 10
 	home.20140203.seq.768 set  9
 	home.20140414.seq.832 set  7
 	home.20140708.seq.896 set  8
 	home.20140815.seq.928 set  6
 	home.20140831.seq.944 set  5
 	home.20140904.seq.948 set  3
 	home.20140908.seq.950 set  2
 	home.20140909.seq.951 set  1
 	home.20140910.seq.952 set  4
 	home.20140911.seq.953 set  1
 	home.20140912.seq.954 set  2
         Found 13 backups as expected up to sequence number 954
         >>>>Marking for recycling home.20140908.seq.950, set number
2

The coverage matches the likelihood of restorations being required - the
coverage tails off exponentially over time.

I found the key to running a Tower of Hanoi scheme is to assign a
sequence number to each backup, from which you can derive a set number.

The dates are for humans - scripts don't have to deal with days of week,
days of months, etc....


We keep one of each set number, except we keep two set ones, and a 
'set 0' as a base set.

With daily backups, any file that has existed for a period of more than
about 1.5*n days in the last 2*n days will be covered, with better
coverage than that for recent files.

A strategy to keep two of every set will easily provide cover for every
file that existed for n days in the last 2*n days.

Kevin: you could use Tower of Hanoi for managing your snapshots...  :-)


Regards

Rob.

Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
Information Management and Technology
CSIRO
T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333
Robert.Bell at csiro.au<mailto:Robert.Bell at csiro.au> | www.csiro.au |
wiki.csiro.au/display/ASC/
Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia

PLEASE NOTE
The information contained in this email may be confidential or privileged.
Any unauthorised use or disclosure is prohibited.  If you have received
this email in error, please delete it immediately and notify the sender by
return email. Thank you.  To the extent permitted by law, CSIRO does not
represent, warrant and/or guarantee that the integrity of this
communication has been maintained or that the communication is free of
errors, virus, interception or interference.

Please consider the environment before printing this email.