Keeping incremental directory-tree snapshots using hardlinks and rsync/wget
From SynologyWiki
Contents |
Introduction
What will this scripts/modifications give you? It will give you a way to take direct advantage of directory-tree snapshots using minimal amount of extra space on your harddrive by using the hardlink capability on your filesystem. It will work on any directory-structure you are keeping up-to-date using rsync -a (like the Synology Manager´s NetBackup-folder), and if you dare to use a custom local-backup-to-USB-script in your crontab, it will also work for your local backup USB-disk (RAID1).
This trick, if used correctly, will take up minimal of extra amount of space, just like other incremental snapshotting tools like: tar. It will not, however, compare files, and store them in a database, like BackupPC does (which is quite CPU and IO-intensive, but effective).
However, any time you directly modify any files, without deleting them first, you will also edit the hardlinked copies in the other snapshots, so it will not work as expected on your public folder, or any shared folders you are using, or modifying directly, ie. through the FileStation. So such editing will be without the comfort of the snapshotting feature, beware!
Preparations before we begin
Before we really begin, you DO have a BACKUP of everything on a separate disk? Yes? If not, make one now, or repent later! ;-)
As always when concerning backup: Always test all possible scenarios BEFORE disaster strikes!
Also, be sure you are using Synology´s native filesystem on your disks, or else, the hardlink-copying-trick will not produce hardlinks at all and this tip will be a fast way of filling your disks with duplicate files instead! This trick will only work with Synology´s native filesystem. You have been forewarned. Better check the disk-usage on all your disks after using this for some time..
I will assume you have installed ssh and/or telnet services on your Synology DiskStation (if you have a hardware firewall, I recommend both, so you are sure to be able to log in if one of the services fail on you). You may use any one of the available guides you find to do this (downloadable files are available on the official Synology page for the DiskStation-series).
Next be sure to have installed the ipkg tool, for easy upgrading of optional packages.
Eg. I executed this shell-script in order to get it all up and running, but you may want to use a later version too:
- sh ds101-bootstrap_1.0-4_powerpc.xsh
Using "ipkg install", I have installed the following packages in order to make life easy. You may want to do it too, so you can follow the instructions later:
- bash_3.2.33-1_powerpc.ipk
- busybox-base_1.10.2-1_powerpc.ipk
- busybox-links_1.10.2-1_powerpc.ipk
- cron_4.1-7_powerpc.ipk
So there. With the initial system and packages up and running, I will assume you are familiar with CLI and know enough not to mess things up, or have some way of restoring it if things fail. If not, then please do not modify your system further.
The directory structure rotation script
First of all you will want make a script, e.g. I put it in:
- /opt/local/bin/rotate_dir_file_date.sh
Btw, if you disagree with vi´s auto-indenting as much as I do, you can turn it off in vi with:
- :set noai
- :set noautoindent
#!/opt/bin/bash
# Rotate directory specified by parameters inserting current date
# Usage: rotate_dir_file_date.sh <rootpath> <dir> <type> <maxnumber>
ROOTPATH=$1
DIR=$2
TYPE=$3
MAXNUMBER=$4
if [ -z "$ROOTPATH" ]; then
echo "Missing argument #1: rootpath to $0" >&2
exit 1
fi
if [ ! -d "$ROOTPATH" ]; then
echo "Missing directory '$ROOTPATH' for: rootpath to $0" >&2
exit 1
fi
if [ ! -d "$ROOTPATH$DIR" ]; then
echo "Missing directory '$ROOTPATH$DIR' for: rootpath and dir to $0" >&2
exit 1
fi
if [ -z "$DIR" ]; then
echo "Missing argument #2: dir to $0" >&2
exit 1
fi
if [ -z $TYPE ]; then
echo "Missing argument #3: type to $0" >&2
exit 1
fi
if [ -z $MAXNUMBER ]; then
echo "Missing argument #4: maxnumber to $0" >&2
exit 1
fi
# Create a new directory appended with date
NEWDIR=@${DIR}.${TYPE}.`date +\%Y\%m\%d`
rm -rf ${ROOTPATH}${NEWDIR} 2>/dev/null
cp -al ${ROOTPATH}${DIR} ${ROOTPATH}${NEWDIR}
# Touch directory to make sure of correct sorting
touch ${ROOTPATH}${NEWDIR}
# Delete the oldest directories (on a system with buggy tail +n command)
SAVEDIRS=`ls -1dt ${ROOTPATH}@${DIR}.${TYPE}.* | head -${MAXNUMBER}`
SAVERX=`echo ${SAVEDIRS} | sed -e 's/ /|/g'`
rm -rf `ls -1dtr ${ROOTPATH}@${DIR}.${TYPE}.* | egrep -v "${SAVERX}"`
|
Make sure you chmod this script with:
- chmod 700 /opt/local/bin/rotate_dir_file_date.sh
I am sure this script can be improved / standardized more, but this does the trick for me. Feel free to improve it!
However, please notice the avoidance of the "tail"-command to solve this, as BusyBox doesn´t give the right implementation of "tail +n", it has been solved using other tools residing under /opt instead (and time it took to find them!).
The main trick here is using "cp -al", which will create hardlink-copies. These snapshots will not take as much extra space as a full copy, if you make sure the directory is being updated using "rsync -a". Mighty useful indeed!
For more details about this trick, please refer to: Easy Automated Snapshot-Style Backups with Linux and Rsync
So what will it do? The script above will create rotations of directory-structure of your choice, prepending with @ to avoid confusion with Synology server-software, and appending with today´s date (although you may want to customize this if you like to include time also). So this script is a bit tailored to the Synology file-toplogy although most of it can be reused on other platforms also, with some careful testing of course.
Because Synology auto-detects fileshares automatically, we´ve chosen to hide the snapshots by prepending the shares with @. You don´t want to remove this character, as Synology remembers the folders, even after you have deleted their respective directories. It turns quite messy very fast as snapshots gets auto-generated, so better hide these directories: out of sight, out of harm. YOU will know how to extract them if anything fails anyways..
The script for rotation operates on 4 parameters, all of which must be passed to the script:
- rootpath: The full path of your root directory appended with slash (/), eg.: /volume1/
- dir: The directory for the directory structure you´d like to rotate, eg.: NetBackup
- type: Any name you want to call the rotation, eg.: daily, weekly, monthly, yearly are obvious choices
- maxnumber: Maximum number of rotations ordered by date, e.g.: 4 will give maximum number of 4 hardlinked copies. When the maximum is reached, any superfluous copies are deleted!
Example crontab
Here´s an example of /opt/etc/crontab (use /etc/crontab on your own risk and headache!). It shows typical use of the directory-structure rotation script on a: daily, weekly, monthly and yearly-basis (very flexible and handy indeed!!). Please notice the use of shell environment variables to locate correct PATHs and using bash, although they may not be necessary.
This is the crontab I have used:
# DO NOT EXPECT CRON TO TAKE CHANGES IN THE CRONTAB-FILE UNTIL 2-3 MINUTES HAVE PASSED # FOLLOW THIS FORMAT STRICTLY. /OPT/BIN/CRON DOES NOT LIKE TOO MUCH VARIATIONS # USE 1 TAB BETWEEN CRONTAB ARGUMENTS INSTEAD OF SPACES AND TRIM ALL LINES TO BE SURE # ALWAYS TEST YOUR CHANGES IN REAL-TIME AFTERWARDS #-------------------------------------------------------------------------------------- SHELL=/opt/bin/bash PATH=/opt/sbin:/opt/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/bin MAILTO="" HOME=/ DIR=safedummy export SHELL PATH MAILTO HOME #-------------------------------------------------------------------------------------- #minute hour mday month wday who command #-------------------------------------------------------------------------------------- # Rotate 'NetBackup' folder under the /volume1 directory, retaining files for up to 4 years: 0 21 * * * root /opt/local/bin/rotate_dir_file_date.sh /volume1/ NetBackup daily 7 15 21 * * 0 root /opt/local/bin/rotate_dir_file_date.sh /volume1/ NetBackup weekly 4 30 21 1 * * root /opt/local/bin/rotate_dir_file_date.sh /volume1/ NetBackup monthly 12 45 21 1 1 * root /opt/local/bin/rotate_dir_file_date.sh /volume1/ NetBackup yearly 4 # Rotate 'usbbackup' folder under the /volumeUSB1 directory, retaining files for up to 4 years: 0 23 * * * root /opt/local/bin/rotate_dir_file_date.sh /volumeUSB1/ usbbackup daily 7 15 23 * * 0 root /opt/local/bin/rotate_dir_file_date.sh /volumeUSB1/ usbbackup weekly 4 30 23 1 * * root /opt/local/bin/rotate_dir_file_date.sh /volumeUSB1/ usbbackup monthly 12 45 23 1 1 * root /opt/local/bin/rotate_dir_file_date.sh /volumeUSB1/ usbbackup yearly 4 # Typical use of rsync -a which is compatible with our rotation of hardlink-copies: 0 19 * * * root rsync -a --delete -e ssh user@server1.example.com:/ /volume1/public/backups/server1.example.com/ # Typical use of wget´s ftp which is NOT compatible with hardlinks (will only transfer newer files based on date-stamp, thus any modifications will alter ALL hardlinked-copies unfortunately..): 10 20 * * * root DIR=/volume1/public/backups/server2.example.com;mkdir -p $DIR;cd $DIR;wget -m -nd --no-passive-ftp -q ftp://user:password@server2.example.com:/ # Typical use of wget´s ftp which is semi-compatible with hardlinks (will remove files and then transfer every file. TODO: Can wget be used to archive files somehow?): #10 20 * * * root DIR=safedummy;DIR=/volume1/public/backups/server2.example.com;mkdir -p $DIR;cd $DIR;rm -rf $DIR/*;wget -m -nd --no-passive-ftp -q ftp://user:password@server2.example.com:/ |
Local backup to USB-disk (RAID1)
The Synology-version of local backup to USB-disk (RAID1) seems to not use the archiving-mode, so the USB-disk will not have correct snapshots either if you use the rotationscript on the /volumeUSB1-partition. To fix this, you may want to use rsync -a manually in /opt/etc/crontab, but you will then also be on your own and not get meaningful reporting through the UI and email. If you decide to do this, remember to disable scheduled local backups so they will not interfere with your custom solution, unless you enjoy getting strange and mixed behaviour in your system.
Here is an example of added lines to /opt/etc/crontab for scheduled rsync -a to your local USB-disk:
# Backup folders to USB-disk using archive option for hardlinks-rotating compatibility 0 6 * * * root rsync -a --delete /volume1/backups /volumeUSB1/usbbackup/ 0 7 * * * root rsync -a --delete /volume1/opt /volumeUSB1/usbbackup/ |
Conclusion
As you can see, this setup is pretty simple and solvable. Within an hour or so, you may be up and running with your snapshotted backups, something which a "backup solution" should really deliver from the start I might argue.
It has some rather strange limitations though, but for incremental networked backups using rsync -a (which is the Synology-default), it is simply genious! Also for the USB RAID1-disk, this will work, with some manual modifications to your system.
I wish it also would work for shared folders, but after all you should have at least a RAID1 and a remote backup anyways, and it´s at the backup it is mostly needed. The solution is really "good enough", and dead simple once you have it up and running.. ;-)
If you decide to modify the snapshot-files, please remember to unlink the file in question first, e.g. by:
# TODO: Any simpler way of making a fresh file from a hardlink?
cp file file.new # Creates a fresh copy of the file
vi file.new # Do your modifications on the file
mv -f file.new file # Overwrite hardlink with the fresh file
|
Then you will not have to deal with the headache of having your identical snapshots altered as well..
Good luck!
Further Resources
- Easy Automated Snapshot-Style Backups with Linux and Rsync
- It is also possible to do this using rsnapshot. The configuration file is a bit tricky (beware of the tabs you need to use), but there is documentation for it. 'rsnapshot' is based on 'rsync'. It works on a DS409.
