DataReplicator to LAN NAS to WAN NAS - For critical data with minimum data transfer

From SynologyWiki
Jump to: navigation, search

Contents

What is this Wiki page about?

This Wiki describes how to establish an automatic routine to backup data on a LAN (Local Area Network) NAS to a WAN (Wide Area Network, e.g. over the web) NAS. It is mainly aimed for users of Synology's Data Replicator III (DRIII) software who backup PC data to the LAN NAS and then (for safety) want that data replicated to a remote (WAN) NAS. This procedure will use a much much smaller amount of WAN data transfer than the operation of DRIII and the Synology Network Backup module would allow without using this modification (see below). It can also be beneficial for LAN NAS to WAN NAS backups even if you do not use DRIII.

This Wiki was created after two friends both bought Synology NAS's and used them so each person has a local and a remote backup of data via each others NAS and the internet.

If you have data you just can't or don't want to loose then you probably already know you should have it stored in a remote location so that it has a separate fire/theft/etc risk to the source of the data. To do that automatically without human intervention normally means transferring the data via a WAN (e.g. the internet) between two rsync enabled devices (e.g. two Synology servers).

Why should I use this modification

In short this routine will dramatically reduce (and monitor) the amount of WAN data transfer used and enable you to control the WAN bandwidth used during a LAN NAS to WAN NAS backup when compared to the Synology provided NetBackup module. This is due to the flexibility provided by being in control of the rsync command used for LAN NAS to WAN NAS backup.

Using this procedure you specify the folders on the LAN NAS to be backed up to the WAN NAS. If you use DRIII then the amount of WAN data reduction is very significant. Using DRIII and standard Synology NAS's you can backup Windows PC data to a LAN NAS and then to a WAN NAS. However because of the way DRIII saves its backup files ***EVERY*** LAN NAS to WAN NAS Network Backup will be at least as big as the TOTAL (i.e. full backup) amount of data being backed up.

This is because DRIII uses a root directory name as a date and time stamp of the backup, every time you run DRIII (either incremental or full backup) a root directory name changes. Therefore rsync (the software tool used by Synology's built in Network Backup Service) sees all of your backup data as being new. Hence if you have 50Gb of data on your PC, and have made no changes to any of it, after running a DRIII backup your LAN NAS to WAN NAS backup will be 50Gb!

If you have made changes to files the situation is even worse, you will be transferring the 50Gb + X*changes_Gb of data, where X is the value you have set for "Number of past versions" in DRIII. As the minimum number of past versions is 1 (this applies even if the "Enable file versions" check box is cleared) you will always be transferring more data via the WAN than the TOTAL amount of data to be backed up from your PC. This is because changed files are additionally linked in a root directory called "Latest" which will also be backed up as whole files rather than as links to the files.

Using this modification procedure you will only transfer by WAN the changed data, for most people this hugely reduces the amount of WAN (internet) bandwidth used.

Additionally this routine provides two further levels of control. The Synology supplied "Network Backup" function does not allow the use of the "-z" (compression) option, and (prior to fw844) forced the use of the "-W" option. By NOT using the "-W" option (in fw844 this is achieved by selecting the "block-level backup") and by using the "-z" compression option (not available in the Synology Network Backup function) you can also further reduce the amount of data transfer over a WAN in most circumstances.

An example of how much less data will be transferred

I have a Windows laptop with 40Gb of files in "my documents". Mostly the files are static, the biggest single file changes are outlook.pst (1.5Gb) and archive.pst (1Gb). These two files change every day and result in a daily DRIII to LAN NAS transfer of usually around 3Gb for all my laptop data. The last DRIII restore point data set on the LAN NAS is therefore 40GB (the size of my "My documents" data), but I have 20 restore points, plus 2 previous file versions of every file. So in total some 1TB of DRIII data as would be seen by the WAN NAS if you didn't use this mod. Additionally my music, photo and video collection held on the LAN NAS (not on my laptop) is a further 200Gb.

Using this mod a typical actual data transfer between LAN NAS and WAN NAS (i.e. via the internet) of the last DRIII restore point (40Gb) plus my music/video/photos is only 55Mb. That is not a typing mistake, it is typically only 50 to 60Mb! This is because:

  1. This routine bypasses the date and time named root directory, so rsync doesn't see all the data as being new
  2. We turn off the -W (whole file) rsync option that Synology uses in their "Network Backup" routine means rsync will analyse each file that has changed and only transmit the changed parts of that file (Note: as of fw844 this is now possible by using the "enable block-level backup" option in the synology supplied "Network Backup" routine).
  3. We use the "-z" option so the changed parts are transmitted in a compressed way.

So for instance my 1.5Gb outlook.pst file, although the file has changed (there are about 100 new emails a day in it) only about 40Mb of the file will have changed, and that is all rsync needs to send to the WAN NAS. That 40Mb compresses down to 20Mb, so as you can see in just that one file this mod reduces the data transfer from 1.5Gb to 20Mb.

So in short, for the whole LAN NAS to WAN NAS backup without this mod each of my LAN NAS to WAN NAS transfers would be 40Gb+, but instead it is about 55Mb and takes the NAS's about 1 hour.

What are the pros/cons/ of using this procedure?

Pros

  1. Optimised WAN data transfer, particularly for DRIII users.
  2. You can limit the WAN bandwidth used enabling you to carry on using your internet connection whilst the backup is in progress. The built in Synology Network Backup routine does not allow you to configure the bandwidth used and consequently for large WAN transfers you would notice that it would absorb almost all of your upload bandwidth during the backup, causing any internet browsing to crawl to a snails pace.
  3. This backup procedure will log the WAN data transfer (in MB) used by the backup. If you are on a restricted monthly broadband data allowance and have large amounts of data on the LAN and WAN NAS it is useful to know how much data transfer the LAN NAS to WAN NAS backup is using.
  4. This backup procedure will email you the results of the backup so you know if it worked or failed and tell you in the email the amount of data transferred.
  5. This procedure also gives examples of backing up other LAN NAS data folders (e.g. music/photo/web).

Cons

  1. This modification procedure is not recommended for complete Linux/Synology virgins as it uses several different Linux aspects. It is written assuming the reader has a basic knowledge of modding/getting around the Synology Servers at Command Line Interface level.
  2. This mod is susceptible to firmware updates, after a firmware update you need to follow the short procedure detailed below in the section What_to_do_after_a_firmware_update.
  3. For DRIII users the LAN NAS will have everything you have configured DRIII to put on it, i.e. all your file versions and restore points. This procedure does not affect the operation of DRIII backing up to your LAN NAS. But to reduce WAN bandwidth usage to the minimum the WAN NAS will only keep the DRIII current data set (the latest restore point set - reflecting the data on your PC last time DRIII was used). The WAN NAS will NOT have any DRIII previous file versions or previous restore points. If you understand basic sh programming you could modify this procedure to have more restore points on the WAN NAS if you want them.
  4. Each Synology NAS and firmware version is different. However there is normally sufficient commonality that many modifications like this one will work on various Synology models and firmware versions. This wiki page was created from my experience of using a DS207+ and CS-407 both with fw 598 but I have now been using this routine for well over 6 months and the NAS's are now on fw 732 as I write this update. In that time I have not had any problems or failures of the routine, I only need to make the two small changes detailed below after a FW update.
  5. Folders containing DRIII data that are to be backed up by this procedure must NOT be managed by the Synology built in "Network Backup" web GUI page. The web GUI reported status for the backup job created in this procedure does not reflect the last LAN NAS to WAN NAS backup made by this procedure. It only reflects the status of the last time you ran it from the web GUI. It will be visible in the Web GUI as any other Backup jobs you have created but its status will not be reported correctly. i.e. the web GUI may report the last backup as "Successful" when in reality the last one could have failed. If you complete this procedure the LAN NAS will instead email you the rsync log so you know whether it was successful or if it failed. If you run the Synology provided Network Backup routine you will erase the DRIII data on the WAN NAS. This is because the built in Synology routine will not follow the symbolic link we create in this mod and will therefore think you have deleted your DRIII data. If you do erase it accidentally (by using the Synology routine by accident) the next rsync backup routine created by this procedure will restore the data but in doing so it will have to transfer all of the erased data over the WAN. If you have a lot of data that may not be practical? If not you will need to put the two NAS's on the same LAN for data re-synchronisation. However, it is unlikely you will forget as this routine will send you an email every time it runs. Also it is unlikely you will want to use the Synology built in routine once you have gone to the effort of using this mod and seen the benefits and flexibility it provides. I've never even come close to making this mistake.
  6. All of your other web GUI created Network backup routines will be unaffected by this routine and can therefore be managed as normal through the Synology web GUI if you want to, but as I stated above that is unlikely and I now use this mod for all my WAN transfers.
  7. This procedure should work for you with the limits stated in this section, but there is no support for this modification. As with any modification there is a risk of making your Synology server either temporarily unusable or temporarily unstable (potentially requiring a full reset and complete data loss, although very very unlikely), all the normal disclaimers apply as stated in this wiki area.
  8. You must check the routine established by this procedure works after you have completed it. If it doesn't work or stops working and you don't notice (e.g. you stop receiving your emails from this routine), you could get to the stage of needing your critical data on the WAN NAS only to find it wasn't backed up! I state it because it is possible, but the routine has never failed for me so far.

Overview of the modification

  1. First we make sure that the PC using DRIII backs up to a folder on the LAN NAS dedicated for just that PC's DRIII backup data, e.g. a folder called "my_DRIII_LAN_backups", i.e. nothing else is placed in that folder, not even other PC's DRIII backups.
  2. Then we create a second folder on the LAN NAS (e.g. "my_LAN_and_WAN_backups") to hold a symbolic link to the data in "my_DRIII_LAN_backups" except it bypasses the date/time stamped directory
  3. The LAN NAS is modded to add a cron job which calls our backup procedure
  4. For DRIII users we create a script that searches for the latest DRIII backup data in a specified directory and creates a symbolic link to the backup data that bypasses the time/date stamped root directory that DRIII uses. This way rsync does not see all the data as having changed, it only sees the real changes you have made to your data.
  5. For security we have to create ssh keys to enable secure automatic rsync connections over encrypted SSH.
  6. We then create a script to call the optimised rsync command for the various folders we wish to backup. In the case of the DRIII data because the built in Synology Network Backup routine does not follow symbolic links (this is good and bad) we specify rsync to follow symbolic links and turn on data compression. If you also want to backup the music photo and video folders we turn off compression as this would be a waste of resources and could actually lead to increased data transfer (trying to compress this already compressed data is not beneficial).
  7. So the LAN NAS can tell you if the backups are taking place and everything is OK or not we install "nail" an email client.
  8. Then we create a script to extract the amount of data transferred from the rsync log and log it separately.
  9. Then we create a script to analyse the rsync log to see if it failed or not and email you the results.

As stated in the intro, this wiki is the result of two friends using each others NAS to provide WAN based backups of each others data, i.e. LAN NAS to WAN NAS "and" WAN NAS to LAN NAS, what I will from now on call "two way backup". However recognising not everyone will want "two way backup" (e.g. a small company using this procedure will probably only want one way - LAN NAS to WAN NAS) this wiki is written so that it describes how to configure for "one way backup" but gives the optional steps for "two way backups".

To keep things as simple as possible in the description of this procedure I refer to one NAS as the LAN NAS and the other as the WAN NAS from the point of view of the person making the mods, i.e. LAN NAS was my NAS and WAN NAS was my friends NAS.

If you want to configure for "two way backup" but are finding the constant switch of references from LAN to WAN and vice versa confusing then I recommend you initially ignore the optional steps described for "two way backups" and just initially configure for one way backup (LAN to WAN only). Once you have configured the one way backup and have seen how it works it will then be much easier to go back and configure the optional "two way backup" parts.

Assumptions

  1. You are comfortable with making your way around a Linux PC or Synology server at the command line level and have a basic understanding of scripts and variables.
  2. For the initial configuration it is recommended you have your PC, the LAN NAS and the eventual WAN NAS all on the same LAN. It is recommended you only move the WAN NAS to its final WAN position after completing the configuration and testing it.
  3. You know how to configure your routers/firewalls at the LAN and WAN points to enable the LAN NAS to WAN NAS backup to proceed.
  4. You have at least an hour spare to make this modification

The Procedure

  1. Using the Synology web based Management GUI, create two shared folders on your LAN NAS, one to receive just the DRIII backup data, e.g. "my_DRIII_LAN_backups" (you can call it aything you want) and another folder to hold the symbolic link to the DRIII data e.g. "my_LAN_and_WAN_backups". Any data you put in the folder "my_LAN_and_WAN_backups" will also get backed up to the WAN NAS. You can use any names you want for these folders but for the rest of this procedure I will use these names, so substitute where necessary. If you want "two way backup" do the same on the WAN NAS.
  2. If you have not already done so, install DRIII on your PC and configure it to make backups to the "my_DRIII_LAN_backups" folder on your LAN NAS. For "two way backup" either get your friend to your house with his PC to do his DRIII backups to the WAN NAS or if he hasn't much data he can do it when the WAN NAS is at the WAN position, or at the end of this procedure you can post the WAN NAS to him, he then makes his DRIII backup to the WAN NAS (from his perspective this is the LAN NAS, but I have written this procedure from your/my perspective) and then post it back to you for you to do a data (rsync) synchronisation and then post the WAN NAS back to him.
  3. Make DRIII do at least two backups to make sure it works. You will see that after the second backup DRIII has created in the directory/folder "my_DRIII_LAN_backups" a directory called [your PC's name] and then within that two directories; 1) the time and date stamped directory containing the current image of your PC's data, and 2) a directory called "Latest" that contains the latest/last versions of all files. As you can see your data appears to be stored twice on the LAN NAS but in fact the contents of the time stamped directory is hard linked to the contents of "Latest". This is fine for the LAN NAS but if using the built in Synology Network Backup routine for the LAN to WAN backup it doesn't copy the Hard Links it just copies the data the Hard Links point to and so on the WAN NAS you literally would have two copies of each file, hence this procedure.
  4. Using the Synology Management web GUI enable the Network Backup Service on both the LAN NAS and the WAN NAS.
  5. Using the Synology Management web GUI create a user "rsync" on both the LAN NAS and WAN NAS (you can use different passwords for each NAS if you want).
  6. Using the Synology Management web GUI create an encrypted synology server Network Backup job on the LAN NAS to backup "my_LAN_and_WAN_backups" to the WAN NAS as user rsync. Do not set any schedule for this backup. Run the backup routine at least once so you can see what it has created on the WAN NAS. For "two way backups" do the same on the WAN NAS.
  7. Enable the SSH Command Line Interface on both the LAN NAS and WAN NAS.
  8. Using the Command Line Interface log in to the LAN NAS as root and create a directory "/volume1/my_scripts" to contain the cronjob scripts. For "two way backups" do the same on the WAN NAS.
  9. cd to /volume1/my_scripts and then use "vi sub_makelinktobackup.sh" to create a new script file and paste the script "sub_makelinktobackup.sh" given below in the section "scripts", in to the file. Set the variables, save the changes and close vi. For "two way backups" do the same on the WAN NAS.
  10. Test the script by using the command "sh /volume1/my_scripts/sub_makelinktobackup.sh" and see the result by using "ls -l /volume1/my_LAN_and_WAN_backups" you should see the symbolic link. Test it by cd'ing in to it and doing a ls command again. For "two way backups" do the same on the WAN NAS if your friend has made his DRIII backup to the WAN NAS, otherwise don't.
  11. Now create a home directory for the user rsync on the LAN NAS, e.g. "mkdir -p /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys", "chown -R rsync /volume1/user_roots/rsync_user".
  12. Edit the "/etc/passwd" file on the LAN NAS using vi and change the root directory for user rsync to "/volume1/user_roots/rsync_user", this is the entry between the last ":" and the second to last ":" on the line that starts with "rsync". Then check that the shell for user rsync is set to "/bin/ash", this is the entry after the last ":" on the line. Save changes and close the file.
  13. On the LAN NAS change to user rsync to test, i.e. "su - rsync", "pwd". Do not exit user rsync yet.
  14. Create the public and private SSH2 keys for rsync user on the LAN NAS, e.g. "cd .ssh/my_LAN_NAS_rsync_user_keys", "ssh-keygen -f my_LAN_NAS_rsync_user_rsa -t rsa" (where my_LAN_NAS can be anything you want so long as it identifies the LAN NAS for you). When prompted for a passphrase leave blank and just press enter, you will be asked twice. You have now created a pair of SSH2 keys.
  15. Exit the user rsync on the LAN NAS to return to user root, i.e. "exit"
  16. Now repeat some of the steps above for the WAN NAS, i.e. login as root, create a home directory for user rsync, edit the "/etc/passwd" file
  17. For "two way backups" just create the public/private SSH keys on the WAN NAS as described above
  18. For "two way backups" exit user rsync on the WAN NAS.
  19. Now you have to move the public key (with the .pub extension) from the LAN NAS to the WAN NAS. The .pub key should be copied to the /volume1/user_roots/rsync_user/.ssh directory on the WAN NAS. Do not use windows drag and drop to move the key else you will end up moving blank files, use ftp or some other means. For me I used the cp command to copy it to a directory I had an ftp share on, and then used ftp to upload it to my PC, then ftp'd from my PC to the WAN NAS's ftp share, and then on the WAN NAS used cp again to get in the right directory. For "two way backup" copy the WAN NAS's .pub key to the LAN NAS.
  20. On the WAN NAS change to user rsync, i.e. "su - rsync" and "cd .ssh", "ls -l". You should now see the LAN NAS's public key (.pub). If you are configuring for "two way backup" you will also see the WAN NAS's own rsa key directory that you created. For "two way backup" do this same step on the LAN NAS.
  21. We now have to add the LAN NAS public key to the WAN NAS file /volume1/user_roots/rsync_user/.ssh/authorized_keys (which probably doesn't exist yet, if so the next command will create it), e.g. on the WAN NAS use the command "cat my_LAN_NAS_rsync_user_rsa.pub >> authorized_keys" where my_LAN_NAS is the name of the rsa key file you previously used. Now delete the public key file (.pub) on the WAN NAS as it isn't needed. For "two way backups" do the same on the LAN NAS.
  22. On the WAN NAS exit rsync user, e.g. "exit". For "two way backups" do the same on the LAN NAS.
  23. Now set access rights for rsync user keys and home directory, i.e. on both the LAN NAS and WAN NAS do "chown -R rsync /volume1/user_roots/rsync_user", "su - rsync", "chmod go-w .", "cd .ssh", "chmod 700 .", "chmod 600 *", "exit". Have you done both the LAN NAS and WAN NAS?
  24. Now test your ssh conection from the LAN to WAN NAS, by entering the following command on the LAN NAS "ssh -v rsync@xxx.xxx.xxx.xxx -i /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys/my_LAN_NAS_rsync_user_rsa" where xxx.xxx.xxx.xxx is the IP address of the WAN NAS. If successful you should see the command prompt of your other NAS. If it was successful type "exit" to close the SSH connection. If it wasn't successful read through the verbose output SSH gave you and see where it failed. If it failed it is most likely because of file permission or naming errors so re-trace through the steps above. For "two way backups" try to establish the connection from the WAN NAS to the LAN NAS.
  25. When rsync (on the LAN NAS) connects to the rsync daemon server (on the WAN NAS) it has to give a password, however it does this securely through the SSH tunnel we have configured. So that you can automate the rsync job you have to place the rsync user password of the WAN NAS in a file where rsync on the LAN NAS can access it. On the LAN NAS enter the commands "cd /root", "vi my_WAN_NAS_rsync_password_file", and then enter the WAN NAS's rsync user password, save the file and close vi. Then "chmod 600 my_WAN_NAS_rsync_password_file" so only user root has access to the file. For "two way backup" do the same for the WAN NAS making the necessary substitutions (i.e. WAN for LAN etc).
  26. Now we can create a script on the LAN NAS to backup to the WAN NAS, "cd /volume1/my_scripts", "vi sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh" and paste the script given below in to it. Make any substitutions if you have used different file/directory names, Save the file and close vi. For "two way backup" do the same on the WAN NAS.
  27. Now test the script on the LAN NAS "sh sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh" to check everything is OK and perform the first backup to the WAN NAS. For "two way backup" do the same on the WAN NAS.
  28. We now need to install nail (an email client) on the LAN NAS. This will enable the LAN NAS to email you the rsync log file. For instructions on how to install and configure nail see nail on wiki page http://www.synology.com/wiki/index.php/A_short_list_of_the_more_useful_900%2B_ipkg_packages.
  29. For "two way backup" also install nail on the WAN NAS
  30. Now we create another script to find how much data we transferred over the WAN and to store it in a log file. On the LAN NAS "cd /volume1/my_scripts", "vi sub_calculate_and_store_total_data_transfer.sh" and paste the contents of the script below in to it. Save the file and close vi. For "two way backups" do the same on the WAN NAS.
  31. Now we create another script that will email us our rsync log files. On the LAN NAS "vi sub_email_rsync_result.sh" and paste the contents of the script below in to it. Save the file and close vi. For "two way backups" do the same on the WAN NAS.
  32. Now test the email script on the LAN NAS, i.e. "sh sub_email_rsync_result.sh" and then check your email account. You should have received your email. For "two way backup" do the same for the WAN NAS.
  33. Now we need to create one final script that will be called by crond and in turn start our sub scripts. On the LAN NAS "cd /volume1/my_scripts", "vi make_symbolic_link_and_rsync_LAN_NAS_to_WAN_NAS.sh" and paste the contents of the script below in to it. Save the file and close vi. For "two way backups" do the same on the WAN NAS.
  34. Now we need to create a crond job schedule to backup the LAN NAS data to the WAN NAS data as frequently as you want, though remember it is over the WAN so depending on how much your data changes it might just be once a day or once a week. To set the crond schedule on the LAN NAS enter "vi /etc/crontab" and add a line to run the make_symbolic_link_and_rsync_LAN_NAS_to_WAN_NAS.sh script. I recommend you schedule your sript to start before your Synology web GUI created Network Backup jobs, an example is given below where the script is scheduled to run once a week at 55 minutes past midnight on Fridays. After making changes you must restart crond to get it to re-read the crontab file, the easiest way is to restart the NAS, i.e. enter "reboot". For "two way backups" do the same on the WAN NAS.
  35. You have now finished 99% of the configuration, and must now fully test your system, to make sure it works.
  36. After fully testing the next step is to prepare for putting the WAN NAS on the WAN. For the script sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh on the LAN NAS to work with the WAN NAS on the WAN (unless you use vpn tunnelling) you will need to comment out the rsync line(s) being used in the LAN to LAN section and remove the comment from the rsync lines in the LAN to WAN section. You must then also change the IP address used in the rsync lines in the script for the one relevant for the WAN NAS over the internet. If your WAN NAS Internet Service Provider only gives you a dynamic IP address you will need to use a dynamic IP address tracking service such as http://www.dyndns.com. After registering you can enter your details on the "Ez-Internet" Web GUI page of the WAN NAS. Then enter your dynamic IP address reference in the LAN NAS script (e.g. replace xxx.xxx.xxx.xxx with myhostname.dyndns.com). For "two way backup" do the same for the LAN NAS and WAN NAS.
  37. Remove the verbose (-v) option from the rsync command in script "sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh" on the LAN NAS so that if you have any file names that contain the word "error" this is not picked up by the script "sub_email_rsync_result.sh" and assumed to be an rsync error message. For "two way backups" do the same on the WAN NAS.
  38. Deploy the WAN NAS to the WAN
  39. On the WAN firewall make sure port 22 (ssh) is forwarded to the WAN NAS as the rsync daemon receives SSH on port 22. If you want to use a different port on the WAN firewall (e.g. port 27952 instead of 22) and your firewall supports port translation you can get the sending rsync command (on the LAN NAS) to use a different port by adding the -p option to the ssh command, e.g. changing "ssh -v" to "ssh -p 27952 -v" in the script sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh on the LAN NAS. Then configure the WAN firewall to translate port 27952 to port 22 and forward it to the WAN NAS. For "two way backup" do the same on the LAN Firewall and the WAN NAS.
  40. Finished

Get your NAS to check your are running DRIII or A.N.Other backup regularly

If you want your NAS to check you are running the backup regularly and email if you are not see How to get your NAS to check you have used DR3 (or A.N.Other Backup software) regularly and email you a reminder if you haven't

Do you use Windows 2000 or earlier or Vista64 and Outlook?

If you do you should read the wiki How to get your NAS to check you are backing up outlook.pst regularly and email you if you haven't. I recommend you also replace the rsync command with the script I have given in that wiki. The wiki checks that your most recent backup held on the LAN NAS contains the file outlook.pst. If it doesn't it skips the backup and emails you. The reason for this is if you use outlook.pst this will possibly be one of the biggest files in the LAN to WAN synchronisation, and will be deleted on the WAN NAS if it isn't on the LAN NAS. If your last DRIII backup set does not include the outlook.pst file (because outlook was open when the backup ran) then the file will be deleted from the WAN NAS. When the next rsync takes place and outlook.pst is back in the last backup set on your LAN NAS you will end up transferring the whole outlook.pst file over the internet to the WAN NAS.

What to do after a firmware update

  1. When you update the firmware the folder "/root/.ssh" is reset. As this is where ssh stores the file "known_hosts" you will need to repopulate this file as detailed above or just login to the LAN NAS CLI and enter the command "ssh -l rsync xxx.xxx.xxx.xxx" where xxx.xxx.xxx.xxx is the ip address of the WAN NAS (if you didn't use the default port remember to include the "-p xxxx" option). SSH will prompt you to say do you want to continue connection to the unknown host. Say yes and it will populate "known_hosts" for you. Do the same on the WAN NAS.
  2. The files in the folder "/root" are reset so you will need to recreate the file(s) my_WAN_NAS_rsync_password_file and my_LAN_NAS_rsync_password_file as detailed above (Step 25 above).
  3. With firmware 839 the file /etc/passwd was reset, so you will need to check the changes in step 12 have not been undone.
  4. With firmware 839 the ash shell and with it ssh became case sensitive, hence the files named "authorized_keys" and "known_hosts" must be named exactly as stated, i.e. no capitalisation.

An example /etc/crontab file

Look at the very last line which is an example for you of what you could enter. You can see I have set it to be started 5 minutes before the other Synology web GUI managed Network backup jobs are scheduled to start. It is up to you, it's just what I did.

#minute hour    mday    month   wday    who     command
53      0       *       *       5       root    /usr/sbin/ntpdate -b time.nist.gov
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "web"
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "video"
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "photo"
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "music"
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "Data"
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "Other_Backup"
54      *       *       *       *       root    sh /volume1/my_scripts/vnstatdump.sh
0       1       *       *       5       root    /usr/syno/bin/synonetbkp -a "DownloadSrv"
55      0       *       *       5       root    sh /volume1/my_scripts/make_symbolic_link_and_rsync_LAN_NAS_to_WAN_NAS.sh

Scripts

"sub_makelinktobackup.sh" script

Below is the /volume1/my_scripts/sub_makelinktobackup.sh script. Change the first two lines under "# Set variables" ONLY, i.e. change the text "my_LAN_and_WAN_backups" and "my_DRIII_LAN_backups" if needed, do NOT change the name of the variables "my_LAN_and_WAN_backups_directory" and "my_DRIII_LAN_backups_directory" by mistake, or anything else.

#!/bin/sh

# Set variables
my_LAN_and_WAN_backups_directory=my_LAN_and_WAN_backups
my_DRIII_LAN_backups_directory=my_DRIII_LAN_backups

# Set my_PC_name variable to the PC name as used by DRIII. This assumes only 1 DRIII client is being used
# If you want more than one PC to be backed up to this directory you will have to hard code the PC name
my_PC_name=`ls /volume1/$my_DRIII_LAN_backups_directory`

# Set d1 variable to the full path of the latest DRIII timestamp directory
d1=`ls -dc /volume1/$my_DRIII_LAN_backups_directory/$my_PC_name/20*|head -n1`

# If previous symbolic link exists delete it
if
  ls /volume1/$my_LAN_and_WAN_backups_directory/$my_PC_name
  then
  rm /volume1/$my_LAN_and_WAN_backups_directory/$my_PC_name
fi

# Make the new symbolic link to the contents of the timestamped directory thereby bypassing it
ln -s $d1 /volume1/$my_LAN_and_WAN_backups_directory/$my_PC_name
echo "ln -s $d1 /volume1/$my_LAN_and_WAN_backups_directory/$my_PC_name"

"sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh" script

Below is the /volume1/my_scripts/sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh script for the LAN NAS. If you also have data on your WAN NAS you want backed up to the LAN NAS then also place this script on the WAN NAS and modify as necessary. You need to replace the xxx.xxx.xxx.xxx's with the appropriate IP address of the other NAS, or once the WAN NAS is at the WAN position if you use DynDNS then replace xxx.xxx.xxx.xxx with "myhostname.dyndns.com". The script also contains an example of optimised rsync command for backing up the photo directory. You could copy/modify this to do the music and video folders if needed. If you want to do other folders that do not mainly consist of compressed data files then add the compression option (-z).

# Build the path used by the built in Synonetbkp routine to keep the same format, e.g. NetBackup/MACHINENAME_MACADDRESS

# 1.Get the MAC address of your NAS's ethernet interface
d1=`ifconfig | egrep 'HWaddr' | sed -e 's/eth0      Link encap:Ethernet  HWaddr //' -e 's/://g'`

# 2.Add the _ character
d1=_$d1

# 3.Add the Machine Name
d1=`uname -n`$d1

# 4.Add the NetBackup/ path
d1=NetBackup/$d1

#echo $d1

#Create a new log file and place the time/date and header into it
echo -e `date` > /volume1/my_scripts/backup_log.txt
echo "my_LAN_and_WAN_backups rsync output" >> /volume1/my_scripts/backup_log.txt

# Now invoke the rsync command, for more info see http://samba.anu.edu.au/rsync/
# -v = --verbose asks rsync to tell you what it is doing
# -a = --archive implies options -rlptgoD (no -H,-A,-X)
# -W = --whole-file if the rsync server and client are on a LAN then there is normally a performance (speed) benefit unless the files are generally large files. If they are not, i.e. over the web, there is normally a disadvantage unless the files are generally small files.
# -z = --compress use if communication is NOT over a LAN and the files format is NOT a compressed type, e.g. music, video, zip etc.
# --bwlimit=200 limits bandwidth to the value in KBPS, in this example 200 KBPS
# -r = --recursive
# -l = --links copies symbolic links as links not files
# -L = --copy-links follows symbolic links and copies them as files/directories
# -p = --perms preserve permissions
# -t = --times preserve modification times
# -g = --group preserves group
# -o = --owner preserves owner
# -D = --devices causes rsync to transfer character and block device files to the remote system to recreate these devices
# --stats causes rsync to output stats on the transfer
# : = connect to rsync comand shell, via ssh, rsh transport
# :: = connect to rsync daemon (default by TCP transport unless -e ssh option is used)
# --port=873 is the default port so not stated

#The rsync commands below doesn't use SSH so leave it commented out, but it is there for reference incase you have problems configuring SSH, don't use this over a WAN
#rsync -v -a -L --delete --timeout=1200 --stats --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/my_LAN_and_WAN_backups rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt
#And below an example for backing up the photo directory if you use photostation (note removed follow links)
#echo -e `date` > /volume1/my_scripts/backup_log.txt
#echo "photo rsync output" >> /volume1/my_scripts/backup_log.txt
#rsync -v -a --delete --timeout=1200 --stats --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/photo rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt

#The rsync commands below are for LAN to LAN use durring initial configuation and testing of this procedure
#The rsync commands below creates a SSH tunnel between client and server, doesn't compress data and doesn't limit bandwidth
rsync -v -a -L --delete --timeout=1200 --stats -e "ssh -v -l rsync -i /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys/my_LAN_NAS_rsync_user_rsa" --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/my_LAN_and_WAN_backups rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt
#And below an example for backing up the photo directory if you use photostation (note removed follow links)
#echo -e `date` > /volume1/my_scripts/backup_log.txt
#echo "photo rsync output" >> /volume1/my_scripts/backup_log.txt
#rsync -v -a --delete --timeout=1200 --stats -e "ssh -v -l rsync -i /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys/my_LAN_NAS_rsync_user_rsa" --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/photo rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt

#The rsync command below is for LAN to WAN use when the WAN NAS is on the WAN
#The rsync command below creates a SSH tunnel between client and server, compresses data (not for photo folder) and limits bandwidth usage (in KBPS)
#rsync -a -z --bwlimit=200 -L --delete --timeout=1200 --stats -e "ssh -v -l rsync -i /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys/my_LAN_NAS_rsync_user_rsa" --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/my_LAN_and_WAN_backups rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt
#And below an example for backing up the photo directory if you use photo station (note removed follow links and removed compresion - don't try to compress if the majority is already compressed data)
#echo -e `date` > /volume1/my_scripts/backup_log.txt
#echo "photo rsync output" >> /volume1/my_scripts/backup_log.txt
#rsync -a --bwlimit=200 --delete --timeout=1200 --stats -e "ssh -v -l rsync -i /volume1/user_roots/rsync_user/.ssh/my_LAN_NAS_rsync_user_keys/my_LAN_NAS_rsync_user_rsa" --password-file=/root/my_WAN_NAS_rsync_password_file /volume1/photo rsync@xxx.xxx.xxx.xxx::$d1 >> /volume1/my_scripts/backup_log.txt

#Now place the date/time in the log file again so you know when it finished
echo -e `date` >> /volume1/my_scripts/backup_log.txt

"sub_calculate_and_store_total_data_transfer.sh" script

Below is the /volume1/my_scripts/sub_calculate_and_store_total_data_transfer.sh script.


#!bash script

#This sub sums the total bytes sent and received lines from backup_log.txt and then stores it in data_transfer_log.txt

cd /volume1/my_scripts

#Find the total bytes sent lines
cat backup_log.txt | grep "Total bytes sent\|Total bytes received" |

#now strip the text infront of the number
sed 's¬Total bytes sent: ¬¬' |
sed 's¬Total bytes received: ¬¬' > temp.txt

#Now sum all the numbers. Note this command will output in scientific notation if total is > 1GB
varsum=`awk '{sum+=$1}END{print sum}' temp.txt`

#Now check if the total was given in scientific notation, e.g contains an 'e' and if so change it back to normall notation
#Except as the cpu cannot deal with so many digits we will make it one decimal place less by only adding 8 0s instead of 9
#and then multiply it by 10 after the division

if echo "$varsum" | grep -q "e"
 then
# echo scientific notation used
  varecheck=1
  varsum=`echo $varsum | sed 's¬e+09¬000¬' | sed 's¬\.¬¬'`
 else
# echo scientific notation not used
  varecheck=0
fi

#echo varecheck= "$varecheck"
#echo sum= "$varsum"

#Now convert $varsum into megabytes
varsum=$(($varsum / 1048576))
if [ $varecheck = 1 ]
 then
  varsum=$(($varsum * 10))
fi

echo megabytes= $varsum
echo `date "+%d %b %Y"` $varsum "MB" >> data_transfer_log.txt

#delete temporary files
rm temp.txt

"sub_email_rsync_result.sh" script

Below is the /volume1/my_scripts/sub_email_rsync_result.sh script.

#This sub routine analyses the result of the rsyns backup and sends an email

#if the backup_log.txt file doesn't contain text "Total Size is" then it failed
#if the backup_log.txt file contains the word "error" then the backup failed (assuming you are not using verbose mode and have a file name containing error)

if
 grep -e "total size is" /volume1/my_scripts/backup_log.txt
  then
   if
    grep -e error /volume1/my_scripts/backup_log.txt
     then
      varsubject="LAN NAS Rsync Backup - Failed"
     else
      varsubject="LAN NAS Rsync Backup"
    fi
  else
  varsubject="LAN NAS Rsync Backup - Failed"
fi

#send the email
varcontent="LAN NAS Backup Report Attached"
echo "$varcontent" | /opt/bin/nail -s "$varsubject" -a /volume1/my_scripts/backup_log.txt -a /volume1/my_scripts/data_transfer_log.txt my_email_name@my_domain.com

#if the email fails nail will create a file dead.letter, test to see if it exists and if so wait 10 minutes and then resend

while ls "/root/dead.letter"
    do
      echo email failed - waiting 10 minutes and then re-sending
      sleep 600
      rm "/root/dead.letter"
      echo "$varcontent" | /opt/bin/nail -s "$varsubject" -a /volume1/my_scripts/backup_log.txt -a /volume1/my_scripts/data_transfer_log.txt my_email_name@my_domain.com
      sleep 10
    done

"make_symbolic_link_and_rsync_LAN_NAS_to_WAN_NAS.sh" script

Below is the /volume1/my_scripts/make_symbolic_link_and_rsync_LAN_NAS_to_WAN_NAS.sh script.


sh /volume1/my_scripts/sub_makelinktobackup.sh
sh /volume1/my_scripts/sub_LAN_NAS_to_WAN_NAS_rsync_backup.sh
sh /volume1/my_scripts/sub_calculate_and_store_total_data_transfer.sh
sh /volume1/my_scripts/sub_email_rsync_result.sh

Other references if you get stuck

The following are pages I found useful whilst learning how to do this procudure

http://samba.anu.edu.au/rsync/

http://www.hackinglinuxexposed.com/articles/20021226.html

http://transamrit.net/docs/rsync/

http://everythinglinux.org/rsync/rsync_content.html

Personal tools
Community Resources