Confusion over backup options

[bioneural]

Hi All,

I'm a bit confused regarding the best way to mirror the contents of a shared folder (or specific shared folders, such as the web directory and Time Machine share) on one DiskStation to a remote DiskStation via the Internet, using DSM 4.0. The Wiki is outdated.

One DS will make a one-way offsite backup to the other; the second DS will in turn make a one-way backup to the first. The backup should ideally be encrypted in transit, incremental (i.e. only what's changed) and scheduled (to use bandwidth in the small hours). The backups don't have to keep versions (like Apple's Time Machine); they just need to be 'current' to mitigate catastrophic loss. Just to be clear, I'm not talking about 'synchronisation' (i.e. two-way) but rather this:

DS1: Ashare >> DS2: Ashare (mirror)
DS2: Bshare >> DS1: Bshare (mirror)

There seem to be several built-in options in DSM and I'm not clear if some of these are merely different ways to access the same functionality:

* Drop-down menu > Backup and Restore > Backup
* Drop-down menu > Backup and Restore > Shared Folder Sync
* Control Panel > Network Backup > Network Backup
* Control Panel > Network Backup > Shared Folder Sync
* Control Panel > Network Backup > Time Backup
* rsync

So, a few questions for the experts:

1. Would someone please enlighten me as to the differences between these options?
2. Is it possible for DS1 and DS2 to backup to each other simultaneously, or is it preferable they don't do this e.g. for performance reasons?
3. Will the preferred option require opening ports on the routers at either end (and if so, which)?

Many thanks, Bruce

[bioneural]

I guess everyone finds this as confusing as I do.

After some experimentation I have learned:

* Time Backup and Shared Folder Sync both required Network Backup to be enabled, which is an 'umbrella term' for the various backup options that perhaps all rely on rsync as the underlying protocol
* Time Backup is unsuitable, because it is limited to one direction (client or server on each DS, but not both)
* Shared Folder Sync won't do, as it replaces shares with the same name on the destination DS (so 'web' replaces the destination 'web' share, breaking web services on the destination)
* Network Backup can be used on it's own without Time Backup or Shared Folder sync, but I wasn't able to get it to work.

You'd think all this would be a lot more 'one click' given the intuitiveness of the rest of DSM. How hard can it be to mirror specific shares on one DS to a specific share/ directory on another DS?

[myCloud]

Please keep this thread going with your findings and the results of your efforts. One of these days we may have a second DiskStation!

[qness]

I'm trying to accomplish basically the same thing.... to backup my primary diskstation to an offsite diskstation. I have a DS-212+ (source) as my primary machine being backed up to a DS-112j (target). They are both on my LAN for initial seeding and testing, but I intend to move the target offsite.

On the target, I enabled the network backup service. It created a share called NetBackup. On the source, I used the Backup Wizard in the Backup and Restore control panel to create a Network Backup to another Synology server. I have about 750GB to backup. I started the job 49 hours ago and it is now about 63% complete. Seems a bit slow to me. I'll post again when I have more info to share.

[bioneural]

Thanks qness.

I've just had another look at this; likewise the two DSs are on my LAN for now (but will soon be 1000 miles apart).

1. On the destination (DS110j) I created a new share with appropriate privileges to receive the network backup
2. On the DS10j I went to Control Panel then Network Backup, and turned on 'Enable network backup service'; Shared Folder Sync and Time Backup are off.
3. On the source DS (DS112) I went to Backup and Restore (not the Control Panel, but in the drop-down) and used the wizard in the Backup tab to select the DS110j by local IP and used the admin account (*** prev I think I tried my user admin account; maybe it requires the system default admin? ***) to authenticate to the share I created above. I guess it might work with NetBackup, the system default, but I presumed that was created as a container for the backup config? Maybe it is intended to contain the backup itself...
4. I selected an unencrypted and an encrypted share (signified by the surrounding @ symbols) on the source DS112
5. Started sync and... backup failed.

The Network Backup log says:

Network Backup failed to backup task [task name] to [destination]. ([23] Some files could not be transferred. Possible reasons: 1) backup user has no permission to access the files, 2) files are deleted, 3) illegal file name, 4) too many folders.)

Of those possibilities only (1) makes sense, so perhaps it doesn't work with encrypted shares... more experiments required.

Assuming I eventually get this to work, I will then have to try out scheduling, and going the other way (DS110j to DS112), or using the dynamic hostname and custom port via the WAN with 873/22 forwarding set up instead of LAN IP.

I think perhaps the next thing to try is setting up the share on the destination DS as a Remote Folder (Tools menu in File Station), so I can view the contents via FS on the source D112 and check all the backed-up files are there!

Because both DSs are on the LAN for now (to do the first full sync) I've not had to deal with port forwarding, but understand it will be necessary to open 873 for rsync and 22 for encrypted transfers on both DSs. Re the latter, there is the suggestion to change the SSH port for the purposes of server hardening:

viewtopic.php?f=160&t=28853

Hmm, risky.

[bioneural]

I just repeated the above, the only change being to deselect the encrypted share. Backup worked almost instantly (7GB test share); those files were actually transferred the first time. Network Backup choked on the encrypted share.

So is the implication that Synology allow you to make encrypted shares that you can't back up, and if you want to backup unencrypted shares over the Internet you have to risk opening port 22 to encrypt the transfer, or do it in the clear? Surely this isn't the situation...

[qness]

Before attempting the Network Backup, I setup local usb backups. I encountered a few problems with file names and ended up reformatting the usb drive from fat32 to ext4 to resolve. I formated the target DS with ext4 also.

As you said, Backup and Restore is in the pull down, not control panel. I used admin as the user, enabled encryption* and block-level backup but not compression. I used the pull down list to identify the target server (the wizard did a search). I'm not sure how this will impact me when moving the target offsite. I plan to test the vpn in the control panel and also try a dynamic dns/port forwarding setup.

* None of the shares are encrypted, but the backup is.

[bioneural]

Network Backup certainly exercises the Synology CPU. Even a LAN transfer (Gigabit Ethernet) without encryption/compression took way longer than it would with FTP. Perhaps Synology put a cap on transfer speeds to protect other services?

I tried a different encrypted share, containing a Windows backup, and this also failed with the report that 'Some files could not be transferred'. Rather unhelpfully the Network Backup log gives no indication as to which of the 78GB of files didn't transfer. This means I cannot vouch for the integrity of the data as I don't know what's missing. Obviously unacceptable.

I was able to backup several unencrypted shares. I was able to change the LAN IP to the dynamic DNS address and perform the backup over the Internet (one way at a time, to avoid port forwarding conflicts on the LAN).

I suppose it would be possible to create a separate backup task for each share, in order to help pinpoint where the trouble may lie, but this still would be like trying to find a needle in a hay stack... However, it seems DSM can only manage one backup task at a time, so they would need to run in series.

I was able to map the destination share on the DS110j as a remote folder on the DS112: in fact I mapped it to the empty NeBackup share the system created (I created an encrypted share as the target). This means I can browse the remote backup from my end in File Station, but it also means I can mount the backup directly in the OS X Desktop. Strangely, using the equivalent settings, I was unable to get the DS110j to mount the share on the DS112 as a remote folder...

Does anyone else find Network Backup unreliable/flakey or too much trouble? A friend of mine uses SyncBack SE on the desktop to manually backup his DS to a remote one via FTP with TLS/SSL. I am thinking to abandon all attempts to use the Synology solution, and use Transmit on the Mac to manually backup. I see several advantages:

1. Speed (utilising the more powerful desktop CPU)
2. Encrypted transfer of encrypted shares
3. Simulation mode before backup
4. Detailed backup log for useful failure feedback
5. Avoid having to open port 22/SSH

Although this is fine for shares, a disadvantage is that it won't backup the DS apps e.g. Photo Station settings, MySQL database etc. So I might have to run Network Backup periodically to do that (which works at least!).

Thoughts?

[myCloud]

I can't help you with backup, but I can with ssh.

You can change the ports ssh and sshd use in their config files in /etc by uncommenting the appropriate line and changing port 22 to something like 22022. After doing so to sshd, you'll need to remember that to get into those boxes with the terminal you need to specify the new port number, i.e. ssh user@domain -p 22022.

I prefer to do port port forward port 22022 to port 22 in the router, rather than change sshd_config and change the port in the app using ssh rather than change ssh_config, but that might not be possible in some apps.

Also remember, any changes you make via the CLI need to be in a "mods" checklist you reapply after a DSM update.

Thanks to both of you for the continuing thread!

[Steph]

Same problem! Here are my observations :

- I backup from a RS812+ DSM 3.2 to a 712+ DSM 4.0 and Remote Backup fails with the same stupid reasons "check logs bla bla bla"
- I used to backup the SAME DATA from the SAME RS812+ to a CS407+ DSM 3.1 with NO ERRORS
- I then did 21 remote jobs (I have been on it for hours!!!) to narrow where the backup would failed

And my conclusion was that there was ONE problematic file : it was because the name of the pdf file was very land AND very deep in many folders. The same file, copied at the same place but remove a few characters in the same and bingo, it works !!!

This would have been SO MUCH EASIER if Synology mentioned where it failed. I have used many many backup software and they all mention this.

I will do a final few testing : I'm going to put back my CS407 and try it again to confirm this first observation. If it's true, then there is maybe a bug in DSM 4.0 ?

Concerning SyncBack, it doesn't use block level transfer . So for me, this is not possible. And you need a computer to be on...

[bioneural]

@Steph, totally agree on the mystifying lack of file/error-specific logging.

I've been continuing to test. Using Network Backup I continue to experience failures, for both encrypted and unencrypted shares:

https://twitter.com/bioneural/status/207787718988668928

One of these backup tasks has now been running for several days on my Gigabit LAN. Curiously, if I use File Station to copy the contents of a share (even an encrypted one) to the second DiskStation, which is mounted as a Remote Folder, it copies successfully and quickly.

I have also been testing the feasibility of manual backup using a desktop FTP client (Panic's Transmit for Mac), which can be configured to juxtapose two 'Remote' windows. I made some screen captures to illustrate the ability to simulate a one-way file sync (copying I believe only files that have changed, but I doubt block-level), and the detailed log shown on job completion indicating what actually happened:

https://twitter.com/bioneural/status/207788788943040512

The same shares that failed via Synology's Network Backup transferred using Transmit, and in a fraction of the time.

If a file can be STORED on a DiskStation, I don't understand why it can't TRANSFER to another DiskStation using the same user privilages. I do find it difficult to believe Synology don't have backup between DiskStations totally nailed. This prospect was actually the reason I bought a second DiskStation, but I have lost the confidence to depend upon the built-in functionality. Network Backup is broken--unless someone can convince me otherwise?

[Steph]

Well there is definitively a bug :

The same data (around 22GB), from the same rackstation to a different Synology on DSM 3.2 and not 4.0, the backup didn't fail !!!

I will write to Synology ...

[bioneural]

A bit more digging... failure after failure.

Because I was unable to compare the files on the DS112 with the encrypted backup of the same on the 'remote' DS110j, here's what I did:

* Copied the original files off the DS112 (decrypted) to a folder the Mac desktop
* Deleted the directories within the share I just copied off the DS112
* Ran a network restore from the DS110j back to the DS112
* Copied the restored files off the DS112 (decrypted) to a second folder on the Mac desktop
* Compared the two desktop folders visually using Transmit
* Ran and automated 'find differences' using BBEdit

I found:
* The Synology 'failed backup' error is legitimate, as there were a number of files missing from the 'restored' folder
* The octal on the files in the original folder was 666 and on the restored folder 777
* There were 6 unique files only in the original folder, and 2 unique files only in the restored folder; the others appeared intact
* The unique files in the restored folder 'almost' matched files in the original folder, but the files names were somehow modified (not by me). For example, the original file 'Bad medicine#356860.pdf' became 'Bad medicine#3569EC.pdf' in the restore folder

I can't explain this and don't know if my experiments help at all, but it would seem to a novice assessment that the Synology backup software is recoding file names when it shouldn't. Hopefully someone from Synology reads these forums and they are working to fix DSM 4!

Postscript: I just ran the backup on this encrypted share 'restoring' the 'restored' files from the DS112 to the backup DS110j. The backup was successful, for the first time! Of course, some of my files are missing (no fear, I have a non-Synology backup!) and who knows how many of them may turn out to be corrupt...

[qness]

@ Bioneural

You mentioned changing the LAN IP to the dynamic DNS address. Do you know of a way to run ddns on a DS, or is it on a client machine?

[bioneural]

@qness

Formerly, I used the setting in my router to sync my public IP address with my free DynDNS domain. However, I wanted to give each DS it's own domain, so disabled this on the router and used the built in Control Panel > ezCloud >DDNS service to give each DS it's own unique domain. DynDNS gives you two free domains on the one account, although there are alternatives including Synology's own MyDS service (see supported DDNS services in the drop-down as above).

By forwarding different ports on each DS, I was able to re-direct both domains to the same public IP, yet still address a specific DS for things like File Station/ management interface. Note that this trick won't work for rsync since we can't change the port from 873, and if you forward this on one DS and then try to do the same on the second, it will fail due to a port conflict (at least with my set-up, using a BeBox in bridge mode and a Time Capsule). Therefore to test backup via the WAN from DS A to B, I had to forward port 873 (and 22 for encryption) on one DS at a time, then swap over to backup from B to A.

Incidentally, there is also a DDNS Updater third-party package available from:

http://package.10trum.de/index.php?id=40&pid=24

...although I've not installed it and don't know how it compares to Synology's offering.

But no, I don't know how to run a home-baked DDNS service on a DS.