Timeout SFTP 4GB limit ext4 Postgresql SSH rsync rsa[Solved]

[LleMikeByw]

18 February 2011 Update: The thread below deals with problems with the default 5 minute timeout on FTP communications - describing how to avoid that by some customisation of the DiskStation, the discovery of a problem with initial setup of the DiskStation which prevented me storing files larger than 4GB in size - and describes how to resolve that by stopping PostGreSQL services and performing a new format of the partition on volume1 allowing VERY large files to be stored. It also gives a lead on how you can perform a FSCK on volume1 to confirm the integrity of your data. I describe how to give non-root users SSH access to the DiskStation - which can allow you to setup SSH to avoid root access. Users will have users group rights in their environment which is much safer for providing access to the DiskStation. I then go on to describe the setup of rsync to allow rsync-ing client PCs without passwords or passphrases using rsa keys. There is a lot of detail here - so you should find much of this useful even if you don't want to deal with all of these issues. Mike


This is my first post to this forum - which I have to say is fairly active and very informative.
I have a problem that I cannot quite resolve despite much reading (about 2 days of deep research).
BACKGROUND
OS: OpenSuse 11.3
Diskstation: Standard DS110j (1 Bay)
Firmware: DSM 3.0-1354
Setup of DS:
DISABLED - Windows File System, Apple File System, Linux NFS
ENABLED: Synology BusyBox 1.16 FTP, Synology Busybox 1.16 SSH
FTP Setup: ACTIVE Port 990, PASSIVE Ports 55536-55663 (Default), Report external IP in Passive Mode: NO, Allow SSL/TLS connection only: YES
CLIENT: Filezilla 3.3.2.1 from OpenSuse 11.3 repository
Connection type: Diskstation and PC connected to same router (EdiMax BR-6104K) to the LAN ports. IPs are 192.168.xxx.xx1 and 192.168.xxx.xx2 (xxx's are obviously numbers in reality).
WHAT WORKS:
FTPES connection to Diskstation is setup fine:
HOST: FTPES://192.168.xxx.xx1
USER: admin
PASSWORD: xxxxxxxxxxxxxxxxx
PORT: 990
Filezilla connects to the FTP server and volume1 directories are displayed fine
Upload of files to Diskstation is accomplished (I've only tried around 4MB so far)
Management of files on Diskstation is accomplished: Create directory, Delete Directory, Move files etc.

WHAT DOES NOT WORK:
Filezilla reports that the Diskstation FTP Server has dropped the connection 421 Timeout after 300s (5mins)

Diskstation Firewall is set up to accept all ports from IPs in the 192.168.xxx.xxx range (Subnet 255.255.255.0).
OpenSuse 11.3 Firewall has been taken down (disabled)

Yet I still get the timeout problem.

I have tried setting Filezilla to try to keep up the connection (FTP Stay-Alive) but this has no effect either.

I have NOT tried forwarding any ports on my router nor opening the firewall on my router because all comms should be on the LAN BEHIND the router's firewall.

The server reports it is dropping the connection (the text appears in green in Filezilla) so I have to suspect the Synology Busybox 1.16 FTP server is the source of the timeout.

Has any body any idea if:
1) There is a setting to establish timeout for the DS Busybox FTP server? I am beginning to look through the DS's config files and may report back.
2) Whether I should be forwarding the relevant ports on my router (but this would make my network open to port requests in the range defined).

I have tried to put as much detail in here as possible so somebody doing a relevant search will catch this problem.
Hope I won't be tackling this problem on my own for too long.
Mike

[LleMikeByw]

Being the only person to post to the FTP forum in the last twenty-four hours must be something of an achievement.
Whatever, necessity is the mother of invention and so it is with much pride that I announce that I have arrived where I wanted to be...
I can log on securely to my DS110j through FTP and am not bumped off by a timeout (I've been up to 40 minutes connected without problem so "Bye-bye, 5 minute timeout!").

However, the route is somewhat more circuitous than I had expected - and it isn't a perfect solution because rather than fix the setup I had - I've gone around the problem.

I have been through most of the setup files in the standard Synology Linux environment and nothing screamed at me "5 minute timeout" consequently I eventually assumed it must be a setting decided by Synology on compilation of the Busybox 1.16 source code for the DSM 3.0-1354 firmware.

This was the culmination of much ssh'ing and even trying CHGFTPA INACTTIMO 0 (IBM FTP) along my path.

Neverthless - I finally decided that there may be a solution though IPKG modding of the DS.

Consequently these are the steps I took (much of this requires root access over ssh to the DS):

1) Install the IKPG Bootstrap for the DS110j located here: http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/stable/syno-mvkw-bootstrap_1.2-7_arm.xsh
2) Note that this is the stable version from the nslu2-linux website (the Wiki talks about downloading the unstable version - they are the same);
3) Follow the instructions in the Synology Wiki for installing IPKG. See here: http://forum.synology.com/wiki/index.php/Overview_on_modifying_the_Synology_Server,_bootstrap,_ipkg_etc
4) Reboot the DS110j.
5) Update the IPKG application list by: ipkg update
6) Download a secure ftp server. What?!?! Haven't I got one when I ticked the FTP using TSL/SSL box in DSM3.0??? Yes, you DO - but it's the Busybox FTP server. Let's go get the ipkg one instead, which works over ssh (on port 22 not port 21): ipkg install openssh-sftp-server
7) Because it's on a different port it doesn't conflict and we are going to change our server anyway (as per the instructions here: http://forum.synology.com/wiki/index.php/How_to_setup_an_sftp-server)
8) Once the installation is complete we need to edit the file: /etc/ssh/sshd_config
9) That file is located on the DS110j not in your version of Linux - you should be accessing the DS110j via ssh, remember? Use "cd .." to move to parent directories.
10) To edit it we need to use vi (a primitive editor - a bit like using a TRS-80 to edit programming lines rather than a C64). So: vi /etc/ssh/sshd_config (There's a 'space' after vi)
11) To edit the file you can use cursor keys to move around. When you find the relevant line you'll need to type "i" or 'insert' to start editing at the location you've put the cursor. Text typed will be entered now. If you use backspace or delete - text will disappear.
12) You need to put a # at the start of the line which says: Subsystem sftp /usr/libexec/sftp-server
13) Now, immediately below type: Subsystem sftp /volume1/@optware/libexec/sftp-server
14) You might need to press 'Esc' to begin moving the cursor again ('Esc' exits "insert" mode). To get the gaps between "Subsystem" and "sftp" and "/" use 'Tab'
15) Once the lines are amended - it's time to save the amended file: 'Esc' :wq
16) Yep - that's 'Esc' 'colon' 'w' 'q' (:=you want to type a command; w=write the file; q=quit vi).
17) See - you've changed the server there...
18) Reboot the DS110j: reboot (Yes - the command exists - just like "poweroff" does, too)
19) The ssh link to the DS110j is lost but we can ssh back in once it has booted up again.

20) Wait for the DS110j to boot up (Status is a green light).

Now then, we've now got a new SFTP server running on our DS110j. To access it - back to Filezilla on your desktop.

21) Filezilla - File - Site Manager - New Site

Host: 192.168.xxx.xx1
Port: 22
Server Type: SFTP
Logon Type: Normal
User: root
Password: xxxxxxxxxxx

The bold text is to bring to your attention the important differences with this login against standard FTP login with the DS.

22) Connect

23) Filezilla communicates with the DS110j and....... we've got a secure link to the DS. Note that the whole disk is available for access because we are connected as root - so don't delete anything unless you really know what you are doing!!!

24) Leave Filezilla connected to the DS as long as you like and there's no problem. Sometimes it says "Disconnected from server" but that shouldn't be your main session and it shouldn't happen mid-file.

The purpose of me obtaining this level of access is to upload files of 20Gig and 10Gig to my DS.

It's possible it would have worked with the standard FTP client but I was unhappy with potential mid-file timeouts and this method allows me to access the whole o/s rather than just the home(s) directory.

I've set Filezilla to preserve timestamps on the uploaded files BUT for some reason the DS drops the seconds figure from the date and time (need to investigate that now).

Nevertheless - this is much better than it was.

Any advice on the "seconds" issue would be much appreciated (it's the icing on the cake )

Mike

[LleMikeByw]

Still talking to myself here

So, I was concerned about the SFTP server on the DS110j not properly recording the timestamp of files stored on the Disk Station.

Well - finally - when I had some time (after reading and re-reading my posts (above)) I got back to some SSHing into the Disk Station.

I entered this command against the files I'd transferred: dir -aen

This is to produce a full listing of the relevant directories with timestamp in seconds and I discovered that the ipkg sftp server IS acting on the Filezilla instructions and recording the timestamp of files as they were recorded on the client PC - including seconds.
Great!

Directories, however, I still need to check.

There is also the issue that when I store files by this method they are owned by root (and the root group) but that's not insurmountable.

Now I need to set up rsync to copy files automatically to the DS from designated backup folders on the attached Linux desktops here and then I'll be on to the webserver and torrent client...

I might even install nethack and do some RPG'ing on the DS

Mike

P.S. Just for the record the original 300 second limit was something designed in by Synology (why not allow the user to set????)...
See the post by a Synology peep here: http://forum.synology.com/enu/viewtopic.php?f=3&t=277&start=75#p40569

[LleMikeByw]

A further update following a little more testing.

I have now used Filezilla to try to store some large files on the DS110j: One of 20Gbytes, one of 6Gbytes and one around 2Gbytes.

The results were that the 2Gb file was stored fine through the ipkg openssh-sftp-server client and Filezilla. I have used md5deep (from the ipkg repository) to check the MD5 sum for the file and it matches exactly that of the original. Success!
Unfortunately, both the 20Gb and 6Gb files were truncated to 4Gb.
This is not a problem caused by Filezilla, nor my PCs nor my routers. Since I have used Filezilla and SFTP to transfer the files successfully (checked with md5sum on both PCs) between two Linux PCs across my network.
So the problem of file size is being caused by the Diskstation.

The possible suspects are:
1) The filesystem format of the drive - created when the firmware and DSM 3.0-1354 were installed. However, I have checked /etc/fstab and it indicates the format of /dev/root/ and /dev/sda3 /volume1 are both ext3. The maximum file size of the format is 2Tb (2,000 Gb). So I do not believe this is the problem.
2) Perhaps the ipkg openssh-sftp-server has a maximum file transfer size of 4Gb? I am currently investigating this.
3) Something else in the setup of the Diskstation - the Firmware (Busybox 1.16), DSM 3.0 or the configuration files is causing the limiting of file sizes to 4Gb. Still to be investigated.

This is a major hurdle for one application of my Diskstation - but I intend to overcome it. Any advice would be much appreciated. I'll let you know if I crack it...

Hourly start up of Diskstation - Hibernation
By the way - for some reason, after updating to DSM 3.0-1354 and installing the ipkg'd openssh-sftp-server I noticed that the Diskstation was starting up every hour and spinning the disk.
As part of my setup I had set the Diskstation to obtain it's system time via NTP from uk.pool.ntp.org (every device I have does).

Well - somehow crontab (/etc/crontab) had been altered (certainly not by me) and had a setting at the "hour" value of 50! Consequently, the Diskstation could never match the "hour" value and hence was checking the NTP server every time it matched the "minute" value - once every hour. I corrected this to a sensible value (between 0-23) and the diskstation is now correctly only booting up once every day to check the time.

I think this is a fault with the Ajax/Javascript used to implement the DSM 3.0-1354 web-application to alter the time (I NTP server) because I vaguely remember changing the time there first before encountering the problem. If there is a problem with that script then it really needs to be fixed fast because unchecked my drive would have been booting up 24 times a day, 168 times a week, 720 times a month or 8760 times a year. This would have severely shortened the life of the drive.

Hope you find this helpful.

Any advice on the 4Gb limit would be much appreciated.

Mike

[LleMikeByw]

Wow!!!! Still the only person posting to this thread... Well... perhaps not many others are trying to store files of 4GB+ .

I have not solved the issue yet - due to Christmas (away from DS) and a new installation of OpenSuse 11.3 (I am deep in Bash and Perl scripts currently - to change the setup of the OpenSuse environment - but enough about that).

This response has two purposes:

1) To keep the topic current;

2) To update the community on what I know now.

The current problem is DS limiting file sizes stored in /volume1 to 4GB (Actually it's anywhere on the DS - but let's focus on /volume1).

In my previous post I suggested three possible reasons for this (See above). The results of my research since suggest that I spoke too soon when dismissing the filesystem (option 1) as being the possible suspect.

I had a closer look through /etc and discovered the mke2fs.conf file is limiting the largefile setup for an ext3 partition e.g. /dev/sda3/volume1, to 4GB.

This appears to be absolutely the problem .

At this juncture, I have surmised that there is only one possible solution to fix this issue:

1) Backup /volume1 (including @optware) to another drive;
2) Without rebooting: Remove the /dev/sda3/volume1 partition;
3) Without rebooting: Change mke2fs.conf to allow files of any size up to 2TB (as allowed under ext3);
4) Without rebooting: Recreate the /dev/sda3/volume1 partition with a revised ext3 filesystem;
5) Without rebooting: Copy back all files to /volume1;
6) Then reboot and hope to be able to store files of greater than 4GB.

There is one matter that I may need to address before doing the above: /opt is linked to /volume1/@optware and the ipkg openssh-sftp-server resides in /volume1/@optware consequently I cannot reboot during the above process. If I do I lose my SFTP server and would have to start from the very beginning by reinstalling my DS. I am considering copying @optware to /dev/root if there is sufficient space and remapping the link... but not going there yet.

Interestingly - in the course of my research I noticed that mke2fs.conf actually has settings for ext4. I intend to investigate that soon too (when I've got that Perl script sorted... ).

If anybody else has undertaken the above strategy previously it would be helpful if they could verify that it will do what I expect.

I'll let you all know how I get on.

Mike

[LleMikeByw]

I have now cracked the 4GB file size limit.
The problem was absolutely caused by Synology Assistant's default setting imposed on the ext3 filesystem when it originally installed the DiskStation operating system.
I have now created a new ext4 filesystem in the /volume1 partition and have successfully stored a file of 20GB - verified with md5deep (Diskstation) and md5sum (PC).
Note that the ext4 filesystem is available with DSM3.0-1354.

Assumptions
1) You have installed the bootstrap to customize your DiskStation with ipkg packages (See above);
2) You have installed ipkg openssh-sftp-server;
3) You are using DSM3.0-1354 (or some variant that supports ext4);
4) I was not running any of the following applications: Media Server, iTunes, Audio Station, Photo Station, Download Station, Surveillance Station;
5) The following were also disabled: Windows file sharing, Apple file sharing, NFS;
6) You are logged into the DiskStation over ssh as root;
7) You are running an SFTP client and logged in as root. My client is Filezilla.

Process
Warning - Do not reboot the DiskStation at any time until you get to my "reboot".
To change an existing ext3 volume to an ext4 volume with the capability to store files >4GB I did the following:

0) To avoid a noisy background to some of this work, switch off the /volume1 crash beep in the web interface of DSM3.0: Control Panel - Power - Beep Control
1) Log into the DiskStation as root using ssh:

Code: Select all

ssh root@192.168.XXX.XX1

2) When the ipkg bootstrap originally ran it created a bind mount from the new optware software stored in /volume1/@optware to /opt. We need to disable this link so that we can copy files from /volume1 unhindered. So:

Code: Select all

umount /opt

3) Because we've now broken the link to the openssh-sftp-server and other ipkg software we need to ensure the software remains available to the system so we copy @optware to /opt temporarily:

Code: Select all

cp -R /volume1/@optware/* /opt

4) If your DiskStation is like my DiskStation a PostGreSQL database will be running to provide information to the web interface functionality and to manage the DiskStation's setup. The database is stored on /volume1 under @database.
To properly backup the /volume1 folders we need to stop the database engine in an elegant fashion. So, all of this must be typed into the console in one continuous line. Don't try copying and pasting from here - type it in:

Code: Select all

su -l admin -c "exec /usr/syno/pgsql/bin/pg_ctl stop -s -m fast"

5) With the database engine stopped we can now access /volume1 unhindered and without causing inconsistencies in the database record. Check that no other active processes are running with admin authority by using:

Code: Select all

ps

(Only root processes should be running and none of these are likely to be updating /volume1)
6) Time to move our data from /volume1. Use Filezilla or your preferred SFTP client to copy all the files from /volume1 to your local PC (I hope you've got a really big drive!). You'll need to be logged in as root to see all of the necessary directories. Make sure you include @database, homes, public, @tmp directories. I copied everything except @optware (We've already got a copy in /opt from 3). This may take some time...
7) Once all the files are copied time to sort out /volume1 so let's free /dev/sda3 (This was the partition mounted to /volume1 for me - it may be different for you. Check it with "mount" before you start). So:

Code: Select all

umount /volume1

If you didn't switch off the /volume1 crash beep then cover your ears. The status light will flash amber now. Don't worry - we know what we are doing...

8 ) I wanted to check the integrity of /volume1 sectors so I issued a filesystem check (e2fsck) at this stage (Not vital to this process):

Code: Select all

e2fsck -cfv /dev/sda3

Warning: Don't issue that command when /dev/sda3 is mounted - you'll wreck your filesystem and probably lose data!

9) Now let's get that new filesystem onto /dev/sda3 (Yeh!!!!). Type this all in, in one continuous line. Don't copy and paste it...

Code: Select all

mke2fs -T ext4 -v -O has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/sda3

Note there's just a space between 'isize' and "/dev"
Watch as the filesystem is created on device /dev/sda3 ... ooooo!!!
10) /dev/sda3 now has a ext4 filesystem (format) and we can store very, very large files on it... so let's begin putting our DiskStation back together:

Code: Select all

mount -t ext4 /dev/sda3 /volume1 -o defaults

Status light will now stop flashing amber.
11) Note that /etc/fstab should be updated automatically by this process - but if it hasn't (check with vi) - alter the line for /dev/sda3 to read:

Code: Select all

/dev/sda3 /volume1 ext4 defaults 0 0

12) Before we can copy files back we need to recreate our sftp server on the DiskStation so:

Code: Select all

mkdir /volume1/@optware

and

Code: Select all

cp -R /opt/* /volume1/@optware

This copies back the ipkg files to volume1
13) Now we can use Filezilla or your sftp client to copy all the folders/files we copied from /volume1 to our PC. This may take some time too... Remember, because you are logged in as root they will all have the ownership: root.root which we'll have to change as and when.
14) We now need to get the PostGreSQL database back up and running, but we need to set the necessary permissions to ensure the database engine runs properly so:

Code: Select all

chown -R admin.admin /volume1/@database

This ensures the database engine runs in the admin user's environment as originally intended.
15) Time to restart the database engine. Type this in, in one line - no copy and paste:

Code: Select all

su -l admin -c "exec /usr/syno/pgsql/bin/pg_ctl -D /var/services/pgsql restart -s -m fast -o \"--config_file=/usr/syno/pgsql/etc/postgresql.conf --hba_file=/usr/syno/pgsql/etc/pg_hba.conf \""


There are no carriage returns in the code. Just type 'space' when you get to the end of each line above.
Make sure you have your spaces in the correct position and don't confuse backslashes with (b)ash esc characters...
The database engine should restart without any problems.
16) Time to clean up the DiskStation - remove stuff that is no longer required:

Code: Select all

rm -Rf /opt/*

Removes our temporary copy of ipkg files which are now back in /volume1/@optware after we did 12 (above) Doing it now ensures the database records the transactions.
17) Get back our /opt mount:

Code: Select all

mount -t bind /volume1/@optware /opt -o bind

18) Let's ensure everything is cleaned-up ship-shape. So let's do a:

Code: Select all

reboot

Ignore messages for packages you didn't have running (e.g. surveillance station)
19) The ssh link is dropped - but we can go back after. We've got some work to do in the DiskStation web interface. So Firefox - Address: 192.168.XXX.XX1 for DiskStation. Log in as admin as usual. Notice the top right-hand side of the window? Notifications? This will record the multiple times the DiskStation told us that volume1 had crashed... just Clear All. While you are here - you might as well put the beep warning back on: Control Panel - Power - Beep Control
20) Log out of the web interface and let's go back to ssh into the DiskStation.

Code: Select all

ssh root@192.168.XXX.XX1
cd ..
mount

/dev/sda3 on /volume1 should now be ext4 with a couple of other settings (e.g. quotas)
21) We can now use Filezilla to store some very, very large files on the DiskStation.
22) Note that all the directories and files are owned by root.root. You'll need to use:

Code: Select all

chown -R user.group /volume1/homes/xxxxxxx

to change the ownership of directories and files.

So that is it - the /volume1 filesystem has been converted to ext4 and I can now store files larger than 4GB on it.

Next step is to get rsync working on it and automatically backup files bypassing the web interface (how limited !).

Important Note
Some other reading of this forum suggests a clean install from scratch with DSM3.0 may lead to ext4 being installed by default - but my DSM3.0 didn't seem to do that when I originally installed the DiskStation operating system (I'll have to check the CD to establish if it was an earlier version). Whatever - this method definitely works and means I can change the filesystem to whatever I want on volume1 so others may find it useful too.

By the way - ipkg provides a version of lsof - this may be useful for you if you find ps discovering additional active processes. You could issue a

Code: Select all

lsof |grep volume1

as well as ps to establish if there are any open files in use on /volume1.

I'll let you know if I find any problems with rsync setup to the DiskStation.

Mike

[cfuttrup]

Hi LleMikeByw - thanks - will have to "chew" in this (lengthy) information for a while, e.g. during the weekend.

[LleMikeByw]

Stage 1 of rsync work
Make sure non-root users can access the diskstation via SSH in an user environment rather than root environment.

Files to edit are:
/etc/ssh/sshd_config
/etc/passwd

Amendments required:

/etc/ssh/sshd_config
Add the following line in /etc/ssh/sshd_config

Code: Select all

AllowUsers = root user1 user2

where user1 and user2 are the names of the relevant users from /etc/passwd.

Note that the case of each character in the user name is relevant: mike does not equal Mike

/etc/passwd
Now edit /etc/passwd to allow the user to login via SSH:

1) Use vi to edit /etc/passwd
2) Change the following in the relevant user's line:

Code: Select all

/var/services/homes/username

becomes

Code: Select all

/volume1/homes/username

3) Change the following in the relevant user's line:

Code: Select all

/sbin/nologin

becomes

Code: Select all

/bin/ash

Now that user can login to the diskstation via SSH and access their home directory and the root (/) folder with "users" rights. So no access to important files e.g. SHADOW.

Just need to setup RSYNC to the user's home folder on the diskstation now...

Mike

P.S. ... and RSA keys...

[LleMikeByw]

Stage 2 of rsync setup

To be honest - following my experience with the setup of the standard SFTP server provided with the Diskstation (which you recall times out after 5 minutes ) I decided not to risk my rsync experiments being influenced by some compiling decision so:

Code: Select all

ipkg install rsync

got me ipkg rsync version 3.0.7 - which was at least more up-to-date than the installed version - which I think was 3.0.4.

Then I setup an rsync session between my PC and the DiskStation.

This was my command (on my Linux PC):

Code: Select all

rsync -vrtplze ssh --progress --stats /home/Mike/Documents Mike@192.168.XXX.XX1:/volume1/homes/Mike/ExampleFolder

Required me to enter password for 'Mike' on DiskStation but I'm experimenting before going full-on RSA authentication... so let's not be too bothered about that...

The command ran fine.

To double check I ran it again including the delete option and after moving a couple of files around...

Code: Select all

rsync -vrtplze ssh --progress --stats --delete /home/Mike/Documents Mike@192.168.XXX.XX1:/volume1/homes/Mike/ExampleFolder

This mostly ran fine but was a little unhappy about a filename with a "\". Removing this file from the transfer resolved the situation and the mirroring completed properly.

So now I'm going to set up RSA authentication and CRON my RSYNC

That's it for now...

Mike

[LleMikeByw]

My rsync backups to the DiskStation will be communicated over SSH and use RSA authentication to ensure automatic backups can be undertaken without the client PC's user knowing the backup is happening and without user interaction.

Consequently, I need to set up the SSH clients and the DiskStation's SSH server to accept password-less logons using RSA public keys.

To do this - we need to ensure the DiskStation will accept RSA public key logons. It appears to be allowed by default but to ensure it absolutely is I edited
/etc/ssh/sshd_config removing the comments ('#') from the following lines:

Code: Select all

RSAAuthentication yes
PubKeyAuthentication yes
AuthorizedKeysFile              .ssh/authorized_keys

I then rebooted the DiskStation so that the SSH server takes up the new settings.

Then - on:

Client PC
1) Login as root - we are using Linux PCs here
2) Enter:

Code: Select all

ssh-keygen -t rsa -b 2048

( Strong 2048-bit encryption of our key and RSA for speed of verification)
3) Follow standard options: Save to /root/.ssh/id_rsa and /root/.ssh/id_rsa.pub
4) Because we don't want the SSH server to prompt us for a passphrase on rsync-ing just carriage return through pass-phrase requests (the key will still be strong)
5) Now our RSA public key is stored on the client PC in /root/.ssh/id_rsa.pub

DiskStation
6) Now login to the DiskStation over ssh as root:

Code: Select all

ssh root@192.168.XXX.XX1

7) It's unlikely that a .ssh folder will exist currently for the user so you'll need to:

Code: Select all

mkdir /volume1/homes/user-directory/.ssh

Obviously 'user-directory' would be Mike in my case.
8 ) We need to copy the id_rsa.pub file to the DiskStation (NOT id_rsa). I used Filezilla to copy it across and it should be stored in the user's .ssh folder on the DiskStation. For me that would be /volume1/homes/Mike/.ssh
9) Now - we need to copy the contents of the id_rsa.pub file to a new file called authorized_keys in /volume1/homes/user-directory/.ssh
10) I simply renamed the file but you could use

Code: Select all

cat /volume1/homes/user-directory/.ssh/id_rsa.pub >> /volume1/homes/user-directory/.ssh/authorized_keys

This appends the client PC's RSA key to the authorized_keys file - useful if it happens to contain other keys.
11) Because I didn't want the user to alter the key - or have rights to alter it I maintained the file-rights to authorized_keys and the .ssh directory as root.root:

Code: Select all

chown -R root.root /volume1/homes/Mike/.ssh

12) But I also wanted the user to be able to login so I ensured that the file could be read by the user (alone) so I issued a:

Code: Select all

chmod 644 /volume1/homes/Mike/.ssh/authorized_keys

13) Log out of the Disk Station by typing:

Code: Select all

exit

Client PC
14) Now the root user of my client PC is able to login over SSH to the user account on the DiskStation as that DiskStation user without issuing a password.

Code: Select all

ssh Mike@192.168.XXX.XX1

15) We should just get the usual BusyBox login message - no passwords required.
16) THE BEST BIT RSYNC commands utilising the SSH option do not require password nor passphrase login. Don't use a passphrase when setting up your RSA key... or it WILL require the passphrase (Duh!!! No automatic rsync-ing...)

Now I just need to establish the structure of my DiskStation folders, what I want to backup
N-N-O-O-O-T-T-T just /home
S-o-o-o-o-o-o
b-o-o-o-o-r-i-n-g-g-g-g-g-g!!!!!

Then I put my rsync bash file somewhere on the client and add a call to it in CRONTAB and I am done.

I'll let you know when it's done...

Mike

[LleMikeByw]

I have been considering whether it is best to backup the whole of an user's home folder - which makes the rsync statement fairly easy - but boring. This avoids the use of 'exclude' and the danger of an user deciding to store data in a unpredictable section of their home folder.

Consequently - despite the boringness - to ensure exhaustive capture of the user's files there is no solution other than to backup the whole home folder.

That means capturing configuration files and other useless gubbins etc. but it seems to me that if I am selective with the restore rather than the backup I am likely to achieve my overall goal - despite the inevitable wasted disk space because of the unnecessarily exhaustive rsyncing.


Meanwhile - OpenSuSE's CRON tasks extend outside of crontab - but, I think, since I have amended most of polkit, iptables and other settings - why stick with the OpenSuSE setup for cron????? Just amend crontab and have done(!).

Alternatively, I have thought about amending the OpenSuSE shutdown and reboot scripts to incorporate my rsync scripts - but backups are to deal with unexpected events.

Complete failures of drives are not generally completely unexpected so this may be unnecessary effort just to ensure completeness of my backups.

BBBBBBBuuuuuuuuuuuuuttttttttttttttttt... it WOULD be neat to alter the shutdown scripts AND amend crontab for, say, two-hourly rsyncs... Hmmmmmmmmhhhhh..... that sounds like what I'll do...

Mike

[LleMikeByw]

60 Minutes Later Update, 25 February 2011: Silly silly me!!!! Start at the very beginning. It's a very good place to begin! 1-2-3 Do-reh-me... The problem was nothing to do with the DiskStation. RESOLV.CONF is absolutely edited correctly and the DiskStation can access the web with a STATIC IP. The problem was the setup of the EdiMax Router. I had forgotten I had setup up MAC Address Filtering as part of the migration to STATIC IPs and I had forgotten to include the DiskStation as one of the "Allowed" MAC addresses. The DiskStation was working before the migration to static IPs so it was likely to work after - however it was the first device that I had spotted the NTP update failure upon - so naturally I thought it was a DiskStation problem. I should have thought that the problem might lie further up the chain - which it did... Silly me. Anyway. Sorted now.


To properly prepare for my automatic backups from client PCs I decided to finally commit my personal network to STATIC IP addresses.

A network topology was designed and I systematically worked through each router and network attached device disabling the DHCP servers and setting up STATIC IP addresses.

Part of the exercise required me to actively specify certain setup on some of my devices e.g. my PCs required active confirmation of DNS servers (because they no longer receive this information via DHCP from the routers). No problem.

However, I have encountered a problem with the DiskStation.

Via the web interface I set up a STATIC IP for the DiskStation and had recorded OpenDNS settings in RESOLV.CONF (IPs 208.67.220.220 and 208.67.222.222). I have also edited /etc/sysconfig/network-scripts/ifcfg-eth0

However, NTP won't access the UK pool (UK.POOL.NTP.ORG) and I cannot use ping (from the DiskStation) to access the web.

This doesn't make sense... Could it be the Firewall??? Or have I got some spurious statement in RESOLV.CONF (Order of settings wrong, perhaps?) Might it be the router's MAC control???

It won't affect the RSYNCing of clients to the DiskStation - but it absolutely affects the accuracy of the DiskStation time setup.

If anybody has any idea why giving the DiskStation a STATIC IP has disabled web access - I would appreciate your comments asap.

Mike

[LleMikeByw]

Time for a short update on this enterprise...

Several days ago now I finalised scripts to write the home folders of several users on several PCs (These PCs have multiple home folders) to the DiskStation using a similar command to the RSYNC command above. The scripts do more than a simple RSYNC and are site specific so I'm afraid I'm not posting them here.

Suffice to say, in OpenSuSE you want to:

1) Write a Bash script (Backup.sh) in vi (say) or Kate and put it in some folder like /sbin (I put it in a subfolder of /sbin):

Code: Select all

#!/bin/bash
#MY special commands went here <g> For me only <G>
#Here's the main command
rsync -vrtplze ssh --safe-links --stats --delete /home/Mike/Documents/ Mike@192.168.XXX.XX1:/volume1/homes/Mike/DesignatedFolder > /root/rsyncrec/rsyncrec.log 2> /root/rsyncrec/rsyncrec-Err-log


The rsync command is all on one line. You've got to save the text lines above in a file called "Backup.sh" - in case it isn't clear. Also note I am doing some record-keeping in /root/rsyncrec. I'd mkdir in advance, me

2) I want to run my backups as root (no interference from users) so:

Code: Select all

chown root.root /sbin/BackupScriptFolder/Backup.sh

3) To be able to call the script, it's got to be executable by root:

Code: Select all

chmod 700 /sbin/BackupScriptFolder/Backup.sh

4) Finally, I want to call the script every 8 hours so I edited crontab (Yes - i know - OpenSuSE allows you to enter scripts in hourly, daily, weekly and monthly cron folders but I cannot be bothered with those - I like to set MY own schedule ) so in crontab - add this last line:

Code: Select all

-17 */8 * * * root -x /sbin/BackupScriptFolder/Backup.sh && /sbin/BackupScriptFolder/Backup.sh

This will run the backup at 17 minutes past the hour every 8 hours a day.

5) The schedule takes about a minute to be adopted by cron.

There you have a CRON'd RSYNC...

I am now busy on the OpenSuSE shutdown scripts.

Mike

[stefhf]

Just to let you know..your posts are very much appreciated and being studied! At least by me
So thank you.

[FlavioB]

Hello Mike, hello all!

I took a look at your posts, Mike, and realised that you'd be the one who helps me out of my hell!
I'm using DS211j with the latest DSM 3.1-1605. I want to "suck" data off of that DS using rsync via ssh from a remote BackupPC (on Linux) server.
I've done this setup with a couple of different OS flavours: Windows 2003 Server, Windows 2008 Server, Mac OS X Server 10.6 and also some Synology DS's.
The DS's which are actually working, are 1x 209j and 1x210j, both of them on DSM-2.3 (don't remember which build).
I managed to configure passwordless login via public/private keypairs and then rsyncing data. So, when I connect to my remote Synology DS with:

ssh root@remoteIP

I get logged in.
Then I can run

rsync rsync://localhost/Data

and get the contents of that rsyncd module listed.

Back to my latest DS211j (DSM 3.1-1605), I'm facing a real problem: I reached the goal of "passwordless login", but rsync is *not* working as expected.
In fact, when I type:

rsync rsync://localhost/Data

I get prompted for the password!!! This behaviour is simply *useless* when running automated tasks!
I tried many ways, but still without success.

Could you, Mike, be helping me out of this?
Where do you think is the culprit?
I was thinking about PAM, but I got no knowledge at all of how PAM authentication works.

Any help will be appreciated!

TIA,
F.