Low perf in NFS and ISCSI vs CIFS in DSM 2.2 and RS407

[rs407]

Hi,
We have some RS407 with DSM 2.2 with raid5 and 700GB Disks.
We have had them for a while and they have done well in providing NAS via CIFS.

Now we are evalueting if Synology can be used as NAS via NFS against ESXI (tested with the free version)
If the RS407 works we plan buying and start testing with RS409+

The read/write performance with CIFS is good and close to the performance chart att http://www.synology.com (>13MB/S write and >25MB/s read,MB=MegaByte) running filecopy from a XP client.

But when I try NFS or ISCSI from two diffrent servers (vmware ESXI 3.5 and Vsphere 4) I get 15-30% of the performance of CIFS, nothing else is loading the servers or the RS407.
I have used test tools and made file copies within the Vm, same resault.
When I use a VM inside the ESXI servers (Windows 2003 server) and connect to the RS407 (maps a networkdrive) over CIFS and make a copy
to a local datastore (local harddrive in server) I get the same performance as when I run CIFS from my client (XP).

This mean that I get good performance over CIFS between the server and RS407 (no network problem) but with NFS and ISCSI the performance are low.

Conclusion:
The NFS (and ISCSI) performance in DSM 2.2 in a RS407 is only 15-30% of the CIFS performance.
It is not a network problem since the same server get good performance with CIFS.
Possible reasons for the low performance
#Its something wrong in the NFS or ISCSI settings between DSM 2.2 and ESXI (NFS have no settings in DSM 2.2,only on/off)?
#DSM 2.2 have no disc cache implemented in NFS and ISCSI as in CIFS (can the low NFS performance actually be the disc performance?)?
#DSM 2.2 NFS and ISCSI implementation is not as good as the CIFS implemantation?

Any ideas or experiance anyone?

[mrn1c3]

I am experiencing exactly the same problem, with ESXI 3.5

CIFS shares run very slow, when copy any data to a VM, whether the VM is on ISCSI.

I have started moving all of my VM's from ISCSI to NFS, to see if the performance increases, I will then turn off ISCSI completely.

I am using 3 x 1tb disks in RAID5.

[vm.franklin]

Greetings all

Please note that I'm looking currently researching into your claims and I'll post a reply when I have finished my performance testing.

In the mean time, can you both list the design specifications of your VMServer, and what kind of HDDs you are using within the Synology DiskStation.

@mrn1c3, I'll need to know what SynoDS you are using.

Also, please make sure you are using 1Gbps Networks in your environments, and for best compatibility, please use 1500MTU.

Hope this helps, and have a good day.

[rs407]

Hi and thanks for the reply.
Here are the data you requested
Host 1
ESXi3.5 Update 4 running on one HP DL380 G3 with one CPU and 3GB mem the wm is a windows 2003 server web edition.
Its only one wm on the vm-host and that windows 2003 server is only running the file tranfer tests.
Host 2
ESXi 4 running on a HP DL380 G5 with 16 GB mem and running a windows 2003 server web edition.
Its only one wm on the vm-host and that windows 2003 server is only running the file tranfer tests.

The performance between the host are simular and they are connected with 1 Gigabit network (MTU 1500) via a Cisco 4510-E switch

disk info
Disk 1 HDS721075KLA330 698.64 GB 35 °C / 95 °F Normal Volume 1
Disk 2 HDS721075KLA330 698.64 GB 36 °C / 97 °F Normal Volume 1
Disk 3 HDS721075KLA330 698.64 GB 36 °C / 97 °F Normal Volume 1
Disk 4 HDS721075KLA330 698.64 GB 35 °C / 95 °F Normal Volume 1
/R

[vm.franklin]

Greetings all

Just as an update - I was able to reproduce the NFS performance behavior which has been described thus far - I'm going to chat with the developers to see what can be done or identify where the configuration error exists.

At this time - there is no guarantee of when a resolution will be available. I'm going to start looking at the iSCSI performance shortly.

EDIT, 2009.10.22: Actually, after further research, it appears that VMware is mounting the NFS share with the "FILE_SYNC" option, which sacrifices performance for data integrity. Please consult with Vmware for further assistance with regards to this topic.

EDIT, 2009.10.23: The lack of iSCSI performance appears to be an a performance error with ESXi itself - as my bench tests from Vista and Fedora (from the same computer) reflect expected performance from the CS407/RS407.

Hope this helps.

[rs407]

Hi,
Thanks for the reply.
NFS
I have search Internet and looked in both ESXI 3.5U4 and ESXI4 and havent found any options to configure the ESXI to use async/unstable options in NFS.
All findings on Internet are based on the "override" in the NAS of this request from the ESXI server, the "override" means that the NAS response back to the server when data are written to NAS memory (instead of to disc) meaning that performance increace on writes.

In other words the NAS implements the Async behaviour regardless of the "sync" request from ESX, will this option be avalible to the DSM?

ISCSI
Do you see diffrens in performance on ISCSI RS407-ESXI and RS407-Vista or Fedora?
In my testing the performance diffrens between NFS and ISCSI are small.

/R

[vm.franklin]

Hi rs407

Can you be more specific about what option you are referring too? Do you have any links that I can review?

I'll have to get back to you with regards to performance numbers.

Good day.

[rs407]

Hi,
Below are some examples that I found
http://sourceforge.net/apps/phpbb/freen ... f=49&t=764
http://davidhajek.blogspot.com/2008/02/ ... mware.html
https://dakeung.com/2008/04/04/vmware-e ... ata-store/
http://forum.open-e.com/showthread.php?t=1147
/R

[vm.franklin]

Greetings rs407

About supporting the 'async' flag within NFS - I'll suggest that to our developers for further consideration

About the iSCSI Performance, I did find a difference between using the variety of clients used (Vista, ESXi, Fedora), but the performance is consistent within each client (example, Fedora, 12MB/Sec writing, 17MB Reading), and since the numbers are consistent, it appears that the behavior is normal.

Please note that the performance of iSCSI or NFS, or what I'm able to achieve is relative to my environment, with the hardware I'm using. The performance you receive may be different.

Hope this helps, and have a good day.

[rs407]

Thanks for your answers vm.franklin!

Bellow you see the performance from the Vsphere 4 host and also a reference test from my XP client.
I even tried with a MS ISCSI initiator installed on the vm MS-2003 server (equal performance as XP).
As you see the performance are only consistant when it relates to CIFS and the RS407 (and local disk), also note that NFS and ISCSI have simular performance.
The CIFS performance from the RS407 are acceptable while NFS and ISCSI is not.

Im not close to get your write numbers on ISCSI and the ESXI, also at my reference test with XP client I not close to your performance.
It could be Nbench settings,parameters in XP and ESXI, or something in the RS407 (in whatever combination),but regardless of the reason behind the low NFS/ISCSI performance the conclusion for us is that we cant use Synology as a ESXI storage at the moment.

Summary of tests NFS/ISCSI compared to CIFS(via the vm MS-2003 server) from ESXI or XP client are that
in write CIFS are 5-10 times faster and in read CIFS are 50% faster.

Note that the write performance on ISCSI via Microsoft ISCSI initiator are almost double as fast as ISCSI via ESXI, but it still 5 times slower than CIFS.
The MS ISCSI have better performance then vmware ESXI.

ESXI
NFS
Write = 10 times slower (maybe the sync_flag will help) than CIFS
Read = equal
ISCSI
Write = 10 times slower (equals the NFS performance) than CIFS
Read = CIFS 50% faster
XP client
ISCSI
Write = almost 5 times slower than CIFS
Read = CIFS 50% faster

Test details
Setup
HW
Server HP DL380 G5
2 CPU x 2,666 Ghz 16GB RAM
Intel Xeon 5150
1 socket with 2 core
Local storage 2*72GB SAS disk in RAID 1

Sw
ESXi 4 Host
VM (HW 7 latest vmware tools)
Windows 2003 server web edition (up-to-date patched) with 1GB of RAM
C: = NFS
E: = ISCSI
F: = Local datastore (local disk in ESXI host)
Z: = CIFS share
All storage exept F: are on RS407

NAS (NFS,ISCSI and CIFS) Synology RS407 DSM 2.2-0942
Disk 1 HDS721075KLA330 36 °C / 97 °F 698.64 GB Normal Volume 1
Disk 2 HDS721075KLA330 37 °C / 99 °F 698.64 GB Normal Volume 1
Disk 3 HDS721075KLA330 37 °C / 99 °F 698.64 GB Normal Volume 1
Disk 4 HDS721075KLA330 36 °C / 97 °F 698.64 GB Normal Volume 1

I used NBench Disk I/O (average 3 test runs) for simplicity
Units in Mbyte/s (1 thread,10MB file)
NFS,ISCSI and Local are via the ESXI host, CIFS direct from the VM MS-2003 server
NFS (C:)
Write = 1,8 - Read = 32
ISCSI(E:)
Write = 2,3 - Read = 22
Local(F:)
Write = 161 - Read = 212
CIFS(Z:)
Write = 20 - Read = 32

Referens test Nbench (1 thread,10MB file) from the vm ms2003 server to the same RS407 via Microsoft ISCSI Initiator 2.08
(only IP via ESXI)
MByte/s
ISCSI
Write = 4,2 - Read 23

Referens test Nbench (1 thread,10MB file) from my XP client to the same RS407
Microsoft ISCSI Initiator 2.08, XP SP3,HP Compaq 6910p,Intel Core2 Duo T770 2,4Ghz, 4GB memory,
MByte/s
ISCSI
Write = 4,2 - Read 20
CIFS
Write = 20 - Read 32

/R

[vm.franklin]

Just as a conclusion to this thread, please refer to the following numbers as what I was able to achieve on....

DS409+
4xHD103UJ -> RAID-5
Firmware DSM 2.2-0942

Hypervisor Computer
E7600
4GB RAM
WD5000AAKS
Intel 82566DM 1Gbps PCIe

Client Computer
T6400
4GB RAM
Realtek 1Gbps
Hitachi 160GB SATA
WD2500BEVT SATA

Client OS
Fedora 2.6.27.37-170.2.104.fc10.x86_64
Windows Vista 6.0.6002 SP2, Ultimate, SUA, NFS Enabled

All numbers are reflected as Megabytes per second, under a 1Gbps Network, 1500MTU, transferring a 4GiB Linux dd=zero file, average of three trials.

Code: Select all

|******************************************************|
|Source      | Fedora | Vista | ESXi  | Xen   | HyperV |
|Destination | ------ >  DiskStation, DS409+  < ------ |
|******************************************************|
|NFS Write   | 27.26  | 22.16 | 6.07  | 24.89 |        |
|NFS Read    | 34.05  | 34.34 | 10.7  | 59.55 |        |
|iSCSI Write | 38.34  | 35.07 | 15.98 | 36.52 | 57.14  |
|iSCSI Read  | 32.02  | 45.06 | 9.57  | 47.35 | 56.19  |
|CIFS Write  | 28.85  | 34.00 |       |       | 34.84  |
|CIFS Read   | 21.11  | 45.07 |       |       | 67.25  |
|******************************************************|


Our developers has reviewed this information and plan on some additional performance tuning to improve the DisksStation's performance for Hypervisor servers.

[rs407]

Hi vm.franklin

Will the "Our developers has reviewed this information and plan on some additional performance tuning to improve the DisksStation's performance for hypervisor servers." be implemented in the Synology DiskStation Manager 2.3-1118.
/R

[vm.franklin]

Greetings rs407

Performance updates of the DiskStation for Hypervisor environments is currently scheduled for a firmware update to be tentatively released in the second half of the 2010 calendar year.

Hope this helps.

[ullerdk]

Code: Select all

|******************************************************|
|Source      | Fedora | Vista | ESXi  | Xen   | HyperV |
|Destination | ------ >  DiskStation, DS409+  < ------ |
|******************************************************|
|NFS Write   | 27.26  | 22.16 | 6.07  | 24.89 |        |
|NFS Read    | 34.05  | 34.34 | 10.7  | 59.55 |        |
|iSCSI Write | 38.34  | 35.07 | 15.98 | 36.52 | 57.14  |
|iSCSI Read  | 32.02  | 45.06 | 9.57  | 47.35 | 56.19  |
|CIFS Write  | 28.85  | 34.00 |       |       | 34.84  |
|CIFS Read   | 21.11  | 45.07 |       |       | 67.25  |
|******************************************************|


Our developers has reviewed this information and plan on some additional performance tuning to improve the DisksStation's performance for Hypervisor servers.

To me it seems like this is a specific ESX problem. I have just ordered a RS407RP+ (before reading this thread), and now I fear that I won't be able to use it for ESX storage (NFS or iSCSI) as the intension was.
/Ulrik

[ullerdk]

Any news on this issue after the release of DSM 2.3?