Comparison of NFS vs. iSCSI - winner: NFS

This room is for the discussion of how the Synology DiskStation can meet the storage needs for Virtual HyperVisors.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
mbu10
Novice
Novice
Posts: 44
Joined: Fri Jul 02, 2010 8:27 am

nfs vs's iscsi

Postby mbu10 » Sun Jan 19, 2014 1:58 am

well just trying vmware 5.0 to the storage on a ssd
using nfs getting steady 100MB a sec with little latency
but using file iscsi to the same disk only get 30 ish
the iops are less than 1/5th on iscsi as well
any ideas
thanks
koochiching
I'm New!
I'm New!
Posts: 2
Joined: Mon Jan 20, 2014 5:08 am

Comparison of NFS vs. iSCSI (file-based)- winner: NFS

Postby koochiching » Mon Jan 20, 2014 6:02 am

I just completed a comparison of HDD performance in a Windows VM using a number of different storage configurations and am posting the results for reference.

Background:
I got my first DS and configured an iSCSI LUN/Target to use with my ESXi server. It worked fine, but I was wondering if I was getting the best performance and how one should ideally divide LUNs and Targets, etc... I went into work and talked to a few guys at Stanford who manage hundreds of VMWare VMs - they said they prefer to use NFS over iSCSI because they thought the configuration was just easier and they didn't see any major advantage to the iSCSI overhead. I thought I'd do some quick testing on my own at home to see how what results I would get.

Configuration:
The ESXi server is a Dell R710 with 24GB RAM and 1 CPU.
The VM is a clean install of Win7 Enterprise with all patches and a 'thickly' provisioned 90GB HDD.
The DiskStation is a DS412+ with 4x3TB WD RED NAS drives configured in a single volume.

Storage Configurations:
1) An iSCSI LUN (file level) on the DS. The DS is connected directly to a dedicated NIC on the ESXi server with MTU set to 9000 on the VM switch, NIC, and the Diskstation to enable jumbo frames. Ideally I would have done a block-level setup but wanted to keep the DS configuration simple and easy to backup...
2) A NFS share on the DS (connected through a Cisco 2600 series switch) to the ESXi server - no jumbo.
3) A NFS share on the DS (connected directly to the ESXi) with MTU at 9000 for jumbo frames.
4) A single 128GB SSD (Kensington V300) connected to a spare SATA port on the ESXi server.
5) A local RAID 0 (3 x 146GB 10K SAS drives) on a Perc 6i controller of the Dell R710. Obviously the DS isn't meant to compare to this, but it is good to have a high-end benchmark.

The Method:
I had created a basic VM with my iSCSI target and installed the ATTO benchmark tool: http://www.attotech.com/disk-benchmark/
I ran the tool, shutdown the VM, and moved it to a different storage source and repeated. When I did the testing, I made sure the DS wasn't being otherwise used.

The Results:
1) iSCSI (jumbo frames)
Image

2) NFS (standard)
Image

3) NFS (jumbo frames)
Image

4) SSD
Image

5) Local RAID0 (3x146GB 10K SAS HDDs)
Image

iSCSI (jumbo frames) vs. NFS (jumbo frames):
While the read performance is similar, the write performance for the NFS was more consistent. The load on the DS also was subjectively lower than when doing the iSCSI work. I'm clearly saturating the single GB nic, wish I had more NICs to try link aggregation...

Local RAID vs. SSD:
You can really see the read enhancement from the RAID 0 with speeds up to 800 Mb/s and writes at 400 Mb/s. The SSD is rated for 400 Mb/s but only showed 240. I think the onboard sata port on my R710 is just Sata2 (with a 300 Mb/s limit) which is probably the bottleneck here. It would be interesting to see what kind of performance you get with the SAS drives in a more realistic raid 5 setup instead of the raid 0, but I only have so much time :-)

Notes:
I tried both 8k and 32k blocks for NFS (as configured under Control Panel Win/Mac/NFS) and there was no appreciable difference. However, if you compare the default vs. jumbo NFS you can see a slight improvement with jumbo frames enabled (9000 MTU).

While the DS configuration for NFS Privileges (under Shared Folder) says you can use asterisks and subnet masks, I ran into trouble until I explicitly defined the ip address for the ESXi server. I found a few references online saying there was some configuration issues with DS and ESXi in this area - if you're having trouble, just set the IP for your ESXi (assuming you have only one :-)

I hope this information is helpful for some folks. Take home for me is that I'm going to scrap my iSCSI setup and just use NFS for my home VMs. The tipper, for me, is the easier NFS configuration.
Last edited by koochiching on Sat Feb 15, 2014 5:00 pm, edited 1 time in total.
sundi712
I'm New!
I'm New!
Posts: 8
Joined: Tue Dec 24, 2013 8:09 pm

Re: Comparison of NFS vs. iSCSI - winner: NFS

Postby sundi712 » Sat Feb 15, 2014 1:23 pm

If possible, change your post subject to 'NFS v File-Based iSCSI'. I am saying this because I just found this out the hard way earlier this week setting up file-based iSCSI with my home ESXi server. I had to use file-based because I'd rather have only one volume for my four HDDs and the synology has multiple purposes. Moving over my VMs via iSCSI was ridiculously slow- after reading up, I enabled NFS and saw an amazing difference. I move everything over to my NFS datastore and killed the iSCSI targets.
However, if iSCSI would've been setup with block-based, network traffic would be much much better. I have a 412+ now, but I want to eventually save up some money to buy a 2-bay synology just for the ESXi server and setup block-based to see the difference.
koochiching
I'm New!
I'm New!
Posts: 2
Joined: Mon Jan 20, 2014 5:08 am

Re: Comparison of NFS vs. iSCSI - winner: NFS

Postby koochiching » Sat Feb 15, 2014 5:06 pm

sundi712 wrote:If possible, change your post subject to 'NFS v File-Based iSCSI'. I am saying this because I just found this out the hard way earlier this week setting up file-based iSCSI with my home ESXi server. I had to use file-based because I'd rather have only one volume for my four HDDs and the synology has multiple purposes. Moving over my VMs via iSCSI was ridiculously slow- after reading up, I enabled NFS and saw an amazing difference. I move everything over to my NFS datastore and killed the iSCSI targets.
However, if iSCSI would've been setup with block-based, network traffic would be much much better. I have a 412+ now, but I want to eventually save up some money to buy a 2-bay synology just for the ESXi server and setup block-based to see the difference.


Done. I too have more in mind for my DS than just VM's so I didn't want to set up a separate volume, etc for block level -- would like to see the results. If it were a dedicated DS box for VMs I would be able to use both ethernet ports and do link aggregation which would probably make the biggest difference in performance. I wish my DS had one more NIC :-). Maybe I should get a usb3 NIC and play around...
User avatar
Barto
Versed
Versed
Posts: 214
Joined: Fri Mar 08, 2013 3:15 pm

Re: Comparison of NFS vs. iSCSI - winner: NFS

Postby Barto » Thu Mar 27, 2014 2:33 pm

What about that fact that you can clone/mirror the iSCSI luns? Also that you can use vaai?

With these features how do you compare NFS
DS1812+, WDRED3TB, Latest DSM
csantos
Novice
Novice
Posts: 44
Joined: Sun Sep 02, 2012 1:13 am

Re: Comparison of NFS vs. iSCSI - winner: NFS

Postby csantos » Wed Mar 18, 2015 1:47 pm

@koochiching,
Just to say thank you!

Was a great job for all that want to use Synology for a VMware enviroment.

:mrgreen:
Regards,

______________________________
Cristiano Santos - Brazil

Synology DS412+ (using SHR)
2x 1.5TB Seagate ST31500341AS
2x 3.0TB WesterDigital WD30EZRX

Return to “Virtual HyperVisors (VMWare/ESXi)”

Who is online

Users browsing this forum: No registered users and 2 guests