Iscsi dropping after DSM 6.0

All questions regarding the Synology system as a iSCSI Target can go here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
JamieSimms
I'm New!
I'm New!
Posts: 2
Joined: Fri Apr 07, 2017 4:32 pm

Re: Iscsi dropping after DSM 6.0

Postby JamieSimms » Fri Apr 07, 2017 5:44 pm

Hello everyone,

I've been watching this post for a while and decided to drop in my experiences as well, since I rarely see anyone with my device posting in the forums.

Last year, my company purchased two RC18015xs+ units, and a RXD1215sas expansion unit, to run in a HA cluster. I updated to 32GB of RAM and 8 NICS per unit, then loaded the expansion unit up with 4TB HGST drives. Should be a great NAS, but unfortunately I have had nothing but issues, specifically with iSCSI.

Our environment contains multiple HP hosts running ESXi 6.0U3 which use this NAS as shared storage via iSCSI. I keep up to date on drivers, patches for ESXi, etc (except ESXi 6.5, loads of issues with this product, I have been warned to steer clear until they correct many issues) We run MPIO via port binding and have verified the configuration via the Synology doc located here: https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Virtualization/How_to_Use_Port_Binding_to_Configure_Multipathing_on_VMware_for_Synology_NAS In addition, I have talked to VMware support and other VMware certified consultants, both of which see that my configuration should work well. Unfortunately, from day one I have been struggling with issues, and I believe these issues to be related to what everyone else here on this post is running into: high iscsi latency, dropped iscsi connections, crashing units, etc. All of which are unacceptable.

I have attempted to work with Synology support a few different times over the past year, and have never once been able to talk to someone directly. Emails may be answered the next day, but in most cases I do not hear back for multiple days, up to a week. Most recently, I have been working on a case for almost a month, and have just had support connect to the unit remotely, this past week. The suggestions from support do not seem to be something that is normally suggested in the VMware community, and I don't believe that it should have to be changed, but I made some of the changes per their request. They said that the best practice was one target per host, which I have not seen anywhere and I wonder if anyone else has. (They did also suggest that I only have one target per LUN...which seems excessive in a shared storage environment and would result in a ton of targets being created.) In addition, it was suggested to disable VAAI, which seems ridiculous. At this point, I am left to try and figure out what the issues may be, just the same as you all are. This is especially frustrating, seeing that we have ~$20k+ invested in a cluster that I cannot reliably use.

After attempting the solution posted above, it appears that unfortunately for anyone with the RC18015xs+ units, we are unable to change the lio version away from 4.x. When this is done, huge glaring ERROR messages appear in the HA manager, and I am unable to mange my cluster properly (reboot active server, switchover, etc). It really appears to me that the change for the lio from 4 to 4.x may have been made to accommodate these multi-server HA clusters.

With all of this said, I do believe that there may be a glimmer of hope though, as at the bottom of my last support email was a blurb about "If nothing of the following helps, we will try to implement different iSCSI engine on the NAS". So maybe there is a new version beyond 4.x in the works?

Anyway, I apologize about the wall of text, but I am hoping that maybe someone out there may have a suggestion, as I wait for my message-in-a-bottle to get back from Synology support...

Thanks!
MrTomasz
Beginner
Beginner
Posts: 24
Joined: Mon Feb 20, 2017 8:01 pm
Location: London

Re: Iscsi dropping after DSM 6.0

Postby MrTomasz » Fri Apr 07, 2017 9:33 pm

I would suggest switching to iSCSI 4.0 engine when devices are *not* in cluster yet... that's how I've got it working...
Izaak99
I'm New!
I'm New!
Posts: 6
Joined: Mon Mar 27, 2017 4:31 am

Re: Iscsi dropping after DSM 6.0

Postby Izaak99 » Mon Apr 10, 2017 7:17 am

After hours of testing horrible iSCSI performance with DSM 6.1 + Vsphere 6 (on their latest and greatest rs4017xs+ boxes), there were some bugs I discovered and passed onto support. Hopefully they get this fixed. I'm already in the process of returning the 2 RS4017xs+ boxes with a different brand as I don't have the time to wait for a patch that might or might not actually fix the latency problems and VMware issues.

What I've noticed is using a block-level LUN (which should have VAAI completely disabled), it still presents to the host that it can support Zero Status acceleration. You can verify this by running "esxcli storage core device vaai status get" on the VSphere host. The output will look something like:

naa.60014055316b198d51f8d4252d8262d9
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported


(Notice Zero Status shows "supported"). Now once I start copying data to this iSCSI LUN, the dmesg logs on the Synology is flooded with "iSCSI:target_core_transport.c:1785:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx2-6b2453a3: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION" and latency hits 500-1500ms. By flooded, I counted around 5000+ messages over writing about 5GB of data. "dmesg | grep "CHECK_CONDITION" | wc -l" will count the # of errors. Opcode 0x93 is directly related to "Zero Blocks/Write Same, which is used to zero-out disk regions." So until they fix that bug, you can either use block-LUNs as RDM-mounted to bypass the false acceleration or use the file-level LUN's. In either case, latency was still in the 30-100ms using those types so it didn't help much.

Oddly, if I didn't create any file-level LUN's beforehand, the only errors I saw were about 15 "iSCSI: reject WRITE_SAME_16 on non-extent LUN" messages back-to-back which googling came back to an old Synology post back in 2013 with the same issue and an unanswered post (https://forum.synology.com/enu/viewtopic.php?t=70678).

Bottom line, I don't know if DSM 6.1 bugged their iSCSI implementation or if the latency was always this bad before, but using these boxes with iSCSI might be ok for a very light loaded lab environment but you're shooting yourself in the foot if you throw these into production. Good luck, I hope they get all these problems ironed out soon!
synology_ukman
Knowledgeable
Knowledgeable
Posts: 377
Joined: Fri Oct 26, 2012 4:51 pm

Re: Iscsi dropping after DSM 6.0

Postby synology_ukman » Mon Apr 10, 2017 12:06 pm

JamieSimms wrote:Hello everyone,

I've been watching this post for a while and decided to drop in my experiences as well, since I rarely see anyone with my device posting in the forums.

Last year, my company purchased two RC18015xs+ units, and a RXD1215sas expansion unit, to run in a HA cluster. I updated to 32GB of RAM and 8 NICS per unit, then loaded the expansion unit up with 4TB HGST drives. Should be a great NAS, but unfortunately I have had nothing but issues, specifically with iSCSI.

Our environment contains multiple HP hosts running ESXi 6.0U3 which use this NAS as shared storage via iSCSI. I keep up to date on drivers, patches for ESXi, etc (except ESXi 6.5, loads of issues with this product, I have been warned to steer clear until they correct many issues) We run MPIO via port binding and have verified the configuration via the Synology doc located here: https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Virtualization/How_to_Use_Port_Binding_to_Configure_Multipathing_on_VMware_for_Synology_NAS In addition, I have talked to VMware support and other VMware certified consultants, both of which see that my configuration should work well. Unfortunately, from day one I have been struggling with issues, and I believe these issues to be related to what everyone else here on this post is running into: high iscsi latency, dropped iscsi connections, crashing units, etc. All of which are unacceptable.

I have attempted to work with Synology support a few different times over the past year, and have never once been able to talk to someone directly. Emails may be answered the next day, but in most cases I do not hear back for multiple days, up to a week. Most recently, I have been working on a case for almost a month, and have just had support connect to the unit remotely, this past week. The suggestions from support do not seem to be something that is normally suggested in the VMware community, and I don't believe that it should have to be changed, but I made some of the changes per their request. They said that the best practice was one target per host, which I have not seen anywhere and I wonder if anyone else has. (They did also suggest that I only have one target per LUN...which seems excessive in a shared storage environment and would result in a ton of targets being created.) In addition, it was suggested to disable VAAI, which seems ridiculous. At this point, I am left to try and figure out what the issues may be, just the same as you all are. This is especially frustrating, seeing that we have ~$20k+ invested in a cluster that I cannot reliably use.

After attempting the solution posted above, it appears that unfortunately for anyone with the RC18015xs+ units, we are unable to change the lio version away from 4.x. When this is done, huge glaring ERROR messages appear in the HA manager, and I am unable to mange my cluster properly (reboot active server, switchover, etc). It really appears to me that the change for the lio from 4 to 4.x may have been made to accommodate these multi-server HA clusters.

With all of this said, I do believe that there may be a glimmer of hope though, as at the bottom of my last support email was a blurb about "If nothing of the following helps, we will try to implement different iSCSI engine on the NAS". So maybe there is a new version beyond 4.x in the works?

Anyway, I apologize about the wall of text, but I am hoping that maybe someone out there may have a suggestion, as I wait for my message-in-a-bottle to get back from Synology support...

Thanks!


I wonder why you didn't buy a SAN for that money if you wanted ISCSI? Far more reliable and stable.
behroez
I'm New!
I'm New!
Posts: 1
Joined: Tue Apr 11, 2017 9:12 pm

Re: Iscsi dropping after DSM 6.0

Postby behroez » Tue Apr 11, 2017 9:19 pm

Hello Guys,

We are still running DSM: DSM 6.0.2-8451 Update 4, because we had a lot of problems sinds we updated to 6.0. We do not dare to upgrade any further sinds all the problems we had so far.
To bad we are running this in a production environement en we have te reboot our synology every week just to be sure that the iscsi won't drop.

This works okey for now, but when we have heavy load (deploying new machines with ESX) then iscsi drops anyway.

we have 2 RS18016XS+ with 2 expansion bay in a cluster setting.

can anyone advise us what to do? This is starting to be a nightmare :-(
MrTomasz
Beginner
Beginner
Posts: 24
Joined: Mon Feb 20, 2017 8:01 pm
Location: London

Re: Iscsi dropping after DSM 6.0

Postby MrTomasz » Wed Apr 12, 2017 5:35 pm

Are you running "new" iSCSI stack v4.x or "old" 4.0?
JamieSimms
I'm New!
I'm New!
Posts: 2
Joined: Fri Apr 07, 2017 4:32 pm

Re: Iscsi dropping after DSM 6.0

Postby JamieSimms » Wed Apr 12, 2017 6:10 pm

synology_ukman wrote:
I wonder why you didn't buy a SAN for that money if you wanted ISCSI? Far more reliable and stable.


Well, regardless of opinion, I am in this position now. While a SAN may have been a better solution in some aspects, it was not in all.

Anyway, I am still hoping someone may run into something that could help things along.

Thanks!
steker
I'm New!
I'm New!
Posts: 3
Joined: Thu Mar 12, 2015 8:56 pm

Re: Iscsi dropping after DSM 6.0

Postby steker » Fri Apr 28, 2017 9:27 pm

Have tried the "fix" and getting the following.

login as: root
root@172.31.0.4's password:
root@xxxxxx:~# synoiscsiep --set-lio-version lio-4.0
ERR(synoiscsiep_int_cs.cpp, 1054)SYNOiSCSIConfigfsChangeCurrentVersion(ISCSI_LIO_4_0_SUPPORT) failed

Has anyone seen this before?
hugie
Trainee
Trainee
Posts: 11
Joined: Thu Dec 08, 2016 1:46 pm

Re: Iscsi dropping after DSM 6.0

Postby hugie » Mon May 08, 2017 11:10 am

Did anyone test the new patches and can verify non-dropping iscsi?

Version: 6.1.1-15101-1

(2017/05/02)
Important Note
The update is expected to be available for all regions within the next few days, although the time of release in each region may vary slightly.
This update will restart Synology NAS models supporting SSD cache.
Fixed Issues
Fixed an issue where domain users/groups might be displayed incorrectly when a file ACL is checked via Samba.
Fixed an issue where users might experience failure or receive no response when updating domain data.
Fixed an issue where volumes might not function normally when read-write SSD cache is used with advanced iSCSI LUNs.
Fixed an issue where encrypted shared folders cannot be unmounted when connections with Samba clients have been established.
Fixed an issue where failure of installation or upgrade might occur in Package Center.
Version: 6.1.1-15101

(2017/05/02)
Important Note
The update is expected to be available for all regions within the next few days, although the time of release in each region may vary slightly.
This update will restart your Synology NAS.
What's New in DSM 6.1.1
This update includes all bug fixes as well as security fixes in the previously released updates since DSM 6.1.
Added support for scheduled RAID scrubbing.
Upgraded Samba to version 4.4.9.
Fixed Issues
Fixed an issue where the OTP might be requested repeatedly when users log in a same trusted device with different accounts.
Fixed an issue where File Station might not work properly when playing slideshows of photos without thumbnails if opened via Application Portal.
Fixed an issue where proxy settings cannot be changed when Let's Encrypt is in use.
Fixed an issue where packages cannot be repurchased after refunds.
Fixed an issue where Auto Block might fail to block addresses when the Allow List is set as an IP range.
Fixed an issue where the default gateway might not be updated when a DHCP client has not received DNS information.
Fixed an issue where firewall rules might not be displayed correctly when users add or remove rules on the same port.
Fixed an issue where UPnP rules might fail to work properly on Synology NAS with a single Ethernet port.
Fixed an issue where the DSM user interface might fail to be properly displayed when DHCP Server receives invalid DNS addresses.
Fixed an issue where L2TP connections might fail to be established in kernel 4.4 when the kernel mode is enabled.
Fixed an issue where an L2TP connection might fail to be established when the server address starts with a number.
Enhanced the usability and display correctness of the Wake on Lan user interface.
Fixed an issue where Synology NAS cannot be properly shut down when a bond has been established between Mellanox ConnectX-3 and Synology E10G17-F2.
Fixed an issue where DSM might show the message of volume crash upon reboot when a single-disk SHR volume has been mounted with cache and transformed into a Disk Group.
Fixed an incorrect volume display on some models where RAID Groups are supported and SHR has been enabled.
Fixed an issue where SHA cannot be created when FS3017 contains over 12 drives.
Fixed an issue where the settings of SMB encrypted transmission cannot be properly configured.
Fixed a connection error that might occur when the name of an SMB shared folder contains consecutive spaces.
Fixed an issue where Key Manager might fail to mount encrypted shared folders automatically on startup.
Enhanced the stability of the iSCSI service.
Fixed an issue where files cannot be written into encrypted shared folders on ext3 file system.
Fixed an issue where Snapshot Replication might fail when compression is enabled for shared folders.
Fixed an issue where DSM cannot be shut down properly on Qoriq or 88f6281/88f6282 platforms when JBOD volumes are in use.
Enhanced the stability of the upgrade process of SHA clusters.
AseK
I'm New!
I'm New!
Posts: 8
Joined: Fri Mar 03, 2017 7:12 pm

Re: Iscsi dropping after DSM 6.0

Postby AseK » Mon May 08, 2017 4:31 pm

Hi hugie,

Downloaded version: 6.1.1-15101-1, but still got issues with iSCSI dropping.
I had to downgrade to lio-4.0 again to get everything working as it should.

So the iSCSI problems are still there.

//AseK
hugie
Trainee
Trainee
Posts: 11
Joined: Thu Dec 08, 2016 1:46 pm

Re: Iscsi dropping after DSM 6.0

Postby hugie » Mon May 08, 2017 8:51 pm

AseK wrote:Hi hugie,

Downloaded version: 6.1.1-15101-1, but still got issues with iSCSI dropping.
I had to downgrade to lio-4.0 again to get everything working as it should.

So the iSCSI problems are still there.

//AseK


Thx a lot for testing.. So there is no reason to upgrade :(
m.msc01
Trainee
Trainee
Posts: 10
Joined: Mon May 16, 2011 3:20 pm

Re: Iscsi dropping after DSM 6.0

Postby m.msc01 » Tue May 09, 2017 9:05 am

Today Synology Release DSM 6.1.1 Update 2 with the Next ISCSI fix :lol:

Version: 6.1.1-15101-2

(2017/05/09)
Important Note
The update is expected to be available for all regions within the next few days, although the time of release in each region may vary slightly.
This update will restart your Synology NAS.
Fixed Issues
Fixed a security vulnerability regarding Linux kernel (CVE-2017-7184).
Fixed an Ubuntu virtual machine system freeze issue on Virtual Machine Manager.
Enhanced the stability of iSCSI service.
jhiggason
I'm New!
I'm New!
Posts: 1
Joined: Fri May 12, 2017 6:41 pm

Re: Iscsi dropping after DSM 6.0

Postby jhiggason » Fri May 12, 2017 6:43 pm

Any word if this update actually fixes the issue? I'm still holding off on upgrading until we get the word....
aMIGa1200
I'm New!
I'm New!
Posts: 2
Joined: Mon Feb 27, 2017 10:55 am

Re: Iscsi dropping after DSM 6.0

Postby aMIGa1200 » Fri May 19, 2017 2:22 pm

Didn't fix it for me but when i lowered mtu from 9000 to 1500 its much more stable (although veeeeery slow).
Izaak99
I'm New!
I'm New!
Posts: 6
Joined: Mon Mar 27, 2017 4:31 am

Re: Iscsi dropping after DSM 6.0

Postby Izaak99 » Fri Jun 30, 2017 11:02 pm

Update seemed to make it worse on the RS4017xs+, now with file-based LUN's I'm getting flooded with the opcode 93 error (where before this was only happening with BLOCK level LUNS). Perhaps they should ditch LIO for SCST and finally put an end to this poorly implemented iSCSI fiasco.

ash-4.3# uname -a
Linux fre-sto-syn2 3.10.102 #15132 SMP Wed Jun 14 12:26:43 CST 2017 x86_64 GNU/Linux synology_broadwell_rs4017xs+
ash-4.3# dmesg | tail
[488447.258753] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.290254] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.339916] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.372300] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.400537] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.431646] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.465558] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.497459] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.528905] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.
[488447.562760] iSCSI:target_core_transport.c:1783:target_setup_cmd_from_cdb iSCSI/iqn.1998-01.com.vmware:fre-srv-esx1-44b9e170: Unsupported SCSI Opcode 0x93, sending CHECK_CONDITION.

Return to “iSCSI Target”

Who is online

Users browsing this forum: No registered users and 1 guest