Possible problem with vSphere 6 and iSCSI MPIO?

This room is for the discussion of how the Synology DiskStation can meet the storage needs for Virtual HyperVisors.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
erpomik
I'm New!
I'm New!
Posts: 7
Joined: Sun Sep 05, 2010 8:38 am

Possible problem with vSphere 6 and iSCSI MPIO?

Postby erpomik » Thu Jun 25, 2015 11:49 pm

Hi

This post is meant as a warning to other users, more than an actual question!

Recently we upgraded one of our small VMware environments to vSphere 6. Soon to experience crashed LUNs and even crashed disks. After two weeks and two disaster recovery scenarios, we decided to go back to vSphere 5.5 as this has been running flawlessly for a long time.

Our environment:
Three DELL R620 servers and two Synology RackStations - One RS3412RPxs and one RS3614RPxs. All Synos are running DSM 5.2 with newest patches. All LUNs are configured as Block-Level iSCSI Targets.
Image

Our experience when running vSphere 6:
When the ESXi host starts up, we see two paths to each LUN on the Syno with LUN IDs like 0, 1, 2, etc. But after a while, the host has quadrupled the paths to each LUN with LUN IDs like 0/256/512/768, 1/257/513/769, etc. And when this happens, all the trouble starts. This is where the Syno gets so busy, that the LUNs are crashing (unrecoverable crashes).

Our conclusion:
After the storm has ceased, we did some investigation and in vSphere Configuration Maximums we found, that Maximum LUN ID has increased from 255 to 1023 (8 bits vs. 10 bits respectively). This might explain the extra "modulus 256" LUNs that the Syno starts to present (numbere 256/512/768 etc.).

A similar post has been placed in the VMware forum https://communities.vmware.com/message/2516864.
Also Synology support has been contacted.
Best regards
Ernst Mikkelsen (VCP5)
cheplyaev.av
I'm New!
I'm New!
Posts: 2
Joined: Fri Jul 10, 2015 9:18 am

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby cheplyaev.av » Fri Jul 10, 2015 9:20 am

Hello, we have same issue.
We had to disable path to "ghost" luns, but they become active after some time again. So we changed round robin to fixed manage path policy. Now it`s fine. Waiting for some fix from VMware or Synology.Image
erpomik
I'm New!
I'm New!
Posts: 7
Joined: Sun Sep 05, 2010 8:38 am

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby erpomik » Tue Jul 14, 2015 10:48 am

cheplyaev.av wrote:So we changed round robin to fixed manage path policy.

Nice workaround. It could have saved me a downgrade, if I had had that idea! 8)
Best regards
Ernst Mikkelsen (VCP5)
cheplyaev.av
I'm New!
I'm New!
Posts: 2
Joined: Fri Jul 10, 2015 9:18 am

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby cheplyaev.av » Tue Jul 14, 2015 11:41 am

erpomik wrote:Also Synology support has been contacted.


Any answer from Synology support? After host reboot disabled "ghost" path sometimes become active again and i have to check it on each host. When "ghost" path is in enable state my DS1513+ become very very slow. Need fix from Synology.
erpomik
I'm New!
I'm New!
Posts: 7
Joined: Sun Sep 05, 2010 8:38 am

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby erpomik » Tue Jul 14, 2015 1:04 pm

cheplyaev.av wrote:Any answer from Synology support? After host reboot disabled "ghost" path sometimes become active again and i have to check it on each host. When "ghost" path is in enable state my DS1513+ become very very slow. Need fix from Synology.

Thursday July 2nd we received the following response to our Synology Support Ticket #609013.
Thank you for your feedback and sorry for this late response.
We are really appreciated for your information, our developers have looking into this issue now.
For now, please use ESXi 5.5 as workaround now, we will sort this out as soon as possible.
Hope this has and please let us know if any further assistance.

Take care and have a good day,
Technical Support
Ricky Lu

To me, this sounds like Synology has almost accepted this to be a problem. However we haven't heard anything from them since.
Best regards
Ernst Mikkelsen (VCP5)
vision100
I'm New!
I'm New!
Posts: 2
Joined: Tue Jul 28, 2015 7:20 pm

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby vision100 » Wed Jul 29, 2015 6:50 pm

Fixed(vmware) workaround thnx
vision100
I'm New!
I'm New!
Posts: 2
Joined: Tue Jul 28, 2015 7:20 pm

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby vision100 » Wed Jul 29, 2015 8:16 pm

after upgrade from esxi 5.5 to esxi 6.0 iscsi LUN

2015-07-27T05:08:59.488Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:09:39.490Z cpu5:32810)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:09:39.490Z cpu5:32810)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:10:19.503Z cpu7:32809)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:10:19.503Z cpu7:32809)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:10:59.516Z cpu2:32811)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:10:59.516Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:11:39.527Z cpu7:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:11:39.527Z cpu7:32810)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:11:39.527Z cpu7:32810)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:12:26.088Z cpu1:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:12:26.089Z cpu2:32809)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:12:26.089Z cpu2:32809)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:13:16.091Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:1 T:1 L:768 : Task mgmt "Abort Task" with itt=0x3fdf4 (refITT=0x3fdf3) timed out.
2015-07-27T05:13:22.017Z cpu4:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:13:22.017Z cpu4:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:13:22.017Z cpu4:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:54612 R: 192.168.1.214:3260]
2015-07-27T05:13:22.017Z cpu1:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:13:22.017Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:1 T:1 L:768 : Task mgmt "Abort Task" with itt=0x3fdf6 (refITT=0x3fdf3) timed out.
2015-07-27T05:13:22.017Z cpu5:32811)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:13:22.017Z cpu5:32811)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:13:22.720Z cpu1:41712)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:13:23.713Z cpu1:33212)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:13:24.727Z cpu1:32822)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:13:25.059Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "ONLINE"
2015-07-27T05:13:25.059Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:13:25.059Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.2.191:52751 R: 192.168.1.214:3260]
2015-07-27T05:14:02.042Z cpu0:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:14:02.042Z cpu1:32810)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:14:02.042Z cpu1:32810)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:14:42.061Z cpu5:32809)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:14:42.061Z cpu5:32809)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:15:22.076Z cpu0:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:15:22.076Z cpu5:32811)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:15:22.076Z cpu5:32811)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:16:02.088Z cpu2:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:16:02.088Z cpu2:32810)WARNING: HBX: 3280: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-07-27T05:16:02.088Z cpu2:32810)WARNING: [HB state abcdef02 offset 3969024 gen 183 stampUS 18308247299 uuid 55b573e2-9c81c485-d98a-6805ca0fedec jrnl <FB 2401002> drv 14.61 lockImpl 3]
2015-07-27T05:18:10.333Z cpu2:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:18:30.718Z cpu6:32996)WARNING: ScsiPath: 7133: Set retry timeout for failed TaskMgmt abort for CmdSN 0x0, status Failure, path vmhba37:C1:T1:L512
2015-07-27T05:18:30.718Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:1 T:1 L:512 : Task mgmt "Abort Task" with itt=0x41719 (refITT=0x41718) timed out.
2015-07-27T05:18:36.614Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:18:36.614Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:18:36.614Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:52751 R: 192.168.1.214:3260]
2015-07-27T05:18:36.614Z cpu2:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:1 T:1 L:512 : Task mgmt "Abort Task" with itt=0x4171b (refITT=0x41718) timed out.
2015-07-27T05:18:36.614Z cpu7:38657)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:18:36.715Z cpu7:33212)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:18:45.160Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:18:45.160Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:18:45.160Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:18203 R: 192.168.1.214:3260]
2015-07-27T05:18:53.441Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:18:53.441Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:18:53.441Z cpu7:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:24287 R: 192.168.1.214:3260]
2015-07-27T05:19:01.749Z cpu5:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:19:01.749Z cpu5:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:19:01.749Z cpu5:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:34938 R: 192.168.1.214:3260]
2015-07-27T05:19:06.363Z cpu5:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "ONLINE"
2015-07-27T05:19:06.363Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:19:06.363Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.2.191:28168 R: 192.168.1.214:3260]
2015-07-27T05:28:19.344Z cpu7:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:33:09.397Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:1 L:512 : Task mgmt "Abort Task" with itt=0x497c8 (refITT=0x497c7) timed out.
2015-07-27T05:33:16.132Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:1 CN:0: TMF Response PDU: Referenced task not found: itt 0x497c8
2015-07-27T05:33:16.132Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000001 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:33:16.132Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 192.168.2.190:32636 R: 192.168.1.214:3260]
2015-07-27T05:33:16.132Z cpu7:42902)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-07-27T05:35:01.557Z cpu2:52261)ALERT: hostd detected to be non-responsive
2015-07-27T05:37:14.756Z cpu6:32996)WARNING: ScsiPath: 7133: Set retry timeout for failed TaskMgmt abort for CmdSN 0x0, status Failure, path vmhba37:C1:T1:L512
2015-07-27T05:37:14.756Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:1 T:1 L:512 : Task mgmt "Abort Task" with itt=0x423ab (refITT=0x423a9) timed out.
2015-07-27T05:37:15.247Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-07-27T05:37:15.247Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-07-27T05:37:15.247Z cpu0:38646)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.191:28168 R: 192.168.1.214:3260]
Tu9a2
I'm New!
I'm New!
Posts: 4
Joined: Wed Nov 11, 2015 8:59 am

Re: Possible problem with vSphere 6 and iSCSI MPIO?

Postby Tu9a2 » Wed Nov 18, 2015 6:29 am

We have plan to build a vSphere 6 environment using HP servers and 1 Syn RS2211RP+ this weekend. Just post here to track if something new happens. Will inform you guys for updated info.

Regards,
Tu9a2

Return to “Virtual HyperVisors (VMWare/ESXi)”

Who is online

Users browsing this forum: No registered users and 2 guests