ESX 4 and iSCSI problem

This room is for the discussion of how the Synology DiskStation can meet the storage needs for Virtual HyperVisors.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.

ESX 4 and iSCSI problem

Postby sepetdav » Tue Dec 29, 2009 11:12 am

Hello everyone!

I use my DS509+ (Firmware 959) as a iSCSI storage for my VMware ESX 4 (vSphere 4.0.0 Build 171294)
VMware is running on a Intel S3210SHLX with QUAD core CPU and two DUAL BCM5704 Ethernet cards connected with a D-Link DGS-1224T Gigabit switch.
The DS509+ is equipped with five 1TB Samsung Spinpoint F1 HD103UJ hard drives.

My problem is with the connection to the iSCSI storage. When the storage is under load, for example if I make a backup within a Virtual Machine and save the backup to the NFS on the DS509+.
VMware will lose the connection after some time (could be a hour or some times half a day)
When connection is lost the only way to solve the problem is to restart the DS509+, I have tried to rescan in VMware but it can't find the storage.
Yesterday I tried to ad the storage again after losing connection, VMware could find the iSCSI target but could NOT connect to it.
After the reboot of the DS509+ it worked fine for like 10 hours during my backup but then it failed again.

Anyone with a solution or workaround?

Regards Peter
sepetdav
I'm New!
I'm New!
 
Posts: 4
Joined: Tue Aug 25, 2009 1:57 pm
Location: Sweden

Re: ESX 4 and iSCSI problem

Postby ladinek » Tue Jan 05, 2010 9:18 am

I have similar problems. We implemented DS209+II for virtual machines images backup under ESX4. First I tried to use NFS, but unstability uccured (large files not copied entirely - see other threads about NFS). Then, I tried iSCSI. I created pre-allocated iSCSI volume, on it VMFS datastore. At first, everything looked great, but after few days when I rechecked, the datastore was unaccessible, I can see the target, but the device over all rescans etc etc. shows 0 bytes capacity. Reboot of the Synology did not helped. I tried latest firmware in Synology, instaled VMWare Update Manager and updated all ESX hosts to latest (Update1+patches), same result. Everything other in the network is OK, and with problems with DS209+II, I set simple machine with Intel Atom and the Openfiler iSCSI target to have 1) comparison 2) backup, and this machine works over a month without a single glitch (iSCSI target, not NFS).
ladinek
I'm New!
I'm New!
 
Posts: 4
Joined: Sat Oct 17, 2009 1:50 pm

Re: ESX 4 and iSCSI problem

Postby sepetdav » Tue Jan 05, 2010 10:23 am

Nice to hear that I'm not alone. I have a case with Synology support and hope they might solve the problem. Can you access the Synology DS shares and web GUI when the iSCSI target is down. I could access other things on the DS in the beginning of this problem but later on when the iSCSI target hang the rest of the DS also hangs. I'm pretty sure the problem is in the DS because when I reboot the DS and rescan the storage (iSCSI) in VMware it will find it again an the virtual machines I had running will continue as nothing happened. Does any of this sound familiar.
MVH Peter "RAID" Davidsson
sepetdav
I'm New!
I'm New!
 
Posts: 4
Joined: Tue Aug 25, 2009 1:57 pm
Location: Sweden

Re: ESX 4 and iSCSI problem

Postby vm.franklin » Tue Jan 05, 2010 11:59 pm

Greetings all

Can you specify the exact details on how you've created the iSCSI Target.
With ESXi - please tell me an example VM you are running, and how long to run the VM/ESXi before the iSCSI Target disappears from ESXi?

Also - is there only one iSCSI Target on the DiskStation?
**Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!**
Library ~ SynologyWiki ~ Synology FAQ ~ Compatibility Lists
Forum Links ~ Forum Policy ~ 3rd-party forums ~ Help us help you ~ Posting Images
Demo Links ~ DSM GUI ~ Photo Station
Downloads ~ Firmware Downloads ~ Beta Program
Support ~ Support Form ~ Submit Kernel Log ~ Synology eNews
vm.franklin
Synology Inc
Synology Inc
 
Posts: 374
Joined: Mon Oct 05, 2009 8:18 pm

Re: ESX 4 and iSCSI problem

Postby ladinek » Thu Jan 14, 2010 9:54 am

After I lost first iSCSI volume as I mentioned before, updated everything, used scheduled power down to reboot everything, created new iSCSI volume of 500GB with Allocate all space now option, re-created iSCSI target, reconnected single ESX, created VMFS datastore and waited overnight for CRON to do its job. However, on the morning, iSCSI datastore seems to be empty (after LONG scanning datastrore...), nothing backed up, and re-running manually my backup script takes very long ending with timeout message.
ladinek
I'm New!
I'm New!
 
Posts: 4
Joined: Sat Oct 17, 2009 1:50 pm

Re: ESX 4 and iSCSI problem

Postby sepetdav » Wed Jan 20, 2010 3:10 pm

Hi!

I created a 500 GB iSCSI Target with pre-allocated disk, CHAP authentication enabled and Allow multiple iSCSI initiators.

I'm using a virtual Machine with Windows 2003 server with Backup Exec installed. I use Backup Exec to backup my other Windows 2003 servers. Backup Exec saves the backup to "disk" in this case a NFS share on same the DS-509.
(I also have a duplicate job that will make a copy of the first backup file to a external disk, I managed to complete the backup once and then I tried to run the duplicate job. The duplicate job copy the disk file to a USB disk attached to the DS. The result was the same, crash/hang of DS)

Last time I tried the DS crashed after 7 minutes, I can ping the DS but the Web GUI and iSCSI target doesn’t respond.

I have a open case at the Synology support and this was the latest answer..

-------------------------------------------------
We were using the same backup software as mentioned in your last thread - Veritas Backup Exec with Virtual Machine.

The DS hanged issue might be caused by the Jumbo Frame; however, since the MTU is disabled, we had no longer discovered this issue so far.

Anyway, we'll pass this request to our product development team and will begin to consider about doing more research on supporting such function. However, this procedure may need some time before our official announcement.

-------------------------------------------------
MVH Peter "RAID" Davidsson
sepetdav
I'm New!
I'm New!
 
Posts: 4
Joined: Tue Aug 25, 2009 1:57 pm
Location: Sweden

Re: ESX 4 and iSCSI problem

Postby vm.franklin » Thu Jan 21, 2010 12:18 am

@ladinek

Your error could not be reproduced - as suggested by sepetdav's post - please try disabling JumboFrames.

Also, please send your kernel log (see sig link) to forum at synology dot com with the email subject of "forum - 19946 - esx4 and iscsi problem" so I know what the kernel log is for. A copy of your cronjob would be helpful as well.

Please note that a solution may not come quickly as we cannot reproduce the error.

Alternatively, you may wish to consider contacting online support (see sig link) for further assistance, as it will put you in touch with our support team directly.

Hope this helps.
**Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!**
Library ~ SynologyWiki ~ Synology FAQ ~ Compatibility Lists
Forum Links ~ Forum Policy ~ 3rd-party forums ~ Help us help you ~ Posting Images
Demo Links ~ DSM GUI ~ Photo Station
Downloads ~ Firmware Downloads ~ Beta Program
Support ~ Support Form ~ Submit Kernel Log ~ Synology eNews
vm.franklin
Synology Inc
Synology Inc
 
Posts: 374
Joined: Mon Oct 05, 2009 8:18 pm

Re: ESX 4 and iSCSI problem

Postby roady001 » Wed Feb 10, 2010 2:12 pm

Just wanted to let you know that we just have had a similar situation:

Model name: DS409+
Current firmware version: DSM 2.2-0942
Firmware date: 2009/09/04

VMware ESXi 4.0.0.208167

Configured an iSCSI target for 750 GB, Allow multiple iSCSI initiators is checked. Maximum receive: 262144, maximum send: 4096. No authentication (as it is just for local LAN use).

We have had the ESXi up and running with multiple virtual machines for like 3 weeks without any problems. last night, the storage was gone from the ESXi, while connection was still live. The synology webinterface was accessible as if nothing happened and I was also able to configure a new iSCSI target of 1 GB (just for testing purposes). Still, the ESXi server could not see this new test iSCSI target.

Some more observations:
The ESXi seemed to hang for quite some time trying to get information from the iSCSI target. Sometimes it failed, other times it showed as 'on-line', but the datastore could not be restored.
First I was blaming the ESXi server and rebooted the machine. I could see the ESXi server connecting again to the Synology (the webinterface reported the server as 'connected'), but the datastore didn't get available and the virtual machines were marked as unaccessible. Then I realized that rebooting the Synology might help and whala: ESXi restored the datastore and the virtual machines (I'm writing this in one of them).

[EDIT]
Today, one day after I wrote the previous, I went into the office and observed that the problem happened again! The ESXi lost connection to the iSCSI. Now I have one virtual machine that wouldn't boot into Windows the first time stating that there were 'hard drive' problems. The second boot fortunately wend smooth again.

New observations:
I now have a strong indication that the problem is due to the network-backup service that accepts an external backup in the night. It is the only thing that changed since 2 days ago. I have another Synology box from an external location that runs a backup to this Synology over the internet (5.00 AM). This is a backup of +/- 30 GB with a huge number of small files. The backup is non-encrypted with byte-level enabled. I found that this morning the Synology was 'sleeping', which of course should never have happened with a live connection to an ESXi server. I'm guessing that the network-backup caused the iSCSI to permanently drop connection and because it didn't have anything to do after the network-backup finished it decided to go into sleep mode. The only way to restore the iSCSI connection to the ESXi is to reboot the Synology
roady001
I'm New!
I'm New!
 
Posts: 2
Joined: Wed Feb 10, 2010 1:49 pm

Re: ESX 4 and iSCSI problem

Postby roady001 » Fri Feb 12, 2010 9:45 am

Another update: I have disabled the backup for last night and today everything is still spinning.
My conclusion for now would be that combined with the previous posters is that the iSCSI will fail under heavy diskload (Synology wide, independent from protocol or transport method).
roady001
I'm New!
I'm New!
 
Posts: 2
Joined: Wed Feb 10, 2010 1:49 pm

Re: ESX 4 and iSCSI problem

Postby vRob » Sun May 23, 2010 5:59 pm

Sorry to bump this thead, but it's quite similar to my problems.

I'm using a DS1010+ (DSM 2.3-1157) with ESX 4.0.0 208167 (Update 1A) and ESXi 4.0.0 Update 1A
Using both Link Agregation and Jumbo packets (MTU 9000).

ESX Server 1: HP DL360 G5, ESX Server 2: IBM x3550 7978
Switch: 3Com Baseline Switch 2924-SFP Plus
no vlans.
Tried running VMs on either host, tried running only one host.
Only one path to the storage.

Using 500 GB iSCSI targets,
Authenication: CHAP
Multiple Sessions: Yes
Tried with and without CRC Checksum; both Header and Data digest.
Tried default values and expanding both Maximum recieve and send segment sizes to the max allowed (262144) on Syno and ESX server.

Running 4 - 6 VMs 32-bit Windows 2003 and XP

Everything seems to work fine, but then.. all VMs stop working, vmkwarning complains:

Code: Select all
May 23 18:14:28 esx001 vmkernel: 0:20:36:47.578 cpu0:4112)WARNING: FS3: 6604: Reservation error: Timeout
May 23 18:15:08 esx001 vmkernel: 0:20:37:27.590 cpu0:4112)WARNING: ScsiDeviceIO: 1266: Failed to issue command (0x16) on device t10.945445000000000000000000100000002008000000000DF0: Timeout


I disabled Jumbo frames on the Synology, and everything goes back to normal, I didn't reboot the Synology device or the ESX server.
After disabling Jumbo Frames (Setting MTU to 1500) vmkwarning:

Code: Select all
May 23 18:25:51 esx001 vmkernel: 0:20:48:10.395 cpu3:4240)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba33:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"


All VMs resumed, mail start flowing again....

I even saw All Paths Dead messages:

Code: Select all
May 23 09:10:04 esx001 vmkernel: 0:11:32:23.735 cpu3:4210)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "t10.945445000000000000000000100000002008000000000DF0" - failed to issue command due to Not found (APD), try again...


I just disabled the Jumbo Frames, so I'll report back in 24 hours or so, to let you know if all VMs keep running or not.

Did I make some configuration errors, are there any best-practices I should follow?.

Thanks!
vRob
I'm New!
I'm New!
 
Posts: 2
Joined: Sun May 02, 2010 10:56 am

Re: ESX 4 and iSCSI problem

Postby vRob » Sat Jun 12, 2010 11:25 am

Sadly I have to add that altough everything seemed ok after disabling the Jumbo Frames. The error has returned. Had it twice the last week.
I need to reboot the DS1010.

Is there something I'm doing wrong?
Are there any logfiles I need to check and/or provide to Synology support?

The thing is that when the device is crashing, file sharing (smb) still works on the device, no high CPU and/or memory usage is shown. But accessing a file on the device is noticably slower. The device seems to need more time to process the file reqest.

I reboot the DS1010, and need to reset all VMs. This will lead to issues within the VMs.

@Synology Support: please look into this, I need a stable plaform and this, just isn't.
vRob
I'm New!
I'm New!
 
Posts: 2
Joined: Sun May 02, 2010 10:56 am

Re: ESX 4 and iSCSI problem

Postby vm.franklin » Mon Jun 14, 2010 4:01 am

@vRob

For communicating with online support directly, please submit your kernel log to our support team, the links to do so are in my signature.
**Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!**
Library ~ SynologyWiki ~ Synology FAQ ~ Compatibility Lists
Forum Links ~ Forum Policy ~ 3rd-party forums ~ Help us help you ~ Posting Images
Demo Links ~ DSM GUI ~ Photo Station
Downloads ~ Firmware Downloads ~ Beta Program
Support ~ Support Form ~ Submit Kernel Log ~ Synology eNews
vm.franklin
Synology Inc
Synology Inc
 
Posts: 374
Joined: Mon Oct 05, 2009 8:18 pm

Re: ESX 4 and iSCSI problem

Postby koulinek » Fri Jun 18, 2010 11:21 am

I have same issue with DS210j and I realized that when iSCSI is disconnected and ESXi did not see anything afer rescan, It is sufficient to disable and enable iSCSI volume on DS210j webinterface.
koulinek
I'm New!
I'm New!
 
Posts: 1
Joined: Fri Jun 18, 2010 11:09 am

Re: ESX 4 and iSCSI problem

Postby pclegen » Sat Nov 27, 2010 12:08 am

i have this exact same problem with my ESXI4 and DS410j.
It can work for days without trouble, even weeks. But suddenly ESXi cant find the iscsi target. Web interface on DS410j is accessible and everything seems OK.
Tried to boot ESXi server without a solution to the problem.
Went into the web interface on DS and disabled and then enablet the iscsi target. Then everything is OK again.
pclegen
I'm New!
I'm New!
 
Posts: 1
Joined: Sat Nov 27, 2010 12:05 am

Re: ESX 4 and iSCSI problem

Postby matarvai » Wed Dec 08, 2010 11:17 am

Has anyone solved this. I have ESXi4.1 connected to DS410 with DSM3.0 and I have same issues. I need to reboot or disable/enable iscsi to get everything working again. This sucks, I also need reliable box with ISCSI.
matarvai
Rookie
Rookie
 
Posts: 35
Joined: Fri Dec 18, 2009 5:50 pm

Next

Return to Virtual HyperVisors (VMWare/ESXi)

Who is online

Users browsing this forum: No registered users and 2 guests