BIG issue with iscsi server on DS509+ !

This room is for the discussion of how the Synology DiskStation can meet the storage needs for Virtual HyperVisors.

BIG issue with iscsi server on DS509+ !

Postby hypervisor » Mon Sep 28, 2009 10:48 pm

Little story first :

I bought a DS509+ to replace an home server hosting iscsi volume for 2 ESX (for 10VM) and cifs shares for various data.
Because i need to migrate the VM and the data, a friend give me 5x 1Tb drives temporary.
After copying the data and the VM, i begin to replace each drive to give them back to my friend.

Here begin the BIG problems :
When i replace the first drive, i used the "repair" option witch (for an unknown reason) disable the iscsi server ! When i re-activate it, the VMFS partition was totally corrupted :

Image

Of course, i had a backup...
After the rebuild finished, i try to delete and re-create the iscsi volume to test again.
Just after remove the iscsi volume, the NAS begin to beep : the volume1 was crashed :

Image

BTW, all drives status is "normal" :

Image

Can i trust this product ?!!!
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby jekyll » Tue Sep 29, 2009 6:44 am

Seagate, WD green / blacks & samsungs in a RAID array...
very bad news from the start...

Thats possibly the root of your problem
CS-407 - DSM 2.2-0953 - 4xWD6400AAKS 'RAID5' : APC Smart UPS 750 & AP9619
DS-107 - DSM 2.2-0953 - WD10EADS [Flashed DS101G+]
DS-106 - DSM 2.2-0953 - WD5000AACS [Flashed DS101G+]
USB Station - Dymo 310 USB : Canon IP4200 : APC UPS 350

DES-1016D : Ellion HMR-700A : Netgear MP101 : 4xBL-C1 IP Cams : APC BK 650MI


A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty

| Back soon - waiting for a new connection |
User avatar
jekyll
g33k
g33k
 
Posts: 1817
Joined: Thu Feb 28, 2008 7:55 pm
Location: Low Burnham, Epworth - UK

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 10:15 am

You mean 5 differents healthy disk could be the cause of a buggy kernel ?

BTW, after a reboot here is the volume 1 :

Image
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby HarryPotter » Tue Sep 29, 2009 11:59 am

RAID 5 with 5 different HD, 4 different brands..... there is just one advice: don't do it!
*Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!*

DS712+ / DSM 4.2 3211 / 2 x HD204UI (RAID0) / LMS 7.7.2-007
DS110+ / DSM 4.2 3211 / HD154UI
DS408 / DSM 4.0 2228 / 3 x ST31000340AS (RAID0) + 1 x HDS721010KLA330
DS207+ / DSM 3.1.1636 / ST31000340AS + WD5000KS
DS-106x / DSM 4.0 2228 (from DS108j) /

2 Squeezebox 3 + Boom
Rapsody N35

APC Smart UPS SUA750i
User avatar
HarryPotter
Honorary Moderator
Honorary Moderator
 
Posts: 12343
Joined: Mon Oct 23, 2006 12:48 pm
Location: Switzerland

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 12:02 pm

I'm about to replace with 5 identical drives but i would like to sucess doing it...
I retry with a provisioned iscsi volume, maybe there is a bug it the auto grow function ?

BTW, if i use NFS server with ESX, the connection with the nfs share is dropped as soon as i enter in the "shared foleder" menu...
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 4:17 pm

HarryPotter wrote:RAID 5 with 5 different HD, 4 different brands..... there is just one advice: don't do it!


It's 3 brand acctually WD, ST and Samsung
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby jekyll » Tue Sep 29, 2009 4:58 pm

hypervisor wrote:It's 3 brand acctually WD, ST and Samsung


What we are trying to point out is....

It is not advised to use different brands or different models of disks.
The RAID array will have trouble as the drives PCB operates in a different manner.

This is why you have a normal status but a failed rebuild... some drives will be timing out before others resulting in any of the drives 'hanging' which in turn will cause a drop.

It's like buying a car and installing a jet engine in the back to work with the original engine in the front...
You need equal devices... of equal cache and build(s), firmware on drives could also result in a drop...
CS-407 - DSM 2.2-0953 - 4xWD6400AAKS 'RAID5' : APC Smart UPS 750 & AP9619
DS-107 - DSM 2.2-0953 - WD10EADS [Flashed DS101G+]
DS-106 - DSM 2.2-0953 - WD5000AACS [Flashed DS101G+]
USB Station - Dymo 310 USB : Canon IP4200 : APC UPS 350

DES-1016D : Ellion HMR-700A : Netgear MP101 : 4xBL-C1 IP Cams : APC BK 650MI


A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty

| Back soon - waiting for a new connection |
User avatar
jekyll
g33k
g33k
 
Posts: 1817
Joined: Thu Feb 28, 2008 7:55 pm
Location: Low Burnham, Epworth - UK

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 6:15 pm

IT does not explain why the iscsi volume was corrupted and NO problem was detected during the rebuild.

Anyway, is there any way to force a consistency check on the RAID volume ?
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby jekyll » Tue Sep 29, 2009 6:32 pm

hypervisor wrote:IT does not explain why the iscsi volume was corrupted and NO problem was detected during the rebuild.

It's corrupt due to compatability issues with various drives in a RAID array.
If you make an iSCSI target with a basic volume... i would bet my last penny that it will be sucsessful!
hypervisor wrote:Anyway, is there any way to force a consistency check on the RAID volume ?

It's an mdadm job!

See this post

Forum search features & google are your friend ;)

Regards


Phill
CS-407 - DSM 2.2-0953 - 4xWD6400AAKS 'RAID5' : APC Smart UPS 750 & AP9619
DS-107 - DSM 2.2-0953 - WD10EADS [Flashed DS101G+]
DS-106 - DSM 2.2-0953 - WD5000AACS [Flashed DS101G+]
USB Station - Dymo 310 USB : Canon IP4200 : APC UPS 350

DES-1016D : Ellion HMR-700A : Netgear MP101 : 4xBL-C1 IP Cams : APC BK 650MI


A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty

| Back soon - waiting for a new connection |
User avatar
jekyll
g33k
g33k
 
Posts: 1817
Joined: Thu Feb 28, 2008 7:55 pm
Location: Low Burnham, Epworth - UK

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 6:54 pm

Thanks, i googled "synology force consistency check" without success :D
can you explain why the iscsi volume was corrupted but not the cifs share ?
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby jekyll » Tue Sep 29, 2009 7:29 pm

hypervisor wrote:can you explain why the iscsi volume was corrupted but not the cifs share ?

To be honest i'm not an expert... just a friendly person offering advise..
However, i would speculate that iSCSI would thrash the RAID array with its format method due to the file system of iSCSI.
And i believe the communication method maybe a little more demanding than CIFS.
It doesn't only boil down to things like that, i guess you have to look at all elements of the equasion.
All it takes is a single drive to hang during a drive stressing period and the array will be dropped.

I myself like to mess around with machines and over time you see patterns in things, it's just a hobby...

The bottom line is... you used different makes & models of drives, as harrypotter confirmed.. Your setup is a recipe for disaster.

I know it's an expensive job buying a 5 x terrabyte drives, but sadly if you value your data your going to have to buy them.

Maybe JBOD may react differently as it's a complete different method of storage!
CS-407 - DSM 2.2-0953 - 4xWD6400AAKS 'RAID5' : APC Smart UPS 750 & AP9619
DS-107 - DSM 2.2-0953 - WD10EADS [Flashed DS101G+]
DS-106 - DSM 2.2-0953 - WD5000AACS [Flashed DS101G+]
USB Station - Dymo 310 USB : Canon IP4200 : APC UPS 350

DES-1016D : Ellion HMR-700A : Netgear MP101 : 4xBL-C1 IP Cams : APC BK 650MI


A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty

| Back soon - waiting for a new connection |
User avatar
jekyll
g33k
g33k
 
Posts: 1817
Joined: Thu Feb 28, 2008 7:55 pm
Location: Low Burnham, Epworth - UK

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 7:42 pm

Actually, those drives are not mine. I got 5 (+1 spare) identical drive in my "old" raid config ;)
Thanks for your answer and your advises.
I tried mdadm --misc --detail /dev/md0 but i got this result : mdadm: cannot open /dev/md0: Permission denied
I guess another process own the device so i can't do anything on it. any idea about that ?
Thanks again and excuse me if i'm a little pissed off but i was about to loose all my data this weekend :shock:
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby HarryPotter » Tue Sep 29, 2009 8:12 pm

"permission denied" on the command line interface is in nearly all cases due to not being logged in as "root" (same password as for "admin")
*Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!*

DS712+ / DSM 4.2 3211 / 2 x HD204UI (RAID0) / LMS 7.7.2-007
DS110+ / DSM 4.2 3211 / HD154UI
DS408 / DSM 4.0 2228 / 3 x ST31000340AS (RAID0) + 1 x HDS721010KLA330
DS207+ / DSM 3.1.1636 / ST31000340AS + WD5000KS
DS-106x / DSM 4.0 2228 (from DS108j) /

2 Squeezebox 3 + Boom
Rapsody N35

APC Smart UPS SUA750i
User avatar
HarryPotter
Honorary Moderator
Honorary Moderator
 
Posts: 12343
Joined: Mon Oct 23, 2006 12:48 pm
Location: Switzerland

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Tue Sep 29, 2009 8:17 pm

HarryPotter wrote:"permission denied" on the command line interface is in nearly all cases due to not being logged in as "root" (same password as for "admin")

Thanks.
Can i use fsck.ext3 to check the /volume1 partition too ?
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Re: BIG issue with iscsi server on DS509+ !

Postby hypervisor » Wed Sep 30, 2009 1:46 am

It appears that the iscsi server is low prioritized when a rebuild is going on.
So my ESX suffer of heavy time out on the datastore where the VM are stored.
Is there any way to rise the priority level of the iscsi server, or lower the rebuild rate ?
hypervisor
Trainee
Trainee
 
Posts: 14
Joined: Mon Sep 28, 2009 10:15 pm

Next

Return to Virtual HyperVisors (VMWare/ESXi)

Who is online

Users browsing this forum: No registered users and 0 guests