DS1813+DS513 - read speeds slow - xfer rate vs utilization

IF your questions pertains to using the file server performance, transfer issues, jumbo frames tuning, post it here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form and check this FAQ:
https://myds.synology.com/support/suppo ... p?lang=enu
https://www.synology.com/en-global/know ... se/faq/586
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

DS1813+DS513 - read speeds slow - xfer rate vs utilization

Postby shoek » Fri Feb 20, 2015 8:16 pm

I've been dealing with slow read speeds on my DS1813+DS513 (loaded full with 13 Hitachi Ultrastar 4TB drives). Synology support has been no help. I've tried all DSM versions from 4.3 to 5.0 to 5.1 with little difference.
I recall speeds being faster (in line with Synology specs) in the past, but can't pin down when or what happened to make it change. I kind of think it could be the addition of the DS513, but I'm not sure. I've played around with network settings - SMB vs SMB2, plain gigabit, gigabit + jumbo frames, 2x gigabit aggregated, etc. and the perf is the same, which gets me thinking that it has something to do with the drives and the RAID implementation of Synology. All apps are disabled and pretty much all services too.

Here is the transfer rate graph of a 30MB Blu-ray ISO. The overall transfer rate was ~20MB/s, with several peaks and valleys as you can see. Time was ~35min. Unacceptable for a RAID system with this many drives.

Image

Notice how at the same time mark where performance was acceptable (80MB/s), the utilization was lower (45%). Everywhere the utilization was 90+%, the performance was abysmal.

Image

Has anyone else experienced this? Is the 1813 just not powerful enough to manage a 13 drive system and still reach 80MB/s speeds?

I should note that writing is much faster than reading, meeting my expectations of 60-80+ MB/s

I'm at a loss here, looking for any ideas anyone may have!

-Steve
dustint
Experienced
Experienced
Posts: 140
Joined: Sun Feb 01, 2015 2:34 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby dustint » Sat Feb 28, 2015 1:32 pm

What are you transferring to? All I see here is info on the Synology, but the other system could be the bottleneck.
1815+; pro-blog.com
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Sat Feb 28, 2015 6:31 pm

Transferring to a Windows 7 PC
When the source of the transfer is a Windows 2008R2 server with a Areca RAID-6 array, the speeds are over 100MB/s on the same content.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Wed Mar 18, 2015 1:48 pm

YOU ARE NOT ALONE.

Also current in the File Server Performance forum- viewtopic.php?f=14&t=98263&p=370226#p370226

When doing large sequential reads, I document the same relationship between utilization % and performance- it tends to bump up on 100%, and copy speed sucks. I'll get about 30-40MB/sec net between the peaks and valleys.

I've had Syno engineers remote in via teamviewer and poke around, but they don't yet know how to solve this. But I can tell you they (or at least the 1 I was working with) have observed this behavior. We've run performance stats on every drive and each drive is solidly within expectations, but the RAID as a whole has performance issues.

I'm sure the 1813 is capable of managing 13 drives and saturating GigE- I've had an 1812 in production for years, and it can approach 200MB/sec in both directions with its bonded GigE. The 1813 is newer- you're suffering the same bug that used mdonion and I am. Both by Synos are on the same DSM build, and my old 1812 is totally fine.

I'd love to gather some more data....
What's your volume format? SHR/SHR2/R5/R6
What are the make, model and size of the disks?
How many disks are in the same volume?
Do you know what build of DSM was installed when the volume was created?
If not, if you know the date it was created, we could probably deduce by DSM release dates, presuming you went to DSM.current before configuring the volume
What packages have you installed?

My working theory is that something is amiss in the creation of RAID volumes under some build of DSM 5.x, which is why my 1812 doesn't have such issues- it's something that happens only when the volume is made. (again, just the opinion of an experienced sysadmin, not a Syno tech)
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Wed Mar 18, 2015 2:36 pm

What's your volume format? SHR/SHR2/R5/R6
What are the make, model and size of the disks?
How many disks are in the same volume?
Do you know what build of DSM was installed when the volume was created?
If not, if you know the date it was created, we could probably deduce by DSM release dates, presuming you went to DSM.current before configuring the volume
What packages have you installed?

SHR2
Hitachi Ultrastar 7K4000 (Enterprise) - 4GB
All 13 (8+5) drives are in the same volume
Original volume was created on the 8 drives in the DS1813+ around July 2013 using latest DSM at the time
Volume was expanded when the DS513 was added in September 2014
Only package installed is Download Station


It is encouraging to hear that Synology engineers have seen the problem. After reading what you suspect may be the root cause, I'm fearful that I'll have to offload TB's of data to another system and rebuild this one as the only way to fix it. In the back of my mind I think it was around the time of adding the DS513 and expanding the volume that the performance degradation began.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Wed Mar 18, 2015 3:47 pm

Your initial volume creation date doesn't help my "bug if created under $DSM_version" theory. One common fact between ours is quantity of drives... yours was fine at 8 disks, but issues at 13. I have 10, as does mdonlon.

I'm also using Hitachi 7K4000 drives.

I can't encourage you to break and reformat your volume- mine was created in December under the then current build of DSM... by browsing through http://usdl.synology.com/download/DSM/5.1/ vs my notes, my build started on 12/2/2014, so the current release then would have been 5.1.5004. I think your expansion would have happened under a 5.0 build.

Not that the issue couldn't exist in 5.0.? and beyond, but that I wouldn't do a DR exercise in hopes that a full rebuild will solve this, you might just replicate the conditions that led to mine not performing properly.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Thu Mar 19, 2015 9:00 pm

How much RAM is in your 1813? It would be silly if an abundance of RAM was related to these cyclical performance issues, but why not? I'm watching the same thing on mine on a multi-Gig read right now...

20 gigs in mine.
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Thu Mar 19, 2015 9:02 pm

Mine has 4GB (up from standard 2)
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Fri Mar 20, 2015 4:02 am

<sigh>.

Drat. That shoots that theory. I've yet to find the common thread, and Synology support is not useful.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Fri Mar 27, 2015 6:01 pm

UPDATE:

There is a common thread- the Hitachi 7K4000 drives. You and I use the SATA builds, mdonlon uses the SAS versions. I now have the Synology engineers investigating a 10 drive array built from Hitachi 7K4000 drives.
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Mon Apr 13, 2015 9:23 pm

I'm wondering if parity checking is happening even though the UI does not show it.

My drive lights are always flashing, even though no clients are attached and the NAS isn't doing anything.

Referencing this post:
http://forum.synology.com/enu/viewtopic.php?f=14&t=99234
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Tue Apr 21, 2015 5:03 pm

I heard back from Synology today:

Hi,

The software developers have came to a conclusion on this issue. That is, the Hitachi with a HUS724040... suffix indeed has slow performance in a RAID consisted of more than 6 disks.

In more specifically, they performed a dd test command on each Hitachi disks to measure the overall I/O speed of RAID, and they found sometimes the command was halted in reading one or two disks and dragged down the overall speed during testing process. However, if a RAID in consisted of less than 6 disks the issue is not likely to happen.

Regards,


Synology America Corp
Support 425-296-3177


So the good news is that Synology says "yep, we can replicate that problem."
The bad news is that the sentence does not continue "and we think we know how to fix it- we'll update you shortly".
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Tue Apr 21, 2015 5:13 pm

Ugh... not what I was hoping to hear!

Is there a way to take an existing 8 drive array (or 8+5 in my case) into multiple smaller arrays while still preserving the data?
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby swy » Tue Apr 21, 2015 6:11 pm

Pretty sure it's not possible to shrink a volume from having data spanned across X disks down to only spanning across X - Y disks. It would be new to me.
shoek
Trainee
Trainee
Posts: 13
Joined: Thu Aug 08, 2013 4:33 am

Re: DS1813+DS513 - read speeds slow - xfer rate vs utilizati

Postby shoek » Tue Apr 21, 2015 6:20 pm

Didn't think so.

Would it be possible to slowly replace the Hitachi Ultrastars with another brand/model of drive. Each time the array would rebuild onto the replaced disk, as I understand it. I could even go to a larger 5TB/6TB drive.

What drive should I consider if I did such a thing? WD Red? ULtrastar were the enterprise class drive, which is why I went with them. I need reliability.

Return to “File Server Performance”

Who is online

Users browsing this forum: No registered users and 4 guests