SSD Cache - Slow uncached read performance

IF your questions pertains to using the file server performance, transfer issues, jumbo frames tuning, post it here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form and check this FAQ:
https://myds.synology.com/support/suppo ... p?lang=enu
https://www.synology.com/en-global/know ... se/faq/586
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
mdonlon
I'm New!
I'm New!
Posts: 4
Joined: Fri Mar 13, 2015 10:14 pm

SSD Cache - Slow uncached read performance

Postby mdonlon » Fri Mar 13, 2015 10:43 pm

I have a RS10613xs+ with 32GB RAM, 10 HDDs configured as a 32TB RAID1 volume and 2 700GB SSDs configured as a read cache.

We are using this device to distribute large files to a team, so sequential read performance is the most important thing we are tuning for. With the SSD cache disabled we max out at about 250MB/s from the RAID volume. With the SSD cache enabled and the data we are reading in the cache, we max out at 800MB/s. This is enough to saturate our network link and is the kind of speeds we were hoping to get.

However, reads from the RAID volume with the SSD cache enabled but with the data not yet populated in the cache are exceptionally slow, about 50MB/s. My expectation is that these kinds of reads would be as fast as reads with the SSD cache disabled, maybe with some small overhead for the caching. Absolute worst case would be the time it takes to read from the HDD plus the time it takes to write to the SSD, but it is significantly slower than that.

We have had to take measures to prime the cache with any new data that is uploaded (since that is most likely going to be downloaded), which isn't ideal. Also whenever some does hit some data that isn't in the cache the throughput isn't high enough to even saturate a single giagabit network link.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: SSD Cache - Slow uncached read performance

Postby swy » Sat Mar 14, 2015 1:29 pm

YOU ARE NOT ALONE.

This sounds uncomfortably like my situation- a 3614RPXS with 10*4TB in R6 with 2*250SSD cache, not yet in deployment due to network copy performance issues. I wonder if you see the same thing as I do: Fire up the resource monitor in DSM, and keep an eye on the volume utilization. Start a big file copy from an SMB client. Do you see the volume utilization sometimes to often shoot up and bounce around 100%? I do, and when it does, it tends to copy like a USB2 drive, about 30-40MB/sec. Disk utilization will probably be a reasonable 10%.

I've had a ticket open with synology for over a month now, had their engineers remoted in on Thursday night doing file copies, monitoring iostat, ect, and it's still not solved.

(My $0.02: if US based level 1 techs have the answer, Synology support is OK. If it's past their abilities and gets the engineers involved... don't be expecting prompt data exchanges. It's PAINFULLY slow when the Taiwan group gets involved.)
mdonlon
I'm New!
I'm New!
Posts: 4
Joined: Fri Mar 13, 2015 10:14 pm

Re: SSD Cache - Slow uncached read performance

Postby mdonlon » Mon Mar 16, 2015 7:26 pm

Yup, we see the same thing, volume utilization pegged at 100% while disk utilization is much lower. Also, usually the quantity of volume IOPS is 10x the disk IOPS. Good to know I'm not alone!
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: SSD Cache - Slow uncached read performance

Postby swy » Tue Mar 17, 2015 1:19 pm

Please- hit up Synology support with a ticket, and point out that you're replicating the issues seen in ticket #435188.

I'm not pleased to know that someone else has the same troubles, but this does tell me that it's improbable that I made a configuration error.

Another question- ssh into the box, and run a mdadm --detail /dev/md1 . Mine comes up as a clean, degraded RAID1, 12 devices, 10 present. I think this can't be intended behavior, but Synology support says it's OK.

Do you see the same?
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: SSD Cache - Slow uncached read performance

Postby swy » Tue Mar 17, 2015 5:10 pm

Also... a 32TB Raid1 across 10 drives? That doesn't make sense. What exactly is your raid configuration?

(I'm gonna guess you meant RAID10- if you had 6TB drives, the math works out)
mdonlon
I'm New!
I'm New!
Posts: 4
Joined: Fri Mar 13, 2015 10:14 pm

Re: SSD Cache - Slow uncached read performance

Postby mdonlon » Tue Mar 17, 2015 8:54 pm

raid5, not raid1. Sorry.

Code: Select all

/dev/md1:
        Version : 0.90
  Creation Time : Fri Dec 31 16:00:09 1999
     Raid Level : raid1
     Array Size : 2097088 (2048.28 MiB 2147.42 MB)
  Used Dev Size : 2097088 (2048.28 MiB 2147.42 MB)
   Raid Devices : 10
  Total Devices : 10
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Tue Mar 17 12:51:43 2015
          State : clean
 Active Devices : 10
Working Devices : 10
 Failed Devices : 0
  Spare Devices : 0

           UUID : 2742b012:65ca95c5:c69de3af:22ddbed7
         Events : 0.19

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sas1p2
       1       8      146        1      active sync   /dev/sas10p2
       2       8      162        2      active sync   /dev/sas11p2
       3       8      178        3      active sync   /dev/sas12p2
       4       8       18        4      active sync   /dev/sas2p2
       5       8       34        5      active sync   /dev/sas3p2
       6       8       66        6      active sync   /dev/sas5p2
       7       8       82        7      active sync   /dev/sas6p2
       8       8       98        8      active sync   /dev/sas7p2
       9       8      130        9      active sync   /dev/sas9p2

/dev/md2:
        Version : 1.2
  Creation Time : Thu May  8 09:58:28 2014
     Raid Level : raid5
     Array Size : 35120666304 (33493.68 GiB 35963.56 GB)
  Used Dev Size : 3902296256 (3721.52 GiB 3995.95 GB)
   Raid Devices : 10
  Total Devices : 10
    Persistence : Superblock is persistent

    Update Time : Tue Mar 17 12:53:02 2015
          State : clean
 Active Devices : 10
Working Devices : 10
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 9b44aa83:e41d43c7:bc77b542:0baf9620
         Events : 11

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sas1p3
       1       8       19        1      active sync   /dev/sas2p3
       2       8       35        2      active sync   /dev/sas3p3
       3       8       67        3      active sync   /dev/sas5p3
       4       8       83        4      active sync   /dev/sas6p3
       5       8       99        5      active sync   /dev/sas7p3
       6       8      131        6      active sync   /dev/sas9p3
       7       8      147        7      active sync   /dev/sas10p3
       8       8      163        8      active sync   /dev/sas11p3
       9       8      179        9      active sync   /dev/sas12p3

swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: SSD Cache - Slow uncached read performance

Postby swy » Tue Mar 17, 2015 10:03 pm

Well, at least I know that my degraded RAID1 state is not the cause and effect of my speed issues.
ndamico
Trainee
Trainee
Posts: 11
Joined: Thu Jul 26, 2012 2:18 am

Re: SSD Cache - Slow uncached read performance

Postby ndamico » Tue May 19, 2015 3:08 am

any updates to this? i'm in the same boat. my rs3615xs+ runs better without the ssd cache.
swy
Rookie
Rookie
Posts: 39
Joined: Fri Aug 10, 2012 9:51 pm

Re: SSD Cache - Slow uncached read performance

Postby swy » Tue May 19, 2015 9:38 pm

Are you using HGST 7k4000 drives? If so, this is now a known issue. Resolution: don't use those drives. Synology and HGST both know about it, but I can't say where they are re a better option.

https://swytechnotes.wordpress.com/2015/04/29/why-you-cant-always-trust-the-hardware-compatibility-list-hcl/

Return to “File Server Performance”

Who is online

Users browsing this forum: No registered users and 3 guests