NFS, BTRFS, and VMWare anyone?

DSM6.0 beta, based on a more efficient system architecture by rolling 64-bit architecture on compatible models and modularizing a greater number of applications, DSM 6.0 delivers significant advancements in productivity, flexibility, and performance.
Key new features in DSM 6.0 include: Btrfs File System, 64-Bit Architecture, Service Modularization, Snapshot & Replication, 802.1X support and much more.
Forum rules
Synology Community is the new platform for the enthusiasts' interaction, and it will soon be available to replace the Forum.
goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Wed Nov 11, 2015 3:50 pm

I submitted a bug report about this a few weeks ago, but I don't think it is registering with Synology support that it is a bug, so I wanted to confirm with the user community that it is a common issue or maybe something wrong with my NAS.

I am running a 3612xs with 10x HDD's in SHR-2. 6x 6TB disks and 4x4TB. I created a single BTRFS volume and created an NFSv3 share for VM's. The volume mounts fine to my hosts. I can browse the datastore on both hosts no problem. I can sVmotion a single VM at a time into the share with varying degrees of success. If I essentially generate any kind of sustained load, I will get errors. If multiple Vm's are being migrated in, they will fail with the error "cannot find file". If multiple VM's are already in the datastore, the hypervisor says it lost the lock file and either reboot (if HA is enabled) or the VM stays powered off. I have removed any network considerations (installed a dedicated switch, dedicated NICs on the host and the NAS, 1500 MTU, no VLAN, no link aggregation/bonding) and still get the errors.

I created a file level iSCSI lun and target inside the same volume and connected it to my hosts and it works fine. It's much slower than NFS (which is why I want NFS, that and NFS is much easier to setup and manage) but it works and I get no errors.

Is anyone using NFS, BTRFS, and VMWare? Did you have any issues? Or is everyone using iSCSI?

dustint
Experienced
Experienced
Posts: 146
Joined: Sun Feb 01, 2015 2:34 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by dustint » Thu Nov 12, 2015 3:00 pm

It seems to me that you would very much want to be using link aggregation, both on the Synology and on your ESXi hosts. Is 1500 MTU still recommended for VMware? Just thinking that if you're running a dedicated storage network for VM storage you'd want 9k packets.

Finally, having mixed drive sizes in your SHR2 pool is definitely not recommended for max performance. I'd recommend you destroy the existing volume and recreate it with only similiar disks.

Oh, one more thing... SSD Cache is your friend. It's entirely possible you're just pushing thoses disks past what they are physically capable of and the SSD Cache will add a lot of IOP potential that will help.

Hope that helps. --- Dustin
1815+; pro-blog.com

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Fri Nov 13, 2015 12:47 am

Sorry... I did have a LAG, I did have jumbo frames, I did have LACP setup for the Synology and the 2 hosts, I did have SSD cache in place. However, when I started experiencing these super strange NFS almost drops, I stripped everything down to the most basic setup to remove any potential problems.

pelliero
I'm New!
I'm New!
Posts: 1
Joined: Sat Nov 14, 2015 12:04 am

Re: NFS, BTRFS, and VMWare anyone?

Unread post by pelliero » Sat Nov 14, 2015 12:20 am

I have exactly the same problem (DS415+ and ESXI 5.5 with NFS database). I have open a ticket but with no result....

They tell me "we guess you should lost some configuration file or forget backup for your VM ESXI to use" and they invite me to contact vmware support.

I didn't follow up. I create an ext4 volume with exactly the sames files and everything is fine. I have tried also iscsi but it was very very slow with btrfs. I am back to ext4 and everything is ok...

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Mon Nov 16, 2015 1:19 pm

YES!! ISCSI is also very problematic as well with the new setup. VERY slow and only one host can connect now. I would really hate to go back to EXT4 because I really would like to have snapshots, but my synology is completely unusable with BTRFS.

evilhamsterman
I'm New!
I'm New!
Posts: 2
Joined: Tue Nov 17, 2015 10:53 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by evilhamsterman » Tue Nov 17, 2015 10:57 pm

I don't have anything to test on, but I do know from experience with BTRFS on Linux servers that is bad for heavy random IO that you would get with VMs or DBs. The copy on write cause heavy fragmentation and low random IOPs and so it is not recommended for those use cases.

You can disable the COW function on a file or directory basis which helps for light loads but EXT4 still performs better.

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Wed Nov 18, 2015 4:38 pm

Yeah, I definitely see the performance issues. I disabled COW when I setup the share originally. It's strange. When I am browsing the datastore, about half the time I get nothing back. No folders, no nothing. I close the window and open it back up and I get results. Which is exactly what VMWare is complaining about with "File not found" or "lock file is missing" when moving VM's or running them on this datastore.

kkuszek
Trainee
Trainee
Posts: 17
Joined: Fri Feb 13, 2015 9:35 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by kkuszek » Wed Nov 18, 2015 9:24 pm

COW is the reason to ever go to btrfs, the big gain. turning off cow makes it not much better than ext4.

btrfs is also not known for it's vm hosting performance.

finally lacp is common but not the best way to get performance for connections to vmware hosts. mpio is the smarter and more redundant way to go. I am not running the beta so I hope it's improved but synology has some serious network performance issues for hosting VM's for me. nfs 4.1 has multipathing and is fantastic, older nfs does not and has a performance hit over iscsi.

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Wed Nov 18, 2015 10:55 pm

I never experienced any NFS performance issues at all. I was able to realize more than 220MB/s thruput with 2 ESX hosts on the previous DSM and NFSv3 datastores. I much prefer NFS to iSCSI, both at home and at work. The Synology at it's core is a file serving device, a NAS. Having it emulate block storage may be ok but file would be easier for it to handle.

Totally agree about COW being why you want BTRFS and that it is not a performance based FS. I had thought to use SSD's to mitigate that, but either 120GB isn't enough or that caching software doesn't really work well to overcome the limitations with BTRFS. I am assuming they are using bcache, which is FS agnostic and thus isn't tuned to deal with the "features" of BTRFS. I had the idea that maybe instead of having 2 volumes/disk groups, 1 SSD and one BTRFS, to have the SSD's are part of the BTRFS and have the COW features happen in cache and then get flushed to SATA as part of the checksum generation process. This would require some SERIOUSLY large SSD's, like 512 or larger but it would work and mitigate much of the performance lag.

heinowalther
I'm New!
I'm New!
Posts: 4
Joined: Sun Mar 31, 2013 6:51 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by heinowalther » Tue Dec 08, 2015 10:46 am

Hi there

I just wanted to say that I have been having the same problems described.
By system also has SSD caching BRTFS and NFS up against VMWARE 6.0.
I also experience CIFS shares that loose their connections, I use OSX, and you are prompted every time the shares are disconnected, so you realise it as it happens... I have seen it when I load the underlying filsystem....
So I guess that BRTFS is at fault here. I think I will have to nuke my BRTFS volumes and go back to ext4 again... sad :-(

/Heino

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Mon Dec 21, 2015 5:30 pm

Ok, so I have had a an open ticket on this with support. They seemed to have resolved the issue for me. I was able to bond the links in the NAS and all my hosts and hammer it with moves. I was able to get over 200MB/s with no time outs or locks being lost. No indication that this will be resolved in the next beta but I think it will be.

heinowalther
I'm New!
I'm New!
Posts: 4
Joined: Sun Mar 31, 2013 6:51 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by heinowalther » Wed Dec 23, 2015 6:39 pm

Could you please specify some details about what you did to fix it ?

Regards,
Heino

goose7791
Rookie
Rookie
Posts: 32
Joined: Wed Dec 24, 2014 8:58 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by goose7791 » Wed Dec 30, 2015 10:45 pm

I had to contact Synology support. They provided an update to the exports.ko file which I put in place on my NAS and then rebooted.

stiansneenjohansen
I'm New!
I'm New!
Posts: 1
Joined: Wed Jun 15, 2016 9:31 pm

Re: NFS, BTRFS, and VMWare anyone?

Unread post by stiansneenjohansen » Wed Jun 15, 2016 9:33 pm

I also have this problem. Is there a fix for it?

Post Reply

Return to “DiskStation Manager 6.0 Beta”