small isilon cluster as small vmware storage

[Dan Pritts]

Hi all,

We're a small site. We have 5-node cluster of X200's. In addition to
general NAS storage usage, we currently run a small vmware cluster of
about 15 VMs with their vmdk's stored on the Isilon, accessed via NFS.
So far, this meets our modest performance needs.

The cluster hums along happily enough. I periodically run

isi statistics drive --nodes=all --top -i5 -r-1

and all the drives show up as <25% busy when we're doing a bulk copy, or
10% busy when we're running backups. In general, they're <5% busy.
time in queue is generally zero, the occasional flash of a few ms on one
drive.

We're looking to expand vmware and move some of our other workloads to
it, and add some capacity. Basically, doubling down on using vmware
with Isilon storage.

When I talked to my EMC rep, he was surprised that I wanted to buy
(more) Isilon for vmware. He said that he'd been told it doesn't
perform very well, and to steer customers to another solution. It's
easy to chalk this up to EMC wanting to sell me a VNX. It's also easy
to say maybe it doesn't perform great, but I only need adequate.
Or...it could really be a bad idea.

So my question is - given modest performance needs, and a desire to not
manage another storage platform, does this sound reasonable? Have any
of you run away screaming from such a situation? I talked to one person
who said they had had problems in the past when the cluster had had to
restripe things. The restripe job locked the vmdk files. VMs hung,
people were sad. However, with 7.1.x code this problem hasn't recurred.

I don't yet have the Isilon VAAI feature licensed although we're
probably going to buy it. Seems like it would be helpful when cloning
VM's; any other particular good points?

I noticed in the best practices document that Isilon recommends thick
provisioned vmdk's. vcenter 6 doesn't (by default) let me create thick
vmdk's on the isilon, only thin. I presume I can work around that.

Any other gotchas?

as background, we looked at vmware virtual SAN - definitely not cheap,
new, and another thing to manage. Also just considered a basic block
storage array, low-end dell stuff. This is cheap, but again, another
thing to manage. The key we keep returning to is another thing to
manage. My boss doesn't want to add anything, and it's probably the
right choice - I want to be able to take vacations.

thanks as always for everyone's feedback.

danno
--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734) 615-9529

[Eugene Lipsky]

We purchased our Isilon for it to be used primarily/exclusively as a file server. Shortly after that I was asked to start using it for VMware mainly because we were short on storage and Isilon was new and there was a lot of space. I fought this for the same reasons why you're considering to stay on Isilon. Not that I don't trust it, I'm sure it would due just fine for most of our workloads but my concern was just like yours, introducing new things to manage. We were already using multiple FC and ISCSI SANs for our VMware farm and I was basically the only one managing it all.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Kunert]

We run about 100 production VMs on our Isilon cluster (4 nodes, X410), as well as using it for bioinformatics analysis with a couple dozen compute nodes hitting it. We even have a large-ish Oracle database that hits it (Oracle has a direct NFS thing that opens its own NFS connections - lots of them, nearly 100). We’ve been running this setup for 5 years, it’s fine. VMware likes NFS storage. I would hate to go back to the days of managing multiple small LUNS. If we need more space for VMs I adjust the quota, mission accomplished.

Most of our VMs are thin provisioned. We haven’t tried VAAI yet. We use Veeam to back up the VMs, which works really well. We run VMware for our administrative computing / public facing VMs, the bioinformatics VMs run on XenServer - which is free and can handle large (120 GB RAM) VMs.

Re: locked VM files - we have had some of those issues in the past, but only with XenServer, VMware was never affected. This is far from a conclusive analysis, but It seems that the older OneFS code would often lock files unnecessarily. XenServer would encounter locked files and the VMs would freeze, while VMware just hummed along. It seems that VMware basically ignores NFS file locks, they use lock files (.lck) instead. Ignoring file locks seems like a bad idea, but as I said the VMs weren’t affected. This would happen during various operations, out of order snapshot deletions, autobalance after adding a new node, etc. We haven’t had any issues like that in the year we’ve been running 7.x on the X410s. That said, it is newer hardware (the issues were on a 72000X cluster) and we haven’t had to add a node yet.

In general I am not in favor of replacing something that works “just because”. If it’s working well and the numbers show it’s not stressed I don’t know why you would change. I suspect your EMC rep sold a lot of Clariions and just doesn’t trust the newfangled stuff.

—
Richard Kunert
IT Manager, University of WI Biotechnology Center

[Dan Pritts]

thanks for the response.  Good to hear others doing the same as I'm planning.

I'm definitely not looking to change "just because" -  I'm not stressing the cluster at all yet, so no surprise it works fine.  I don't expect I'll really be stressing the Isilon going forward (or I wouldn't be doing this), but I will definitely be putting more load on.

thanks
danno

Richard Kunert

February 4, 2016 at 10:34 AM

We run about 100 production VMs on our Isilon cluster (4 nodes, X410), as well as using it for bioinformatics analysis with a couple dozen compute nodes hitting it. We even have a large-ish Oracle database that hits it (Oracle has a direct NFS thing that opens its own NFS connections - lots of them, nearly 100). We’ve been running this setup for 5 years, it’s fine. VMware likes NFS storage. I would hate to go back to the days of managing multiple small LUNS. If we need more space for VMs I adjust the quota, mission accomplished.

Most of our VMs are thin provisioned. We haven’t tried VAAI yet. We use Veeam to back up the VMs, which works really well. We run VMware for our administrative computing / public facing VMs, the bioinformatics VMs run on XenServer - which is free and can handle large (120 GB RAM) VMs.

Re: locked VM files - we have had some of those issues in the past, but only with XenServer, VMware was never affected. This is far from a conclusive analysis, but It seems that the older OneFS code would often lock files unnecessarily. XenServer would encounter locked files and the VMs would freeze, while VMware just hummed along. It seems that VMware basically ignores NFS file locks, they use lock files (.lck) instead. Ignoring file locks seems like a bad idea, but as I said the VMs weren’t affected. This would happen during various operations, out of order snapshot deletions, autobalance after adding a new node, etc. We haven’t had any issues like that in the year we’ve been running 7.x on the X410s. That said, it is newer hardware (the issues were on a 72000X cluster) and we haven’t had to add a node yet.

In general I am not in favor of replacing something that works “just because”. If it’s working well and the numbers show it’s not stressed I don’t know why you would change. I suspect your EMC rep sold a lot of Clariions and just doesn’t trust the newfangled stuff.

—
Richard Kunert
IT Manager, University of WI Biotechnology Center

Dan Pritts

February 3, 2016 at 6:08 PM

Hi all,

We're a small site.  We have 5-node cluster of X200's.  In addition to general NAS storage usage, we currently run a small vmware cluster of about 15 VMs with their vmdk's stored on the Isilon, accessed via NFS.  So far, this meets our modest performance needs.

The cluster hums along happily enough.  I periodically run

    isi statistics drive --nodes=all --top -i5 -r-1

and all the drives show up as <25% busy when we're doing a bulk copy, or 10% busy when we're running backups.  In general, they're <5% busy.  time in queue is generally zero, the occasional flash of a few ms on one drive.

We're looking to expand vmware and move some of our other workloads to it, and add some capacity.  Basically, doubling down on using vmware with Isilon storage.

When I talked to my EMC rep, he was surprised that I wanted to buy (more) Isilon for vmware.  He said that he'd been told it doesn't perform very well, and to steer customers to another solution.   It's easy to chalk this up to EMC wanting to sell me a VNX.  It's also easy to say maybe it doesn't perform great, but I only need adequate.  Or...it could really be a bad idea.

So my question is - given modest performance needs, and a desire to not manage another storage platform, does this sound reasonable?  Have any of you run away screaming from such a situation?  I talked to one person who said they had had problems in the past when the cluster had had to restripe things.  The restripe job locked the vmdk files.  VMs hung, people were sad.  However, with 7.1.x code this problem hasn't recurred.

I don't yet have the Isilon VAAI feature licensed although we're probably going to buy it.  Seems like it would be helpful when cloning VM's; any other particular good points?

I noticed in the best practices document that Isilon recommends thick provisioned vmdk's.  vcenter 6 doesn't (by default) let me create thick vmdk's on the isilon, only thin.  I presume I can work around that.

Any other gotchas?

as background, we looked at vmware virtual SAN - definitely not cheap, new, and another thing to manage.  Also just considered a basic block storage array, low-end dell stuff. This is cheap, but again, another thing to manage.  The key we keep returning to is another thing to manage.  My boss doesn't want to add anything, and it's probably the right choice - I want to be able to take vacations.

thanks as always for everyone's feedback.

danno

[Dan Pritts]

I came across this e-mail today and thought I'd report back.  We ended up moving forward with the Isilon as vmware storage. We now have 6x X200/X210's with 12x2TB disks (no SSD).  3 of the X nodes and the vmware servers are connected at 10G, the rest of the cluster at 1G.  VMware NFS mounts go to the 10G nodes, and NAS workloads go to the 1G nodes. 

We support about 50 (almost all Linux) VMs with a moderate workload on this system, as well as our ongoing NAS-based  workloads (again, moderate performance requirements).  We have kept our biggest I/O consumers on physical hardware, just a few machines (eg, production db & search engine). 

We did not license VAAI; it was not available for vsphere 6 when we bought, and we haven't missed it.

Mostly, we do not use thick-provisioned VMDK's (which is listed as best practice).  Two reasons  for this.  (1) The vcenter GUI doesn't support creating thick VMDK's on NFS storage, so you have to do it by hand if you want it.  (2) The obvious better disk space utilization.

Performance is adequate.  All drives are still generally less than 10% busy.  The lack of SSD means that NFS performance for metadata wasn't great before.  It hasn't improved, but isn't noticeably worse.  In practice this is an annoyance rather than a real problem. 

thanks
danno

Dan Pritts

February 3, 2016 at 6:08 PM