Use cases of Quadstor in the community

[Andreas Schlager]

Hi all,

it would be interesting to know how quadstor is used in the community.
Does it run in a production environment, serving storage for databases and f.e. ERP systems?
Any business critical applications?
Or in Lab/test-bed environments / development?
It also of interest how your quadstor setup is - f.e. multi-node cluster or sync-mirror, iSCSI or FC attached LUN's (in- and out)

Thanks for your feedback!
-Andreas.

[Dmitry Polezhaev]

Hi, Andreas!

As for me, I'm using QUADStor as FreeBSD-based virtual appliance in the lab, to prepare the solution for possible customers. 

Thus: ERP - no, business critical - no. 

But: iSCSI - yes, FC out - yes (up to quad-FC-path), sync mirror HA - yes, and a lot of stress-testing in various configurations (even with Oracle RAC over the QUADStor HA :) ).

Cheers,

Dmitry

[Dmitry Polezhaev]

Another example: just tried QUADStor+Linux tandem with 2 software RAIDs, plus old HP MSA500 with 10x 73GB HDDs, directly attached via HBA. 1 week uptime as VMWare 6 FC storage - everything is fine. Hope, this image will describe the system configuration enough.

[Dmitry Polezhaev]

After about 3 weeks of moderate load laboratory system crushed. The incident timeline was:

- Investigation of the crash point - 1 day. 3x 147 HDDs and a 5-drive U320 SCSI cage are not with us now.

- Investigation of the software damage - 1 day. The boot mirror with one HDD in the died cage has destroyed GRUB data and OS partition map configuration. It was simpler to completely reinstall the OS, than to fix it.

- Bringing up the remaining software array with 73 GB drives - 15 minutes. MSA500 data had no corruption at OS level.

- QUADStor installation, database repair (as described in the knowledgebase), reboot, FC access setting - 15 minutes. All remaining VMWare datastores were recovered without data loss.

Verdict: the software layer stability appeared much over hardware. QUADStor repair procedure is really simple. Thanks to authors!

[Paul Reid]

We've been using Quadstor on top of CentOS 7 to create hyper-converged VMware environments.

We've been testing a dual host setup using VMware high availability and DRS on top of a hyper-converged QuadStor share-nothing cluster that is mirroring 1 2TB SSD and 1 2TB spindle per host over a 40Gbps Ethernet-over-Infiniband link. QuadStor fails over flawlessly and performance is incredible. Each host has both QuadStor instances entered into its iSCSI configuration, with the IP of the local QuadStor VM as its preferred target.

We initially had some freezing problems due to vSphere 6 having some new requirements for iSCSI targets, but a quickly released QuadStor update completely solved the problem and everything has been running perfectly since.

The dual host setup will go into production in the next few weeks. We have several virtual machines running on the cluster (a 2 node pfSense cluster, a pair of domain controllers, an 2 node Exchange 2013 cluster, a backup VM that runs Altaro VMBackup, a web server, a mail server, an FTP server, and several test VMs). The cluster sees a fair bit of disk traffic during backups, and never complains.

We're super impressed with QuadStor, and will likely start rolling out some similar clusters to our test bed in the near future. At that time, we'll be including QuadStor support agreements with the clusters, so that the QuadStor project is properly supported, the same way we do with other software we roll out for customers (pfSense, for example).

[Andreas Schlager]

Hi Paul,

I'm interested to see a follow-up with your experiences when you're in production for a month or two if possible.

Many thanks!

KR
-Andreas.

[Paul Reid]

Hi, Andreas,

The machine has been humming away on the bench for several weeks now. Everything was running great, until I spotted a new version of QuadStor (3.2.11) and started to upgrade it. Everything went south at that point. One of my VMFS volumes was completely trashed by the upgrade. The other volume looks ok, so far. We'll see if the synchronization it's doing right now is going to trash that one, too. If it does, I'll have to start over from scratch, since that will be the VMs drive and the backup drives both gone.

If the backup drive lives, I'll be able to build up a new backup VM, attach the backup VMDK, and restore all of my VMs - but at this point, I don't know if that will be the case. It appears to be completely re-synchronizing the entire 2TB drive, at the moment, for whatever reason, and the ESX hosts are both virtually unresponsive, so I can't check on much. All I can do is watch the drive lights and eyeball the QuadStor web interface - which tells me it's re-synchronizing, but there's no progress indicator, so I have no idea how long it's going to take.

It would seem that if you get a system that's running solidly, leave it that way. I was stupid enough to fix something that wasn't broken, and now it's a mess. I know better - I just figured it was a smart thing to upgrade, since the latest version fixes a kernel panic on large vDisks. I never saw the kernel panic problem, however - perhaps 2TB wasn't enough to trigger the problem.

[Dmitry Polezhaev]

Paul, the next possible evolution checkpoint for Your mirrored cluster might be VMWare SRM. However, SRM requires and agent presence, which is not available for QuadStor.

[Paul Reid]

Hi, Dmitry,

SRM would be something over and above just mirroring some storage - it's a totally different animal. And it's expensive. And it won't support a VM like QuadStor, nor would it like a VM that has an HBA passed through to it, like the QuadStor VM does.

I have a backup solution in place, called Altaro VMBackup, but my backups were going to a QuadStor mirrored vdisk - thus the concern that my backups were going to be corrupted by the resync. The backup solution is still in place, but I have instructed Altaro to also send a copy of the backups to an "offsite" target, as well - which is actually just a NAS (that of course doesn't use QuadStor) connected to the server, just in case.

As it turned out, the QuadStor mirror where my backups were going didn't get trashed by the resync, and I was able to just wipe the corrupted VMs on my other QuadStor volume and restore them from the backups, so I was able to recover without having to rebuild my VMs from scratch. Now, with the backup copy on the NAS, if the QuadStor instance goes completely south and I lose the vdisk with my backups on it as well as my vdisk with the VMs, I can just blow the whole thing away and restore from the NAS. In fact, if I am in a rush, I can spin up critical VMs directly from the NAS while I get the QuadStor vdisks set up again for a proper restore.

[Dmitry Polezhaev]

Greetings, Paul!

On Tuesday, April 11, 2017 at 7:20:14 PM UTC+3, Paul Reid wrote:

And it won't support a VM like QuadStor, nor would it like a VM that has an HBA passed through to it, like the QuadStor VM does.

I'm not agree, because the successfull example already exists. Nutanix has the storage, running in a VM, and it has the SRM agent. The HBA passthrough is not mandatory, commonly saying, and QuadStor on FreeBSD perfectly runs in VM (I played a lot with this tandem). Thus, technologically, QuadStor is near to be a building block for hyperconvergent system, but now it may miss some particular features for particular applications (like SRM agent).

On Tuesday, April 11, 2017 at 7:20:14 PM UTC+3, Paul Reid wrote: 

I have a backup solution in place, called Altaro VMBackup.... In fact, if I am in a rush, I can spin up critical VMs directly from the NAS while I get the QuadStor vdisks set up again for a proper restore.

In my sandbox the Veem is much popular, but the multi-layer availability solution (storage mirroring + offsite backup) makes the great respect from my side. Well done!