Virtual Cluster down, no quorum

144 views

Skip to first unread message

Thomas Bartz

unread,

Jul 27, 2022, 3:33:48 PM7/27/22

to Isilon Technical User Group

Hi everybody,

first of all thanks for adding me to this group.
I have an issue with my virtual Isilon cluster I've build up for test purposes...

It is/was a 2 node cluster with(unfortunatly) no quorum configured.

After a power outage, the boot disk of one of those virtual Isilon Node got scrambled... and it didn't boot up again.

Because of that, one node is now missing and the complete cluster is down

here is what the cluster/surviving node status is:

"isi status"

Warning: This node is not connected to the cluster.

Health Throughput (bps) HDD Storage SSD Storage

---+--------------------+-----+-----+-----+-----+-----------------+-----------------

1|192.168.4.175 | OK | 0| 0| 0 | 132G/ n/a( n/a) | 0/ n/a( n/a)

2|192.168.4.176 |D--- | n/a| n/a| n/a | n/a/ n/a( n/a) | n/a/ n/a( n/a)

---+---------------+-----+-----+-----+-----+-----------------+-----------------

Cluster Totals: | n/a| n/a| n/a | n/a/ n/a( n/a) | n/a/ n/a( n/a)

"isi devices node smartfail --node-lnn 2" - and

"isi devices node smartfail --node-lnn 2 --force" are not working

-> Unable to commit node smartfail update:Error committing config: Could not create a quorum of connections.

"isi devices node stopfail"

You are about to stopfail node lnn 1. Are you sure? (yes/[no]): yes

Cannot stopfail lnn 1 because it is not currently smartfailing.

"sysctl efs.gmp.group"

efs.gmp.group: <37a4e0> (2) :{ 1:0-6, down: 2 }

So my question is, does somebody know how I could either bring that data/OneFS partition up to get the data out of that box

OR how I could add a quorum to that cluster

OR how I could add a node to that cluster

OR add those disks to a newly created cluster to move the data off that Isilon... (but from what I've seen, on the initialization Isilon is writing some GUID to the disks and later looking for that GUID

I already tried to manually add a node to that cluster, but this wasn't successfull because the cluster is offline...