Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.

Dismiss

Random Event 1135

467 views

Skip to first unread message

D1Artagnan

unread,

Apr 16, 2009, 8:09:01 PM4/16/09

I'm looking for an advice how to troubleshoot Event 1135.

Scenario:
2x nodes Windows 2008 Sp1 x64 Failover Cluster (Node and File Share Majority)
Exchange 2007 Sp1 CCR
Cluster nodes and witness are on a VMware 3.5, connected to FC SAN
Additional software: McAfee Group Shield 7 Sp1 for Exchange, SCOM2007
client, SMS 2003 Advanced client and ARC Server Backup Agent for Exchange ver
12.1

Problem description:
Event 1135: Cluster node 'STLAKLMB01' was removed from the active failover
cluster membership....

This event is logged on both Active and Passive cluster nodes. In addition
the Passive node reports
Event 1069: Cluster resource 'File Share Witness (\\STLAKLXCH03\Quorum)' in
clustered service or application 'Cluster Group' failed
and
Event 1564: File share witness resource 'File Share Witness
(\\STLAKLXCH03\Quorum)' failed to arbitrate for the file share
'\\STLAKLXCH03\Quorum'. Please ensure that file share '\\STLAKLXCH03\Quorum'
exists and is accessible by the cluster.

This happened 2 times in the last one week (11:30 PM and 1:06 AM). Downtime
in both cases was about 2 minutes after which the Passive node reconnected
and the cluster recovered. The impact was that 4 out of the 6 (2 out of 6 in
the first case) Exchange 2007 storage groups failed to recover the
replication after the failure and my only option was to re-seed them in the
morning.

The stange thing here is that there aren't any events that may suggest
network failure. Furthermore the failed (passive) node keeps reporting that
both networks Public and Heartbeat are up. No other servers or infrastructure
components have registered any network otages at the time of the events.

Q1: How do I troubleshoot this failure - are there any additional logs or
tools I could use to capture more information?

Q2: How to configure the Failover Cluster to delay shutting down the
cluster. All current settings are default

Your help is much appreciated

John Toner [MVP]

unread,

Apr 17, 2009, 11:57:09 AM4/17/09

1) If you go to a command line and issue a "cluster log /g" command, this
will generate a cluster.log file in the c:\windows\cluster\reports folder
that might provide additional information. Also, you can check the Failover
Cluster Operational logs for messages regarding network
messages...operational logs are under Diagnostics > Applications and Service
Logs > Microsoft > Windows > FailoverClustering

2) You cannot delay the shutdown of the cluster, but you can perform some
tweaks that might help delay the amount of time it takes to get to the point
where it is determined that the node is not available by adjusting the
heartbeat settings.

The default heartbeat value is that a heartbeat signal is sent once every
second (1000 milliseconds) and when a node misses a series of 5 heartbeats,
another node will initiate failover. You can adjust these values in Windows
2008 clusters by using the following commands:

cluster /prop SameSubnetDelay=<value>
cluster /prop SameSubnetThreshold=<value>

If your cluster nodes are on separate subnets, you would adjust the
following values instead:

cluster /prop CrossSubnetDelay=<value>
cluster /prop CrossSubnetThreshold=<value>

You can type cluster /prop to see your current settings.

Regards,
John

Visit my blog: http://msmvps.com/blogs/jtoner

"D1Artagnan" <D1Art...@discussions.microsoft.com> wrote in message
news:BA7BB4E4-F5B5-4FDF...@microsoft.com...

D1Artagnan

unread,

Apr 22, 2009, 6:03:02 PM4/22/09

Hi John,

Thank you for your help

Cluster.log on both nodes were not very useful. The log on the active node
has not logged events between 1.04 and 22.04. The log on the passive node has
some events logged on 4th and 8th April. Both logs have no events logged for
the time of the failures.

Failover Cluster Operational Log also appears to have missed some periods of
time although not that large - no events were logged between 1:08 AM on 17.04
and 3:29 PM on 20.04. The first time stamp coincides with the time when the
cluster recovered from a failure, the second timestamp is when the backup
started

Windows System Event log seems to be the most useful. I'm not sure if the
cluster service has crashed and that caused the disconnection to the active
node, or the node has lost connectivity to the quorum and that caused the
cluster service to terminate. It also looks like there is some pattern in the
time of the fault: Occurrences in the last 2 weeks are

23.04 - From 1:05:18 AM to 1:07:51 AM
17.04 - From 1:06:18 AM to 1:07:55 AM
14.04 - From 11:30:37 PM to 11:33:09 PM

Regards,
Ilian

Windows System Log
-----------------------------------------------------------------------------------
Level Date and Time Source Event ID Task Category
Information 23/04/2009 1:07:54
a.m. Microsoft-Windows-Time-Service 37 None The time provider NtpClient is
currently receiving valid time data from nzsakldc01.nzsakl.bhp.com.au
(ntp.d|0.0.0.0:123->152.153.40.60:123).
Information 23/04/2009 1:07:51 a.m. Tcpip 4201 None The system detected that
network adapter Local Area Connection* 12 was connected to the network, and
has initiated normal operation.
Information 23/04/2009 1:07:51 a.m. Tcpip 4201 None The system detected that
network adapter Local Area Connection* 12 was connected to the network, and
has initiated normal operation.
Information 23/04/2009 1:07:52
a.m. Microsoft-Windows-Time-Service 37 None The time provider NtpClient is
currently receiving valid time data from nzsakldc01.nzsakl.bhp.com.au
(ntp.d|0.0.0.0:123->152.153.40.60:123).
Information 23/04/2009 1:07:51 a.m. Service Control Manager 7036 None The
Cluster Service service entered the running state.
Warning 23/04/2009 1:07:01
a.m. Microsoft-Windows-Time-Service 131 None NtpClient was unable to set a
domain peer to use as a time source because of DNS resolution error on
'nzsakldc01.nzsakl.bhp.com.au'. NtpClient will try again in 15 minutes and
double the reattempt interval thereafter. The error was: No such host is
known. (0x80072AF9).
Critical 23/04/2009 1:06:55
a.m. Microsoft-Windows-FailoverClustering 1564 File Share Witness
Resource File share witness resource '' failed to arbitrate for the file

share '\\STLAKLXCH03\Quorum'. Please ensure that file share
'\\STLAKLXCH03\Quorum' exists and is accessible by the cluster.

Error 23/04/2009 1:06:56 a.m. Service Control Manager 7031 None The Cluster
Service service terminated unexpectedly. It has done this 1 time(s). The
following corrective action will be taken in 60000 milliseconds: Restart the
service.
Error 23/04/2009 1:06:56 a.m. Service Control Manager 7024 None The Cluster
Service service terminated with service-specific error 5925 (0x1725).
Information 23/04/2009 1:06:55 a.m. Service Control Manager 7036 None The
Cluster Service service entered the stopped state.
Critical 23/04/2009 1:06:49
a.m. Microsoft-Windows-FailoverClustering 1177 None "The Cluster service is
shutting down because quorum was lost. This could be due to the loss of
network connectivity between some or all nodes in the cluster, or a failover
of the witness disk.
Run the Validate a Configuration wizard to check your network configuration.
If the condition persists, check for hardware or software errors related to
the network adapter. Also check for failures in any other network components
to which the node is connected such as hubs, switches, or bridges."
Error 23/04/2009 1:06:48
a.m. Microsoft-Windows-FailoverClustering 1069 Resource Control
Manager Cluster resource 'File Share Witness (\\STLAKLXCH03\Quorum)' in
clustered service or application 'Cluster Group' failed.
Critical 23/04/2009 1:06:47
a.m. Microsoft-Windows-FailoverClustering 1564 File Share Witness
Resource File share witness resource 'File Share Witness