System.ObjectDisposedException in C# Driver

[Rob Emery]

Hiya,

We think we've hit upon an issue in the Datastax C# cassandra driver 3.8.0, what we think occurred is as follows:

A query was in-flight and due to our maximum-request duration a ThreadAbortException which broke some part of the connection state as a large proportion of queries after the ThreadAbortException kept throwing this exception:

```
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
  at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
  at System.Security.Cryptography.Utils.HashData(SafeHashHandle hHash, Byte[] data, Int32 cbData, Int32 ibStart, Int32 cbSize)
  at System.Security.Cryptography.MD5CryptoServiceProvider.HashCore(Byte[] rgb, Int32 ibStart, Int32 cbSize)
  at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
  at Cassandra.RPToken.RPTokenFactory.Hash(Byte[] partitionKey)
  at Cassandra.Cluster.GetReplicas(String keyspace, Byte[] partitionKey)
  at Cassandra.TokenAwarePolicy.<NewQueryPlan>d__6.MoveNext()
  at Cassandra.Requests.RequestHandler.GetNextHost()
  at Cassandra.Requests.RequestHandler.<GetNextConnection>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout)
  at Cassandra.Session.Execute(IStatement statement)
  ...
  blahh blahh our code here
```

Looking at RPToken.cs we think it basically has to be to do with _md5 possibly being interrupted between the if on line 71 (https://github.com/datastax/csharp-driver/blob/master/src/Cassandra/RPToken.cs#L71).

I've had a look through the changelogs etc and I can't see this issue having been reported previously. We triaged the issue by killing the process and restarting it, that fixed it right up.

Is this an issue you've seen before? We've tried to get a reproducible test case, but failed thus far so we think it must be *very* rare.

Many Thanks,
Rob

[João Reis]

Hi,

I don't think I've seen that issue before but looking at RPTokenFactory, it seems like what you're describing is probably the issue. It's relying on ThreadStatic for thread safety in regards to the MD5 instance and ThreadStatic might not be the most reliable solution in an async/await context. I'll try to get a reproducible test case as well.

It might be a rare issue because most clusters use the Murmur Partitioner (I'm assuming) and also because it might only happen when a ThreadAbortException occurs. I'm kind of confused as to why this exception would be thrown... I don't think the driver would ever throw this exception so it's more likely that it is the client application itself throwing it. Is the application using ASP.NET?

Best regards,

João Reis

[João Reis]

I'm not able to reproduce this unfortunately. In order for this to happen, it would mean that two threads used the same _md5 field (MD5 is not thread safe) but that should not be possible with ThreadStatic and there's no async/await operations involved with the _md5 field. If you are able to come up with a reproducible test case you can open a JIRA issue (https://datastax-oss.atlassian.net/browse/CSHARP) or just reply here in this thread.

Best regards,

João Reis

[Rob Emery]

Indeed, that is the situation that we found as well.

I'm pretty sure that the TAE is totally unrelated to Cassandra, this is an ASP.NET application and it's being thrown when the request exceeds the ScriptTimeout.  We had an issue with the amount of load we were under at the point when things went south so we think the queries just started getting cut off; we do have a memory dump of the process so we can inspect the references as the issue persisted until we recycled the application, so effectively the actual reference in _md5 was broken.

I don't think the code itself is wrapped in Tasks or similiar, although it might be; I'll check the codebase to see if we've async'd any part of it outside of the driver itself.

[Joao Reis]

_md5.ComputeHash() ( https://referencesource.microsoft.com/#mscorlib/system/security/cryptography/hashalgorithm.cs,a71a3cba2c656040 ) calls Initialize() which disposes the internal handle and creates a new one ( https://referencesource.microsoft.com/#mscorlib/system/security/cryptography/md5cryptoserviceprovider.cs,80152d7eff9b1f70 ). If a ThreadAbortException occurs after the Dispose() but before the new handle is created, then the MD5 instance will become unusable because of its invalid state.

There's a lot of driver code that shares state between requests and threads for performance reasons so it's very likely that this exception could cause a failure in another part of the driver the next time it happens even if we attempt to make this specific class abort-safe. Bottom line is that ThreadAbortException is essentially evil and should be avoided at all costs so my advice would be to increase the ScriptTimeout to avoid this exception if possible.

Best regards,

João Reis |  Senior Software Engineer - Drivers

joao...@datastax.com  |  datastax.com

A big Get Focused ‘thank you’

Why you should partner with an Agile company

Phone: 0800 021 0888   Email: cont...@codeweavers.net
Codeweavers Ltd | Barn 4 | Dunston Business Village | Dunston | ST18 9AB

Registered in England and Wales No. 04092394 | VAT registration no. 974 9705 63 

  

--
To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.