ObjectDisposedException in R4B18

[Marcelo Volmaro]

From time to time I'm getting the following exception on B18

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'ReturnBuffer should not be disposed after it's parent operation context was disposed'.
   at Sparrow.Json.JsonOperationContext.ReturnBuffer.ThrowParentWasDisposed()
   at Sparrow.Json.JsonOperationContext.ReturnBuffer.Dispose()
   at Sparrow.Json.JsonOperationContext.ParseToMemoryAsync(Stream stream, String documentId, UsageMode mode, Nullable`1 token)
   at Raven.Client.Http.RavenCommand`1.<ProcessResponse>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Raven.Client.Http.RequestExecutor.<ExecuteAsync>d__69`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Raven.Client.Http.RequestExecutor.<ExecuteAsync>d__69`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Raven.Client.Documents.Session.AsyncDocumentSession.<LoadAsync>d__48`1.MoveNext()

This is not something that I can reproduce consistently. It may take a minute, 10 minutes or an hour. Once this happens, Raven gets in an unstable state and any subsequent requests get aborted with the same error. I have to restart my app to continue working.

This is on an .net core app (netcoreapp1.1 compiled as win7-x64).

The session is not disposed when the LoadAsync is being called (already checked that). 

[Oren Eini (Ayende Rahien)]

And to double check things, you haven't disposed the document store as well, I assume.

Does this happen when under load, after some idle time? 

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcelo Volmaro]

The document store is a singleton, so no, it is not disposed. And no, this is not under load.

The only "strange" thing that this has is that I have a cache in front of this request, so instead of issuing the request on the same context as .net core, this runs on a null context/maybe other thread.

So, the line that tries to access the DB is _tokenCache.GetOrCreateAsync(id, key => base.LoadAsync(key)).AsTask();

_tokenCache is a wrapped dictionary. GetOrCreateAsync returns a ValueTask (that's why the AsTask()). Sometimes the cache refresh works, sometimes doesn't (and that's where this fails).

The GetOrCreateAsync doesn't has anything strange and I've been using in production for some time and never failed. Just for you to check, here you have some code from that library:

public ValueTask<TValue> GetOrCreateAsync([NotNull] TKey key, [NotNull] Func<TKey, Task<TValue>> valueFactory)

    {

      this.ThrowArgumentNullException((object) key == null, nameof (key), (string) null);

      this.ThrowArgumentNullException(valueFactory == null, nameof (valueFactory), (string) null);

      this.AssertNotDisposed();

      CachedValue<TKey, TValue> cachedValue;

      if (this._entries.TryGetValue(key, out cachedValue) && cachedValue != null && !cachedValue.CheckExpired(DateTimeOffset.UtcNow.Ticks))

        return new ValueTask<TValue>(cachedValue.Value);

      return this.AwaitableFactory(valueFactory, key);

    }

private async ValueTask<TValue> AwaitableFactory(Func<TKey, Task<TValue>> valueFactory, TKey key)

    {

      await this._semaphoreSlim.WaitAsync();

      try

      {

        CachedValue<TKey, TValue> entry;

        if (this._entries.TryGetValue(key, out entry) && entry != null)

        {

          if (!entry.CheckExpired(DateTimeOffset.UtcNow.Ticks))

            return entry.Value;

          this._entries.Remove(key);

        }

        TValue obj = await valueFactory(key);

        TValue value = obj;

        obj = default (TValue);

        this._entries.Add(key, CachedValue<TKey, TValue>.New(key, value, this._expirationMode, this._expiration));

        return value;

      }

      finally

      {

        this._semaphoreSlim.Release();

        this.StartScanForExpiredItems();

      }

    }

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Is it possible that you are mixing things from multiple sessions? It looks like this can do that. In particular, what are you caching here, the document?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Marcelo Volmaro]

Yes, I'm caching the document. But I can't see how things can be mixed up. The pipeline goes like (simplified):

1) User make a request

2) MVC creates an instance of the document session (via AddScoped in the IServicesCollection). That instance MUST be the same for the whole requests. If this is wrong, well, MVC got it wrong somewhere.

3) In my code I check if I have an instance of the document in the cache. If I don't, I simply call LoadAsync on the document session provided by MVC.

4) Cache the instance if needed, return it.

There is no way I can be mixing things up in my code. Also, since this is happening "on my machine" while I'm working on the app, there is only one request going on here (mine).

[Oren Eini (Ayende Rahien)]

Can you see if this happens if you make a request, wait a minute and 10 seconds and try it again? 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Marcelo Volmaro]

I just left the app open for an hour, hit refresh and everything works well. As I mentioned, this is not something that I can easily reproduce, but once it happens, everything breaks out.

[Marcelo Volmaro]

So, today happened again. I took some SS that may be useful. What I find strange is those two threads, one with the code from the request and another that has the Raven request. On normal operations, that second thread (the one to the right)  never appears on the parallel stack.

[Oren Eini (Ayende Rahien)]

Um... this might be a thread hopping issue.

From the stack trace, it looks like you using both sync and async calls, and I'm assuming you do that with ConfigureAwait(false);

In that case, you might hop between async contexts, I think. Depending on how your IOC deals with it , you might get an already disposed session.

Can you add a debug assert of the _isDisposed field on the session (it is private, get it via reflection) and check it before you use it there?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Marcelo Volmaro]

Not sure what are you referring to "sync" calls. All my calls are async all the way up. There is no .Wait() or .Result on any Task. As for ConfigureAwait, yes, in all my library code I have set it to false (since I don't know the target platform). In MVC I don't even bother, as MVC core doesn't uses a synchronization context at all, unlike asp.net (more at http://blog.stephencleary.com/2017/03/aspnetcore-synchronization-context.html if you are interested). The IOC is the standard that comes with mvc.core (Microsoft.Extensions.DependencyInjection).

I will check the _isDisposed and let you know it's state at that time...