LoadAsync never returns

[Lee Baker]

I have been banging my head against this today and I am not sure where to go.

I have implemented a user session system using Raven as the store and intermittently when I try to call the LoadAsync method (awaited) it the code never progresses. I am watching the calls to Raven in fiddler and I can see that no database calls are made in this situation.

My best guess is I am getting the document store into a funny state but I can't reproduce the issue whenever I want and I am not sure where to start looking to provide more information (or a failing test)

Another issue I am facing is the SaveChangesAsync in the "Extend" and "Logout" methods throw an "Underlying connection is closed" exception, also intermittently.

If anyone has any pointers I would be very grateful.

using System;

using System.Security.Cryptography;

using System.Text;

using System.Threading.Tasks;

using Raven.Client;

    public class RavenGuestSessionDataService : IGuestSessionDataService

    {

        private readonly IDocumentStore _store;

        private const int DefaultSessionExpiryMinutes = 360000;

        private const int ShortSessionExpiryMinutes = 20;

        private const int IdByteLength = 16;

        private static readonly RandomNumberGenerator RandomGen = RandomNumberGenerator.Create();

        public RavenGuestSessionDataService(IDocumentStore store)

        {

            _store = store;

        }

        public async Task<PhoenixGuestSession> CreateDemoSession(int guestId, DemoType type)

        {

            return GetPhoenixContractSessionDocument(await CreateSession(guestId, type, ShortSessionExpiryMinutes, null));

        }

        public async Task<GuestSession> CreateImpersonateSession(int guestId, string baseSessionId)

        {

            return GetContractSessionDocument(await CreateSession(guestId, null, ShortSessionExpiryMinutes, baseSessionId));

        }

        public async Task<GuestSession> CreateSession(int guestId)

        {

            return GetContractSessionDocument(await CreateSession(guestId, null, DefaultSessionExpiryMinutes, null));

        }

        private async Task<DataService.Document.GuestSession> CreateSession(int guestId, DemoType? type, int periodMinutes, string baseSessionId)

        {

            using (var ravenSession = _store.OpenAsyncSession())

            {

                var sessionId = new StringBuilder();

                if (type != null)

                {

                    // demo session Id

                    sessionId.Append("demo" + type.ToString().ToLower() + "-" + GetId());

                }

                else if (baseSessionId != null)

                {

                    // impersonate session Id

                    sessionId.Append("investor-" + GetId());

                }

                else 

                {

                    sessionId.Append("guest-" + Guid.NewGuid());

                }

                var session = new DataService.Document.GuestSession

                                  {

                                      GuestId = Guest.Metadata.GetId(guestId),

                                      Expiry = DateTimeOffset.Now.AddMinutes(periodMinutes),

                                      SessionId = sessionId.ToString(),

                                      DemoType = (int?)type,

                                      PeriodMinutes = periodMinutes,

                                      BaseSessionId = baseSessionId

                                  };

                await ravenSession.StoreAsync(session);

                await ravenSession.SaveChangesAsync();

                return session;

            }

        }

        private static string GetId()

        {

            var bytes = new Byte[IdByteLength];

            RandomGen.GetBytes(bytes);

            return bytes.ToBase62();

        }

        public async Task<GuestSession> GetSession(string sessionId)

        {

            Logger.Current.Log(LogLevel.Info, "Start getting session - " + sessionId);

            using (var ravenSession = _store.OpenAsyncSession())

            {

                var session = await GetSessionDocument(sessionId, ravenSession);

                var guestSession = session == null ? null : GetContractSessionDocument(session);

                Logger.Current.Log(LogLevel.Info, "Completed getting session - " + sessionId);

                return guestSession;

            }

        }

        public async Task<PhoenixGuestSession> GetPhoenixSession(string sessionId)

        {

            using (var ravenSession = _store.OpenAsyncSession())

            {

                Logger.Current.Log(LogLevel.Info, "Start getting phoenix session - " + sessionId);

                var session = await GetSessionDocument(sessionId, ravenSession);

                Logger.Current.Log(LogLevel.Info, "Completed getting phoenix session - " + sessionId);

                return session == null ? null : GetPhoenixContractSessionDocument(session);

            }

        }

        public async Task RevokeSession(string sessionId)

        {

            using (var ravenSession = _store.OpenAsyncSession())

            {

                var session = await GetSessionDocument(sessionId, ravenSession);

                if (session == null)

                {

                    return;

                }

                ravenSession.Delete(session);

                await ravenSession.SaveChangesAsync();

            }

        }

        public async Task<GuestSession> ExtendSession(string sessionId)

        {

            using (var ravenSession = _store.OpenAsyncSession())

            {

                var session = await GetSessionDocument(sessionId, ravenSession);

                if (session == null)

                {

                    return null;

                }

                session.Expiry = DateTimeOffset.Now.AddMinutes(session.PeriodMinutes);

                await ravenSession.SaveChangesAsync();

                return GetContractSessionDocument(session);

            }

        }

        public async Task UpdateNextGlobalAuthCheck(string sessionId, DateTimeOffset nextCheck)

        {

            using (var ravenSession = _store.OpenAsyncSession())

            {

                var session = await GetSessionDocument(sessionId, ravenSession);

                if (session == null)

                {

                    return;

                }

                session.NextGlobalAuthCheck = nextCheck;

                await ravenSession.SaveChangesAsync();

            }

        }

        private async Task<DataService.Document.GuestSession> GetSessionDocument(string sessionId, IAsyncDocumentSession ravenSession)

        {

            var session = await ravenSession.LoadAsync<DataService.Document.GuestSession>(DataService.Document.GuestSession

                                                                                                              .Metadata

                                                                                                              .GetId(sessionId));

            return session;

        }

        private static GuestSession GetContractSessionDocument(DataService.Document.GuestSession session)

        {

            return new GuestSession

            {

                Expiry = session.Expiry,

                SessionId = session.SessionId,

                GuestId = Guest.Metadata.GetIntId(session.GuestId),

                NextGlobalAuthCheck = session.NextGlobalAuthCheck

            };

        }

        private static PhoenixGuestSession GetPhoenixContractSessionDocument(DataService.Document.GuestSession session)

        {

            return new PhoenixGuestSession

            {

                Expiry = session.Expiry,

                SessionId = session.SessionId,

                GuestId = Guest.Metadata.GetIntId(session.GuestId),

                NextGlobalAuthCheck = session.NextGlobalAuthCheck,

                DemoType = (DemoType?)session.DemoType, // TODO: This better

                BaseSessionId = session.BaseSessionId

            };

        }

    }

[Lee Baker]

The exception stack for the underlying connection issue is

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

   --- End of inner exception stack trace ---

   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)

   --- End of inner exception stack trace ---

   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

   --- End of inner exception stack trace ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.HttpJsonRequest.<>c__DisplayClass13.<<SendRequestInternal>b__12>d__15.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.HttpJsonRequest.<RunWithAuthRetry>d__1d`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.HttpJsonRequest.<SendRequestInternal>d__1a.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.HttpJsonRequest.<WriteAsync>d__5d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<>c__DisplayClass1b3.<<BatchAsync>b__1b1>d__1b6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.ReplicationInformerBase`1.<TryOperationAsync>d__24`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.ReplicationInformerBase`1.<ExecuteWithReplicationAsync>d__17`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<ExecuteWithReplication>d__268`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

   at Raven.Client.Document.Async.AsyncDocumentSession.<SaveChangesAsync>d__b6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at PPS.Phoenix.DataAccess.RavenGuestSessionDataService.<RevokeSession>d__1b.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at PPS.Phoenix.Web.Code.Auth.PhoenixAuthProvider.<Logout>d__13.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException(Task task)

   at System.Threading.Tasks.Task.Execute()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Nito.AsyncEx.AsyncContext.Run(Func`1 action)

   at PraemiumAPI.Common.Web.Component.Auth.Mvc.PraemiumDbAuthorizeAttribute.AuthorizeCore(HttpContextBase httpContext)

   at System.Web.Mvc.AuthorizeAttribute.OnAuthorization(AuthorizationContext filterContext)

   at System.Web.Mvc.ControllerActionInvoker.InvokeAuthorizationFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__19(AsyncCallback asyncCallback, Object asyncState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TResult](AsyncCallback callback, Object state, BeginInvokeDelegate beginDelegate, EndInvokeDelegate`1 endDelegate, Object tag, Int32 timeout)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state)

   at System.Web.Mvc.Controller.<BeginExecuteCore>b__1c(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state)

   at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)

   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[Oren Eini (Ayende Rahien)]

What build are you using? Where are you running this code?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lee Baker]

Hi Oren,

This code is running in a asp.net webapi project running in Azure.

We are using version 3.0.3528 of the RavenDB.Client package.

We are using version 3 server build #3528

[Lee Baker]

I have coded around the problem it seems. I was using a wrapper class to make async calls syc, and I think that is what was causing this issue. 

[Oren Eini (Ayende Rahien)]

Yes, you cannot do that unless you are very careful.

If you want sync code, use the sync session, which did this careful coding

[Lee Baker]

Thanks Oren,

It seems I am still getting the underlying connection issue when calling async code from an async MVC controller. I haven't seen this issue running in my development environment but when I release the code to test I get this semi regularly (still am un-sure on how to reproduce it consistently though)

   at PPS.Phoenix.DataAccess.RavenGuestSessionDataService.<CreateSession>d__a.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at PPS.Phoenix.DataAccess.RavenGuestSessionDataService.<CreateDemoSession>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at PPS.Phoenix.Web.Code.Auth.PhoenixAuthProvider.<StartDemo>d__22.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at PPS.Phoenix.Web.Controllers.DemoController.<Demo>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass37.<BeginInvokeAsynchronousActionMethod>b__36(IAsyncResult asyncResult)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d()

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f()

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.<BeginInvokeAction>b__1c()

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)

   at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)

   at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)

   at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)

   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

[Lee Baker]

I have changed my code to all be synchronous and I am still getting the following exception

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Raven.Client.Extensions.TaskExtensions2.ResultUnwrap[T](Task`1 task)

   at Raven.Client.Document.DocumentSession.SaveChanges()

   at PPS.Phoenix.DataAccess.RavenGuestSessionDataService.ExtendSessionSynchronous(String sessionId)

   at PraemiumAPI.Common.Web.Component.Auth.Mvc.PraemiumDbAuthorizeAttribute.AuthorizeCore(HttpContextBase httpContext)

   at System.Web.Mvc.AuthorizeAttribute.OnAuthorization(AuthorizationContext filterContext)

   at System.Web.Mvc.ControllerActionInvoker.InvokeAuthorizationFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__19(AsyncCallback asyncCallback, Object asyncState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TResult](AsyncCallback callback, Object state, BeginInvokeDelegate beginDelegate, EndInvokeDelegate`1 endDelegate, Object tag, Int32 timeout)

   at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state)

   at System.Web.Mvc.Controller.<BeginExecuteCore>b__1c(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state)

   at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)

   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)

   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)

   at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)

   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

...

[Oren Eini (Ayende Rahien)]

What is your topology? Are you running on the same server? Different servers?

Is RavenDB restarting in the meantime? Do you have firewall in the middle?

--

[Lee Baker]

Hi Oren,

I am currently running wireshark on the RavenDB server to see if I can see what is going on.

Our setup is:

• RavenDB running in an Azure VM
  
• Website running as an Azure Cloud service 
  
• Both of these are on the same virtual network, but that is all I know. 
  
• RavenDB is not restarting.
  

I am trying to find out if its an azure networking issue, but it only seems to affect us when there has been no traffic for a while (not sure of an exact time) and we are calling SaveChanges

None of our queries or loads seem to have this issue.

...

[Lee Baker]

I have also turned on trace logging in RavenDB so hopefully I see something from Raven next time this issue happens.

In the PCAP I am seeing a resest command sent from Raven back to my client

...

[Oren Eini (Ayende Rahien)]

At a guess, I would say that the problem is that the connection between the two machine is being broken because of idle time.

RavenDB use HTTP, and that uses keep alive.

So we don't have to make re-create the TCP connection each and every time.

If Azure is breaking connections, that explains it.

See here: 

https://social.msdn.microsoft.com/Forums/azure/en-US/2fd134d3-d18f-407d-848c-2f8fc75c2624/an-existing-connection-was-forcibly-closed-by-the-remote-host?forum=WAVirtualMachinesVirtualNetwork

In particular: 

) Azure will forcibly close a connection that is idle for 1 minute: this is what is terminating our connection

--

[Lee Baker]

TL/DR: When connecting to an Azure VM from a machine inside the same Azure virtual network, when the server closes the connections due to idle timeout it doesn't send all the RST packets back to the client. However when connecting via the external IP address all RST packets are sent. This causes the raven client to try and communicate over an expired connection and throw exceptions when trying to write (although it handles reads gracefully)

Detail:

The error we are seeing is that when trying to write a document, after a 2 minute idle period, RavenDB server tells us the connection we are trying to use is closed and throws an exception.

The issue is an Azure networking issue, but it is different to the one found above.

Our setup is to have an Azure VM running the RavenDB server and a website existing on a cloud service. Both of these services are in the same Azure virtual network (so have IP addresses in the same subnet).

Based on our initial testing we believed the issue might be that the server is closing the connections but the client isn't receiving the reset packets.

We ran wireshark on both ends (the VM and the cloud service VM) to watch what traffic was passing through each service while we performed the following tests:

• Run application from dev machine (azure emulation) pointing to RavenDB (external address).  The resets occur, but it’s resetting all connections which doesn’t cause the exception
• Run application from dev machine (azure emulation) within same network in Azure using internal IP.  We have the same exception occurring caused by inconsistent resets.
• Moved application to a VM and run in IIS (opposed to managed/emulated cloud service). We have the same exception occurring caused by inconsistent resets.
• Change the connection string of the application in IIS to external IP of VM/RavenDB.  The resets occur, but it’s resetting all connections which doesn’t cause the exception.
• Change the connection string on the cloud service to use external IP of VM/RavenDB.  The resets occur, but it’s resetting all connections which doesn’t cause the exception.

The outcome of this was that when connecting to the RavenDB VM by the internal IP address the VM doesn't send all RST packets when it times-out the connections however all these packets are all coming through when connecting by the external IP address. 

This causes the raven client to think the connections are still open and try to communicate over the closed connection and when doing writes, it throws an exception (the server has closed the connection). When doing reads it seems the client handles this error gracefully and just re-establishes the connection and tries again.

We have raised this with Microsoft and hopefully will get some form of resolution from them.

This was a very frustrating issue to debug and hopefully this explanation will help some others.

...

[Oren Eini (Ayende Rahien)]

Are you using RavenDB replication?

In that situation, RavenDB will actually retry failed connections, so that might be a good idea to recover from this error.

--