Changes API Disconnecting in RavenDB 4.0

[Lucas bode]

When using the Changes API, after a period of 10 minutes or so of no activity, the changes subscription will disconnect. Here is a snippet of code I am using.

var subscription = Store.Changes();
subscription.ConnectionStatusChanged += (sender, args) => Console.WriteLine($"Raven Connection Status Changed to {Store.Changes().Connected} - {DateTime.Now}");
subscription.ForDocumentsInCollection<T>()
    .Subscribe(async change => handle(await GetUpdatedEntity(change)));

Produces this output...

Here is what I am using. 

RavenDB - 4.0.3-patch-40031 (Inside Linux docker container)

.Net Core app using RavenDB.Client - 4.0.3-patch-40031

Any thoughts on why this happens? 

[Lucas bode]

I also observed that the admin interface uses the changes API when you are viewing a document. I did a test to see if this connection gets lost the same way that it happens in my application. And after about ten minutes it did lose the connection. So when I updated that document in another tab I didn't get the notification within the UI that tells me the document has been modified. So that result kind of leads me to the conclusion that it may be some sort of bug?

[Oren Eini (Ayende Rahien)]

It might be, yes.

Can you provide us with the full details about your system?

What is the exact versions? OS version?

Are you running the client on the same machine as the server?

Any firewall / AV in the middle?

Are you using authentication?

The client is running CoreCLR or full framework?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lucas bode]

What is the exact versions? OS version?

• We are running the following docker image - ravendb/ravendb:4.0.3-patch-40031-ubuntu.16.04-x64
• Occurred locally for me on a Windows machine - (Version 10.0.16299 Build 16299)
• Occurred on our Linux host (CentOS Linux 7), Kernel - (Version 3.10.0-693.17.1.el7.x86_64)

Are you running the client on the same machine as the server?

• Yes

Any firewall / AV in the middle?

• No

Are you using authentication?

• No, using the Unsecured option.

The client is running CoreCLR or full framework?

• I believe just the CoreCLR...

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Tal Weiss]

This is confirmed, you can track the issue at http://issues.hibernatingrhinos.com/issue/RavenDB-10781.

Note that I'm not sure regarding the idleness of the database been important since the connection is not idle even if no changes are reported to the client (an empty report is been sent every 5 seconds)

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hibernating Rhinos Ltd  

Tal Weiss l Core Team Developer l Mobile:+972-54-802-4849

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811l Skype: talweiss1982

RavenDB paving the way to "Data Made Simple" http://ravendb.net/ 

[Lucas bode]

Is there a particular time frame when this could possibly be resolved? The current state of the issue sort of makes the feature unusable at this time.

Hibernating Rhinos Ltd  

[Oren Eini (Ayende Rahien)]

Hi,

We are working on figuring out exactly what is going on. At this point, we are sniffing at the TCP level, because this is very strange.

We have an application level heartbeat that is supposed to keep  this alive.

We are currently checking if this might be related to options.Limits.MinResponseDataRate, but we are still not sure

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Tal Weiss]

Hi Lucas,

I'm just wondering are you connecting to the server using localhost when this the changes break?

[Lucas bode]

Awesome, thank you for the update. I guess I was just kind of wondering if it was actively being looked at or if we would have to come up with a replacement solution in the interim. 

[Lucas bode]

Yes I have used localhost to connect, but the issue seems to be present pretty universally wherever I try it. 

So as an example, I ran this exact command to spin up my RavenDb container -

docker run -d -p 8080:8080 ravendb/ravendb

And I connect to that container in my application via "localhost:8080". Using that test I lost my connection in ~16 minutes.

[Oren Eini (Ayende Rahien)]

Can you try capturing this via tcpdump on docker, or wireshark?

Something is killing the connection, we aren't sure what

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Tal Weiss]

Hi Lucas,

I have looked at the code and raven client is suppose to reconnect if it detected a failure in the web socket.

And it appears that it works:

.

.

.

Put on Users/0000000000000000004-A

Raven Connection Status Changed to False - 3/28/2018 12:34:05 PM

Raven Connection Status Changed to True - 3/28/2018 12:34:32 PM

.

.

.

Note that the delay is because it was waiting on a breakpoint in the client.

What version of RavenDB client are you using?

[Tal Weiss]

Also leaving changes open seems to show that the disconnection correlates to the database idle time.

notice that on the latest client it re-connects right away.

Try setting  databases.maxidletimeinsec (https://ravendb.net/docs/article-page/4.0/csharp/server/configuration/database-configuration#databases.maxidletimeinsec)

To a longer period than 15 sec

[Tal Weiss]

15 minutes

[Lucas bode]

For some reason my setup doesn't seem to be reconnecting after that timeout. I went ahead and created a little sample app that spins up a ravenDB (4.0.3-patch-40033) container and a .net core console app using the raven client package (4.0.3-patch-40033). It's basically an app that connects to the changes api and saves a document. Every time it saves a document I increment the time until saving the next document by 15 seconds so you can see the changes api working and then disconnecting.

If you go to the root of the project there should be a docker compose file. If you run a compose command it will spin up both. If you check the logs of the "changesapitest" container you should see the relevant logs. In the compose setup, I set the MaxIdleTimeInSec to 30 seconds in order to expedite the behavior in question. I had a coworker run the same command and he observed the same behavior as I see.

As you can see, the changes api responds as expected when a new document is added. But when the time between document saves increases above the idle time, the changes api disconnects and fails to reconnect. And subsequently the changes api event no longer fires. I've attached a .zip file of the app. I figured this would be the easiest way to reproduce the same environments between us. 

Also - I'm running Docker 18.03.0-ce-win59 (16762) on a Windows 10 machine.

15 minutes

Hibernating Rhinos Ltd  

Tal Weiss l Core Team Developer l Mobile:+972-54-802-4849

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811l Skype: talweiss1982

RavenDB paving the way to "Data Made Simple" http://ravendb.net/ 

[Tal Weiss]

Running your app against raven ubuntu docker image.

I suspect it has to do with the .Net core version you're running.

It appears that the guys at Microsoft has issues with WebSockets and they requested that I'll upgrade to the latest .Net core version and I did.

You can verify your.Net version from the command prompt: dotnet --version

Please check the latest version 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lucas bode]

Did you try executing with the docker compose command? (docker-compose up -d --build --force-recreate) My hope was to use that to isolate the issue so we don't have to go back and forth between differences in environments. It's important to use that compose command because I am setting the "Databases.MaxIdleTimeInSec" to 30 to help produce the error earlier. If you run the app against a RavenDB instance with the default value, it will take much longer to get to that 15 minute split between times when the documents are saved. It looks like your log output doesn't wait that long, there is only 4 minutes between saves and doesn't show the disconnect.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/87wcFw4eoCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

[Tal Weiss]

no I have not, I'm not sure what you're referring to, can you explain how to run ' docker-compose up -d --build --force-recreate'?

[Tal Weiss]

BTW it has been running for the past hour and a half and still didn't disconnect 

[Lucas bode]

If you navigate to the root of the project in a command prompt there should be a "docker-compose.yml" file present. If you run the command "docker-compose up -d --build --force-recreate" it will execute the compose file and spin up a RavenDB container and a container with my test app inside it. The test app will be using the RavenDB container that was spun up. You should then be able to run "docker logs {containerID}" to watch the behavior.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/87wcFw4eoCw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tal Weiss]

Okay, running both from the container.

How can I tell what is the .Net core version used by the container? 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tal Weiss]

NM, microsoft/aspnetcore-build:2.0.

[Tal Weiss]

Okay got it to fail, finally.

I'm trying it again with an updated .NET core version inside the container

still fails.

I believe that this is a .NET core/ docker/ ubuntu issue I'll have somebody more familiar with those technologies to have a look

[Lucas bode]

I tried updating the .Net core images to use newer versions and it was still failing for me as well.

[Tal Weiss]

Okay,

This issue relates to https://github.com/dotnet/corefx/issues/24430

The client in the docker is not aware that the socket has been closed and this is why it doesn't try to re-connect.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lucas bode]

I'm don't believe it is an issue isolated to docker. I just ran the app using a fresh RavenDB windows instance. (RavenDB-4.0.3-patch-40034-windows-x64, not in docker) 

Here are my results. (Note the dotnet --version should be pretty recent)

[Lucas bode]

Or are you saying that it is an issue regardless of docker?

NM, microsoft/aspnetcore-build:2.0.

15 minutes

--

<p style="margin:0cm 0cm 0.0001pt;direction:ltr;font-size:11pt;font-family:Calibri

[Lucas bode]

Did you see my last reply stating that the issue can still be reproduced outside of a docker container?

So is the plan to wait until the fix is released in a future version of .net core? Maybe it would be a good idea to point out the issues with this feature in the documentation until it can be fixed so other developers can avoid some of the investigation work I've had to do.

[Oren Eini (Ayende Rahien)]

.Net 2.1 is right around the corner, and it will have the fixes for this. 

We'll update the docs accordingly.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Lucas bode]

Okay that would be great. I was under the impression from the github issue linked above (https://github.com/dotnet/corefx/issues/24430) that the fix would not be included in 2.1 and would be pushed to a future release, so that's the only reason I was worried.

Thanks,

Lucas

NM, microsoft/aspnetcore-build:2.0.

15 minutes

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811l Skype: <a value="+97246227811" styl

[Tal Weiss]

I have investigated this deeply today.

Firstly a fix I have issued that prevents the unloading of a database if there is an opened changes api connection "fixes" the problem in your test case. 

It's actually not fixing the problem but avoiding it, the issue can still occur if the socket closes due to a real network problem. 

I have actually ruled out my conclusion, although the error is what expected the manner of the failure isn't. 

I believe we can make our code more robust to fix the issue entirely without having to depend on the dot net core fix (although if they fix it it will also resolve the problem). 

It appears we try to access the socket from the OnError method but it is already in a faulty state. 

[Tal Weiss]

Hi Lucas,

I'm glad to say that we have finally pinned point what exactly is happening and managed to fix it, the fix should make it to the next stable build planned for next week.

To whom this concern the problem was that the cancelation tokens in the server's WebSocket were not respected and we had to unload the database after a grace period.

We ended up disposing of the WebSockets on the server side causing it to miss the disconnection handshake, this would throw on the client causing us to report Error in the IObserver, killing the changes API connection.

The change made was to not throw if we are able to reconnect.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lucas bode]

Cool, I am happy to hear it! This feature will help us out a ton. Thanks for all the work you've done on it.

-Lucas