Waited x ms for the query to return non stale result.

135 views
Skip to first unread message

Derek den Haas

unread,
Jan 9, 2020, 8:23:36 AM1/9/20
to RavenDB - 2nd generation document database
Ok so when a user saves something, I'm refreshing the list with WaitForNonStaleResults(),

now I get this exception:
Waited 430 ms for the query to return non stale result.

but I'm using:
query = query.Customize(a => a.WaitForNonStaleResults(TimeSpan.FromSeconds(5)));


Why is it throwing an exception?
Using client 4.2.4 (since the last two are giving me another error).
Using server 4.2.6

Oren Eini (Ayende Rahien)

unread,
Jan 9, 2020, 8:26:30 AM1/9/20
to ravendb
Looks very strange, actually. 
Can you show the full error? Is this reproducible for you?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/23dc97e8-280e-4e50-9c88-6dbd37a2bc2e%40googlegroups.com.


--
Oren Eini
CEO   /   Hibernating Rhinos LTD
Skype:  ayenderahien
Support:  sup...@ravendb.net

Derek den Haas

unread,
Jan 9, 2020, 8:57:14 AM1/9/20
to RavenDB - 2nd generation document database
No, it does this SOMETIMES (might be that it is no longer stale at the moment I refresh), I log all the exceptions in the application, and I'm 100% sure I'm using WaitForNonStaleResults(5 seconds). I'm not using WaitForNonStale anywhere else in the application.

I thought maybe you fill the timespan differently from what I'm currently using.

Here the whole exception:

        "Message": "Waited 430 ms for the query to return non stale result.",
        "Type": "TimeoutException",
        "Source": "   at Raven.Client.Documents.Session.Operations.QueryOperation.EnsureIsAcceptable(QueryResult result, Boolean waitForNonStaleResults, Nullable`1 duration, InMemoryDocumentSessionOperations session) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\Operations\\QueryOperation.cs:line 354\n   at Raven.Client.Documents.Session.Operations.QueryOperation.EnsureIsAcceptableAndSaveResult(QueryResult result, Nullable`1 duration) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\Operations\\QueryOperation.cs:line 294\n   at Raven.Client.Documents.Session.Operations.QueryOperation.EnsureIsAcceptableAndSaveResult(QueryResult result) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\Operations\\QueryOperation.cs:line 285\n   at Raven.Client.Documents.Session.AsyncDocumentQuery`1.ExecuteActualQueryAsync(CancellationToken token) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\AsyncDocumentQuery.cs:line 1012\n   at Raven.Client.Documents.Session.AsyncDocumentQuery`1.InitAsync(CancellationToken token) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\AsyncDocumentQuery.cs:line 1002\n   at Raven.Client.Documents.Session.AsyncDocumentQuery`1.ExecuteQueryOperationInternal(Nullable`1 take, CancellationToken token) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\AsyncDocumentQuery.cs:line 962\n   at Raven.Client.Documents.Session.AsyncDocumentQuery`1.ExecuteQueryOperation(Nullable`1 take, CancellationToken token) in C:\\Builds\\RavenDB-Stable-4.2\\42021\\src\\Raven.Client\\Documents\\Session\\AsyncDocumentQuery.cs:line 944\n   at DirectCode.Session.Queries.QueryDatabaseWithLocalTransformerByAttributes`4.ExecuteAsync(IServiceProvider provider, QueryRequestContext request)\n   at DirectCode3.Controllers.ApiController.Query(QueryRequest model) in /opt/atlassian/pipelines/agent/build/src/DirectCode3/Controllers/ApiController.cs:line 221\n   at lambda_method(Closure , Object )\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\n   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\n   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n   at DirectCode.ErrorLogging.Middlewares.ExceptionHandlerMiddleware.Awaited(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) in /opt/atlassian/pipelines/agent/build/src/_Framework/DirectCode.ErrorLogging/Middlewares/ExceptionHandlerMiddleware.cs:line 97"

Op donderdag 9 januari 2020 14:26:30 UTC+1 schreef Oren Eini:
Looks very strange, actually. 
Can you show the full error? Is this reproducible for you?

On Thu, Jan 9, 2020 at 3:23 PM Derek den Haas <derek...@gmail.com> wrote:
Ok so when a user saves something, I'm refreshing the list with WaitForNonStaleResults(),

now I get this exception:
Waited 430 ms for the query to return non stale result.

but I'm using:
query = query.Customize(a => a.WaitForNonStaleResults(TimeSpan.FromSeconds(5)));


Why is it throwing an exception?
Using client 4.2.4 (since the last two are giving me another error).
Using server 4.2.6

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

Derek den Haas

unread,
Jan 9, 2020, 9:02:44 AM1/9/20
to RavenDB - 2nd generation document database
P.s. from what I see it's at the server which is having the 5 seconds "pause" problem, which I reported in another topic. I first updated to 4.2.6 (asked by a collegue of you) and afterwards the first few days it wasn't there anymore, but we are now 14 days into this version, and it seems to be back.

Though the error looks wrong, I guess the actual time is longer than 430ms (> 5 seconds). Maybe the error of my database is somewhere between the Stopwatch.Start() of WaitForNonStale (don't know if this is client side or server side) and the Stopwatch.Start of the query itself.

Op donderdag 9 januari 2020 14:57:14 UTC+1 schreef Derek den Haas:

Egor Shamanaev

unread,
Jan 12, 2020, 8:32:11 AM1/12/20
to rav...@googlegroups.com
Hi Derek,

What is the other exception that you are getting on the latest client version ?

Are there any exceptions on server side when you run this query? Could you please also provide us the admin logs ? To do that you have to open the admin logs page, issue the query on the client side, and send us the export (you can send it to sup...@ravendb.net)

Please also attach the admin logs from the moment you are getting the 5 seconds "pause" problem, as for this issue what is the uptime of the server ? Does a restart solve the problem ?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/c190a813-4501-45fa-80f1-af0588b11ca4%40googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Jan 12, 2020, 8:39:06 AM1/12/20
to ravendb
Can you also try to capture the stack traces at that time? Or a minidump?

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/CAJvo4-mDYx%2BwocsgrPG6UDUZCyyg5VZ7c%2B-0NtNFBiQqrs5SeA%40mail.gmail.com.

Derek den Haas

unread,
Jan 13, 2020, 3:25:33 AM1/13/20
to RavenDB - 2nd generation document database
Egor,
This was the other issue I'm having with 4.2.5 and .6: https://groups.google.com/forum/#!topic/ravendb/sj54ZzRtKG0

Both,
I will try to fetch logs of a query, though it's not happening all the time, so it's a bit hard to get stack traces or a minidump of it, it's 1 out of 100 requests (approximately).

Op zondag 12 januari 2020 14:39:06 UTC+1 schreef Oren Eini:

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Jan 13, 2020, 4:50:20 AM1/13/20
to ravendb
The other issue was resolved and merged, you should be able to get it from the nightly

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/08c16414-98a0-4bd2-b152-86cfd05c0bef%40googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Jan 13, 2020, 4:50:34 AM1/13/20
to ravendb
As for this issue, can you reproduce this on a standalone machine? 
Or only on your production server?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/08c16414-98a0-4bd2-b152-86cfd05c0bef%40googlegroups.com.

Derek den Haas

unread,
Jan 13, 2020, 7:17:55 AM1/13/20
to RavenDB - 2nd generation document database
I'm currently trying to get it to reproduce on a standalone machine using the database I think produces the error. I'm creating a script to simulate actions done, but it looks like it hangs due to a single document insert which makes 3 indexes stale which hangs the server, which will make queries run "slow" (any query, since the whole server hangs, also queries not waiting for non stale).

I will come back when I have this working.

Op maandag 13 januari 2020 10:50:34 UTC+1 schreef Oren Eini:

Oren Eini (Ayende Rahien)

unread,
Jan 13, 2020, 7:33:01 AM1/13/20
to ravendb
Is it possible that you are swapping? Writes won't block reads in RavenDB

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/7f288f85-128e-40f9-96b7-26b19cd0d262%40googlegroups.com.

Derek den Haas

unread,
Jan 13, 2020, 9:36:44 AM1/13/20
to RavenDB - 2nd generation document database
No not possible :). It's using 10 out of 20 gb memory and I haven't added a swap space to linux. Docker also had this problem, I switched to a standard VPS because I thought it might be due to the docker environment.

By the way, during those 5 seconds also the dashboard of ravendb doesn't update (memory/cpu usage won't update for 5 seconds), so ravendb itself seems to be hanging. It looks like it's being triggered by a document insert. When only querying, it never hangs.


Op maandag 13 januari 2020 13:33:01 UTC+1 schreef Oren Eini:

Oren Eini (Ayende Rahien)

unread,
Jan 13, 2020, 10:22:51 AM1/13/20
to ravendb
I can't think of anything that would cause it.
Can you do a proc dump on this when this happen? I hope that this would allow us to pin point what is causing the problem

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/5bef3133-9a34-486b-97b1-ee72571d5d2e%40googlegroups.com.

Derek den Haas

unread,
Jan 14, 2020, 1:01:50 AM1/14/20
to RavenDB - 2nd generation document database
I'm trying to create a script locally to mimic the behavior of a typical load on that server (doc inserts + doc queries). If I can mimic that and also get the same issue, I can share the login credentials to that machine with you... It will be on a stand alone server :).

Op maandag 13 januari 2020 16:22:51 UTC+1 schreef Oren Eini:

Oren Eini (Ayende Rahien)

unread,
Jan 14, 2020, 3:04:28 AM1/14/20
to ravendb
That would be awesome, thank you very much

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/2f096ee7-20cd-4168-b910-6156d44ae107%40googlegroups.com.

Derek den Haas

unread,
Feb 5, 2020, 1:26:29 AM2/5/20
to RavenDB - 2nd generation document database
Have you found it yourself?


We are still trying to replicate it, though this looks very much the same...

Op dinsdag 14 januari 2020 09:04:28 UTC+1 schreef Oren Eini:

Derek den Haas

unread,
Feb 5, 2020, 1:30:29 AM2/5/20
to RavenDB - 2nd generation document database
If so, is there a ticket I can follow to see when it is fixed? We hate to restart the database every now and then.

Op woensdag 5 februari 2020 07:26:29 UTC+1 schreef Derek den Haas:

Egor Shamanaev

unread,
Feb 6, 2020, 10:34:25 AM2/6/20
to rav...@googlegroups.com
Hi,

https://issues.hibernatingrhinos.com/issue/RavenDB-14555  

The changes are in the latest nightly, please check if this fixes your issue.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/51308c10-f7c1-4764-9d6d-76ec7010f366%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages