ForceReadFromMaster doesn't seem to be being honoured - 3.5.0-rc-35157

[Nick]

I open a session to a database which is in a cluster with OpenSessionOptions as follows

new OpenSessionOptions

{

        Database = "DBName",

        ForceReadFromMaster = true

}

I then load a document, make changes and try to save and get a concurrency exception every time.

Looking at it further the etag of the document loaded is the etag from the secondary server in the cluster (i.e. not the leader) the problem is obviously we're then trying to save to the leader with an invalid etag

Failover behaviour is ReadFromAllWriteToLeaderWithFailovers

This seems to have started since upgrading to build 3.5.0-rc-35157 yesterday.

Seems the session is not honouring the ForceReadFromMaster setting.

Nick

[Maxim Buryak]

Hi,

Can you provide more details about the cluster configuration?

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick]

Hi,

2 nodes in the cluster, both voting nodes, one nominated as leader.

Here is a gist of the client side document store set up

https://gist.github.com/nickchampion/0d07d733c6722dd9b687f250b4e5b6c6

Note that in this code for the environment in question the following config properties are

ReplicationEnabled = true

IndexReplicationEnabled = true

ReplicateReplicatedDocuments = true

Here's screenshot of global replication config and cluster.

Anything else you want?

[Maxim Buryak]

Hi,

I was not able to reproduce the issue, could you please try writing a unit test that reproduces the issue(see attached test as an example)

Meanwhile, looking at your code, it looks a bit strange that you define the failover servers yourself. What happens if you remove that section?

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Nick]

I will try to get something to you today.

On defining the failover servers manually, interesting Oren recommended this approach in another topic so the clients always know the topology in case the configured leader is not available when the client starts up, are you saying we should use a different approach?

[Oren Eini (Ayende Rahien)]

That is an extreme paranoid settings.

On most cases, it is enough to specify any of the servers, the client will read the topology and remember that.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

[Nick]

Ok thanks.

Removing the manual configuration of the failover servers makes no difference.

Changing the FailoverBehavior obviously does, if I have failover behaviour set ReadFromAllWriteToLeader or ReadFromAllWriteToLeaderWithFailovers I can reproduce the concurrency error every time.

I will try and get a failing test to you later today

[Nick]

Oren - on your comment about extreme paranoid, we run RavenDB in Azure which as you probably know take servers out periodically for maintenance which can often last 10-20 minutes, is it not the case that if a client restarted during scheduled maintenance of the primary server we configure for the clients it wouldn't be able to start up?

Seems to me it would be prudent to always specify the failover servers to avoid such a situation.

Thanks

Nick

[Nick]

I am not able to reproduce this using your test, one thing I did notice though was that you had not specified optimistic concurrency for the session.

However the test here (connecting to physical servers with the cluster config specified above) consistently fails

https://gist.github.com/nickchampion/54fe02ce54b4c7315ef87d678b4fb94c

If it helps I can give you access to the server so you can run the test yourself or look at the set up. Let me know and I will email you details.

Here is the fiddler trace from running the test, you can see the initial session which creates the doc writes to the leader (raven-sg-2) then the next session initially reads from the secondary server (non leader) despite specifying ForceReadFromMaster, then writes, as expected, to the leader causing concurrency exception.

[Maxim Buryak]

Hi,

Thanks, the issue was reproduced. You can track it at: http://issues.hibernatingrhinos.com/issue/RavenDB-4845

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Maxim Buryak]

Hi, 

Latest 3.5 unstable contains the fix to the problem we've discussed

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Nick]

Hi,

Thanks for the update, just wondering if someone could get back to me on the comment further up the thread...

"Oren - on your comment about extreme paranoid, we run RavenDB in Azure which as you probably know take servers out periodically for maintenance which can often last 10-20 minutes, is it not the case that if a client restarted during scheduled maintenance of the primary server we configure, the client wouldn't be able to start up?

Seems to me it would be prudent to always specify the failover servers to avoid such a situation."

Thanks

[Oren Eini (Ayende Rahien)]

If the client is started _for the very first time_.

If they are just restarted, they remember the previous values.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811