Read from all servers and write behaviour?

44 views
Skip to first unread message

ZNS

unread,
Apr 21, 2017, 9:05:53 AM4/21/17
to RavenDB - 2nd generation document database
Hi.

We have a master -> slave replication scenario. We are configuring our store to use the ReadFromAllServers failover behaviour. However we are seeing that some deletions of documents don't get replicated. We have a job that runs every night. It queries an index for docs to be deleted then deletes them using session load/delete. However the deletion doesn't seem to be replicated. The next time the query runs and hits the slave it gets hits on documents that should have been deleted (and are deleted on the master). Could this be a bug or is this to be expected when using ReadFromAllServers? #35183

I'd like to write a test for this, but don't know if it's possible to test replication?

Tal Weiss

unread,
Apr 21, 2017, 9:20:46 AM4/21/17
to RavenDB - 2nd generation document database
You can write a test by inheriting from https://github.com/ravendb/ravendb/blob/v3.5/Raven.Tests.Common/ReplicationBase.cs this should take care of most of the things you need to setup.
Have a look at other tests inheriting from it.
Deletions should be replicated to all replicas if you can reproduce this then it is surely a bug.
We implement replication of deletion by sending tombstones of the documents, those should reside in the master until they are replicated to all destinations.
We are not aware of any issue with replication on v3.5 so a failing test is welcomed.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Hibernating Rhinos Ltd  cid:image001.png@01CF95E2.8ED1B7D0

Tal Weiss l Core Team Developer Mobile:+972-54-802-4849

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811l Skype: talweiss1982

RavenDB paving the way to "Data Made Simplehttp://ravendb.net/ 

ZNS

unread,
Apr 22, 2017, 7:20:58 AM4/22/17
to RavenDB - 2nd generation document database

After looking at this a bit closer I think it might be a custom trigger that is causing this. We have a delete trigger which executes "DisableAllTriggersForCurrentThread()". I tried to write a test for it but I can't get the trigger to execute.. perhaps you can take a look at it?

https://gist.github.com/ZNS/7564c7146b986a3cbb13ce3eda765b52


On Friday, April 21, 2017 at 3:20:46 PM UTC+2, Tal Weiss wrote:
You can write a test by inheriting from https://github.com/ravendb/ravendb/blob/v3.5/Raven.Tests.Common/ReplicationBase.cs this should take care of most of the things you need to setup.
Have a look at other tests inheriting from it.
Deletions should be replicated to all replicas if you can reproduce this then it is surely a bug.
We implement replication of deletion by sending tombstones of the documents, those should reside in the master until they are replicated to all destinations.
We are not aware of any issue with replication on v3.5 so a failing test is welcomed.
On Fri, Apr 21, 2017 at 4:05 PM, ZNS <ulrik.a...@gmail.com> wrote:
Hi.

We have a master -> slave replication scenario. We are configuring our store to use the ReadFromAllServers failover behaviour. However we are seeing that some deletions of documents don't get replicated. We have a job that runs every night. It queries an index for docs to be deleted then deletes them using session load/delete. However the deletion doesn't seem to be replicated. The next time the query runs and hits the slave it gets hits on documents that should have been deleted (and are deleted on the master). Could this be a bug or is this to be expected when using ReadFromAllServers? #35183

I'd like to write a test for this, but don't know if it's possible to test replication?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Oren Eini (Ayende Rahien)

unread,
Apr 23, 2017, 5:22:14 AM4/23/17
to ravendb

Hibernating Rhinos Ltd  

Oren Eini l CEO Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

ZNS

unread,
Apr 23, 2017, 6:20:35 AM4/23/17
to RavenDB - 2nd generation document database

Thanks, but I tried that first but that didn't work either. I have other succesful tests with triggers but they use the EmbeddableDocumentStore. Not sure If I can use an embeddable document store as master for a replication test?

Oren Eini (Ayende Rahien)

unread,
Apr 23, 2017, 6:21:38 AM4/23/17
to ravendb
Try using non embedded, IIRC
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

ZNS

unread,
Apr 23, 2017, 6:28:58 AM4/23/17
to RavenDB - 2nd generation document database

Huh? The test as it is uses the "GetNewServer()" and "NewRemoteDocumentStore()" methods. Which I assume are non embedded, or are there other non embedded stores to use?
https://gist.github.com/ZNS/7564c7146b986a3cbb13ce3eda765b52

Oren Eini (Ayende Rahien)

unread,
Apr 23, 2017, 6:32:22 AM4/23/17
to ravendb
Yes, don't use "testdb", use the default one, and it will work there.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Tal Weiss

unread,
Apr 23, 2017, 6:46:23 AM4/23/17
to RavenDB - 2nd generation document database
I think that disabling triggers for the current thread could very well disturb replication especially if you use the trigger to delete documents.
Here is were we generate the tombstones:
This may not be invoked if the trigger runs on the same thread,
Also there are replication history put triggers that could also not run because you are preventing them and that will lead to missing replication history that could lead to conflicts and/or missing data on the destination (not only deletions).

ZNS

unread,
Apr 23, 2017, 1:25:20 PM4/23/17
to RavenDB - 2nd generation document database

By default do you mean the system database?

ZNS

unread,
Apr 23, 2017, 1:53:40 PM4/23/17
to RavenDB - 2nd generation document database

The test I've written does succeed, but I still think this might be an issue, since it depends on which thread triggers are run on. So I'll remove the disabling from our triggers and check that we don't trigger triggers infinitely. Next issue is with documents on the slave server which should have been deleted. Are these safe to simply delete? What happens for example if a document is put back on the master with the same id as a previously deleted one?

Oren Eini (Ayende Rahien)

unread,
Apr 24, 2017, 1:14:58 AM4/24/17
to ravendb
If you restore the same document id, it will cause a conflict on replication.
You can set the slave to unconditionally accept writes from the master.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

ZNS

unread,
Apr 24, 2017, 4:32:29 AM4/24/17
to RavenDB - 2nd generation document database

Is this by using the conflict resolution setting? We've set it to "Latest" which I guess should work.

Oren Eini (Ayende Rahien)

unread,
Apr 24, 2017, 4:39:37 AM4/24/17
to ravendb
That would work, yes
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages