SQLReplication gets stale

[Robert Edin]

Hi,

I have struggled with SQLReplication that gets stale.

Now I think I see a pattern that might be it.

Scenario:

My client program has written new docs to raven and they are replicated to SQL for a period.

After restarting the client and the new instance is storing new docs to raven SQL replication stops.

Resetting the replication, restating raven server etc  does not help.

If I empty the SQL DB, documents stored before the restart of the client writing raven are replicated again, but non of the new.

I notice the following:

The replication status doc has the following

"LastDocEtag": "01000000-0000-0008-0000-000000002F12"

And the documents that are not replicated has:

"LastDocEtag": "01000000-0000-0009-0000-000000xxxxxx"

Could it maybe be the case that higher bits are not honored when comparing etags?

BR, Robert

[Robert Edin]

Forgot to say that I use v2596

[Oren Eini (Ayende Rahien)]

No, that should most certainly work.

What does the log say about this?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Edin]

Log attached.

[Oren Eini (Ayende Rahien)]

We just fixed a similar problem, can you test with the latest codebase?

[Robert Edin]

Build 2598 seems to have fixed this :)

[Robert Edin]

Unstable behaviour.... it is not fixed yet :(

The observation is repeated. The replication stops at a "border" beween 

two different runs by the client generating docs.

I reset the SQL-replication, and the I can watch the Raven/SqlReplication/Status

get updated when it rescan documents until it stops exactly where it did

before the reset. And this point is a mentioned below when the third group

in the etag is changed. 

But wait!!!  In the log I find this:

Raven.Storage.Esent.StorageActions.DocumentStorageActions, Document with key '01000000-0000-0006-0000-000000000321' was not found

 

But this document exist!!! Or at least a document with this etag. It is the doc at the point where replication stopped!

A littlle later in the log Raven.Storage.Esent.StorageActions.DocumentStorageActions also claims that

various documents with names like GUID, dateTimeOffset and partial names of existing document names are missing. Ughhh...

[Oren Eini (Ayende Rahien)]

Document with that KEY doesn't exists. That doesn't mean anything about the etag.

But it is interesting that something is search for etag by key. It shouldn't.

Can we meet tomorrow and check this again?

[Robert Edin]

I save the state and Skype you tomorrow. What time is good for u?

2013/6/13 Oren Eini (Ayende Rahien) <aye...@ayende.com>

--
You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/Y8APfNWMYB0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

9 AM GMT+2

[Robert Edin]

I will try to skype you again on monday.

When I look at the release note for RC 2603 it looks like you have done work that may address this, right?

[Oren Eini (Ayende Rahien)]

Yes

[Robert Edin]

I have tried v2605 today.

It looks really promising. The SQL replication seems much more stable now.

Interestingly my problem with "too many steps in script" seems also been solved. 

The only problem I had was that the server throws out-of-memory exceptions after 

running with replication on while writing new docs to the db.

This PC is only a 32-bit with lots of stuff running, so memory is limited, but Raven has never thrown out-of-memory before.

[Oren Eini (Ayende Rahien)]

Hm,

Can you get a dump of it when it is using a lot of memory?

[Robert Edin]

The DMP file is waiting in Skype. I'm behind a FW that locks me out from most kind of file sharing.

[Oren Eini (Ayende Rahien)]

I need you to also send c:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll

[Robert Edin]

The file is waiting in Skype

[Robert Edin]

The file is still  waiting in Skype

Den tisdagen den 18:e juni 2013 kl. 10:28:40 UTC+2 skrev Oren Eini:

[Oren Eini (Ayende Rahien)]

I am not at the office, and I downloaded it there.

Can you have the two files available in a place I can download them?

[Robert Edin]

A cellphone with unlimited data plan bypasses the f***** FW....

https://dl.dropboxusercontent.com/u/1894671/mscordacwks.dll

https://dl.dropboxusercontent.com/u/1894671/Raven.Server.DMP.zip

[Kijana Woodard]

Security. Can't do your job, but everything is secure. :-D

[Oren Eini (Ayende Rahien)]

How many replication tasks do you have?

How complex are the scripts?

[Oren Eini (Ayende Rahien)]

I was able to figure out that those are the things that took the most mem

802,195 System.Collections.Generic.LinkedListNode<Jint.Expressions.Statement>
890,727 Jint.PropertyBags.MiniCachedPropertyBag
890,727 Jint.DictionaryPropertyBag
890,727 System.Collections.Generic.Dictionary<System.String,Jint.Native.Descriptor>


890,731 System.Collections.Generic.Dictionary+Entry<System.String,Jint.Native.Descriptor>
912,212 System.Int64
1,065,529       Jint.Expressions.Identifier
1,311,553       Jint.Native.ValueDescriptor
1,584,252       System.Int32[]


1,764,324       Jint.Debugger.SourceCodeDescriptor
2,060,804       Raven.Json.Linq.RavenJValue
2,322,465       System.String
3,528,648       Jint.Debugger.SourceCodeDescriptor+Location

the problem is that I look at the code and I can't figure out how this can happen. We don't keep those things around.

I guess that this might be just stuff that the GC didn't have time to cleanup yet, or that the heap is really fragemented, but that shouldn't happen anyway.

Do you have very complex scripts? In general, we should have about 20 instances of a single script, which doesn't account for those values.

[Robert Edin]

Two scripts. They are not trivial, but not very complex. I can send them tomorrow.

[Oren Eini (Ayende Rahien)]

Can you send me a db export?

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Edin]

I set up a "kit" for you tomorrow.

You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/Y8APfNWMYB0/unsubscribe.

[Oren Eini (Ayende Rahien)]

thanks

[Robert Edin]

Details mailed to aye...@ayende.com

thanks

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "ravendb" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/Y8APfNWMYB0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Oren Eini (Ayende Rahien)]

Thanks, I am in a conference now, may take a while longer to go over it.

[Robert Edin]

Enjoy your conference :)