RavenDB 3.0 internal replication

Derek den Haas

unread,

May 15, 2014, 4:13:01 AM5/15/14

to rav...@googlegroups.com

I want to distribute global documents to other databases, for example, in my set-up, all users will have the same "Companies" and "Articles" to begin with.

It's a normal replication, but RavenDB is fully working and only sending this:

Request #1,165: GET - 0 ms - EasyFlor-Dev77 - 200 - http://192.168.11.1:

8081/databases/EasyFlor-Dev77/replication/lastEtag?from=http:%2F%2Ftransip-vps:8

081%2Fdatabases%2FEasyFlor&currentEtag=01000000-0000-0002-0000-00000000075F&dbid

=1abdefba-3855-4cc4-bd9d-46a184c6fc1e

Request #1,166: GET - 0 ms - EasyFlor-Dev77 - 200 - http://192.168.11.1:

8081/databases/EasyFlor-Dev77/replication/lastEtag?from=http:%2F%2Ftransip-vps:8

081%2Fdatabases%2FEasyFlor&currentEtag=01000000-0000-0002-0000-00000000075F&dbid

=1abdefba-3855-4cc4-bd9d-46a184c6fc1e

Request #1,167: GET - 0 ms - EasyFlor-Dev77 - 200 - http://192.168.11.1:

8081/databases/EasyFlor-Dev77/replication/lastEtag?from=http:%2F%2Ftransip-vps:8

081%2Fdatabases%2FEasyFlor&currentEtag=01000000-0000-0002-0000-00000000075F&dbid

=1abdefba-3855-4cc4-bd9d-46a184c6fc1e

Request #1,168: GET - 0 ms - EasyFlor-Dev77 - 200 - http://192.168.11.1:

8081/databases/EasyFlor-Dev77/replication/lastEtag?from=http:%2F%2Ftransip-vps:8

081%2Fdatabases%2FEasyFlor&currentEtag=01000000-0000-0002-0000-00000000075F&dbid

=1abdefba-3855-4cc4-bd9d-46a184c6fc1e

(same etag over and over again and no documents are replicating)

So this is from RavenDB 3.0 to the same RavenDB 3.0 instance (other database) #3219.

If you don't want these errors since you are still in development, I will only log them internally and check them when we consider migrating to RavenDB 3.0.

Derek den Haas

unread,

May 15, 2014, 4:18:29 AM5/15/14

to rav...@googlegroups.com

Some extra information:
2 indexes (without loaddocument, excluding the 2 ravenDB indexes of conflicts and "allDocuments")

200.000 documents (3kB on average)

Op donderdag 15 mei 2014 10:13:01 UTC+2 schreef Derek den Haas:

Oren Eini (Ayende Rahien)

unread,

May 15, 2014, 10:21:50 AM5/15/14

to ravendb

Can you explain the setup fully?

You have 1 RavenDB 3.0 server.

There are 2 (or more?) dbs on it.

All of them with Replication enabled.

You have setup replication between one of those databases to the other(s).

However, you aren't see those documents show in the other db?

Did you replicate the documents to the source db?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office: + 972-4-622-7811

Fax: + 972-153-4622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Derek den Haas

unread,

May 16, 2014, 2:44:35 AM5/16/14

to rav...@googlegroups.com

I haven't replicated the documents to the source db, because that will mean I have to set "changed and replicated".

Setup source db:

"Raven/ActiveBundles": "Compression;DocumentExpiration;Replication;PeriodicBackups"

Setup destination db:

"Raven/ActiveBundles": "Compression;DocumentExpiration;Replication;PeriodicBackups"

Source DB has 200.000 documents

Destination DB has 0 documents

SourceDB replication config:

{

"Destinations": [

{

"Url": "http://192.168.11.1:8081",

"Username": null,

"Password": null,

"Domain": null,

"ApiKey": null,

"Database": "DestinationDB",

"TransitiveReplicationBehavior": "None",

"IgnoredClient": false,

"Disabled": false,

"ClientVisibleUrl": null

}

],

"Source": "1abdefba-3855-4cc4-bd9d-46a184c6fc1e"

}

That's about it. This will eat 100% CPU and is still trying to get the same batch of items and never a document will get replicated.

Another issue, one of my indexes isn't indexing:

The index (stale, shouldn't contain ANY items at all...):

docs.EntityAttachmentMappings.Select(result => new {

Id = result.__document_id,

URL = result.URL

})

and I only have one other index (this one is working):

docs.Colors.Select(result => new {

Id = result.__document_id,

Name = result.Name["all"],

RHSCode = result.RHSCode,

Query = new string[] {

result.Name["all"],

result.RHSCode

}

})

The only steps I've done:

- Create an all new RavenDB environment, and copy Essent dll to the Server/ dir

- Added 2 dbs (Source and ReplicationDB)

- Insert 200.000 documents using BulkInsert (20 different entity types)

- Added 2 extra indexes (just to test if it's indexing properly)

- Added Replication to another database (on the same RavenDB instance)

I got the following issues:

- Indexing seems stuck (it says "starting" all the time and isn't disabled)

-- No errors in any log whatsoever

- 1 index is fully indexed, another is stuck

- replication not working at all, just eating all CPU and doing the same thing over and over again

Op donderdag 15 mei 2014 16:21:50 UTC+2 schreef Oren Eini:

Derek den Haas

unread,

May 16, 2014, 2:47:08 AM5/16/14

to rav...@googlegroups.com

PS raven is doing something:

Found a total of 0 documents that requires indexing since etag: 00000000-0000-0000-0000-000000000000: ()

every minute or so...

Op donderdag 15 mei 2014 10:13:01 UTC+2 schreef Derek den Haas:

I want to distribute global documents to other databases, for example, in my set-up, all users will have the same "Companies" and "Articles" to begin with.

Oren Eini (Ayende Rahien)

unread,

May 16, 2014, 2:53:18 AM5/16/14

to ravendb

What build are you using?

What do you copy the esent dll?

Derek den Haas

unread,

May 16, 2014, 10:42:16 AM5/16/14

to rav...@googlegroups.com

3219, Asseblies/ wasn't linked to RavenDB. Setting that DLL in the parent folder (/Server/) was a workaround to make RavenDB work.

But if you've fixed a lot of things in a more recent build, I can check that version.

Op vrijdag 16 mei 2014 08:53:18 UTC+2 schreef Oren Eini:

Oren Eini (Ayende Rahien)

unread,

May 16, 2014, 6:52:42 PM5/16/14

to ravendb

Yes, please try the latest.

Derek den Haas

unread,

May 17, 2014, 8:57:58 AM5/17/14

to rav...@googlegroups.com

#3222:

Created a database:

Could not finalize request properly Compilation Errors:

Line 8, Position 18: Error CS0234 - The type or namespace name 'Documents' does

not exist in the namespace 'Lucene.Net' (are you missing an assembly reference?)

So removed the databases again, and put the file Lucene.Net.Dll to the parent folder.

Tested documents startsWith:

session.Advanced.Stream<T>(startsWith: "EntityMappings/Articles/") is failing for large result sets, returning 0 documents (no errors whatsoever).

Raven/DocumentsByEntityName

__document_id: EntityMappings/VBNArticle/*

Will give me 40.000 results.

Smaller sets of data will give me the documents using startsWith.

Also tried:

var query = provider.GetCommands().StreamDocs(startsWith: id);

while(query.MoveNext())

{

// Fetch metadata

var metadata = (RavenJObject)query.Current["@metadata"];

// Serialize document

var document = query.Current.JsonDeserialization<T>();

// Add the Id to the property (if property exists)

var documentIdProperty = document.GetType().GetProperty("Id");

if(documentIdProperty != null)

documentIdProperty.SetValue(document, metadata["@id"].Value<string>());

yield return document;

}

Isn't working as well... and I don't want to rely on the index, since that one might be stale.

Op zaterdag 17 mei 2014 00:52:42 UTC+2 schreef Oren Eini:

Derek den Haas

unread,

May 17, 2014, 9:04:42 AM5/17/14

to rav...@googlegroups.com

And replication not working; this time it's doing something else:

Request #7,157: GET - 2 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182755

Request #7,158: GET - 2 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182756

Request #7,159: GET - 2 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182757

Request #7,160: GET - 1 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182758

Request #7,161: GET - 1 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182759

Request #7,162: GET - 1 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182760

Request #7,163: GET - 1 ms - - 200 - http://192.168.11.1:8081

/databases?pageSize=1024&_=1400328182761

And that's all; no replication stats.

On restart of RavenDB same error as previous, it's doing something, but don't know what. Not adding any documents.

Op donderdag 15 mei 2014 10:13:01 UTC+2 schreef Derek den Haas:

I want to distribute global documents to other databases, for example, in my set-up, all users will have the same "Companies" and "Articles" to begin with.

Michael Yarichuk

unread,

May 18, 2014, 10:55:15 AM5/18/14

to rav...@googlegroups.com

Hi,

I've checked the issues with RavenDB build 3223, and found:

a) replication worked correctly on 200k docs, between two instances of this build

b) streaming with startsWith works correctly on 200k (tried that with attached unit test)

c) index with the following definition also works - tried it on the generated data from the unit test

docs.Articles.Select(doc => new{

Id = doc.__document_id,

Name = doc.Name

});

Can you test with the latest build of RavenDB and tell if you are still seeing those issues?

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

Rhino-Logo

StreamStartsWithTests.cs

Derek den Haas

unread,

May 18, 2014, 12:09:37 PM5/18/14

to rav...@googlegroups.com

a) I use the same RavenDB server on the same port, but replicating from one database to another.

- Still not working

b) I try to create a failing test, because on some names it's failing, and on some it isn't.

By the way, it's always failing on the same "startingWith" names, and always working on other names, so it's quite strange.

For example:

EntityMappings/VBNArticles/ won't work, there are 40.000 documents in this one, but non will return

EntityMappings/Categories/ is working, only 30 documents

EntityMappings/CompanyRoles/ isn't working, only 4 documents

on just exporting those 4 documents and insert them in another database, it is working

on cleaning RavenDB, set up a new database, and try it again, the same will fail again... so it's quite consistant, but it's pretty strange. I hope I have some time tomorrow to create a failing test for you. Else I can always give you credentials to the database.

c) well the ravenIndex is working. __document_id: EntityMappings/VBNArticles/* will give me 40.000 documents, but startingWith gives me 0 documents :/

Op zondag 18 mei 2014 16:55:15 UTC+2 schreef Michael Yarichuk:

Michael Yarichuk

unread,

May 18, 2014, 12:22:18 PM5/18/14

to rav...@googlegroups.com

Interesting. Can you share dump of the data? I would like to reproduce this on my machine.

Derek den Haas

unread,

May 18, 2014, 1:14:55 PM5/18/14

to rav...@googlegroups.com

I will create a dump tomorrow, just deleted the test DB (including the database folder).

Op zondag 18 mei 2014 18:22:18 UTC+2 schreef Michael Yarichuk:

Derek den Haas

unread,

May 21, 2014, 2:31:48 AM5/21/14

to rav...@googlegroups.com

Got one on 3227:

FAILING:

StartsWith:

EntityMappings/PackagingSort/

WORKING

Index Raven/DocumentsById:

__document_id:EntityMappings/PackagingSort/*

As you can see, hopefully, startswith is failing to find any documents using that prefix (but will find loads of others!), while the index is finding the documents.

EntityMappings/VBNArticle/ for example is working...

I will send the database to you by wetransfer, to your given mail address. Please let me know your findings.

Op zondag 18 mei 2014 19:14:55 UTC+2 schreef Derek den Haas:

Derek den Haas

unread,

May 21, 2014, 2:39:57 AM5/21/14

to rav...@googlegroups.com

P.s. on a reboot of the RavenDB service, it's still forcible resetting some indexes (they are empty).

P.s.2.

Replication is still failing

From: 192.168.11.1:8081 Database: EasyFlor

To: 192.168.11.1:8081 Database: EasyFlor-Test

It will only sync the same Etag over and over again, using 99% CPU and never synchronize a single document.

Request # 389: GET - 0 ms - EasyFlor-Test - 200 - http://192.168.11.1:80

81/databases/EasyFlor-Test/replication/lastEtag?from=http:%2F%2Ftransip-vps:8081