I have a weird situation ... I am beginning to test Sharding and Replication for when we are ready to go live with a new application which should be shortly after 2.0 is released.
I have 1 server 4 databases ... 2 Databases are set up Shards, 2 are replicated instances of each of the 2 Shards. Replication is set to go both ways, with Changes Only.
I persisted the indexes to all 4 database.
I have about 8 documents total spread out in 5 collections
I have 17 indexes total... About every 3 - 5 seconds.. it goes from 17 indexes stale... to 0 Indexes stale ... to 1 index stale (EntityByName) ... then 17 indexes stale... and the whole process repeats itself.. basically every 10 - 15 seconds it goes through that cycle.
I am wondering if I have a configuration incorrect or what. When I added the indexes it took a great amount of time when normally with a normal DocumentStore the indexes persist within only a second or 2...
Would I be better off creating different instances of RavenDB on different ports to test this configuration?
On Fri, Nov 16, 2012 at 12:10 AM, Troy <tzar...@gmail.com> wrote:
> I have a weird situation ... I am beginning to test Sharding and
> Replication for when we are ready to go live with a new application which
> should be shortly after 2.0 is released.
> I have 1 server 4 databases ... 2 Databases are set up Shards, 2 are
> replicated instances of each of the 2 Shards. Replication is set to go both
> ways, with Changes Only.
> I persisted the indexes to all 4 database.
> I have about 8 documents total spread out in 5 collections
> I have 17 indexes total... About every 3 - 5 seconds.. it goes from 17
> indexes stale... to 0 Indexes stale ... to 1 index stale (EntityByName) ...
> then 17 indexes stale... and the whole process repeats itself.. basically
> every 10 - 15 seconds it goes through that cycle.
> I am wondering if I have a configuration incorrect or what. When I added
> the indexes it took a great amount of time when normally with a normal
> DocumentStore the indexes persist within only a second or 2...
> Would I be better off creating different instances of RavenDB on different
> ports to test this configuration?
I put Fiddler on the server... and there is no traffic.
If I bring up Studio and look at a non-replicated database, there is not really any traffic ... I bring up in Studio one of the sharded databases... and I constantly see stats?noCache=xxx ... I assume sinece things are stale it might constantly be checking to see if the stale index number has changed for the statistics.
Is there a different way you wanted me to inspect the http traffic?
On Thursday, November 15, 2012 6:04:40 PM UTC-5, Oren Eini wrote:
> That is interesting, can you check what the http traffic to this looks > like?
> On Fri, Nov 16, 2012 at 12:10 AM, Troy <tza...@gmail.com <javascript:>>wrote:
>> I have a weird situation ... I am beginning to test Sharding and >> Replication for when we are ready to go live with a new application which >> should be shortly after 2.0 is released.
>> I have 1 server 4 databases ... 2 Databases are set up Shards, 2 are >> replicated instances of each of the 2 Shards. Replication is set to go both >> ways, with Changes Only.
>> I persisted the indexes to all 4 database.
>> I have about 8 documents total spread out in 5 collections
>> I have 17 indexes total... About every 3 - 5 seconds.. it goes from 17 >> indexes stale... to 0 Indexes stale ... to 1 index stale (EntityByName) ... >> then 17 indexes stale... and the whole process repeats itself.. basically >> every 10 - 15 seconds it goes through that cycle.
>> I am wondering if I have a configuration incorrect or what. When I added >> the indexes it took a great amount of time when normally with a normal >> DocumentStore the indexes persist within only a second or 2...
>> Would I be better off creating different instances of RavenDB on >> different ports to test this configuration?
The document called: "Raven/Replication/Sources/http://web01:8182/databases/Members.USA.R01" is CONSTANTLY being changed by the server... the etags are incrementing... and that causes the indexes to constantly go in and out of stale.
On Thursday, November 15, 2012 6:23:32 PM UTC-5, Troy wrote:
> I put Fiddler on the server... and there is no traffic.
> If I bring up Studio and look at a non-replicated database, there is not > really any traffic ... I bring up in Studio one of the sharded databases... > and I constantly see stats?noCache=xxx ... I assume sinece things are stale > it might constantly be checking to see if the stale index number has > changed for the statistics.
> Is there a different way you wanted me to inspect the http traffic?
On Thursday, November 15, 2012 9:02:13 PM UTC-5, Troy wrote:
> More info:
> The document called: "Raven/Replication/Sources/ > http://web01:8182/databases/Members.USA.R01" is CONSTANTLY being changed > by the server... the etags are incrementing... and that causes the indexes > to constantly go in and out of stale.
> On Thursday, November 15, 2012 6:23:32 PM UTC-5, Troy wrote:
>> I put Fiddler on the server... and there is no traffic.
>> If I bring up Studio and look at a non-replicated database, there is not >> really any traffic ... I bring up in Studio one of the sharded databases... >> and I constantly see stats?noCache=xxx ... I assume sinece things are stale >> it might constantly be checking to see if the stale index number has >> changed for the statistics.
>> Is there a different way you wanted me to inspect the http traffic?
On Fri, Nov 16, 2012 at 4:02 AM, Troy <tzar...@gmail.com> wrote:
> More info:
> The document called: "Raven/Replication/Sources/http://web01:8182/databases/Members.USA.R01"
> is CONSTANTLY being changed by the server... the etags are incrementing...
> and that causes the indexes to constantly go in and out of stale.
> On Thursday, November 15, 2012 6:23:32 PM UTC-5, Troy wrote:
>> I put Fiddler on the server... and there is no traffic.
>> If I bring up Studio and look at a non-replicated database, there is not
>> really any traffic ... I bring up in Studio one of the sharded databases...
>> and I constantly see stats?noCache=xxx ... I assume sinece things are stale
>> it might constantly be checking to see if the stale index number has
>> changed for the statistics.
>> Is there a different way you wanted me to inspect the http traffic?
Vlad,
Actually, it won't be obvious to catch. Things _work_, it is just that we
have extra stuff happening.
We run an extensive series of tests / workout for a lot of the things that
we do, but we don't run them for every build.
Only on stable / RC builds.
On Fri, Nov 16, 2012 at 7:49 PM, Vlad K <eqplug...@gmail.com> wrote:
> Once I added data/indexes to db I got the same issue as mentioned here.
> Happy it's fixed in 45.
> This brings us to a question - are there actually any tests for being run
> for replication bundle because this issue should've been obvious to catch?
On Friday, November 16, 2012 1:32:14 PM UTC-5, Oren Eini wrote:
> Vlad, > Actually, it won't be obvious to catch. Things _work_, it is just that we > have extra stuff happening. > We run an extensive series of tests / workout for a lot of the things that > we do, but we don't run them for every build. > Only on stable / RC builds.
> On Fri, Nov 16, 2012 at 7:49 PM, Vlad K <eqpl...@gmail.com <javascript:>>wrote:
>> Once I added data/indexes to db I got the same issue as mentioned here. >> Happy it's fixed in 45.
>> This brings us to a question - are there actually any tests for being run >> for replication bundle because this issue should've been obvious to catch?
They don't.
The problem was something else, it was that we updated the _etag_, and due
to another issue, we needed to update what the last etag is even if we
don't replicate.
Unfortunately, this resulted in what is effectively a distributed infinite
loop, only when you have master/ master relations.
That is actually quite hard to notice, because the end result is that
everything still works, it is just that the system keeps working.
On Fri, Nov 16, 2012 at 9:14 PM, Vlad K <eqplug...@gmail.com> wrote:
> Fair enough, I just figured you'd have a test to make sure that system
> documents don't take part in document replication.
> On Friday, November 16, 2012 1:32:14 PM UTC-5, Oren Eini wrote:
>> Vlad,
>> Actually, it won't be obvious to catch. Things _work_, it is just that we
>> have extra stuff happening.
>> We run an extensive series of tests / workout for a lot of the things
>> that we do, but we don't run them for every build.
>> Only on stable / RC builds.
>> On Fri, Nov 16, 2012 at 7:49 PM, Vlad K <eqpl...@gmail.com> wrote:
>>> Once I added data/indexes to db I got the same issue as mentioned here.
>>> Happy it's fixed in 45.
>>> This brings us to a question - are there actually any tests for being
>>> run for replication bundle because this issue should've been obvious to
>>> catch?
