Database Performance and Maintenance

[Chris]

Is there any maintenance, like defrag that should be run on a Raven
data file? It seems like inserts are getting slower. I'm not sure if
it's the growing size of the DB or the upgrade from 573 to 616, but
inserting 200,000 documents used to take <30 seconds and now it's a
couple of minutes.

I have also grown the number of indexes from 2 to 5, but they are
still set to lazy update. I broke up one index that covered two entity
types into two separate indexes. I found no difference in query
performance, so would having two be any slower to update than the one?

[Oren Eini (Ayende Rahien)]

inline

On Tue, Jan 31, 2012 at 4:21 PM, Chris <cdh...@gmail.com> wrote:

Is there any maintenance, like defrag that should be run on a Raven
data file? It seems like inserts are getting slower. I'm not sure if
it's the growing size of the DB or the upgrade from 573 to 616, but
inserting 200,000 documents used to take <30 seconds and now it's a
couple of minutes.

We changed the default size of the hilo generator, which can be the cause of that slowdown.

If you really want, you can execute esentutl /d on the database, to force a defragement (not it has to be off), but it isn't likely, I would assume the hilo max size has more to do with it.

 

I have also grown the number of indexes from 2 to 5, but they are
still set to lazy update. I broke up one index that covered two entity
types into two separate indexes. I found no difference in query
performance, so would having two be any slower to update than the one?

Doesn't matter in query time, slightly more expensive at indexing time, but you probably don't care.

[Chris]

I attempted the same volume of inserts into a fresh database and it
took several minutes. It does seem that changes in 616 are causing
write performance issues.

So far, in my testing, I'm seeing a 10x increase in time to create a
batch of new documents.

Is that in line with your expectations with the new hilo generator?

On Jan 31, 9:34 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Chris H.]

I deleted all data and started fresh to get rid of the dup Default database which I think was causing problems. 

A subsequent import of over 800,000 documents took a little over 13 minutes which is inline with what I originally saw on 573. Maybe a little slower, but not 10x.

Thanks, Oren.

[Oren Eini (Ayende Rahien)]

Can you reproduce the slow inserts? Is there anything that you can think that would be causing this?

If you care to do a big import job, you might want to do this:

var multiTypeHiLoKeyGenerator = new MultiTypeHiLoKeyGenerator(store, 1024);

store.Conventions.DocumentKeyGenerator = o => multiTypeHiLoKeyGenerator.GenerateDocumentKey(store.Conventions, o);

That will make it go to the DB only once every 1024, not every 32, as is the default.

[Chris H.]

The HiLoKeyGenerator code seems to speed things up a good bit. From
around 61k/min to 100k/min. I actually set it to 10240 since most
operations are on over one hundred-thousand documents at a time. Do
you have any advice on the use/abuse of this method?

I've dropped and restarted populating the database from scratch,
taking note of metrics this time around, so I'll report back when I
get back up to around 2 million docs if I see any changes.

On Feb 1, 6:31 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

Chris,

We set it to 32 mostly because we want to avoid skipping to high whenever you restart the app.

For bulk scenarios, set it as high as you want, sure.

[Chris Marisic]

For doing the bulk insert, is it only possible to edit the document store conventions as opposed to changing things on a specific session?

It seems reasonable that you would want to be able to check out a big block of keys for doing a bulk load of 10K+ 100K+ things without it impacting every other session.

[Oren Eini (Ayende Rahien)]

Chris,

Create your own HiLoGenerator, and explicitly assign the ids, it will work.

[Chris Marisic]

Excellent, bookmarked this post.

I could definitely see why very big bulk imports would be slower if it constantly has to poll the server for new hi values.

[Chris H.]

So, if I have one class that creates a new Document store, but two
separate import methods that could use different KeyGenerators, will
the separate sessions use the Conventions configured when the session
is created or do I need separate DocumentStore instances for each
KeyGenerator configuration?

Also, I changed to using Async calls and now I get this error. Worked
fine synchronously. Is there any other consideration besides changing
the session type and save changes method calls? It only occurs a
couple times in the thousand calls to StoreChangesAsync.

System.InvalidOperationException: Collection was modified; enumeration
operation may not execute.
at
System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()
at Raven.Json.Linq.DictionaryWithParentSnapshot.get_Keys() in c:
\Builds\raven\Raven.Abstractions\Json\Linq\Dictionary
WithParentSnapshot.cs:line 86
at Raven.Json.Linq.DictionaryWithParentSnapshot.get_Count() in c:
\Builds\raven\Raven.Abstractions\Json\Linq\Dictionar
yWithParentSnapshot.cs:line 236
at Raven.Json.Linq.RavenJObject.get_Count() in c:\Builds\raven
\Raven.Abstractions\Json\Linq\RavenJObject.cs:line 30
at Raven.Json.Linq.RavenJToken.DeepEquals(RavenJToken other) in c:
\Builds\raven\Raven.Abstractions\Json\Linq\RavenJTo
ken.cs:line 266
at Raven.Json.Linq.RavenJToken.DeepEquals(RavenJToken t1,
RavenJToken t2) in c:\Builds\raven\Raven.Abstractions\Json\
Linq\RavenJToken.cs:line 220
at
Raven.Client.Document.InMemoryDocumentSessionOperations.EntityChanged(Object
entity, DocumentMetadata documentMeta
data) in c:\Builds\raven\Raven.Client.Lightweight\Document
\InMemoryDocumentSessionOperations.cs:line 904
at
Raven.Client.Document.InMemoryDocumentSessionOperations.<PrepareForEntitiesPuts>b__18(KeyValuePair`2
pair) in c:\B
uilds\raven\Raven.Client.Lightweight\Document
\InMemoryDocumentSessionOperations.cs:line 783
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at
Raven.Client.Document.InMemoryDocumentSessionOperations.PrepareForEntitiesPuts(SaveChangesData
result) in c:\Build
s\raven\Raven.Client.Lightweight\Document
\InMemoryDocumentSessionOperations.cs:line 783
at
Raven.Client.Document.InMemoryDocumentSessionOperations.PrepareForSaveChanges()
in c:\Builds\raven\Raven.Client.Li
ghtweight\Document\InMemoryDocumentSessionOperations.cs:line 776
at
Raven.Client.Document.Async.AsyncDocumentSession.SaveChangesAsync() in
c:\Builds\raven\Raven.Client.Lightweight\Do
cument\Async\AsyncDocumentSession.cs:line 207
at Vovici.CMDB.Helpers.MTALogImporter.ImportProcessLog(String path)
in C:\perforce\netops\learning\Vovici.CMDB.Helper
s\Vovici.CMDB.Helpers\MTALogImporter.cs:line 146 SaveChanges exception
caught at record 5120.

On Feb 2, 12:42 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>

wrote:
> Chris,
> Create your own HiLoGenerator, and explicitly assign the ids, it will work.
>
>
>
>
>
>
>
> On Thu, Feb 2, 2012 at 5:55 PM, Chris Marisic <ch...@marisic.com> wrote:
> > For doing the bulk insert, is it only possible to edit the document store
> > conventions as opposed to changing things on a specific session?
>
> > It seems reasonable that you would want to be able to check out a big
> > block of keys for doing a bulk load of 10K+ 100K+ things without it
> > impacting every other session.
>
> > On Thursday, February 2, 2012 10:24:41 AM UTC-5, Chris H. wrote:
>
> >> The HiLoKeyGenerator code seems to speed things up a good bit. From
> >> around 61k/min to 100k/min. I actually set it to 10240 since most
> >> operations are on over one hundred-thousand documents at a time. Do
> >> you have any advice on the use/abuse of this method?
>
> >> I've dropped and restarted populating the database from scratch,
> >> taking note of metrics this time around, so I'll report back when I
> >> get back up to around 2 million docs if I see any changes.
>
> >> On Feb 1, 6:31 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
> >> wrote:
> >> > Can you reproduce the slow inserts? Is there anything that you can
> >> think
> >> > that would be causing this?
>
> >> > If you care to do a big import job, you might want to do this:
>

> >> > var multiTypeHiLoKeyGenerator = new MultiTypeHiLoKeyGenerator(**store,
> >> 1024);
> >> > store.Conventions.**DocumentKeyGenerator = o =>
> >> > multiTypeHiLoKeyGenerator.**GenerateDocumentKey(store.**Conventions,

[Oren Eini (Ayende Rahien)]

Chris,

I don't understand the 1st question, and it looks like you are actually working with the session while it is making an async operation.

That is not allowed.

[Chris H.]

Yes, the session was persistent. I updated to to put each batch in one
session and that got me about a 10x improvement on performance over
the basic SaveChanges.

Thanks again.

On Feb 3, 1:00 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:
> Chris,

> I don't understand the 1st question, and it looks like you are actually

> working with the session *while* it is making an async operation.

> > Raven.Client.Document.InMemoryDocumentSessionOperations.<PrepareForEntities Puts>b__18(KeyValuePair`2

[Oren Eini (Ayende Rahien)]

For your information, this will auto tune itself starting in build 640

[Chris H.]

What will auto-tune? The hilo generator?

On Feb 5, 11:31 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>

[Oren Eini (Ayende Rahien)]

Yes