RavenDB 4.0

[Derek den Haas]

Tried to import a 3.0 database using Export / Import documents only;

Database on 3.0 in size: 1GB with doc count: 240.000 (containing indexes!)

Database on 4.0 > 12 GB with doc count: 78.000

Dump itself is around 40mb

Is this expected? Or should I never try to load a ravendump from 3.x to 4.0 (since you've stated somewhere that your new update process would probably mean to upgrade the database using a script).

P.s. I tried the Linux build on Ubuntu (Debian gave an error on libstdc++ (think it was looking at the wrong directory, didn't investigate further), 

might be helpfull to add settings_posix.json, since you aren't using the other config file. Luckely this file is floating around in your Git.

[Oren Eini (Ayende Rahien)]

No, it shouldn't be this big.

We are taking some space ahead of time, and indexes have a minimum size of 16MB now, but what you are talking about isn't really expected.

In the DB Status, you have Storage details, can you look at that?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Derek den Haas]

It was now only containing 40.000 documents, so used a little less space. You can have the ravendump itself, it happens all the time; can send it to you by skype/mail/etc.

Op maandag 19 december 2016 07:17:05 UTC+1 schreef Oren Eini:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

We'll be grateful if you can send it to sup...@ravendb.net.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Derek den Haas]

Sent to you by mail. Good luck finding the solution.

P.s. we are also rewriting our codebase to learn from our experience using RavenDB, which should be finished around summer next year. Would it be wise to start with RavenDB 4.0, or is that one more than a year away from being released (I know you must hate this kind of ETA questions). Since I'm better in mastering Linux above Windows (ok I simply hate Windows (seen to much sh*t, though the current state is much better), might consider the new win server 2016 core? image without GUI), I will also revamp our servers to linux, so if you want to say anything about it, it would be highly appreciated.

Also interested in your opinion to the current state of .net CORE on linux, is it hanging together by bits and pieces or is it in a kind of solid state (since they are still talking about "preview-3")

Op maandag 19 december 2016 10:19:02 UTC+1 schreef Oren Eini:

[Oren Eini (Ayende Rahien)]

Thanks, I reproduced this. It looks like for some reason a document with size of ~4KB text was transformed into size 256KB.

The problem seems to be that you have a lot of unique property names.

It looks like we are writing all the unique property names, across the entire batch, in each document metadata.

This has been fixed. For reference, here is the fix:

https://github.com/ravendb/ravendb/commit/fd4539d1b134ee7b403c4aeb3b9bb470dfae6a3d

.NET Core on Linux seems to be very stable from our point of view. In other words, so far pretty much all of the crashes we have seen have been our fault :-)

The current plan is to have 4.0 release in about 6 months or so.

After this fix, importing our data results in:

{

• $type: "Raven.Server.Smuggler.Documents.Data.ImportResult, Raven.Server",
• DocumentsCount: 264,782,
• RevisionDocumentsCount: 0,
• IndexesCount: 0,
• TransformersCount: 0,
• IdentitiesCount: 9,
• Warnings: [ ],
• Message: null,
• ElapsedMilliseconds: 429,772,
• Elapsed: "00:07:09.7725354"

}

The size of the DB is 2 GB, the reason for that is that you have > 100 collections, and we reserve some space per collection (IIRC, that is about 2 - 8 MB by default) whic is held so documents of the same collections will be physically together.

[Oren Eini (Ayende Rahien)]

Okay, this had to do with the way we reserve space for collections. We assume that you have a relatively few collections with large number of docs.

Your scenario is having a lot of very small collections.

The current time is about a minute and a half, and ~750MB or so on release.

{

• $type: "Raven.Server.Smuggler.Documents.Data.ImportResult, Raven.Server",
• DocumentsCount: 264782,

• RevisionDocumentsCount: 0,
• IndexesCount: 0,
• TransformersCount: 0,
• IdentitiesCount: 9,
• Warnings: [ ],
• Message: null,

• ElapsedMilliseconds: 82900,
• Elapsed: "00:01:22.9000333"

}

[Derek den Haas]

It's an unfilled database, most collections will grow over time

- Purchases

- DistributedPurchases

- Debtors

- Suppliers

- Etc. etc.

Great to hear it's fixed, hope to see a build in the near future, might throw some other collections to it and see if it'll stick. 

Btw, did Raven/DocumentsByEntityName die?

Thanks about your opinion to Linux (and dotnet stability), still don't know why I want to switch to Linux (must be old memories about AD and DNS problems (and therefore AD problems) that i'm alwasy eager to return to Linux). And ofcourse your planning of RavenDB 4, which fits our release schedule :).

Op dinsdag 20 december 2016 12:27:17 UTC+1 schreef Oren Eini:

[Oren Eini (Ayende Rahien)]

Yes, we killed Raven/DocumentsByEntityName in favor of a better alternative.

And we'll have a build this week

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Okay, for reference, this allowed us to find several different performance issues, thanks for that.

Importing the data now results in a db whose size is 766.99 mb and takes 25 seconds.

[Derek den Haas]

Same file, new build (40006) will give a 5 gb database, just fyi. EntityMappings is the largest collection (3,56gb)

Op woensdag 21 december 2016 11:51:40 UTC+1 schreef Oren Eini:

[Derek den Haas]

Small other thing, wanted to delete this collection, though select complete collection not yet implemented and delete all results by index is giving: (might be expected, but so you know)

There is no handler for path: DELETE /databases/EasyFlor-New/bulk_docs/EntityMappings?query=Id%3AEntityMappings%2F*&pageSize=128&allowStale=false

Op vrijdag 13 januari 2017 12:18:55 UTC+1 schreef Derek den Haas:

[Oren Eini (Ayende Rahien)]

This isn't working yet, so yes, expected.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Are you sure?

We are talking about:

Dump of EasyFlor-New, 2016-12-18 23-35

File size 39.5MB

This is what it looks like on my end:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Derek den Haas]

Didn't had that dump anymore, created a new one (same database), which is 40.5 MB, should only contain some minor new documents (so I thought).

I've just send it to the previous given mail address (support)

Op zondag 15 januari 2017 13:39:11 UTC+1 schreef Oren Eini:

[Oren Eini (Ayende Rahien)]

264,873 docs

[Derek den Haas]

I'll try to get the same results, was using your new build on a linux environment. I'll give it a spin later today or tomorrow (depends on when I got my new test environment to work)

Op dinsdag 17 januari 2017 08:58:40 UTC+1 schreef Oren Eini:

[Oren Eini (Ayende Rahien)]

I'm testing that on Windows, mind.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Shouldn't be any difference, though