[RavenDB] Governor on Query Results?

[jdn]

Rob Ashton suggested I bring this to the group (off of a comment I
made on his blog).

"Let’s say there are 100,000 books in the document store and we invoke
the following code:

1: Book[] books = documentSession.Query<Book>()
2: .ToArray();
How many books do you expect for there to be in that
collection?......RavenDB..... automatically limits the number of
results returned back. Both the .NET client and server have this
behaviour built into them and this means you’ll only get (at the
moment), 128 objects coming back for the above query. This is equally
true for all types of queries"

I can't even begin to believe this idea wasn't completely rejected out
of hand. Isn't this like saying "select * from myTable" in T-SQL
should have a governor built in because some idjit might issue it
against the transaction table that has 474 million rows?

Is this really built in? How could it be overriden besides fork the
source to get rid of this egregious design flaw?

Help, I'm confused.

TIA
jdn

[Ayende Rahien]

This is built in, yes.

The client set the page size to 128 by default, but you can change that if you want.

The server set the max size for paging at 1024, this isn't overridable.

If you really want that, I'll accept a patch that make this configurable, but there is going to be a default, and it is going to be small.

Some references:

http://nhprof.com/Learn/Alerts/UnboundedResultSet

http://relation.to/Bloggers/HandlingFailureWithHibernate

Especially:

http://ayende.com/Blog/archive/2007/11/04/Release-It.aspx

There is also a huge difference between a relational database and document database.

You don't really process that many documents and you do rows. Both because of the nature of the access patterns and because the sort of things you would do that are replaced by indexed.

[jdn]

With all due respect, all of those links are totally irrelevant, but
I'm glad you confirmed that I hadn't lost my mind that Ayende had gone
from running with scissors to tying his user's shoelaces.

How do you know what my use cases are? I have a financial services
client, and can easily imagine reasonable scenarios where 128 is
useless and 1024 is probably as well. I'm aware that there is a
difference between relation and document DB.

You are seriously crippling your product based on chapters from
'Release It!'? That is so disappointing (that's the mildest word I
can use). People make dumb mistakes, and good developers *learn* from
them.

I was really looking forward to RavenDB, but I don't feel like
spending extra money on anything that cripples *my* ability to do what
*I* want, when *I* want, with *my* software development. I have an
MSDN subscription that fulfills that, thank you. Surprised that you
would go this route. Are you making everything sealed while you are
at it?

I could just use the OSS version and rip out all that, but then I'd
have to remember to do that every single time an improved release came
out. Feh.

Consider building a Hibernating Rhinos RavenDB Profiler and include it
in the commercial license, instead of assuming your end users are
clueless children. Even if we are.

At least make your public API documentSession.CrippledQuery so that
they don't spend time wondering what the current magical numbers are
that prevent the code they are writing from doing what it should be
doing.

jdn

[Ayende Rahien]

Jdn,

May I suggest you will take some time to watch the stars, smell the flowers, or whatever you do that helps you relax?

No one died, no one is seriously injured.  Nothing is wrong.

I understand that you disagree quite strongly with this decision. If you feel so strongly about it, please read the 4th statement in my previous reply. 
I'll quote it again:

"If you really want that, I'll accept a patch that make this configurable, but there is going to be a default, and it is going to be small."

The code in question is located in: ravendb\Raven.Database\Server\Responders\HttpExtensions.cs

Line 208 - 209

[jdn]

LOL.

No one died when EFv1 was built so that you couldn't do POCO either
(and yes, I know, it is not from a technical perspective The Same
Thing (TM)). Etc. etc. etc.

It's your product. Hopefully, you'll make:
Book[] categoryBooks =
documentSession.Query<Book>("BookByCategory") .WaitForNonStaleResults() .Where(String.Format("Category:
{0}", category)).Take(129).OrderBy("Popularity").ToArray();
throw an exception instead of mysteriously failing to do what the API
says it is doing.

Maybe make it:
Book[] categoryBooks =
documentSession.Query<Book>("BookByCategory") .WaitForNonStaleResults() .Where(String.Format("Category:
{0}",
category)).Take(129).AndIReallyMeanIt.OrderBy("Popularity").ToArray();


jdn

[Ayende Rahien]

Jdn,

You code would just work.

[Ayende Rahien]

g/You/Your/s

[jdn]

Only if I knew that I had overridden the default.

IMHO, a public API should do exactly what it says it is doing. If I
mean take(129) (or take(1025) whichever value it is), then it should
do that. I shouldn't have to read the documentation to know that I'm
going to get a different result or that there even is a default that
changes the result of what a public API indicates it is doing.

Does this mean end users can do Bad Things(TM) if they don't know any
better? Yes. But the defaults are simply what you think are the
right defaults, and you don't know what even marginally intelligent
end users might want to do with various use cases that they know best
about.

Every product contains defaults, it isn't that you are including
defaults that makes me want to smell the stars or watch the flowers
(lol), it is that you are changing the semantics of an API call, and a
freaking Query API call to boot, to do something non-obvious, and all
because you think you need to prevent an end user from doing something
that could be harmful in a release it sort of way.

I don't need to read Release It to know calling ToArray or ToList or
ToIguana on a collection might not be performant, and I'm on the
dumber end of the end user scale.

jdn

[Ayende Rahien]

What I find most interesting is that instead of taking the time to do a three minute fix, the location of which I have already given you, so you don't even have to search for it, you seems content to rail at me the mailing list.

Send me a patch, or stop complaining.

[jdn]

sigh. Thanks for missing the point. I don't *want* a patch for this,
there shouldn't *be* a patch since the design is *wrong*.

I don't know enough about how the administration piece (if there is
such a think) works, to know how to change:

if (pageSize > 1024)
pageSize = 1024;

to include a default here, because I don't see where the 128 value
gets overriden (probably in another class, do you know which off-hand
to model it off of?)

But okay.

Peace, out.

jdn

[jdn]

To ignore the default I mean, that doesn't break the existing
semantics. However Rob overrode/ignored the existing default of 128.

jdn

[Ayende Rahien]

I disagree with you that the design is wrong. I strongly believe that active defaults are the best way to reach good software.

The 128 default is set in IndexQuery, line 11.

The configuration system for Raven is RavenConfiguration, which is accessible from the callers of GetPageSize using this expression: Database.Configuration

You need to wire a new property and make it load the value from the app settings.

You'll be able to find samples of how this is done in the RavenConfiguration constructor.

[Ayende Rahien]

Sorry, I am not following.

[jdn]

Active defaults appropriately placed. We disagree (strongly) on
which.

I'll look at the places indicated. Give me a day or X, new dev
machine being set up.

Thanks.

jdn

> > jdn- Hide quoted text -
>
> - Show quoted text -

[jdn]

Patch was submitted, and with some minor modifications, Ayende has
committed in code to allow the 1024 max page size to be overriden
through configuration.

jdn

> > - Show quoted text -- Hide quoted text -