"SyntaxError: Unexpected end of input" Querying view with 5M documents

[Nick Wood]

I'm running a query against a view that emits ~5M rows. The key that's being emitted for each row is 32 characters and the value is < 10 characters. I'm using the NodeJS 2.0.8 client and running a query with 'skip' set to 1,600,000 (give or take 100,000) and with 'limit' set to 10,000 (I've also tried 100,000 with the same results).

Sometimes it works. Other times it gives the following error after about 1 minute 15 seconds:

undefined:0
SyntaxError: Unexpected end of input
at parse (native)

I've tried increasing the operationTimeout to 5 minutes with no apparent affect.

I can't find anything in the logs.  Any ideas?

A second related question is that it seems like setting the limit to 10k when pulling from the beginning of the view (i.e. skip is set to 0) causes the view query to return results very quickly.  But when setting 'skip' to a much higher value, the query takes a lot longer, even though 'limit' is still set to 10k.  I assumed the query time would be linear with the same limit regardless of where in the full data set the results are coming from.  Is this expected behavior?

Nick

[Andrey Rogozhnikov]

Nick, did you find a way to query records with "large" skips? I've experienced similar behaviour: a view with hundreds of millions of records, query.skip about 10M or 50M, and the message:

undefined:1

SyntaxError: Unexpected end of input
    at parse (native)

[Andrey Rogozhnikov]

The solution turned out to be simple.

The index of a view is a B-tree, so looking for range between two IDs is of logarithmic complexity, and that's ok, but skipping exact count of records requires linear count of operations. I.e., skipping 5M records requires O(5M) operations, but search for two bounds in a database of 1G elements requieres O(30) operations.