1000 results limit?

231 views
Skip to first unread message

Leonel Togniolli

unread,
Jun 5, 2011, 3:35:12 PM6/5/11
to hnse...@googlegroups.com
Aparently there's a 1000 results limitation when paging through searches. I wanted to run an analysis on all submitted stories so that limits me somewhat. Is there an alternative?

I could probably create a new search after I reach the limit filtering by submitted date but I'd prefer to avoid it if possible.

Zack Maril

unread,
Jun 5, 2011, 3:41:18 PM6/5/11
to hnse...@googlegroups.com
I think you are looking for the start keyword. Just keep incrementing by a thousand until you hit them all. 
Ex. "&start=0", "&start=1000", etc. 

andres

unread,
Jun 5, 2011, 3:41:48 PM6/5/11
to HNSearch
Currently, the HNSearch API is focused on search so it's probably not
the right tool for your analysis. It sounds like you need access to
the full data set.

It would be cool to add map/reduce features to ThriftDB in the future
but that will take a while!

Andres

Leonel Togniolli

unread,
Jun 5, 2011, 3:46:21 PM6/5/11
to hnse...@googlegroups.com
Actually that's exactly what I meant. If start is too high, you'll get:

{"message": "Validation error: max limit is 100, max start+limit is 1000", "__class__": "ClientErrorResponse"}

Zack Maril

unread,
Jun 5, 2011, 3:53:38 PM6/5/11
to hnse...@googlegroups.com
That sucks. You could always use dates then, but I don't think there is a way to get a response without adding in a query. And there is a filter in place methinks that strips out high frequency words. http://api.thriftdb.com/api.hnsearch.com/items/_search?q=the

Still trying to find a way to pull everything. 

Andres Morey

unread,
Jun 5, 2011, 4:04:29 PM6/5/11
to hnse...@googlegroups.com
Zack,

So sorry but the API is not designed for your use case. The limits are in place because of practical considerations with full-text search. Google has a result limit as well for similar reasons.

When we add map/reduce capabilities to ThriftDB you'll probably be able to perform your analysis using the API.

Andres

Zack Maril

unread,
Jun 5, 2011, 4:20:11 PM6/5/11
to hnse...@googlegroups.com
What use case? I am just trying to figure out what can be done; hacking an API seems like an activity that involves doing things with the API that it maybe wasn't strictly designed for.  If there were map/reduce capabilities, it would be too easy. 

Andres Morey

unread,
Jun 5, 2011, 6:20:06 PM6/5/11
to hnse...@googlegroups.com
Good point. Hack away.
Reply all
Reply to author
Forward
0 new messages