1000 results limit?

[Leonel Togniolli]

Aparently there's a 1000 results limitation when paging through searches. I wanted to run an analysis on all submitted stories so that limits me somewhat. Is there an alternative?

I could probably create a new search after I reach the limit filtering by submitted date but I'd prefer to avoid it if possible.

[Zack Maril]

I think you are looking for the start keyword. Just keep incrementing by a thousand until you hit them all. 

Ex. "&start=0", "&start=1000", etc. 

[andres]

Currently, the HNSearch API is focused on search so it's probably not
the right tool for your analysis. It sounds like you need access to
the full data set.

It would be cool to add map/reduce features to ThriftDB in the future
but that will take a while!

Andres

[Leonel Togniolli]

Actually that's exactly what I meant. If start is too high, you'll get:

{"message": "Validation error: max limit is 100, max start+limit is 1000", "__class__": "ClientErrorResponse"}

[Zack Maril]

That sucks. You could always use dates then, but I don't think there is a way to get a response without adding in a query. And there is a filter in place methinks that strips out high frequency words. http://api.thriftdb.com/api.hnsearch.com/items/_search?q=the

Still trying to find a way to pull everything. 

[Andres Morey]

Zack,

So sorry but the API is not designed for your use case. The limits are in place because of practical considerations with full-text search. Google has a result limit as well for similar reasons.

When we add map/reduce capabilities to ThriftDB you'll probably be able to perform your analysis using the API.

Andres

[Zack Maril]

What use case? I am just trying to figure out what can be done; hacking an API seems like an activity that involves doing things with the API that it maybe wasn't strictly designed for.  If there were map/reduce capabilities, it would be too easy. 

[Andres Morey]

Good point. Hack away.