how to increase max_results

[fragk...@gmail.com]

Hi

I am wondering whether there is anyway to increase the max_results
value.
It seems that the maximum number of results I can retrieve is 1000.

Can one increase this value? If no, then can I retrieve, say the first
10000 ,
documents of some category by retrieving chunks of 1000?

many thanks

Frag.

[julius.lucks]

Hi Frag,

Thanks for your interest and the email. Right now the limit of 1000
results is enforced by the backend database that the API runs off of.

You should be able to receive chunks of documents by using the 'start'
parameter and incrementing that in successive calls - see http://export.arxiv.org/api_help/docs/user-manual.html#_calling_the_api
for more information.

Depending on what your use case is though, you may want to look at the
ORE interface which allows for more bulk-type downloads. See http://www.openarchives.org/ore/1.0/toc

Thanks,

Julius

---------------------------------------------------------------------------------------
http://www.openwetware.org/wiki/User:Julius_B._Lucks
----------------------------------------------------------------------------------------

[Fragkiskos Papadopoulos]

Many thanks Julius, I'll check out the ORE interface.

However, I tried the 'start' parameter but could not make it work.

See, say you want to get 2000 results. I thought that by setting  start=0 max_results=1000 for the first chunk

and start=1001 max_results=2000 for the second, I would be able to get all the 2000. 

But it turns out that I do not get anything for the second chunk.

Perhaps the start value should be by default less than (or equal to) max_results. Since max_results cannot change

the 'start' index cannot exceed 1000, unless I am missing something ? :)

thanks,

frag.

[julius.lucks]

Hi Frag,

Can you paste in the exact url's you are trying?

Thanks,

Julius

---------------------------------------------------------------------------------------

http://www.openwetware.org/wiki/User:Julius_B._Lucks

----------------------------------------------------------------------------------------

[Fragkiskos Papadopoulos]

sure,

try for example this:

http://export.arxiv.org/api/query?search_query=cat:cond-mat&start=1000&max_results=1100

it won't return any results. I'm I doing something wrong?

many thanks!

[julius.lucks]

Hi Frag,

Actually you are not doing anything wrong and this has to do with the way the backend is plugged into the API.  Is there a way you could use the sortBy and sortOrder parameters of the API call to be able to refine your search?

Maybe if you describe your use case, we can think of a work-around.

Thanks,

Julius

---------------------------------------------------------------------------------------

http://www.openwetware.org/wiki/User:Julius_B._Lucks

----------------------------------------------------------------------------------------

[Fragkiskos Papadopoulos]

I see :)

I am trying to get the all the results (or only authors and titles) from some category, say cond-mat.

Ideally I would like to constrain my search to only a time period, e.g. 1 year. 

So, I was trying to get all 2008 results (around 11000 entries) from the cond-mat category.

I appreciate the help a lot!

[Paul Ginsparg]

well actually he's doing something wrong with
start=1000&max_results=1100
since he presumably meant max_results=100
(i.e. from 1000 - 1099)

but there is a bug anyway, things like
start=900&max_results=10

gives ten results, but

start=995&max_results=10

gives only 5, and starts above 1000 give nothing, so it clearly
doesn't know how to start beyond 1000

[Fragkiskos Papadopoulos]

thanks Paul, 

yes initially I tried what you say, but I thought that was wrong too as I did not get any results either for the reason

you also say:)

[Thorsten]

see my post today on new search parameters and other improvements

the problem described below was fixed yesterday 5/3/09

actually the limit was intentionally hard-coded, but at the very least
there should have been a useful error message instead of a silently
enforced upper limit. max_results is now limited to 30,000 and a
larger value will result in an error.

thanks for the feedback and for alerting us to the problem
T.

On May 1, 6:09 pm, Fragkiskos Papadopoulos <fragkisk...@gmail.com>
wrote:

> thanks Paul,
> yes initially I tried what you say, but I thought that was wrong too as I
> did not get any results either for the reason
> you also say:)
>

> On Fri, May 1, 2009 at 4:59 PM, Paul Ginsparg <ginsp...@cornell.edu> wrote:
>
> > well actually he's doing something wrong with
> > start=1000&max_results=1100
> > since he presumably meant max_results=100
> > (i.e. from 1000 - 1099)
>
> > but there is a bug anyway, things like
> > start=900&max_results=10
>
> > gives ten results, but
>
> > start=995&max_results=10
>
> > gives only 5, and starts above 1000 give nothing, so it clearly
> > doesn't know how to start beyond 1000
>

> > On 5/1/09, julius.lucks <julius.lu...@gmail.com> wrote:
> > > Hi Frag,
>
> > > Actually you are not doing anything wrong and this has to do with the way
> > > the backend is plugged into the API.  Is there a way you could use the
> > > sortBy and sortOrder parameters of the API call to be able to refine your
> > > search?
>
> > > Maybe if you describe your use case, we can think of a work-around.
>
> > > Thanks,
>
> > > Julius
>
> > ---------------------------------------------------------------------------------------
> > >http://www.openwetware.org/wiki/User:Julius_B._Lucks
>
> > ----------------------------------------------------------------------------------------
>
> > > On May 1, 2009, at 4:39 PM, Fragkiskos Papadopoulos wrote:
> > > sure,
>
> > > try for example this:
>

> >http://export.arxiv.org/api/query?search_query=cat:cond-mat&start=100...

>
> > > it won't return any results. I'm I doing something wrong?
>
> > > many thanks!
>

> > > On Fri, May 1, 2009 at 4:02 PM, julius.lucks <julius.lu...@gmail.com>

> > wrote:
>
> > > > Hi Frag,
>
> > > > Can you paste in the exact url's you are trying?
>
> > > > Thanks,
>
> > > > Julius
>
> > ---------------------------------------------------------------------------------------
> > > >http://www.openwetware.org/wiki/User:Julius_B._Lucks
>
> > ----------------------------------------------------------------------------------------
>
> > > > On May 1, 2009, at 3:49 PM, Fragkiskos Papadopoulos wrote:
>
> > > > Many thanks Julius, I'll check out the ORE interface.
>
> > > > However, I tried the 'start' parameter but could not make it work.
>
> > > > See, say you want to get 2000 results. I thought that by setting
> >  start=0
> > > max_results=1000 for the first chunk
> > > > and start=1001 max_results=2000 for the second, I would be able to get
> > all
> > > the 2000.
>
> > > > But it turns out that I do not get anything for the second chunk.
>
> > > > Perhaps the start value should be by default less than (or equal to)
> > > max_results. Since max_results cannot change
> > > > the 'start' index cannot exceed 1000, unless I am missing something ?
> > :)
>
> > > > thanks,
>
> > > > frag.
>

> > > > On Fri, May 1, 2009 at 2:29 PM, julius.lucks <julius.lu...@gmail.com>

> > > wrote:
>
> > > > > Hi Frag,
>
> > > > > Thanks for your interest and the email.  Right now the limit of 1000
> > > > > results is enforced by the backend database that the API runs off of.
>
> > > > > You should be able to receive chunks of documents by using the
> > 'start'
> > > > > parameter and incrementing that in successive calls - see
> > >http://export.arxiv.org/api_help/docs/user-manual.html#_calling_the_api
> > > > >  for more information.
>
> > > > > Depending on what your use case is though, you may want to look at
> > the
> > > > > ORE interface which allows for more bulk-type downloads.  See
> > >http://www.openarchives.org/ore/1.0/toc
>
> > > > > Thanks,
>
> > > > > Julius
>
> > ---------------------------------------------------------------------------------------
> > > > >http://www.openwetware.org/wiki/User:Julius_B._Lucks
>
> > ----------------------------------------------------------------------------------------
>

[Fragkiskos Papadopoulos]

thanks a lot for all the help and super-fast responses!

frag.