Total Results changed?

[Jimbo Jones]

Hello

As I recall, the "opensearch:totalResults" used to report the total
number of records that matched the search. Now it appears to report
the number in the current batch.

For example:

http://export.arxiv.org/api/query?search_query=all:electron

has
<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
opensearch/1.1/">10</opensearch:totalResults>

Whereas

http://export.arxiv.org/api/query?search_query=all:electron&start=6000&max_results=8000

has

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
opensearch/1.1/">8000</opensearch:totalResults>
<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/
opensearch/1.1/">6000</opensearch:startIndex>

<opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/
opensearch/1.1/">8000</opensearch:itemsPerPage>

SO the totalResult is the max_results, not the total of all results
(which in this case would be > 14,000)

Any thoughts on how to get the total of all results?

Thanks, Jim

[Thorsten S]

Hi Jimbo,

thanks for alerting us to the problem. This has now been corrected and
the total number of results is correctly returned again.


http://export.arxiv.org/api/query?search_query=all:electron&start=6000&max_results=8000
returns
...
<opensearch:totalResults
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">54602</opensearch:totalResults>
<opensearch:startIndex
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">6000</opensearch:startIndex>

<opensearch:itemsPerPage
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">8000</opensearch:itemsPerPage>

...

Keep those bug reports coming
T.

[Jimbo Jones]

Thanks, I'll check it out.

On May 8, 1:34 pm, Thorsten S <thorsten.schwan...@gmail.com> wrote:
> Hi Jimbo,
>
> thanks for alerting us to the problem. This has now been corrected and
> the total number of results is correctly returned again.
>

> http://export.arxiv.org/api/query?search_query=all:electron&start=600...

> returns
> ...
>   <opensearch:totalResults
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">54602</opensearch:totalResults>
>   <opensearch:startIndex
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">6000</opensearch:startIndex>
>   <opensearch:itemsPerPage
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">8000</opensearch:itemsPerPage>
> ...
>
> Keep those bug reports coming
> T.
>

> On Tue, May 5, 2009 at 8:39 AM, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Hello
>
> >  As I recall, the "opensearch:totalResults" used to report the total
> > number of records that matched the search. Now it appears to report
> > the number in the current batch.
>
> >  For example:
>
> >    http://export.arxiv.org/api/query?search_query=all:electron
>
> >  has
> > <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">10</opensearch:totalResults>
>
> >  Whereas
>

> >    http://export.arxiv.org/api/query?search_query=all:electron&start=600...

[Christian Hesketh]

--
J. Christian Hesketh M.Sc.
CEO - Ion Channel Media Group
2028 Harvard Avenue, Suite 103
Montreal, QC
CANADA H4A 2V9

Tel: +1(514)245-8107
Fax: +1(514)313-5675

http://www.IonChannelMedia.com

[Jimbo Jones]

Hi

My unit tests are still failing.

1. I get the correct first results. For example.
http://export.arxiv.org/api/query?search_query=ti:epsilon&start=0&max_results=10
returns:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<link href="http://arxiv.org/api/query?
search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10"
rel="self" type="application/atom+xml"/>
<title>ArXiv Query:
search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10</
title>
<id>http://arxiv.org/api/JulGnBrHKLln2O5o+vBGCaylCro</id>
<updated>2009-05-13T00:00:00-04:00</updated>

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">400</opensearch:totalResults>

<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">0</opensearch:startIndex>

<opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:itemsPerPage>
<entry>
..... 10 Entries
</entry>
</feed>

2. Trying to get the next 10 results (i.e. results 11-20 of the 400),
I get (from http://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&max_results=10
)

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<link href="http://arxiv.org/api/query?
search_query=ti:epsilon&amp;id_list=&amp;start=10&amp;max_results=10"
rel="self" type="application/atom+xml"/>
<title>ArXiv Query:
search_query=ti:epsilon&amp;id_list=&amp;start=10&amp;max_results=10</
title>
<id>http://arxiv.org/api/HSngF0IY9C2Fz3ozYTl+TuPlf9g</id>
<updated>2009-05-13T00:00:00-04:00</updated>

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">400</opensearch:totalResults>

<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:startIndex>

<opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:itemsPerPage>
</feed>

Ie. no results

What's wrong? This worked previously.

Thanks, Jim

On May 8, 1:34 pm, Thorsten S <thorsten.schwan...@gmail.com> wrote:

> Hi Jimbo,
>
> thanks for alerting us to the problem. This has now been corrected and
> the total number of results is correctly returned again.
>

> http://export.arxiv.org/api/query?search_query=all:electron&start=600...

> returns
> ...
>   <opensearch:totalResults
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">54602</opensearch:totalResults>
>   <opensearch:startIndex
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">6000</opensearch:startIndex>
>   <opensearch:itemsPerPage
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">8000</opensearch:itemsPerPage>
> ...
>
> Keep those bug reports coming
> T.
>

> On Tue, May 5, 2009 at 8:39 AM, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Hello
>
> >  As I recall, the "opensearch:totalResults" used to report the total
> > number of records that matched the search. Now it appears to report
> > the number in the current batch.
>
> >  For example:
>
> >    http://export.arxiv.org/api/query?search_query=all:electron
>
> >  has
> > <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">10</opensearch:totalResults>
>
> >  Whereas
>

> >    http://export.arxiv.org/api/query?search_query=all:electron&start=600...

[Thorsten S]

your second query requests a zero size slice

max_results is the total number of results the query will return.
start is the offset into that resultset at which display starts

so to retrieve results 11-20 you need &max_results=20&start=10

Cheers
Thorsten

[Jimbo Jones]

Hi

My unit tests are still failing.

1. I get the correct first results. For example.
http://export.arxiv.org/api/query?search_query=ti:epsilon&start=0&max_results=10
returns:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<link href="http://arxiv.org/api/query?
search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10"
rel="self" type="application/atom+xml"/>
<title>ArXiv Query:
search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10</
title>
<id>http://arxiv.org/api/JulGnBrHKLln2O5o+vBGCaylCro</id>
<updated>2009-05-13T00:00:00-04:00</updated>

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">400</opensearch:totalResults>

<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">0</opensearch:startIndex>

<opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:itemsPerPage>
<entry>
..... 10 Entries
</entry>
</feed>

2. Trying to get the next 10 results (i.e. results 11-20 of the 400),
I get (from http://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&max_results=10
)

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<link href="http://arxiv.org/api/query?
search_query=ti:epsilon&amp;id_list=&amp;start=10&amp;max_results=10"
rel="self" type="application/atom+xml"/>
<title>ArXiv Query:
search_query=ti:epsilon&amp;id_list=&amp;start=10&amp;max_results=10</
title>
<id>http://arxiv.org/api/HSngF0IY9C2Fz3ozYTl+TuPlf9g</id>
<updated>2009-05-13T00:00:00-04:00</updated>

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">400</opensearch:totalResults>

<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:startIndex>

<opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/

opensearch/1.1/">10</opensearch:itemsPerPage>
</feed>

Ie. no results

What's wrong? This worked previously.

Thanks, Jim




On May 8, 1:34 pm, Thorsten S <thorsten.schwan...@gmail.com> wrote:

> Hi Jimbo,
>
> thanks for alerting us to the problem. This has now been corrected and
> the total number of results is correctly returned again.
>

> http://export.arxiv.org/api/query?search_query=all:electron&start=600...

> returns
> ...
>   <opensearch:totalResults
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">54602</opensearch:totalResults>
>   <opensearch:startIndex
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">6000</opensearch:startIndex>
>   <opensearch:itemsPerPage
> xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">8000</opensearch:itemsPerPage>
> ...
>
> Keep those bug reports coming
> T.
>

> On Tue, May 5, 2009 at 8:39 AM, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Hello
>
> >  As I recall, the "opensearch:totalResults" used to report the total
> > number of records that matched the search. Now it appears to report
> > the number in the current batch.
>
> >  For example:
>
> >    http://export.arxiv.org/api/query?search_query=all:electron
>
> >  has
> > <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">10</opensearch:totalResults>
>
> >  Whereas
>

> >    http://export.arxiv.org/api/query?search_query=all:electron&start=600...

[Jimbo Jones]

Hi

Then you've changed the definition of max_results. From the API
document http://export.arxiv.org/api_help/docs/user-manual.html in the
section "start and max_results paging"

<quote>
Within the total results set, start defines the index of the first
returned result, using 0-based indexing. max_results is the number of
results returned by the query. For example, if wanted to step through
the results of a search_query of all:electron, we would construct the
urls:

http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=10
(1)
http://export.arxiv.org/api/query?search_query=all:electron&start=10&max_results=10
(2)
http://export.arxiv.org/api/query?search_query=all:electron&start=20&max_results=10
(3)

1. Get results 0-9
2. Get results 10-19
3. Get results 20-29
</quote>

Your example #2 corresponds to what I am doing in my example. Like I
said, this used to work.

Going forward, is this the way the API will work? Or will the API go
back to what is documented?

Thanks, Jim

On May 13, 11:18 am, Thorsten S <thorsten.schwan...@gmail.com> wrote:
> your second query requests a zero size slice
>
> max_results is the total number of results the query will return.
> start is the offset into that resultset at which display starts
>
> so to retrieve results 11-20 you need &max_results=20&start=10
>
> Cheers
> Thorsten
>

> On Wed, May 13, 2009 at 8:41 AM, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Hi
>
> >  My unit tests are still failing.
>
> >  1. I get the correct first results. For example.

> >http://export.arxiv.org/api/query?search_query=ti:epsilon&start=0&max...

> > returns:
>
> > <?xml version="1.0" encoding="utf-8"?>
> > <feed xmlns="http://www.w3.org/2005/Atom">
> >  <link href="http://arxiv.org/api/query?
> > search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10"
> > rel="self" type="application/atom+xml"/>
> >  <title>ArXiv Query:
> > search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10</
> > title>
> >  <id>http://arxiv.org/api/JulGnBrHKLln2O5o+vBGCaylCro</id>
> >  <updated>2009-05-13T00:00:00-04:00</updated>
> >  <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">400</opensearch:totalResults>
> >  <opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">0</opensearch:startIndex>
>
> >  <opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">10</opensearch:itemsPerPage>
> >  <entry>
> >  ..... 10 Entries
> >  </entry>
> > </feed>
>
> >  2. Trying to get the next 10 results (i.e. results 11-20 of the 400),

> > I get (fromhttp://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&ma...

[Jimbo Jones]

Sorry, messed up post.

What I meant to say:

You've changed the meaning of max_results from what is documented. In

http://export.arxiv.org/api_help/docs/user-manual.html in the section

"start and max_results paging", you have

<quote>
Within the total results set, start defines the index of the first
returned result, using 0-based indexing. max_results is the number of
results returned by the query. For example, if wanted to step through
the results of a search_query of all:electron, we would construct the
urls:

http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=10
(1)
http://export.arxiv.org/api/query?search_query=all:electron&start=10&max_results=10
(2)
http://export.arxiv.org/api/query?search_query=all:electron&start=20&max_results=10
(3)

1. Get results 0-9
2. Get results 10-19
3. Get results 20-29
</quote>

Example 2 in the document corresponds to what I am doing. As I said,
this used to work.

Going forward, will this API be stable or will it go back to how it is
documented? I would prefer that it goes back, because then existing
code that is written to the specification will continue to work.

Thanks Jim

On May 13, 11:18 am, Thorsten S <thorsten.schwan...@gmail.com> wrote:

> your second query requests a zero size slice
>
> max_results is the total number of results the query will return.
> start is the offset into that resultset at which display starts
>
> so to retrieve results 11-20 you need &max_results=20&start=10
>
> Cheers
> Thorsten
>

> On Wed, May 13, 2009 at 8:41 AM, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Hi
>
> >  My unit tests are still failing.
>
> >  1. I get the correct first results. For example.

> >http://export.arxiv.org/api/query?search_query=ti:epsilon&start=0&max...

> > returns:
>
> > <?xml version="1.0" encoding="utf-8"?>
> > <feed xmlns="http://www.w3.org/2005/Atom">
> >  <link href="http://arxiv.org/api/query?
> > search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10"
> > rel="self" type="application/atom+xml"/>
> >  <title>ArXiv Query:
> > search_query=ti:epsilon&amp;id_list=&amp;start=0&amp;max_results=10</
> > title>
> >  <id>http://arxiv.org/api/JulGnBrHKLln2O5o+vBGCaylCro</id>
> >  <updated>2009-05-13T00:00:00-04:00</updated>
> >  <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">400</opensearch:totalResults>
> >  <opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">0</opensearch:startIndex>
>
> >  <opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/
> > opensearch/1.1/">10</opensearch:itemsPerPage>
> >  <entry>
> >  ..... 10 Entries
> >  </entry>
> > </feed>
>
> >  2. Trying to get the next 10 results (i.e. results 11-20 of the 400),

> > I get (fromhttp://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&ma...

[Thorsten S]

I have corrected the API to work as documented.

I was thinking lower and upper boundary, not offset and length.
Thanks for pointing out the error
T.

[Jimbo Jones]

Hi

I see that you've fixed it? Both of these

http://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&max_results=10

and

http://export.arxiv.org/api/query?search_query=all:electron&start=10&max_results=10

now seem to work as documented

Thanks, JIm

On May 13, 11:46 am, Jimbo Jones <james.a.gra...@gmail.com> wrote:
> Sorry, messed up post.
>
> What I meant to say:
>

> You've changed the meaning of max_results from what is documented. Inhttp://export.arxiv.org/api_help/docs/user-manual.htmlin the section

> "start and max_results paging", you have
>
> <quote>
> Within the total results set, start defines the index of the first
> returned result, using 0-based indexing. max_results is the number of
> results returned by the query. For example, if wanted to step through
> the results of a search_query of all:electron, we would construct the
> urls:
>

> http://export.arxiv.org/api/query?search_query=all:electron&start=0&m...
> (1)http://export.arxiv.org/api/query?search_query=all:electron&start=10&...
> (2)http://export.arxiv.org/api/query?search_query=all:electron&start=20&...

[Jimbo Jones]

Nearly there ...

One more fix:

The third example from the documentation

http://export.arxiv.org/api/query?search_query=all:electron&start=20&max_result=10

gives

Error
Wed, May 13, 2009 12:00 AM
max_results < start. This doesn't make sense

Thanks, Jim

On May 13, 11:52 am, Jimbo Jones <james.a.gra...@gmail.com> wrote:
> Hi
>
>   I see that you've fixed it? Both of these
>

> http://export.arxiv.org/api/query?search_query=ti:epsilon&start=10&ma...
>
> and
>
> http://export.arxiv.org/api/query?search_query=all:electron&start=10&...

>
> now seem to work as documented
>
>   Thanks, JIm
>
> On May 13, 11:46 am, Jimbo Jones <james.a.gra...@gmail.com> wrote:
>
> > Sorry, messed up post.
>
> > What I meant to say:
>

> > You've changed the meaning of max_results from what is documented. Inhttp://export.arxiv.org/api_help/docs/user-manual.htmlinthe section

[Thorsten S]

thanks for being persistent.

also fixed
T.