Sort by Date

[jdeslip]

I have been working on an Android app to use with the arXiv. I'll be
releasing it to the market soon; however, I am wondering if you guys
ever implemented the sort by date API (i.e. so I don't have to
download all results and then sort it). I read on the FAQ that you
are working on it, but haven't seen an update...

Cheers!

Jack

[Thorsten S]

> --
> You received this message because you are subscribed to the Google Groups "arXiv api" group.
> To post to this group, send email to arxi...@googlegroups.com.
> To unsubscribe from this group, send email to arxiv-api+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/arxiv-api?hl=en.
>
>

Hi Jack,

glad to hear someone is working on an Android app. I was going to look
at that myself next time I have some spare cycles.
The sortBy and sortOrder parameters should do what you are asking for.

Let me quote from an email I sent to another user a while back, all of
this should still apply. Please don't hesitate to ask questions should
you run into problems, and please also send us a link to the finished
app.

I'll see to it that the help pages get updated to reflect this added
functionality.

Cheers
T.

* you can specify date ranges with key "submittedDate" of
"lastUpdatedDate" as part of the querystring, e.g.

search_query=lastUpdatedDate:[200712310900+TO+200712310959]

the date/time string is interpreted as GMT

* you can specify optional "sortBy" and "sortOrder" parameters

sortBy = [relevance, submittedDate, lastUpdatedDate]
sortOrder = [ascending, descending]

For example, to find all the papers last updated between 09:00am -
09:59am GMT on December 31st, 2007, sorted by submission date, issue
this query:

http://export.arxiv.org/api/query?search_query=lastUpdatedDate:[200712310900+TO+200712310959]&sortBy=submittedDate&sortOrder=ascending

you'll notice that arxiv:0705.1827 was first submitted in May 2007 and
replaced during the specified time-window in December 2007, so a
similar query specifying a submittedDate time-window will not list
that particular paper

http://export.arxiv.org/api/query?search_query=submittedDate:[200712310900+TO+200712310959]&sortBy=submittedDate&sortOrder=ascending

To get all of the papers submitted in December 2007 which have
"electron" in the title, use

http://export.arxiv.org/api/query?search_query=lastUpdatedDate:[200712010000+TO+200712312359]+AND+ti:electron&start=0&max_results=500&sortBy=submittedDate&sortOrder=ascending

Here we use the "start" and "max_results" parameters to return the
first 50 matches (slices must be limited to < 2000).

The response feed tells us that there are 97 matches total
<opensearch:totalResults
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">97</opensearch:totalResults>
so you could retrieve the remaining 47 via parameter settings
&start=50&max_results=100

Let us know of any problems with this interface.

Best regards
Thorsten Schwander

[Jack Deslippe]

Great thanks!

One issue.  I notice that there are some issues with the cond-mat category.  

http://export.arxiv.org/api/query?search_query=submittedDate:[200912010000+TO+200912312359]+AND+cat:math-ph&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending  

(math-ph) works fine but 

http://export.arxiv.org/api/query?search_query=submittedDate:[200912010000+TO+200912312359]+AND+cat:cond-mat&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending

(cond-mat) returns nothing.  

Also,

http://export.arxiv.org/api/query?search_query=cat:cond-mat&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending

Shows the most recent papers as 2004 (while - math-ph shows up to date).  Is the cond-mat collection to big for the API queries I am doing?  Or, am I just doing something stupid?

-Jack

[Thorsten S]

Hi Jack,

cond-mat appears to require a subject-class, e.g. cat:cond-mat.stat-mech

http://export.arxiv.org/api/query?search_query=cat:cond-mat.stat-mech&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending

and cond-mat.quant-gas

http://export.arxiv.org/api/query?search_query=cat:cond-mat.quant-gas&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending

etc.

I believe this is an oversight on our end. I'll have a look at the
code. Thanks for alerting us to this issue

Cheers
T.

[Thorsten S]

while addressing another question in this group I realized that a
simple wildcard (cat:cond-mat*) will capture papers with main category
"cond-mat" and those with a more specific sub-category. e.g.
"cond-mat.soft"

http://export.arxiv.org/api/query?search_query=cat:cond-mat*&start=0&max_results=500&sortBy=submittedDate&sortOrder=descending

It works this way, because the search is implemented as a simple full
text string search on individual metadata fields with basic wildcard
functionality

Cheers
T.

[Juan Jose Garcia-Ripoll]

I have noticed that the search by date leaves out crosslisted manuscripts.

Say that I design a search for today. This includes all papers submitted up to the day before 20:00. That is ok, but it will not show cross listed papers, which have be probably submitted _earlier_.

This makes it very difficult to reproduce the equivalent of the "new" list in the arXive.

[Thorsten S]

arXiv doesn't log a separate time-stamp for crosslist additions or
changes. However, the modification time for the metadata is updated
whenever metadata is modified, and crosslisting counts as
modification. Therefore a search on modification time instead of
submission date will be close to what you ask for. The key is
"lastUpdatedDate"

e.g.
http://export.arxiv.org/api/query?search_query=cat:quant-ph+AND+lastUpdatedDate:[201103142000+TO+201103152000]&max_results=50&sortBy=submittedDate&sortOrder=descending

gives 29 results, which are almost the same as today's
http://arxiv.org/list/quant-ph/new with 31 results.

I don't know, yet, why there is a difference, though.

Cheers
T.

[Thorsten S]

On Wed, Mar 16, 2011 at 12:06 PM, Thorsten S
<thorsten....@gmail.com> wrote:
> arXiv doesn't log a separate time-stamp for crosslist additions or
> changes. However, the modification time for the metadata is updated
> whenever metadata is modified, and crosslisting counts as
> modification. Therefore a search on modification time instead of
> submission date will be close to what you ask for. The key is
> "lastUpdatedDate"
>
> e.g.
> http://export.arxiv.org/api/query?search_query=cat:quant-ph+AND+lastUpdatedDate:[201103142000+TO+201103152000]&max_results=50&sortBy=submittedDate&sortOrder=descending
>
> gives 29 results, which are almost the same as today's
> http://arxiv.org/list/quant-ph/new with 31 results.
>
> I don't know, yet, why there is a difference, though.

the incremental index update appears to omit submissions/updates
received via the sword interface
After this problem is addressed the above /api request should have the
same content as the /new listing

Cheers
T.

[Juan Jose Garcia-Ripoll]

I am having problem today with requests over the weekend. The following list

export.arxiv.org/api/query?search_query=submittedDate:[201103312000+TO+201104011959]+AND+cat:quant-ph

does not return anything, while it should have been today's quant-ph list. Am I wrong?

Juanjo

[Thorsten S]

Thanks for reporting this issue

There is a problem with the index on the api server. It is missing
last Friday's updates. I am rebuilding it now and it should be ready
shortly.

Cheers
T.