num-found seems to be floating

[Paolo Simeone]

Dear all,

I am noticing some confusing result with my requests from v3.0 API.

This is live from my (Python) debugger. Is this something expected or am I getting something wrong in my request?

(Pdb) url_search = 'https://pub.orcid.org/v3.0/search/?q=*:*+AND+profile-last-modified-date:[2020-11-16T00:00:00Z TO 2020-11-23T00:00:00Z]&start=0&rows=100'

(Pdb) res = requests.get(url_search, data=payload, headers=headers)

(Pdb) ET.fromstring(res.text.encode('utf-8')).get('num-found')

'483763'

(Pdb) res = requests.get(url_search, data=payload, headers=headers)

(Pdb) ET.fromstring(res.text.encode('utf-8')).get('num-found')

'483730'

(Pdb) url_search = 'https://pub.orcid.org/v3.0/search/?q=*:*+AND+profile-last-modified-date:[2020-11-16T00:00:00Z TO 2020-11-23T00:00:00Z]&start=100&rows=100'

(Pdb) res = requests.get(url_search, data=payload, headers=headers)

(Pdb) ET.fromstring(res.text.encode('utf-8')).get('num-found')

'483730'

(Pdb) url_search = 'https://pub.orcid.org/v3.0/search/?q=*:*+AND+profile-last-modified-date:[2020-11-16T00:00:00Z TO 2020-11-23T00:00:00Z]&start=483700&rows=100'

(Pdb) res = requests.get(url_search, data=payload, headers=headers)

(Pdb) ET.fromstring(res.text.encode('utf-8')).get('num-found')

'483730'

(Pdb) url_search = 'https://pub.orcid.org/v3.0/search/?q=*:*+AND+profile-last-modified-date:[2020-11-16T00:00:00Z TO 2020-11-23T00:00:00Z]&start=483700&rows=100'

(Pdb) res = requests.get(url_search, data=payload, headers=headers)

(Pdb) ET.fromstring(res.text.encode('utf-8')).get('num-found')

'483682'

[Pedro Costa]

Hi Paolo,

We keep live data in Solr so query results are always dynamic. We just store the latest last-modified-date, not a history of last-modified-dates, so the results of a query such as the one in your post are expected to change. So, for example, a given record's last-modified date might fall within the timeframe of your query today, but I update that record today and you run the query again, the last-modified-date for that record no longer falls within your timeframe.

We have a lambda file generated daily which contains the last-modified-date for all records. You could save that file every day and then get the data from it. This could be a solution to what you're trying to do. Please note this is available only to ORCID members.

Kind regards,

Pedro Costa

QA Lead

https://orcid.org/0000-0003-4566-1411