DSpace 7 API search by lastModified date

[Omar Villa]

Hi,

I want to know if using the new DSpace 7 API there is a way to perform a search to get only those records that have been created and / or updated from a certain date, I see among the information obtained with the API (endpoint /api/core/items) that there is the "lastModified" field which is useful to know this, but I would like to filter it without having to get all the items first, the closest I have found is that I can do a search using the metadata dc.date.issued something like:

/api/discover/search/objects?f.dateIssued=%5B2019 TO 2021%5D,equals

But I would rather like a result like the one obtained with OIA using a query like the following:

https://example.com/oai/request?verb=ListRecords&metadataPrefix=oai_dc&from=2019-08-01&until=2021-09-15

Thanks,

Omar

[Ben Heartland]

Hi,

Thanks for raising this Omar, this would be a welcome addition for us too. Currently, we first use OAI to find recently changed items, then use the 6.x REST API to get their metadata. Having both operations consolidated under the new core API would be simpler and more robust.

All the best,

Ben

[Tim Donohue]

Hi Omar & Ben,

The DSpace 7 REST API can support​ Solr/Lucene style queries, provided that it's URL encoded.  So, this sort of search by lastModified date is already supported for Items​ as the lastModified date of an Item is indexed in Solr (as the Solr field "lastModified"), see:
https://github.com/DSpace/DSpace/blob/main/dspace-api/src/main/java/org/dspace/discovery/indexobject/ItemIndexFactoryImpl.java#L141

The syntax is a bit odd, as you must both use a Solr date range search & ensure it is URL encoded. 

• Solr date range query syntax: https://solr.apache.org/guide/8_0/working-with-dates.html
• YOU MUST URL encode special characters in this syntax, e.g. the square brackets [] and any other special characters, etc.

But, once you do all that, it works.  Here's an example using the demo REST API:

https://api7.dspace.org/server/api/discover/search/objects?query=lastModified%3A%5B2021-01-01T00:00:00Z%20TO%202021-09-01T00:00:00Z%5D

That's hard to read, but un-encoded it says:

"query=lastModified:[2021-01-01T00:00:00Z TO 2021-09-01T00:00:00Z]"

In other words, everything modified between Jan 1 and Sept 1 of this year.

I'll admit, I cannot guarantee this will work perfectly for all Solr date queries (e.g. I'm having difficulty getting it to work locally for an open-ended query using an asterisk), but the basics do work.

(NOTE: This only​ works for Items, as currently DSpace doesn't support a "lastModified" date concept for Communities or Collections, etc.)

Tim

---

From: 'Ben Heartland' via DSpace Technical Support <dspac...@googlegroups.com>
Sent: Tuesday, September 21, 2021 4:48 AM
To: DSpace Technical Support <dspac...@googlegroups.com>
Subject: [dspace-tech] Re: DSpace 7 API search by lastModified date

 

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/f1a49926-57cb-43c0-b8cb-21f3b516492dn%40googlegroups.com.

[Ben Heartland]

Thanks for the quick reply Tim, that's great to know!

Best wishes,

Ben

[Omar Villa]

Sorry, I think I clicked on reply to the author.

I really appreciate it Tim, your explanation will help me a lot.

Best,

Omar

[Omar Villa]

Hello, I have a couple of questions related to this same topic, as far as I see the endpoint /api/discover/search allows the results to be sorted by:

score

dc.title

dc.date.issued

dc.date.accessioned

For example:

/api/discover/search/objects?sort=score

Would it be possible to sort them by the date of the last modification? I have tried something like the following, but it didn't work for me:

/api/discover/search/objects?sort=query:lastModified

/api/discover/search/objects?sort=lastModified

/api/discover/search/objects?sort=query

Finally, what does the option to sort by "score" refer to, i.e., does it have any relation to item statistics such as number of downloads or something like that that determines their score?

Best,

Omar

[Tim Donohue]

Hi Omar,

All the search options under that `/api/discover/search/` endpoint are completely configurable in our discovery.xml configuration file.  For example, the sort options for the default configuration can be configured here: https://github.com/DSpace/DSpace/blob/main/dspace/config/spring/api/discovery.xml#L155-L166

Sort options can be configured separately for different "configurations" as well (this corresponds to the "configuration" param in the REST Contract: https://github.com/DSpace/RestContract/blob/main/search-endpoint.md).  So, for example, here's a separate list of sort options for the "administrativeView" configuration: https://github.com/DSpace/DSpace/blob/main/dspace/config/spring/api/discovery.xml#L434-L445

You'll notice each of those configs refer to a sort "bean" later in the discovery.xml.  Here's where those are configured: https://github.com/DSpace/DSpace/blob/main/dspace/config/spring/api/discovery.xml#L2143-L2214

More documentation on configuring Discovery (which corresponds to the /api/discover/ endpoints) for your own local needs can be found in our documentation at: https://wiki.lyrasis.org/display/DSDOC7x/Discovery

Finally, the "score" sort option is just the default sort that Apache Solr provides (as all our search/browse is powered by Apache Solr).  In Solr, that default sort option is based on Solr's own relevance score.  It can be tuned at the Solr level (see solr docs), but we don't heavily tune it by default in DSpace. https://solr.apache.org/guide/8_0/relevance.html

Hopefully that helps some, but let us know on this list if you have other questions.

Tim

---

From: 'Omar Villa' via DSpace Technical Support <dspac...@googlegroups.com>
Sent: Tuesday, November 30, 2021 4:47 PM

To: DSpace Technical Support <dspac...@googlegroups.com>

Subject: Re: [dspace-tech] Re: DSpace 7 API search by lastModified date

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/d3bf2484-f687-43e1-9788-08e0ce915278n%40googlegroups.com.

[Omar Villa]

Hi Tim,

Thank you very much for the help, now I understand the "score" sorting option. I will review the documentation and configurations you indicate to adjust it to what I need.

Regards,

Omar