oai-pmh metadataPrefix

91 views
Skip to first unread message

aussda....@gmail.com

unread,
May 22, 2018, 7:01:51 AM5/22/18
to Dataverse Users Community
Hi Everyone,

Is it possible to customize the metadataPrefix value for OAI harvesting? (https://data.aussda.at/oai?verb=ListMetadataFormats). CESSDA is looking to create a common data catalog and would like to standardize the prefixes across servers to "oai_ddi25" or "oai_ddi32" (as opposed to "oai_ddi"). I looked around our server for an xml template or other settings call but couldn't find anything. 

Thanks for the help. Best, Frank

Philip Durbin

unread,
May 22, 2018, 7:33:47 AM5/22/18
to dataverse...@googlegroups.com
No, it's not possible to change the prefix from "oai_ddi" unless you fork the code, which we don't recommend. It's hard coded here: https://github.com/IQSS/dataverse/blob/v4.8.6/src/main/java/edu/harvard/iq/dataverse/export/OAI_DDIExporter.java#L28

You are welcome to open a GitHub issue about this.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/faeb504d-a1f3-4cb6-a799-83380355683b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

aussda....@gmail.com

unread,
May 22, 2018, 8:30:28 AM5/22/18
to Dataverse Users Community
Phil,

Thanks for the quick reply. I thought this might be the case. A related question: it the metadata transformation to DDI performed by the harvest server hard-coded as well? We have to implement CESSDA's new metadata model (called CMM) in the coming months. I understand that we can adapt the Dataverse internal metadata scheme to capture the new elements at ingest, but not sure how to carry this forward to the DDI elements exported for harvesting.

Thanks as always for the help. Best, Frank


On Tuesday, May 22, 2018 at 1:33:47 PM UTC+2, Philip Durbin wrote:
No, it's not possible to change the prefix from "oai_ddi" unless you fork the code, which we don't recommend. It's hard coded here: https://github.com/IQSS/dataverse/blob/v4.8.6/src/main/java/edu/harvard/iq/dataverse/export/OAI_DDIExporter.java#L28

You are welcome to open a GitHub issue about this.
On Tue, May 22, 2018 at 7:01 AM, <aussda....@gmail.com> wrote:
Hi Everyone,

Is it possible to customize the metadataPrefix value for OAI harvesting? (https://data.aussda.at/oai?verb=ListMetadataFormats). CESSDA is looking to create a common data catalog and would like to standardize the prefixes across servers to "oai_ddi25" or "oai_ddi32" (as opposed to "oai_ddi"). I looked around our server for an xml template or other settings call but couldn't find anything. 

Thanks for the help. Best, Frank

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
May 22, 2018, 9:50:27 AM5/22/18
to dataverse...@googlegroups.com
Yes, each export format is hard coded. For example, there's a recent pull request to export metadata in DataCite XML format that's compliant with OpenAIRE guidelines: https://github.com/IQSS/dataverse/pull/4664

Of course, that's just for *export*. The list of export formats is at http://guides.dataverse.org/en/4.8.6/admin/metadataexport.html

To add support for *harvesting* is a second stage for any format. Once the pull request above for DataCite XML export has been merged, the next step will be to make it be available via OAI-PMH (harvesting): https://github.com/IQSS/dataverse/issues/4318

I hope this helps. Please keep the questions coming.

Phil

On Tue, May 22, 2018 at 8:30 AM, <aussda....@gmail.com> wrote:
Phil,

Thanks for the quick reply. I thought this might be the case. A related question: it the metadata transformation to DDI performed by the harvest server hard-coded as well? We have to implement CESSDA's new metadata model (called CMM) in the coming months. I understand that we can adapt the Dataverse internal metadata scheme to capture the new elements at ingest, but not sure how to carry this forward to the DDI elements exported for harvesting.

Thanks as always for the help. Best, Frank


On Tuesday, May 22, 2018 at 1:33:47 PM UTC+2, Philip Durbin wrote:
No, it's not possible to change the prefix from "oai_ddi" unless you fork the code, which we don't recommend. It's hard coded here: https://github.com/IQSS/dataverse/blob/v4.8.6/src/main/java/edu/harvard/iq/dataverse/export/OAI_DDIExporter.java#L28

You are welcome to open a GitHub issue about this.
On Tue, May 22, 2018 at 7:01 AM, <aussda....@gmail.com> wrote:
Hi Everyone,

Is it possible to customize the metadataPrefix value for OAI harvesting? (https://data.aussda.at/oai?verb=ListMetadataFormats). CESSDA is looking to create a common data catalog and would like to standardize the prefixes across servers to "oai_ddi25" or "oai_ddi32" (as opposed to "oai_ddi"). I looked around our server for an xml template or other settings call but couldn't find anything. 

Thanks for the help. Best, Frank

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/5631841d-8690-42dd-b62b-9f5e6172169e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

aussda....@gmail.com

unread,
Jun 1, 2018, 4:06:56 AM6/1/18
to Dataverse Users Community
Hi Phil,

Thanks very much. We're having another oai harvest issue that we can't quite figure out. We just noticed that our oai server returns noRecordsMatch when the set parameter is not included in the request. For example, compare this:


to this:


Is there a setting someplace to instruct the server to serve records even if the set parameter is not specified? We are using dataverse 4.6.2. Thanks very much for the help.

Best, Frank





On Tuesday, May 22, 2018 at 3:50:27 PM UTC+2, Philip Durbin wrote:
Yes, each export format is hard coded. For example, there's a recent pull request to export metadata in DataCite XML format that's compliant with OpenAIRE guidelines: https://github.com/IQSS/dataverse/pull/4664

Of course, that's just for *export*. The list of export formats is at http://guides.dataverse.org/en/4.8.6/admin/metadataexport.html

To add support for *harvesting* is a second stage for any format. Once the pull request above for DataCite XML export has been merged, the next step will be to make it be available via OAI-PMH (harvesting): https://github.com/IQSS/dataverse/issues/4318

I hope this helps. Please keep the questions coming.

Phil
On Tue, May 22, 2018 at 8:30 AM, <aussda....@gmail.com> wrote:
Phil,

Thanks for the quick reply. I thought this might be the case. A related question: it the metadata transformation to DDI performed by the harvest server hard-coded as well? We have to implement CESSDA's new metadata model (called CMM) in the coming months. I understand that we can adapt the Dataverse internal metadata scheme to capture the new elements at ingest, but not sure how to carry this forward to the DDI elements exported for harvesting.

Thanks as always for the help. Best, Frank


On Tuesday, May 22, 2018 at 1:33:47 PM UTC+2, Philip Durbin wrote:
No, it's not possible to change the prefix from "oai_ddi" unless you fork the code, which we don't recommend. It's hard coded here: https://github.com/IQSS/dataverse/blob/v4.8.6/src/main/java/edu/harvard/iq/dataverse/export/OAI_DDIExporter.java#L28

You are welcome to open a GitHub issue about this.
On Tue, May 22, 2018 at 7:01 AM, <aussda....@gmail.com> wrote:
Hi Everyone,

Is it possible to customize the metadataPrefix value for OAI harvesting? (https://data.aussda.at/oai?verb=ListMetadataFormats). CESSDA is looking to create a common data catalog and would like to standardize the prefixes across servers to "oai_ddi25" or "oai_ddi32" (as opposed to "oai_ddi"). I looked around our server for an xml template or other settings call but couldn't find anything. 

Thanks for the help. Best, Frank

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Jun 1, 2018, 6:22:39 AM6/1/18
to dataverse...@googlegroups.com
Which records would you want to be served with no "set" parameter?

I'm not very familiar with OAI-PMH so I'm not sure if what you're describing is a bug or not. Someone else on this list might know.

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/c56e3b42-15e1-49ab-ae58-1fb0bfc1cb69%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Laura Huisintveld

unread,
Jun 1, 2018, 7:19:35 AM6/1/18
to Dataverse Users Community
Hello Frank,

I have encountered this as well, we updated recently to dataverse version 4.8.6. and know I do get a result for my OAI-PMH queries without specifying a set:
for example:

I guess one of releases after 4.6.2 contains the fix, maybe someone else on the forum knows?

Best,
Laura


Op vrijdag 1 juni 2018 10:06:56 UTC+2 schreef aussda....@gmail.com:

Philip Durbin

unread,
Jun 1, 2018, 7:36:42 AM6/1/18
to dataverse...@googlegroups.com
Ah, yes, something like https://data.aussda.at/oai?verb=GetRecord&metadataPrefix=oai_ddi&identifier=doi:10.11587/HUQSXB works just fine for a single dataset. Thanks, Laura.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b104dcd3-a306-4a05-9d87-5cc1aeb62f4b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

aussda....@gmail.com

unread,
Jun 1, 2018, 7:45:56 AM6/1/18
to Dataverse Users Community
Hi Phil,
This came up in the context of the new CESSDA data catalog, which is harvesting data from of of its members. My experience with oai is limited so I'm not sure if this is normal behavior either. I should however have checked the query against the Harvard installation before posting to this forum. It produced the same noRecordsMatch result without an identified set, strongly suggesting that the issue in not in fact an issue: https://dataverse.harvard.edu/oai?verb=ListIdentifiers&metadataPrefix=oai_ddi

Thanks in any case for your quick reply. Best, Frank


To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Laura Huisintveld

unread,
Jun 1, 2018, 7:46:35 AM6/1/18
to Dataverse Users Community
Yes, it works for the 'getRecord' request. But if you want to use the 'ListIdentifiers' query, it won't give you a result if you do not specify a set.

According to the OAI-PMH principles, 'set' is only a optional argument here, it should not be mandatory:



Op vrijdag 1 juni 2018 13:36:42 UTC+2 schreef Philip Durbin:
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Jun 1, 2018, 7:54:50 AM6/1/18
to dataverse...@googlegroups.com
Sure, it looks like all 13 installations of Dataverse on the "List of Dataverse installation OAI-PMH (Harvesting) URLs and sets" spreadsheet contain a set to harvest, so maybe it's a norm or good practice to create a set: https://docs.google.com/spreadsheets/d/12cxymvXCqP_kCsLKXQD32go79HBWZ1vU_tdG4kvP5S8/edit?usp=sharing

That spreadsheet was created following the discussion at https://groups.google.com/d/msg/dataverse-community/fqyVxWZSDRA/q5mf4PvZBQAJ

I hope this helps,

Phil


To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/727e8d0f-03eb-47ad-8fa7-8ad322bfb2e3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

aussda....@gmail.com

unread,
Jun 1, 2018, 8:06:29 AM6/1/18
to Dataverse Users Community
This does help, thanks. May I add AUSSDA to this list?
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Jun 1, 2018, 9:29:49 AM6/1/18
to dataverse...@googlegroups.com
Sure, I went ahead and added AUSSDA for you. Please leave a comment on that spreadsheet if you'd like a set to be added. Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e7c07225-904f-4c04-a647-8859abc8a323%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Juan Corrales

unread,
Jun 2, 2018, 8:42:45 AM6/2/18
to Dataverse Users Community
Hi all,

  oai-pmh protocol request that ListRecords and ListIdentifiers with not parameter (set), should be return all records:


  To have a 100% oai-pmh response, there are some goods oai-pmh validator portals http://validator.oaipmh.com/ , http://re.cs.uct.ac.za/

  Best,

Juan


Philip Durbin

unread,
Jun 4, 2018, 7:35:21 AM6/4/18
to dataverse...@googlegroups.com
Thanks for pointing this out, Juan. "Is there a way to allow harvest not using the OAI sets?" was asked over at https://github.com/IQSS/dataverse/issues/4659 I just copied your reply into a new comment.

Phil


--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d9c9bcc7-6c46-4cb4-b09b-6a668f3bf776%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.