Add harvesting client fail

103 views
Skip to first unread message

Michel Bamouni

unread,
Aug 16, 2018, 10:40:35 AM8/16/18
to Dataverse Users Community
Hello,

When I try to add zenodo(https://zenodo.org/oai2d) as harvesting client , I get the error : "Invalid URL. Failed to establish connection and receive a valid server response."
In the glassfish server logs, I see the following detail : "metadataformats: failed;Failed to execute listmetadataformats; No valid response received from the OAI server."
When I run the request to get zenodo oai provider metadata (https://zenodo.org/oai2d?verb=ListMetadataFormats), it works fine.

For information, in find the zenode oai base url in zenodo developers web site : http://developers.zenodo.org/#oai-pmh

Best regards,

Michel

Philip Durbin

unread,
Aug 16, 2018, 11:14:16 AM8/16/18
to dataverse...@googlegroups.com
I *think* I got a bit farther with the following config as seen from http://localhost:8080/api/harvest/clients/zenodo but it still doesn't work. Once I click the "run" button on the harvesting client I created I get (over and over) "Failed to find a global identifier in the OAI_DC XML record" from line 229 of ImportGenericServiceBean.java (I'm on c53ef2a on the "develop" branch). I'm not sure anyone has ever using Dataverse to harvest from Zenodo's OAI-PMH server. Maybe I'm doing something wrong but it's non-obvious to me how to fix it. Can you please create an issue at https://github.com/IQSS/dataverse/issues about this?

Thanks!

Phil

p.s. The config I tried:


{
  "status": "OK",
  "data": {
    "nickName": "Zenodo",
    "dataverseAlias": "zenodo",
    "type": "oai",
    "harvestUrl": "https://zenodo.org/oai2d",
    "archiveUrl": "https://zenodo.org",
    "archiveDescription": "This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data.",
    "metadataFormat": "oai_dc",
    "set": "user-zenodo",
    "schedule": "none",
    "status": "inProgress",
    "lastHarvest": "Thu Aug 16 10:59:14 EDT 2018",
    "lastResult": "INPROGRESS",
    "lastSuccessful": "N/A",
    "lastNonEmpty": "N/A",
    "lastDatasetsHarvested": "N/A",
    "lastDatasetsDeleted": "N/A",
    "lastDatasetsFailed": "N/A"
  }
}

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/1e886228-bc50-4658-8cfa-49204fedcc61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Condon, Kevin M

unread,
Aug 16, 2018, 11:21:39 AM8/16/18
to dataverse...@googlegroups.com


Does having a stylesheet in the response pose a problem?



From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> on behalf of Philip Durbin <philip...@harvard.edu>
Sent: Thursday, August 16, 2018 11:14 AM
To: dataverse...@googlegroups.com
Subject: Re: [Dataverse-Users] Add harvesting client fail
 
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/CABbxx8HQNX4rdhb%3Dxia7ianpj80gR5ccQp1Fe89Jrcn8AUm_sg%40mail.gmail.com.

Michel Bamouni

unread,
Aug 16, 2018, 11:26:20 AM8/16/18
to Dataverse Users Community
Hi Phil,

If understand well, it's possible de configure a harvesting clients by using a config file. If true, How I can see the documentation for this?

Michel
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Aug 16, 2018, 11:41:38 AM8/16/18
to dataverse...@googlegroups.com
I created that harvesting client using the GUI and only used the API call I mentioned to get a "dump" of the config from the database. From looking at the code it does seem to be possible to create harvesting clients using the API ( https://github.com/IQSS/dataverse/blob/v4.9.2/src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java#L145 ) but I can't find any documentation for it. If you'd like some documentation for this, please create a separate GitHub issue for this.

Thanks!

Phil


To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bb986e5a-1335-452d-8220-c58439ec4b4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michel Bamouni

unread,
Aug 16, 2018, 11:50:15 AM8/16/18
to Dataverse Users Community

Ok I undestand.

You say that you successfully create zenodo Harvesting but But my issue is that I can't create this client on my dataverse.
I get the erreor "Invalid Url" like in the joined pictures. So Does you meet this error ?
Michel
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Aug 16, 2018, 12:03:05 PM8/16/18
to dataverse...@googlegroups.com
Huh. I don't get that error. I'm able to click "Next" and then the Glassfish server.log says this: "8 metadata formats total"

I'm not sure why you're seeing something different and I can't remember if we fixed a bug in this part of the code recently. Again, I'm basically running code that's a little bit ahead of Dataverse 4.9.2, which was recently released.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/0c96fe8a-521f-4c91-b5c2-6b7ff9925153%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jamie Jamison

unread,
Aug 17, 2018, 4:01:56 PM8/17/18
to Dataverse Users Community
I get the same error with 4.9.2

dataverse-metadata-harvesting-error.png

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Aug 17, 2018, 4:12:36 PM8/17/18
to dataverse...@googlegroups.com
Well, https://dataverse.harvard.edu/dataverse/ssda_ucla is not what "Server URL" is expecting. It's expecting https://dataverse.harvard.edu/oai instead.


To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/5c58fd46-04f8-417f-ae3b-bcd1eacbd529%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jamie Jamison

unread,
Aug 17, 2018, 4:20:13 PM8/17/18
to Dataverse Users Community
Thank you - yet another beginner mistake!
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Aug 17, 2018, 4:27:09 PM8/17/18
to dataverse...@googlegroups.com
No worries. You may be interested in this "List of Dataverse installation OAI-PMH (Harvesting) URLs and sets" spreadsheet: https://docs.google.com/spreadsheets/d/12cxymvXCqP_kCsLKXQD32go79HBWZ1vU_tdG4kvP5S8/edit?usp=sharing

These are installations of Dataverse that have indicated that they are happy to let people harvest from them.


To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/b48653e6-fb09-4a6f-ab66-6f05645282d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

steven.f...@gmail.com

unread,
Sep 10, 2018, 10:31:33 AM9/10/18
to Dataverse Users Community
Hi Philip,

I get the same message as you for "https://zenodo.org/oai2d" on dataverse 4.8.6 and 4.9.1 in the server logs:

"Failed to find a global identifier in the OAI_DC XML record"

Could you solve the problem? Should we create a new issue on github?

thank you very much


Le jeudi 16 août 2018 17:14:16 UTC+2, Philip Durbin a écrit :
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Sep 10, 2018, 5:18:31 PM9/10/18
to dataverse...@googlegroups.com
Hi Steven,

No, I don't have a fix for the problem. Yes, please do create an issue at https://github.com/IQSS/dataverse/issues when you get a chance.

Thanks!

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/fd58d5c0-00b1-4448-88bd-4caae7e8ff96%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.