Curation stalls but publishing to RDA DOI management succeeds

[David Lewis]

Hi everyone,

We are having issues with ReDBox 1.6.2 records that have been submitted and waiting for curation.

The DOI minting has completed successfully and the records are appearing on the ANDS DOI management page.

The records which we have minted DOIs for are all stuck at "(waiting on curation)"

dc:relation.vivo:Dataset.0.dc:identifier

    10.5072/33/58d496a4c2282 (waiting on curation) (OPTIONAL)

dc:creator.foaf:Person.0.dc:identifier

    curtin.edu.au/parties/people/69 (waiting on curation)

locrel:prc.foaf:Person.dc:identifier

    curtin.edu.au/parties/people/69 (waiting on curation)

foaf:Organization.dc:identifier

    curtin.edu.au/parties/groups/43 (waiting on curation)

Could someone please advise me on what is happening and how I can resolve it?

I am pretty new to ReDBox, so I'm not sure what file / section to provide here.

Thanks
David Lewis

[David Lewis]

Here is a segment from our transactionManager.log file. I hope this helps.

2017-07-24 15:01:39,754 transactionManager ERROR  CurationManager      Cannot resolve ID '10.5072/33/58d496a4c2282'
2017-07-24 15:01:39,754 transactionManager ERROR  CurationManager      Error processing related objects '9debb6688ee9abcd69dcbad1d1157393':
com.googlecode.fascinator.api.transaction.TransactionException: Cannot resolve identifer: 10.5072/33/58d496a4c2282
        at com.googlecode.fascinator.redbox.plugins.curation.redbox.CurationManager.checkChildren(CurationManager.java:979) [plugin-transaction-curation-redbox-1.6.2.jar:na]
        at com.googlecode.fascinator.redbox.plugins.curation.redbox.CurationManager.checkChildren(CurationManager.java:932) [plugin-transaction-curation-redbox-1.6.2.jar:na]
        at com.googlecode.fascinator.redbox.plugins.curation.redbox.CurationManager.curation(CurationManager.java:720) [plugin-transaction-curation-redbox-1.6.2.jar:na]
        at com.googlecode.fascinator.redbox.plugins.curation.redbox.CurationManager.parseMessage(CurationManager.java:1437) [plugin-transaction-curation-redbox-1.6.2.jar:na]
        at com.googlecode.fascinator.common.transaction.GenericTransactionManager.parseMessage(GenericTransactionManager.java:172) [fascinator-common-1.1.5.jar:na]
        at com.googlecode.fascinator.messaging.TransactionManagerQueueConsumer.onMessage(TransactionManagerQueueConsumer.java:382) [fascinator-core-1.1.5.jar:na]
        at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1088) [activemq-all-5.3.0.jar:5.3.0]
        at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:127) [activemq-all-5.3.0.jar:5.3.0]
        at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:197) [activemq-all-5.3.0.jar:5.3.0]
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) [activemq-all-5.3.0.jar:5.3.0]
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) [activemq-all-5.3.0.jar:5.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_141]
        at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141]

[Vicki Picasso]

Hi David,

Is it possible that your records are waiting action in TIM? To complete curation process?

cheers, Vicki

VICKI PICASSO | Senior Librarian, Research Support Services
University Library | Academic Division | The University of Newcastle (UON)
University Drive | Callaghan | NSW | 2308 | Australia

T: +61 2 492 15861   E: vicki....@newcastle.edu.au

 

UON | A global leader in innovation | Ranked in the top 250 universities in the world*

*Times Higher Education World University Rankings 2016 and QS World University Rankings 2016

 

---

From: redbo...@googlegroups.com <redbo...@googlegroups.com> on behalf of David Lewis <d.l...@curtin.edu.au>
Sent: 24 July 2017 17:20
To: ReDBox
Subject: [ReDBox] Curation stalls but publishing to RDA DOI management succeeds

 

--
-- Website: http://www.redboxresearchdata.com.au
 
You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en
---
You received this message because you are subscribed to the Google Groups "ReDBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grant Jackson]

Hi David,

I presume this problem is in your test environment (since 10.5072 is a test prefix).

I have not minted DOIs in v1.6.2. I understand that DOIs can be minted within ReDBox-Mint either before or after curation, so I wasn't aware that DOI minting would have any affect on curation. Are you saying that if you mint a DOI before curation that dataset is curated with the errors you specified, but if you curate a dataset which has no DOI that dataset is curated ok?


I'm wondering if something else (not the DOI) is holding up the curation. Eg1. Does Mint need to connect to NLA to get an NLA identifier? Eg2. Are you using Related Data fields (which I believe have an issue affecting curation)?

Other ideas:
- Have you successfully minted DOIs before which do not affect curation?
- Can you curate (publish) a dataset which has no DOI?
- Any related errors in main.log for redbox or mint?

Cheers, Grant

On Mon, Jul 24, 2017 at 5:45 PM, Vicki Picasso <vicki....@newcastle.edu.au> wrote:

Hi David,

Is it possible that your records are waiting action in TIM? To complete curation process?

cheers, Vicki

VICKI PICASSO | Senior Librarian, Research Support Services
University Library | Academic Division | The University of Newcastle (UON)
University Drive | Callaghan | NSW | 2308 | Australia

T: +61 2 492 15861   E: vicki.picasso@newcastle.edu.au

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- Website: http://www.redboxresearchdata.com.au
 
You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo+unsubscribe@googlegroups.com.

[David Lewis]

Hello Vicki.

Thanks for your response.

I've checked the Mint system-config.json file and the nlaIntegration.endabled is set to false, so it looks like we're not integrating with Trove / NLA.

The curation I think is referring to local curation.

Interestingly, the new People and Groups entries don't turn up in the OAI-PMH feed, so I'm wondering if there is a problem with Mint.

Thanks and cheers!
David Lewis

On Monday, July 24, 2017 at 4:15:55 PM UTC+8, Vicki Picasso wrote:

Hi David,

Is it possible that your records are waiting action in TIM? To complete curation process?

cheers, Vicki

VICKI PICASSO | Senior Librarian, Research Support Services
University Library | Academic Division | The University of Newcastle (UON)
University Drive | Callaghan | NSW | 2308 | Australia

T: +61 2 492 15861   E: vicki.picasso@newcastle.edu.au

[David Lewis]

Hello Grant

Thanks for your response.

I inherited the management of the ReDBox + Mint server from another staff who left recently, so I'm still trying to wrap my head around ReDBox and Mint.

I had no idea that 1.6.2 was a SNAPSHOT until I looked at the version carefully.

We have been able to successfully mint DOIs on the same server previously, but there was a caveat that if the process was waiting on curation for too long, there was a problem that needed to be attended to.

I'm scratching my head and posterior to figure out what could be wrong. I only see the stack trace in the transactionManager log file that I posted above, but I have no idea what it means.

Cheers, David.

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- Website: http://www.redboxresearchdata.com.au
 

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo...@googlegroups.com.

[Grant Jackson]

Hi David,

I have used 1.6.1 and 1.8 with DOIs without problems, so it is feasible that 1.6.2 is ok. However, I'm not sure your issue is related to DOIs.

Assuming this is your test environment, I would try simple tests (with slightly increasing complexity) to try to determine which step is causing the problem. Eg.

- Attempt to publish a dataset without any links to people in Mint (eg. just type in a name in the People tab) & without a DOI. Does it curate?
- If that works, attempt to publish a dataset with a link to one person in Mint (preferably outside of any group) & without a DOI. Does it curate?
- If that works, attempt to publish a dataset with a link to one person in Mint within a group & without a DOI. Does it curate?
- Try adding a DOI to one (or each?) of the above datasets. Is the DOI creation successful?
- Try repeating the first 3 tests but mint a DOI & verify it is successful before curating (ie. before clicking publish). Do they curate?

Also, you said:

> Interestingly, the new People and Groups entries don't turn up in the OAI-PMH feed...

Not certain what you mean by "new" but people and groups will not appear in OAI-PMH until they are successfully curated. Are you saying that if your dataset has links to people and groups which have been recently added to Mint, this will cause the dataset curation to fail? If so, perhaps something about your Mint config or CSV data has changed.

Hope this helps.

Cheers, Grant

T: +61 2 492 15861   E: vicki....@newcastle.edu.au

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo+unsubscribe@googlegroups.com.

[David Lewis]

Hello Grant

Thanks for the tips. I'll pull the guys in the project team together to perform the trouble shooting.

To answer your question, the 'new' people and groups record pertain to the latest data set that we are attempting to import into ReDBox and publish to RDA. The new data set has references to a new group and new person as the author of the new research data.

We'll try out the test cases and fill you in on our progress.

Thanks and cheers, David.

T: +61 2 492 15861   E: vicki.picasso@newcastle.edu.au

[Grant Jackson]

Hi David,

If you find the issue seems to be related to loading person or group data into Mint, there are some references below which might help.

http://www.redboxresearchdata.com.au/documentation/system-administration/administering-mint/loading-data
http://www.redboxresearchdata.com.au/documentation/system-administration/administering-mint/loading-data/loading-people-data
http://www.redboxresearchdata.com.au/documentation/system-administration/administering-mint/loading-data/loading-group-data
https://groups.google.com/forum/#!searchin/redbox-repo/toby$20$20Ingesting$20records$20into$20the$20Mint$20via$20csv|sort:relevance/redbox-repo/r_tvotMIZow/vu3yc4ezkqcJ
https://groups.google.com/forum/#!searchin/redbox-repo/toby$20$20Ingesting$20records$20into$20the$20Mint$20via$20csv|sort:relevance/redbox-repo/rbOCjOs-dFM/zDI9UmZsuSEJ

Cheers, Grant

T: +61 2 492 15861   E: vicki....@newcastle.edu.au

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo+unsubscribe@googlegroups.com.

[Andrew Brazzatti]

Hi David,

All the above mentioned is great advice. One thing I've noticed is that you're using the DOI to reference a related dataset. If I remember correctly Mattias, made some changes to the templates so that the RIF-CS was using it as the record's identifier but I don't think that would have impacted the internal curation process hence the stacktrace you have in your logs.

Something like this is what you probably see on the dataset record you're attempting to relate t​o. If so, you need to use the persistent ID value rather than the DOI. 

Thanks,
Andrew

T: +61 2 492 15861   E: vicki....@newcastle.edu.au

You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

---
You received this message because you are subscribed to the Google Groups "ReDBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redbox-repo+unsubscribe@googlegroups.com.

[David Lewis]

Hi Grant

Thank you very much! I've just come back from one day of leave.

I've just started going through the documentation. I managed to locate and read the first two links myself but I didn't know about the other three.

I think we've discovered the problem. The ReDBox system-config.json had an error with the root record's DOI. I think what is happening is a curation traffic jam. The bad DOI appears to be stuck in the "waiting for curation" state and will never curate (since it's bad), so the remaining user and group references will also be jammed in the same state. Does this make sense?

As always I'm still learning.

Many thanks for your help and guidance!

Cheers, David

[David Lewis]

Hi Andrew

In general, I suspect that it's got to do with an invalid DOI. I believe it is a problem within the ReDBox system-config.json file.

The team I'm working with will be rectifying this DOI today and testing one record to see if we are on the right track.

Many thanks for chipping in with your advice!

Cheers, David

[Grant Jackson]

Hi David,

It is true that if one of the records will not curate, then none of the records in the group of related records (ie. dataset, people and groups) will complete their curation. However, I don't really have any experience with a "bad" DOI. I think you could test your theory in a way similar to what I described before by checking that you can publish simpler records (eg. which have no DOI prior to publication).

Also, unless you have resolved the root cause of your problem, I think it is worth checking Andrew's suggestion since he knows infinitely more about ReDBox than I.

Thanks.

Cheers, Grant

--

[David Lewis]

Hello Vicki, Grant, Andrew

Good news! We've managed to solve the problem. Let me break it down:

1) The root record (there is one) had a different DOI from the one in the ReDBox system-config.json file for the new-alerts dataset baseline section
2) Each record that was imported and assigned the invalid DOI as a parent record
3) Therefore each record was unable to curate because the metadata record referenced by the incorrect DOI was not present in our ReDBox system
4) Because each record is stuck in the curation state it will not publish

Once we corrected the DOI in the system-config.json, restarted ReDBox, deleted off the records that hadn't been processed yet, and then re-imported the records, our first test record went through curation to published within a few seconds.

So thank you everyone for your thoughts and guidance. As I said, we are still learning.

Cheers to all of you,
David Lewis

[Grant Jackson]

Excellent, that's great David.

It sounds like your system might be a little more complicated than ours due to your use of new-alerts for dataset submission.

Cheers, Grant

--