Harvesting problems: harvesting clients enter "FAILED" state

[Piotr Wyrostek]

Dataverses versions: 5.12.1, 5.13

After defining a client on our dataverse working as a data integrator, at first it works as expected. Pressing "run harvesting" gives the "last results":

SUCCESS - n harvested, m deleted, k failed.

However, after some time (maybe after the first "scheduled" run) the "last results"

of that client displays: FAILED.

Pressing "run harvesting" again and again leaves the "last results" constantly in FAILED state.

We have upgraded dataverse from 5.12.1 to 5.13, but the problem remains.

The server log states:

[#|2023-03-21T10:12:36.873+0100|WARNING|Payara 5.2022.3|javax.enterprise.resource.webcontainer.jsf.flash|_ThreadID=90;_ThreadName=http-thread-pool::jk-connector(1);_TimeMillis=1679389956873;_LevelValue=900;_MessageID=jsf.externalcontext.flash.response.already.committed;|
  JSF1095: The response was already committed by the time we tried to set the outgoing cookie for the flash.  Any values stored to the flash will not be available on the next request.|#]

The only solution so far is to delete the client and define it again.

But we have about 90 clients defined and every day some of them change the state to FAILED.

Looking up for some advice.

Regards, Piotr Wyrostek

[Don Sizemore]

Hello,

On your Dataverse client machine there should be a log file for each harvesting client run beneath domain1/logs/ — do they say anything more helpful?

Thank you,

Don

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e7e73123-25ea-4dfa-b7f7-e752e597cc66n%40googlegroups.com.

[Piotr Wyrostek]

Below is the contents of the newest log of one of failing clients:

----------------------------------------------------------------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE log SYSTEM "logger.dtd">
<log>
<record>
  <date>2023-03-22T08:23:39.443517Z</date>
  <millis>1679473419443</millis>
  <nanos>517000</nanos>
  <sequence>73945</sequence>
  <logger>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.weip_2023-03-22T09-23-39</logger>
  <level>INFO</level>
  <class>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean</class>
  <method>logBeginOaiHarvest</method>
  <thread>262</thread>
  <message>from=Thu Feb 02 10:19:05 CET 2023</message>
</record>
<record>
  <date>2023-03-22T08:23:39.529895Z</date>
  <millis>1679473419529</millis>
  <nanos>895000</nanos>
  <sequence>73946</sequence>
  <logger>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.weip_2023-03-22T09-23-39</logger>
  <level>SEVERE</level>
  <class>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean</class>
  <method>doHarvest</method>
  <thread>262</thread>
  <message>Exception processing harvest, server= https://agh.rodbuk.pl/oai,format=dataverse_json io.gdcc.xoai.serviceprovider.exceptions.InvalidOAIResponse OAI responded with code: badArgument</message>
</record>
<record>
  <date>2023-03-22T08:23:39.530447Z</date>
  <millis>1679473419530</millis>
  <nanos>447000</nanos>
  <sequence>73947</sequence>
  <logger>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.weip_2023-03-22T09-23-39</logger>
  <level>SEVERE</level>
  <class>edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean</class>
  <method>logException</method>
  <thread>262</thread>
  <message>io.gdcc.xoai.serviceprovider.exceptions.InvalidOAIResponse OAI responded with code: badArgument
StackTrace:
io.gdcc.xoai.serviceprovider.parsers.ListIdentifiersParser.hasNext(ListIdentifiersParser.java:41)
io.gdcc.xoai.serviceprovider.handler.ListIdentifierHandler.nextIteration(ListIdentifierHandler.java:64)
io.gdcc.xoai.serviceprovider.lazy.ItemIterator.hasNext(ItemIterator.java:31)
io.gdcc.xoai.serviceprovider.lazy.ItemIterator.&lt;init&gt;(ItemIterator.java:22)
io.gdcc.xoai.serviceprovider.ServiceProvider.listIdentifiers(ServiceProvider.java:80)
edu.harvard.iq.dataverse.harvest.client.oai.OaiHandler.runListIdentifiers(OaiHandler.java:254)
edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.harvestOAI(HarvesterServiceBean.java:236)
edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.doHarvest(HarvesterServiceBean.java:175)
edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.doAsyncHarvest(HarvesterServiceBean.java:104)
jdk.internal.reflect.GeneratedMethodAccessor697.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:588)
org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:408)
com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4835)
com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:665)
com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:834)

[Don Sizemore]

It looks like the harvesting server in this case is running version 5.12.1 (the date bug is server-side). This server (https://agh.rodbuk.pl/) will want to upgrade to 5.13, then your harvesting run should succeed.

Thank you,

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/7aad5df4-8486-4bf9-bd5c-5de043e446fcn%40googlegroups.com.

[Piotr Wyrostek]

Don, thank you very much for the suggestion. I will let you know about the results after the upgrade.

Regards,

Piotr