Intermittent XSync error

54 views
Skip to first unread message

Simon Doran

unread,
Oct 27, 2022, 4:33:54 PM10/27/22
to xnat_discussion

Hi Everyone (and particularly Mohana!),


  I've recently started using XSync for a project:


Source: XNAT 1.8.2.2

Destination: XNAT 1.8.2.2

Sync'ing: just imaging sessions

Anonymisation:

version "6.2"

removeAllPrivateTags


  I've initiated 4 sync tasks with the following outcomes:

XSync_stats.png

 

The failures have occurred on different imaging sessions each time, with the imaging session where the first failure occurred on 26 Oct seeming to transfer fine on 27 Oct. In each case, the failure is characterised by a freeze in data transfer for several hours, followed by a resumption later in the day, with no intervention by me.


The error reported by XSync in the email sent to me is:


org.nrg.xsync.exception.XsyncStoreException: Unable to store for subject 0489 experiment 20190501_102000_Ingenia at org.nrg.xsync.local.XsyncExperimentTransfer.syncExperiment(XsyncExperimentTransfer.java:183) at org.nrg.xsync.local.RemoteSubject.pushExperiment(RemoteSubject.java:523) at org.nrg.xsync.local.RemoteSubject.syncExperiments(RemoteSubject.java:468) at org.nrg.xsync.local.RemoteSubject.sync(RemoteSubject.java:203) at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.syncSubject(ProjectChangeDiscoverer.java:443) at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.sync(ProjectChangeDiscoverer.java:186) at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.call(ProjectChangeDiscoverer.java:107) at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.call(ProjectChangeDiscoverer.java:53) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)


  The observed behaviour of pausing for a long period is entirely consistent with xsync.log, which has this:


2022-10-27 12:47:32,019 [xsync-thread-4] ERROR org.nrg.xsync.services.remote.RemoteRESTServiceImpl - importXar will be reattempted in 7200 seconds.

but unfortunately, the cause of the error is not detailed enough to be diagnostic:

2022-10-27 14:47:32,078 [xsync-thread-4] ERROR org.nrg.xsync.services.remote.RemoteRESTServiceImpl - Exception thrown during importXar

org.springframework.web.client.ResourceAccessException: I/O error on PUT request for "https://**********/data/user/cache/resources/20221027_14473222/files/1666860450911_***************.xar": Error writing request body to server; nested exception is java.io.IOException: Error writing request body to server

Any thoughts as to where I could go next to diagnose this, particularly given that as soon as XSync gives up on this imaging session after three tries, all the rest of the imaging sessions sync just fine?

Best wishes,

Simon


        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:675)

        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:622)

        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:540)

        at org.nrg.xsync.services.remote.RemoteRESTServiceImpl.uploadXarToCache(RemoteRESTServiceImpl.java:252)

        at org.nrg.xsync.services.remote.RemoteRESTServiceImpl.importXarWithoutRetry(RemoteRESTServiceImpl.java:184)

        at org.nrg.xsync.services.remote.RemoteRESTServiceImpl.importXar(RemoteRESTServiceImpl.java:143)

        at org.nrg.xsync.connection.RemoteConnectionManager.importXar(RemoteConnectionManager.java:348)

        at org.nrg.xsync.local.XsyncExperimentTransfer.storeXar(XsyncExperimentTransfer.java:445)

        at org.nrg.xsync.local.XsyncExperimentTransfer.syncExperiment(XsyncExperimentTransfer.java:181)

        at org.nrg.xsync.local.RemoteSubject.pushExperiment(RemoteSubject.java:523)

        at org.nrg.xsync.local.RemoteSubject.syncExperiments(RemoteSubject.java:468)

        at org.nrg.xsync.local.RemoteSubject.sync(RemoteSubject.java:203)

        at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.syncSubject(ProjectChangeDiscoverer.java:443)

        at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.sync(ProjectChangeDiscoverer.java:186)

        at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.call(ProjectChangeDiscoverer.java:107)

        at org.nrg.xsync.discoverer.ProjectChangeDiscoverer.call(ProjectChangeDiscoverer.java:53)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

Caused by: java.io.IOException: Error writing request body to server

        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3599)

        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3582)

        at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:263)

        at org.springframework.util.StreamUtils.copy(StreamUtils.java:143)

        at org.springframework.http.converter.ResourceHttpMessageConverter.writeContent(ResourceHttpMessageConverter.java:110)

        at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:102)

        at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:45)

        at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:228)

        at org.springframework.http.converter.FormHttpMessageConverter.writePart(FormHttpMessageConverter.java:378)

        at org.springframework.http.converter.FormHttpMessageConverter.writeParts(FormHttpMessageConverter.java:358)

        at org.springframework.http.converter.FormHttpMessageConverter.writeMultipart(FormHttpMessageConverter.java:347)

        at org.springframework.http.converter.FormHttpMessageConverter.write(FormHttpMessageConverter.java:260)

        at org.springframework.http.converter.FormHttpMessageConverter.write(FormHttpMessageConverter.java:92)

        at org.springframework.web.client.RestTemplate$HttpEntityRequestCallback.doWithRequest(RestTemplate.java:899)

        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:659)

        ... 19 common frames omitted

2022-10-27 14:47:32,079 [xsync-thread-4] ERROR org.nrg.xsync.local.XsyncExperimentTransfer - Could not sync experiment

Mohana Ramaratnam

unread,
Oct 28, 2022, 1:45:35 AM10/28/22
to xnat_di...@googlegroups.com
Hi Simon,

Have you investigated logs on the destination side to see if around the time of the sync from source if something goofy was happening on the destination side?

Mohana

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/f3a3499e-0c77-44bb-b8ce-320c4ae7282cn%40googlegroups.com.

Simon Doran

unread,
Oct 28, 2022, 8:45:11 PM10/28/22
to xnat_discussion
Hi Mohana,

  Ah, thanks for that. I didn't put 2 and 2 together to realise that there were equivalent logs on the destination server. I will take a look.

  Best wishes,

Simon

Reply all
Reply to author
Forward
0 new messages