Sync tool I/O error: Too many open files

[David Chandek-Stark]

I am seeing a lot of sync errors that look like this in sync.log:

>>>

ERROR  2017/01/19 11:17:04 [pool-3-thread-7] (SyncWorker.java:68) [run()] - Exception syncing file /srv/perkins/duracloud/spaces/ddr-fcr3/content/datastreamStore/

0/9/59/info%3Afedora%2Fduke%3A170844%2Fproperties%2Fproperties.3 was File not found: /srv/perkins/duracloud/spaces/ddr-fcr3/content/datastreamStore/0/9/59/info%3A

fedora%2Fduke%3A170844%2Fproperties%2Fproperties.3

java.lang.RuntimeException: File not found: /srv/perkins/duracloud/spaces/ddr-fcr3/content/datastreamStore/0/9/59/info%3Afedora%2Fduke%3A170844%2Fproperties%2Fpro

perties.3

        at org.duracloud.sync.endpoint.MonitoredFile.computeChecksum(MonitoredFile.java:98) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.MonitoredFile.getChecksum(MonitoredFile.java:83) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.DuraStoreChunkSyncEndpoint.addUpdateContent(DuraStoreChunkSyncEndpoint.java:199) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.addUpdateContent(DuraStoreSyncEndpoint.java:304) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.doAddContent(DuraStoreSyncEndpoint.java:262) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.syncFileAndReturnDetailedResult(DuraStoreSyncEndpoint.java:229) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.mgmt.SyncWorker.run(SyncWorker.java:65) ~[duracloudsync-4.2.0.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

Caused by: java.io.FileNotFoundException: /srv/perkins/duracloud/spaces/ddr-fcr3/content/datastreamStore/0/9/59/info%3Afedora%2Fduke%3A170844%2Fproperties%2Fprope

rties.3 (Too many open files)

        at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_111] 

        at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_111]

        at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_111]

        at org.duracloud.common.util.ChecksumUtil.generateChecksum(ChecksumUtil.java:39) ~[duracloudsync-4.2.0.jar:na]

        at org.duracloud.sync.endpoint.MonitoredFile.computeChecksum(MonitoredFile.java:95) ~[duracloudsync-4.2.0.jar:na]

        ... 9 common frames omitted

<<<

The file referenced does in fact exist and is not new. I am using the "clean start" option for the sync tool (v4.2.0).

Is it possible that the code is not closing file descriptors?

Thanks,

David

[jackhill]

[Jack Hill]

On Thu, 19 Jan 2017, David Chandek-Stark wrote:

> /srv/perkins/duracloud/spaces/ddr-fcr3/content/datastreamStore/0/9/59/info%3Afedora%2Fduke%3A170844%2Fproperties%2Fprope
> rties.3 (Too many open files)
>         at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_111] 
>         at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_111]
>         at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_111]
>         at org.duracloud.common.util.ChecksumUtil.generateChecksum(ChecksumUtil.java:39) ~[duracloudsync-4.2.0.jar:na]
>         at org.duracloud.sync.endpoint.MonitoredFile.computeChecksum(MonitoredFile.java:95) ~[duracloudsync-4.2.0.jar:na]
>         ... 9 common frames omitted
>
> <<<
>
> The file referenced does in fact exist and is not new. I am using the "clean start" option for the sync tool (v4.2.0).
>
> Is it possible that the code is not closing file descriptors?
>
> Thanks,
> David

It looks like we're currently using the default ulimit of 4096 for open
files. Given what the sync tool I would not be surprised if instead of
running into a leak, we just need to increase the limit. Is there advice
on what a reasonable limit should be? Perhaps 40,000 ?

Best,
Jack (I work with David)

[Bill Branan]

Hi David and Jack,

I expect that Jack is correct here. A ulimit of 4096 is likely just too low given the number of files that the SyncTool is processing. The value you set will depend on the capacity of the machine, but as a starting point, I'd suggest trying 65536 (which is 2^16), then adjust from there.

best,

Bill

--
You received this message because you are subscribed to the Google Groups "DuraCloud Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-users+unsubscribe@googlegroups.com.
To post to this group, send email to duracloud-users@googlegroups.com.
Visit this group at https://groups.google.com/group/duracloud-users.
For more options, visit https://groups.google.com/d/optout.