Import stalls once each of the worker threads have died with exception "03180088 Failed to copy content from file: Caused by: java.nio.channels.ClosedByInterruptException"

[Aaron Trewern]

After running for a while each of the worker threads appears to die with an exception like the one below.

Once each of the threads has dies the process makes no further progress and does not finish.

The message implies that there was a problem processing the content file.

I don't think it is a permissions issue since the import folder has global read permissions on all files and folders and thousands of content files are processed successfully.

The lowest level cause suggests that the thread copying the file content was interrupted.

Not sure where to look next.

2018-04-17 23:52:38,820  ERROR [bulkimport.impl.Scanner] [BulkImport-Importer-0002] BULKIMPORT: Bulk import from 'Default' failed.
 org.alfresco.extension.bulkimport.impl.ItemImportException: Unexpected exception:
 class org.alfresco.service.cmr.repository.ContentIOException: 03170011 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/17/23/52/2e562c61-f2a0-4737-8545-39d817ece981.bin, mimetype=application/vnd.ms-outlook, size=1155072, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL HONG KONG/2008002963 - 29 Jul/711302.msg
While importing item: 711302.msg (1 version):
    HEAD: <content> <metadata>
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:230)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchImpl(BatchImporterImpl.java:184)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$200(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$2.execute(BatchImporterImpl.java:161)
    at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchInTxn(BatchImporterImpl.java:152)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$000(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$1.doWork(BatchImporterImpl.java:130)
    at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:555)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatch(BatchImporterImpl.java:124)
    at org.alfresco.extension.bulkimport.impl.Scanner$BatchImportJob.run(Scanner.java:472)
    at org.alfresco.extension.bulkimport.impl.BulkImportThreadPoolExecutor$1.run(BulkImportThreadPoolExecutor.java:136)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 03170011 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/17/23/52/2e562c61-f2a0-4737-8545-39d817ece981.bin, mimetype=application/vnd.ms-outlook, size=1155072, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL HONG KONG/2008002963 - 29 Jul/711302.msg
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:491)
    at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportItemVersion.putContent(FilesystemBulkImportItemVersion.java:211)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContent(BatchImporterImpl.java:627)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContentAndMetadata(BatchImporterImpl.java:488)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersion(BatchImporterImpl.java:435)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importFile(BatchImporterImpl.java:390)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:212)
    ... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
    at org.alfresco.repo.content.AbstractContentAccessor$CallbackFileChannel.write(AbstractContentAccessor.java:429)
    at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
    at java.nio.channels.Channels.writeFully(Channels.java:101)
    at java.nio.channels.Channels.access$000(Channels.java:61)
    at java.nio.channels.Channels$1.write(Channels.java:174)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at org.alfresco.repo.content.LimitedStreamCopier.copyStreamsLong(LimitedStreamCopier.java:94)
    at org.alfresco.repo.content.AbstractContentWriter.copyStreams(AbstractContentWriter.java:509)
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:486)
    ... 20 more

2018-04-18 01:09:33,180  ERROR [bulkimport.impl.Scanner] [BulkImport-Importer-0001] BULKIMPORT: Bulk import from 'Default' failed.
 org.alfresco.extension.bulkimport.impl.ItemImportException: Unexpected exception:
 class org.alfresco.service.cmr.repository.ContentIOException: 03180088 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/18/1/9/0ea1987b-d6a5-45b7-b3c8-97a52ce1a255.bin, mimetype=application/vnd.ms-outlook, size=0, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL TEMASEK/2013000812 - 21 Apr/2045462.msg
While importing item: 2045462.msg (1 version):
    HEAD: <content> <metadata>
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:230)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchImpl(BatchImporterImpl.java:184)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$200(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$2.execute(BatchImporterImpl.java:161)
    at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchInTxn(BatchImporterImpl.java:152)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$000(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$1.doWork(BatchImporterImpl.java:130)
    at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:555)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatch(BatchImporterImpl.java:124)
    at org.alfresco.extension.bulkimport.impl.Scanner$BatchImportJob.run(Scanner.java:472)
    at org.alfresco.extension.bulkimport.impl.BulkImportThreadPoolExecutor$1.run(BulkImportThreadPoolExecutor.java:136)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 03180088 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/18/1/9/0ea1987b-d6a5-45b7-b3c8-97a52ce1a255.bin, mimetype=application/vnd.ms-outlook, size=0, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL TEMASEK/2013000812 - 21 Apr/2045462.msg
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:491)
    at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportItemVersion.putContent(FilesystemBulkImportItemVersion.java:211)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContent(BatchImporterImpl.java:627)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContentAndMetadata(BatchImporterImpl.java:488)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersion(BatchImporterImpl.java:435)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importFile(BatchImporterImpl.java:390)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:212)
    ... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
    at org.alfresco.repo.content.AbstractContentAccessor$CallbackFileChannel.write(AbstractContentAccessor.java:429)
    at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
    at java.nio.channels.Channels.writeFully(Channels.java:101)
    at java.nio.channels.Channels.access$000(Channels.java:61)
    at java.nio.channels.Channels$1.write(Channels.java:174)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at org.alfresco.repo.content.LimitedStreamCopier.copyStreamsLong(LimitedStreamCopier.java:94)
    at org.alfresco.repo.content.AbstractContentWriter.copyStreams(AbstractContentWriter.java:509)
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:486)
    ... 20 more

2018-04-17 14:56:11,610  ERROR [bulkimport.impl.Scanner] [BulkImport-Importer-0007] BULKIMPORT: Bulk import from 'Default' failed.
 org.alfresco.extension.bulkimport.impl.ItemImportException: Unexpected exception:
 class org.alfresco.service.cmr.repository.ContentIOException: 03170212 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/17/14/56/3cea48e6-23d2-41a7-a156-54bcd9e03f56.bin, mimetype=application/vnd.ms-outlook, size=15482880, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL HONG KONG/2013004125 - 16 Oct/2587829.msg
While importing item: 2587829.msg (1 version):
    HEAD: <content> <metadata>
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:230)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchImpl(BatchImporterImpl.java:184)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$200(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$2.execute(BatchImporterImpl.java:161)
    at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatchInTxn(BatchImporterImpl.java:152)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.access$000(BatchImporterImpl.java:69)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl$1.doWork(BatchImporterImpl.java:130)
    at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:555)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importBatch(BatchImporterImpl.java:124)
    at org.alfresco.extension.bulkimport.impl.Scanner$BatchImportJob.run(Scanner.java:472)
    at org.alfresco.extension.bulkimport.impl.BulkImportThreadPoolExecutor$1.run(BulkImportThreadPoolExecutor.java:136)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 03170212 Failed to copy content from file:
   writer: ContentAccessor[ contentUrl=store://2018/4/17/14/56/3cea48e6-23d2-41a7-a156-54bcd9e03f56.bin, mimetype=application/vnd.ms-outlook, size=15482880, encoding=UTF-8, locale=en_GB]
   file: /home/atrewern/export_folder/A/APL HONG KONG/2013004125 - 16 Oct/2587829.msg
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:491)
    at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportItemVersion.putContent(FilesystemBulkImportItemVersion.java:211)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContent(BatchImporterImpl.java:627)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersionContentAndMetadata(BatchImporterImpl.java:488)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importVersion(BatchImporterImpl.java:435)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importFile(BatchImporterImpl.java:390)
    at org.alfresco.extension.bulkimport.impl.BatchImporterImpl.importItem(BatchImporterImpl.java:212)
    ... 14 more
Caused by: java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
    at org.alfresco.repo.content.AbstractContentAccessor$CallbackFileChannel.write(AbstractContentAccessor.java:429)
    at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
    at java.nio.channels.Channels.writeFully(Channels.java:101)
    at java.nio.channels.Channels.access$000(Channels.java:61)
    at java.nio.channels.Channels$1.write(Channels.java:174)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at org.alfresco.repo.content.LimitedStreamCopier.copyStreamsLong(LimitedStreamCopier.java:94)
    at org.alfresco.repo.content.AbstractContentWriter.copyStreams(AbstractContentWriter.java:509)
    at org.alfresco.repo.content.AbstractContentWriter.putContent(AbstractContentWriter.java:486)
    ... 20 more