C-Store stucks during dicom import

[Денис Смирнов]

Hi!

I have a huge dvd archive with MR images and I import them to Orthanc with a c# program based on fo-dicom. Sometimes import process stucks and in Orthanc log (--trace level) I get
​
I0626 17:44:56.890514 FilesystemStorage.cpp:118] Creating attachment "1cf65351-6b5b-4a5e-8098-5b62aeb6600a" of "DICOM" type (size: 1MB)
I0626 17:44:56.925272 FilesystemStorage.cpp:118] Creating attachment "e2f17ab8-c3b0-442e-b454-5f94bd260a61" of "JSON summary of DICOM" type (size: 1MB)
E0626 17:47:57.099443 RunnableWorkersPool.cpp:82] Native exception while handling some runnable object

First of all I've checked attachements in database with uuids from log. They were not created.

Secondly, I analyzed traffic with WireShark (dicom filter). It seems that client sucessfully sends c-ctore to Orthanc. Orthanc get all c-store packets but doesn't answer with dicom c-store response. And doesn't save the last attachment.

Thirdly. I tryed to resend to Orthanc images that caused a problem (may be there was something wrong with them). Files seemed ok cause during second attemp they successfully saved to Orthanc.

So sometimes during big imports something happens to Orthanc. It stucks and it doesn't save images to disk (that's why no cstore response and import hangs). Also ther is no error/warning information in my logs on trace level. Files seems to be ok (they can be reimported to Orthanc with the same tool). Orthanc is build from the last source version (release, not debug). OS - Ubuntu xenial. PG plugin (latest version build from source). Any ideas what can cause a problem?

P.S. If you need aditional information tell me please)

[Alain Mazy]

Hi David,

Thanks for the report.

At first look, since this is not reproducible at the file level, this looks like a resource exhaustion or ...  However, the out-of-memory exception shall already be logged correctly.

I have just added 3 lines to that file to try to get a bit more detail about the error but, honestly, I'm not sure this will help.  https://bitbucket.org/sjodogne/orthanc/commits/4d500a555aade2e91e0421be08585e7ce9a749af

Could you eventually give it a try with this new code and get back to us if you get anything different in the logs ?

If this does not help, could you eventually try to import your files with another tool like DCMTK ?  Or, could you eventually share your fo-dicom code so we can try to reproduce on our side ?

Best regards,

Alain.

--

Alain Mazy / CTO 
a...@osimis.io / +32 494 31 67 27

	OSIMIS S.A.  Rue bois Saint-Jean 15/1BE-4102 Seraing  www.osimis.io

[Sébastien Jodogne]

Hello,

If you try to re-import the same set of files, does Orthanc also throws this error message?

If so, it strongly calls for an error within some of the DICOM files. Please send us a (possibly anonymized) sample DICOM file that triggers this error.

You could also try with a barebone Orthanc installation, without any plugin or Lua script, in order to check whether the issue is related to your PostgreSQL setup or to the core of Orthanc.

In any case, we need a full debugger trace to understand from where the problem comes. First compile Orthanc and all its associated plugins in debug mode (i.e. add the "-DCMAKE_BUILD_TYPE=Debug" option to cmake), then invoke "gdb" similarly to:

# gdb --ex "catch t" --ex r --ex bt --args ./Orthanc --verbose Configuration.json

Execute your import script, and copy/paste the backtrace produced by gdb.

HTH,

Sébastien-

[Денис Смирнов]

Hello,

Thanks for 3 line commit for logging an exception. Finally I reproduced an error. The problem was in file system corruption:

E0629 11:16:50.952267 StoreScp.cpp:220] Exception while storing DICOM: The directory to be created is already occupied by a regular file

E0629 13:07:36.835250 RunnableWorkersPool.cpp:82] std::exception while handling some runnable object: boost::filesystem::create_directory: Structure needs cleaning: "/data/orthanc/e2/f1"

 

среда, 28 июня 2017 г., 23:00:01 UTC+10 пользователь Sébastien Jodogne написал:

[Alain Mazy]

Hello,

Indeed, this error is signaling that your file system is corrupted and that you need to run fsck tool or something like that.

Can you confirm that running such a tool fixed your problem ?

[Денис Смирнов]

Hello,

Yes, I'll report you about results. I think to report you on the next week because file system corruption happened to 10Tb iscsi with 3,2Tb of dicom images. So I can start only at Saturday evening (when diagnostic center closes until Monday). I plan to mount corrupted iscsi as a readonly layer and make a low level backup with dd/ddrescue of 10Tb before restoring file system. There is a small chance to loose all data and I am a little paranoid about it. Sending 10Tb of data over 1Gb LAN is not a quick task... so report only next week)

30 июня 2017 г., в 20:24, Alain Mazy <a...@osimis.io> написал(а):

[Денис Смирнов]

Hello,

Fsck fixed my problem. It removed corrupted inodes anf now all dicom files save successfully during big imports.

30 июня 2017 г., в 23:17, Денис Смирнов <dart...@gmail.com> написал(а):