RSNA CTP standalone applications basic steps

[Razvan Bocu]

Hello,

I have installed the CTP standalone application. The administrative service runs on port 10000 and I am able to access the interface. Nevertheless, I still don't understand how to proceed in order to perform basic operations. An an example, I would like to pass some DICOM files to the system and get back the anonymized versions, or any other similar DICOM-related operations. I noticed the scripts editor interface, but I still haven't figured out how to actually use those scripts. Are there some instructions on how to effectively use the system? I have read the existing Wiki, but it is still unclear to me. At least, please provide some pointers on how to put the system at work and perform DICOM related operations. Thank you in advance!

[Razvan Bocu]

I managed to make the file import and anonymization service operational. I still haven't figured out if the storage service supports the specification of an external storage space (ftp, http, etc.) for the root parameter? At the moment, the imported files are saved on the server's local storage space, is it possible to customize this?

[Razvan Bocu]

One last question, how do I configure the system to also generate thumbnails based on the uploaded DICOM files? 

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

The top-level article on CTP is http://mircwiki.rsna.org/index.php?title=MIRC_CTP.

 

The key concept in understanding CTP is the pipeline. A pipeline is an ordered sequence of pipeline stages. A pipeline stage is a component that performs some kind of processing on a file as it flows down the pipeline.

 

There are four types of pipeline stages:

 

1. An import service obtains files from an external source, which might be an external system like a PACS or a workstation, or it might be a storage system like an a directory tree.

 

2. A processor performs some kind of computational function on a file which might or might not result in the file being modified. An anonymizer is an example of a processor, but there are many other types.

 

3. A storage service stores a copy of a file in some kind of storage system.

 

4. An export service sends either a copy of a file or just information abstracted from the file to an external system.

 

As described in the article referenced above, CTP contains many built-in pipeline stages of each of the four types.

 

CTP is configured through an XML file, config.xml. The best way to configure CTP is to use the configuration editor in the Launcher program, as described in http://mircwiki.rsna.org/index.php?title=The_CTP_Launcher_Configuration_Editor. With the editor, you can construct pipelines from scratch, or you can insert any of the pre-configured pipelines listed in the Pipeline menu. (Typically, after inserting a pre-configured pipeline, you will want to delete any stages that are unnecessary in your application and modify some of the pre-configured stage attributes to you liking.) The editor contains help text for each of the parameters of each pipeline stage.

 

CTP can be started using the Launcher program or it can be installed as a service. When setting up a new CTP site or adding a new pipeline, it is usually convenient to use the Launcher during testing0.

 

Here is a simple example. Suppose you want to send DICOM images from your PACS to CTP and have CTP anonymize them and store them in a local directory. The pipeline would consist of a DicomImportService, a DicomAnonymizer, and a FileStorageService.

 

As described in http://mircwiki.rsna.org/index.php?title=The_CTP_FileStorageService_Web_Server, the FileStorageService provides some flexibility in the storage hierarchy. In large collections, the type attribute can be used to group studies by year, month, week, or even day. The root attribute specifies the location of the storage tree within which the studies are stored. The attribute can be specified as a relative or absolute path. When storing on disks located elsewhere on the network, most people have simply mounted the remote disk and referenced it as if it were local. It might be possible to specify a fully qualified network location – I have never tested that.

 

The FileStorageService  has two other capabilities that you might find useful:

 

-- If you specify a port attribute, it will provide an HTTP server on the port, allowing you to see the images and examine their DICOM elements. (You will need to give yourself the read privilege in the CTP User Manager. See sections 3 and 5 in http://mircwiki.rsna.org/index.php?title=CTP_Authentication_and_Privileges for more details.)

-- If you specify a jpeg child element for the FileStorageService (from the Children menu in the editor), it will create and store JPEG images of any size you configure.

 

There are many articles on the Wiki covering most aspects of CTP. See http://mircwiki.rsna.org/index.php?title=MIRC_CTP_Articles for a list.

 

JP

--
You received this message because you are subscribed to the Google Groups "RSNA MIRC CTP/TFS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsnas-ctpmirc-user...@googlegroups.com.
To post to this group, send email to rsnas-ctpmir...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsnas-ctpmirc-user-group.
For more options, visit https://groups.google.com/d/optout.

[Razvan Bocu]

Thank you for the informative post, I shall study it carefully.

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[Razvan Bocu]

Hello,

I am still not able to generate the jpeg thumbnails. The config.xml has the following contents, but the jpeg files do not appear after the import is finished, although I can visualize the contents of the DICOM files in the File Storage Interface. Maybe, is there something that I miss, how could I generate the thumbnails? Ideally, they would be placed in the same folder with the anonymized DICOM files, after the import operation. ImageIO Tools are installed on the server side. Thank you!

<Configuration>

  <Server port="24080"/>

  <Pipeline name="Main Pipeline">

 

     <ImportService

               name="dicomImport"

               id="dicomImport"

               class="org.rsna.ctp.stdstages.DicomImportService"

               root="roots/dicomImportService"

               port="24104"

               logConnections="no"

               suppressDuplicates="no" />

    

     <Processor

            name="pixel-checker"

            id="pixel-checker"

            class="org.rsna.ctp.stdstages.DicomFilter"

            root="roots/pixel-checker"

            script="scripts/pixel-checker-filter.script"

            quarantine="quarantines/pixel-checker" />

     

    <Processor

            name="idmap"

            class="org.rsna.ctp.stdstages.IDMap"

            root="roots/idmap" />

 

   <StorageService

            name="storage"

            id="storage"

            class="org.rsna.ctp.stdstages.FileStorageService"

            root="/DCM_ImportSystem/Imported_DCM_Data/"                      

            type="month"

            timeDepth="0"

            acceptDuplicateUIDs="yes"

            acceptDicomObjects="yes"

            acceptXmlObjects="no"

            acceptZipObjects="no"

            acceptFileObjects="no"

            returnStoredFile="yes"

port="24105"

            quarantine="quarantines/storage" >

<jpeg frame="first" wmax="10000" wmin="96" q="-1" />

</StorageService> 

       

     <Anonymizer

       name="DicomAnonymizer"

       class="org.rsna.ctp.stdstages.DicomAnonymizer"

       root="de-id/dicom-anonymizer"

       script="scripts/DicomAnonymizer.script"                                 

       quarantine="quarantines/dicom-anonymizer" />

 

  </Pipeline>

 

</Configuration>

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[Razvan Bocu]

I think I made it. Probably, the inability to generate thumbnails was generated by the fact that the server which was hosting the CTP toolkit was not having Image IO Tools installed. On that system there is JRE 8 installed, but when I try to install Image IO Tools using the installer on the RSNA site, it says that at least JRE1.4 should be present. The operating system on that server is Window Server 2012. Any idea on how I could make image display and thumbnail generation operational in that environment?

Sorry for posting this message series, but I am excited by CTP, and I would like to underestand it as soon as possible.

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[Sean Doyle]

Is it the 32 bit version of Java? That could also be an issue. JAI doesn't work with 64 bit Java.

Another issue that you might run into later is that I've had issues with automatic Java updates messing up JAI. So - everything is working but a month later you get mysterious failures because the *.jar or *.dll files are missing. So - I keep copies of the *.jar and *.dll files around to re-install if they disappear. 

I don't know about Windows Server 2012; I know it works fine in Windows Server 2008 and Windows 7. 

--

You received this message because you are subscribed to the Google Groups "RSNA MIRC CTP/TFS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rsnas-ctpmirc-user-group+unsub...@googlegroups.com.
To post to this group, send email to rsnas-ctpmirc-user-group@googlegroups.com.

[Razvan Bocu]

Hello,

Would it be possible to tell the file storage pipeline to store the anonymized(eventually thumbnails) files on a remote server, other than the local storage space on the CTP server? In general, what storage options exist for the generated files, apart from the server's own storage space?

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[Sean Doyle]

We store the data on a CIFS file server. We're currently running CTP on a Windows server under a user account and this user has access to the file server (as determined by Active Directory credentials). If the directory you wanted to write out to is 

\\servername\folder

you have to enter

\\\\servername\\folder

into the CTP configurations.

The only issue that we've encountered is that one of pipeline stages is slow writing to network servers. I suspect that it's a buffered I/O issue in one component. What we did to work around this was to have the root for the pipeline be on the local machine and just write the final output to the network share. This works extremely well. 

--

[Razvan Bocu]

Hello,

Meantime, I tried to deploy the system un Ubuntu Server 14.04, I installed the Java Image IO Tools, but the thumbnails generation fails at import times, and the import pipeline also crashes. I attach the picture depicting the error chain to this message. Any idea on how to correctly deploy the system in Ubuntu 14.04? I used the java -jar Runner.jar to start it.

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

The NoClassDefFoundError for the StreamSegmentMapper class is telling you that the ImageIO Tools are not installed.

 

Check that these files are present:

 

{Java JRE Directory}/lib/ext/jai_imageio.jar

{Java JRE Directory}/lib/ext/clibwrapper_jiio.jar

{Java JRE Directory}/i386/libclib_jiio.so

 

JP

From: Razvan Bocu

Sent: Wednesday, September 21, 2016 7:51 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

 

 

--
You received this message because you are subscribed to the Google Groups "RSNA MIRC CTP/TFS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rsnas-ctpmirc-user...@googlegroups.com.
To post to this group, send email to rsnas-ctpmir...@googlegroups.com.

[Razvan Bocu]

Hello,

Is there any way to configure the CTP anonymization process so that the original tree folder structure that is processed is kept as much as possible on the output(anonymized) output. At what degree is this possible and how? 

Thank you very much!

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

CTP storage services do not know the source of the objects they process, so they can't mimic the structure in what they save.

 

If you want to anonymize a tree structure, one way to do it would be to copy the whole tree, then run DicomEditor. DicomEditor uses the same anonymizer as CTP. It can walk a directory tree and anonymize all the DICOM files it finds.

 

The installer is available at: http://mirc.rsna.org/download/DicomEditor-installer.jar.

 

Simple instructions are available at: http://mircwiki.rsna.org/index.php?title=DicomEditor.

 

When you run the program, it will display a window like this:

 

 

The directory tab describes some of the features in more detail. Depending on how the files to be anonymized are named, you may have to click the ".dcm" button in the footer and specify other extensions. (If you specify [dcm], it will select files whose names look like UIDs.)

 

You can configure the anonymizer on the Anonymizer tab. The default script is one that is used by some projects that use CTP, but you should make sure it does what you want before starting the anonymization.

 

If you have made a copy of the input files as a backup, then uncheck the "Change names of modified files" button. That will cause the anonymizer to overwrite the files as it goes along.

 

Once you're ready, navigate to the rot of the tree, check the "Include subdirectories" box, and click the Anonymize button.

I hope this helps.

 

JP

 

rom: Razvan Bocu

Sent: Wednesday, October 12, 2016 9:21 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

[Razvan Bocu]

Thank you for this answer. One more question, would it be possible to somehow generate an IDMap using the DicomEditor, so that anonymized PatientIDs, for example, are associated to their original values, like the main CTP import system does?

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

DicomEditor doesn't have an ID mapping feature.

 

All I could suggest is to process the original data twice – once with CTP to build the ID map, and once with DicomEditor to produce the anonymized objects in the directory structure you want to preserve. To make it work, you will have to ensure that you use the same anonymizer script for both. It's not a pretty solution, but if you really need to preserve the original directory structure, that's all I can suggest.

 

JP

 

From: Razvan Bocu

Sent: Thursday, October 13, 2016 6:13 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

 

[Razvan Bocu]

Thank you for the valuable efforts and suppory. I have another question. Let us suppose that a certain dcm file is present in two or more directories of a dcm data submission that is processed by a pipeline. The uniqueness of the UIDs can be assured by the proper setting in the config.xml file. The question is, can the import process be configured in such a way so that the possible duplicate dcm files are detected, and subsequent uploads of the duplicates get blocked or at least flagged?

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

There is no way to suppress the processing of objects with the same SOPInstanceUID as previously processed objects.

 

The DicomImportService has a configuration parameter that allows it to make a log entry when it receives an object with the same SOPInstanceUID as any of the last 10 objects it has received. This feature is designed to be used when testing a new configuration or when you suspect untoward behavior of your PACS export function. There is also a parameter to prevent such objects from being queued for processing. Since the depth is only 10 objects, it won't meet your requirement in the general case.

 

It would be possible to implement an extension to the ArchiveImportService to suppress duplicates. What import service are you using? (Or are you using FileSender to select the objects to send to CTP?)

 

JP

 

From: Razvan Bocu

Sent: Thursday, October 20, 2016 2:57 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

 

[Razvan Bocu]

I use the FileSender to send the files to the CTP backend. What would be involved into enabling the extension to the ArchiveImportService to suppress the duplicates?

I currently use the following config.xml file for the CTP backend. I am open to any suggestions that may help at suppressing the file duplicates.

<Configuration>
  <Server port="24080"/>
  <Pipeline name="Main Pipeline">



 
     <ImportService
               name="dicomImport"
               id="dicomImport"
               class="org.rsna.ctp.stdstages.DicomImportService"
               root="roots/dicomImportService"
               port="24104"
               logConnections="no"
               suppressDuplicates="no" />  
   
     <Processor
            name="pixel-checker"
            id="pixel-checker"
            class="org.rsna.ctp.stdstages.DicomFilter"
            root="roots/pixel-checker"
            script="scripts/pixel-checker-filter.script"
            quarantine="quarantines/pixel-checker" />
 
    <Processor

            name="IDMap"
            class="org.rsna.ctp.stdstages.IDMap"
            root="directory_path" />

 
   <StorageService
            name="storage"
            id="storage"
            class="org.rsna.ctp.stdstages.FileStorageService"

            root="directory_path"                      
            type="month"
            timeDepth="0"
            acceptDuplicateUIDs="no"
            acceptDicomObjects="yes"
            acceptXmlObjects="no"
            acceptZipObjects="yes"

            acceptFileObjects="no"
            returnStoredFile="yes"
 port="24105"
            quarantine="quarantines/storage" >

 
 <!--<jpeg frame="first" wmax="10000" wmin="96" q="-1" />-->

 
 </StorageService>
 
 
 
       
     <Anonymizer
       name="DicomAnonymizer"
       class="org.rsna.ctp.stdstages.DicomAnonymizer"
       root="de-id/dicom-anonymizer"
       script="scripts/DicomAnonymizer.script"                                
       quarantine="quarantines/dicom-anonymizer" />
   
 
  </Pipeline>
 
</Configuration>

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

If you're using FileSender to send the files, then the best thing to do is to add the duplicate suppression feature to FileSender.

 

I'll take a look at it later today.

 

JP

 

From: Razvan Bocu

Sent: Friday, October 21, 2016 4:06 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

 

[Razvan Bocu]

Thank you very much, I am looking forward to it. 

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

I put a new FileSender on the RSNA site at:

 

    http://mirc.rsna.org/download/FileSender-installer.jar

 

This one is version 24 (shown in the header bar of the window).

 

This version has a new checkbox in the lower right pane:

 

 

When checked, the Skip duplicate SOPInstanceUIDs for DICOM box causes FileSender to skip files that have the same SOPInstanceUID as files that have already been transmitted in the current send request. It only applies to DICOM send requests (not HTTP).

 

When the box is unchecked, FileSender sends all files it encounters.

 

JP

[Razvan Bocu]

Excellent,

Thank you very much. Could we also see the code change on github or elsewhere? I would also like to learn, this is the reason I ask.

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[John Perry]

The source is checked into my GitHub repository:

 

    https://github.com/johnperry/FileSender

 

JP

 

From: Razvan Bocu

Sent: Friday, October 21, 2016 8:32 AM

To: RSNA MIRC CTP/TFS User Group

Subject: [MIRC CTP/TFS User Group] Re: RSNA CTP standalone applications basic steps

 

[Razvan Bocu]

Excellent, I'll study the code carefully. 

Thank you!

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[Razvan Bocu]

Hello,

Although the question's subject is slightly offtopic, I thought that it is preferable not to open a new thread just for this question. Thus, I would like to ask, is there any possibility that CTP would process NIfTI files (.nii, .nii.gz) in terms of anonymization and/or header metadata(tags) extraction? If not, what would be, according to your opinion, the optimal software alternative in order to process the NIfTI files? Or, at least, is there any reliable NIfTI to DICOM converter? 

Thank you!

On Tuesday, September 6, 2016 at 4:40:10 PM UTC+3, Razvan Bocu wrote:

[David Clunie]

NIfTI files don't contain any identifying information:

https://nifti.nimh.nih.gov/pub/dist/src/niftilib/nifti1.h

At least not unless the additional BIDS JSON file is present.

I have a crude NIfTI to DICOM converter in my toolkit, which
requires you to supply limited demographic metadata as input:

http://www.dclunie.com/pixelmed/software/javadoc/com/pixelmed/convert/NIfTI1ToDicom.html

David