Need to import LOTS of images!!

[Robby Otts]

I've set up a new dcm4chee system on Ubuntu 12.04.  This will be a "backup" system for users to view images remotely via the web.  I need to import 1000s of images and the dcmsnd utility timed out, or rather I got an OutOfMemory java error, while scanning the images before importing.

Is there a better way to get these images imported into my database rather than using the dcmsnd utility?

Right now I have the external USB hard drive mounted to my linux server.....just need to get them all imported properly.

Any assistance would be great!

Thanks,

Robby Otts

[nicolas...@gmail.com]

You need more memory than the default value (64 MB).

In your shell, type:

export JAVA_OPTS=-Xms128m -Xmx512m

If it doesn't work, insert VM memory arguments in the last line of the dcmsnd file:
"%JAVA%" -Xms128m -Xmx512m %JAVA_OPTS% -cp "%CP%" %MAIN_CLASS% %ARGS%

[Robby Otts]

I tried the export you recommended....export JAVA_OPTS=-Xms128m -Xmx512m

and got the following:

.Exception in thread "main" java.lang.OutOfMemoryErro r: Java heap space

        at java.io.UnixFileSystem.resolve(UnixFileSystem.java:108)

        at java.io.File.<init>(File.java:250)

        at java.io.File.listFiles(File.java:1140)

        at org.dcm4che2.tool.dcmsnd.DcmSnd.addFile(DcmSnd.java:1069)

        at org.dcm4che2.tool.dcmsnd.DcmSnd.addFile(DcmSnd.java:1073)

        at org.dcm4che2.tool.dcmsnd.DcmSnd.main(DcmSnd.java:835)

root@ubuntu:/opt/toolkit/dcm4che-2.0.28/bin#

Is that the same error as before?

On Sun, Apr 6, 2014 at 2:44 AM, <nicolas...@gmail.com> wrote:

You need more memory than the default value (64 MB).

In your shell, type:

export JAVA_OPTS=-Xms128m -Xmx512m


If it doesn't wort, insert VM memory arguments in the last line of the dcmsnd file:

"%JAVA%" -Xms128m -Xmx512m %JAVA_OPTS% -cp "%CP%" %MAIN_CLASS% %ARGS%

On Saturday, April 5, 2014 7:59:24 PM UTC+2, Robby Otts wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "dcm4che" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dcm4che/C0WDIleJCGg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dcm4che+u...@googlegroups.com.
To post to this group, send email to dcm...@googlegroups.com.
Visit this group at http://groups.google.com/group/dcm4che.
For more options, visit https://groups.google.com/d/optout.

[nicolas...@gmail.com]

Did you try the second solution, I would  be surprised if it doesn't work.

[Robby Otts]

The line you recommended is in the dcmsnd.bat file, to be used on Windows machines.  I'm running this on a linux machine and this is the last line in the dcmsnd file:

# Execute the JVM

exec $JAVA $JAVA_OPTS -cp "$CP" $MAIN_CLASS "$@"

[fleetwoodfc]

I usually have a script that will traverse the image directory and calls dcmsnd in such a way that it only has to process a limited batch of images e.g. process 1 month at a time or 1 year. Obviously this depends upon your directory structure.

To unsubscribe from this group and all its topics, send an email to dcm4che+unsubscribe@googlegroups.com.

[Alvaro [Andor]]

If you have lots and lots of images it will be almost impossible unless
you devote loads of gigabytes of ram to the tool.

Usually, and supposing your in a Linux/Unix/Mac environment, I would use
xargs or parallel.

xargs is a quite powerful tool that will serve your dcmsnd tool with
only a certain number of files each time (lets say 200), and run dcmsnd
any times needed until all the parameters are gone.

You won't run, lets say:

./dcmsnd DCM4...@1.1.1.1:11112 /pacs/archive

That would go through all your files (i.e. 50 million) and serve all
files as a parameter to dcmsnd. Serving 50 million parameters to a
command would be crazy, and also dcmsnd will try to read metadata for
all 50 million files before sending, and that would blow out your memory
(that's the problem you had).

So, I usually would do something like this:

find /archive -type f -print0 | xargs -0 ./dcmsnd DCM4...@1.1.1.1:11112

That would search in your /archive folder, look for all things of type
'file', and pass it as parameters to dcmsnd, around 100 or 200 files a
time. xargs will take care of not overflowing the number of parameters
your os will allow at a time. The -print0 and -0 parameters are there to
avoid problems with spaces and special characters.

Also, you may want to send several threads in parallel to improve
performance or send only pieces of your pacs, so open different
terminals, or different 'screen' terminals and on each separate a
different year or month to send:

find /archive/2013/01 -type f -print0 | xargs -0 ./dcmsnd
DCM4...@1.1.1.1:11112
find /archive/2013/02 -type f -print0 | xargs -0 ./dcmsnd
DCM4...@1.1.1.1:11112
find /archive/2013/03 -type f -print0 | xargs -0 ./dcmsnd
DCM4...@1.1.1.1:11112
find /archive/2013/04 -type f -print0 | xargs -0 ./dcmsnd
DCM4...@1.1.1.1:11112
find /archive/2013/05 -type f -print0 | xargs -0 ./dcmsnd
DCM4...@1.1.1.1:11112
.
.
.


I wouldn't do more than 4 threads at a time. Watch your network
broadband and don't choke other services! ;)

--
Alvaro "Andor" Gonzalez
an...@pierdelacabeza.com
GnuPG KeyID 0xa1e486f0

[Robby Otts]

Great info!   I'll give this a try tonight.

Thanks a bunch

Rob

[Robby Otts]

This seems to work pretty well, however, I'm still running out of memory. I'm trying to understand more how java works as I'm more of a unix admin and don't quite understand the how java functions in relation to memory.

Here is the exception I'm getting after about 20 minutes of imports:

2014-04-11 21:39:30,711 ERROR -> (WorkManager(2)-39) [org.dcm4chee.arr.listeners.mdb.ReceiverMDB] Failed processing -

java.lang.ArrayIndexOutOfBoundsException

2014-04-11 21:39:30,760 INFO  DCMSND->DCM4CHEE (TCPServer-1-1) [org.dcm4chex.archive.ejb.session.StorageBean] inserting instance FileMetaIn

fo[uid=1.3.12.2.1107.5.2.6.22723.30000010120114504735900000595

        class=1.2.840.10008.5.1.4.1.1.4/MR Image Storage

        ts=1.2.840.10008.1.2/Implicit VR Little Endian

        impl=1.2.40.0.13.1.1.1-dcm4che-1.4.31]

I first exported the following:  export JAVA_OPTS="-Xms128m -Xmx512m"

Then run the following command for my imports:  find /data/dir2/dir21 -type f -print0 | xargs -0 ./dcmsnd DCM4CHEE@localhost:11112

I have 100's of directories and I'm trying to split them up as much as possible but still have 1000s of images to go through.

Any other suggestions???

Many thanks in advance!

Robby

[Alvaro [Andor]]

I0m neither an expoert on Java, but, That doesn't look like the typical out-of-memory error to me. more like a misunderstanding on what's being read or parsed from the file...

Does the dcm4chee/pacs on the other side gives any error?
Can somebody lend a hand on this?

Thanks

To unsubscribe from this group and all its topics, send an email to dcm4che+u...@googlegroups.com.

To post to this group, send email to dcm...@googlegroups.com.
Visit this group at http://groups.google.com/group/dcm4che.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dcm4che+u...@googlegroups.com.