Threads generation - Memory heap exception

[Jorge Urzúa]

For our project we integrate the DROID library. Droid runs in the same Java VM of our project. The  integration of DROID is done by the class IANUSDroidInterface (this class is attached). We have big set of files that we want validate with DROID (30.000 files, about 700 GB). The problem is that when we call the method  IANUSDroidInterface.validate, DROID creates a threads and it does not stop them. The attached picture shows that our system keeps stable until we call DROID.  The amount of threads created by DROID increase until we reach a memory heap exception.

My questions:

1. Why after the validation of a file by DROID, the thread stays alive?
2. How can we deal with this use?
3. There is a better way to integrate DROID?

[Radek Hubner]

Hello Jorge,

Thanks for your email. I'm afraid that I don't have any good news. When droid was first design, we didn't expect high integration with other software. This has changes and we are working to refactor core parts of droid to enable easy integration of droid with other systems.

I quickly check your code and I don't see any significant problem. Where I can see problem is DROID, ProfileInstanceManagerImpl.java:242, we start 2 executor services, which are never terminated. In past lot of resources was spend to make DROID more multithreaded and also able to throttle IO resources, pause identification, etc... All this work wasn't done best.

Depends on your requirements, there are few ways how to deal with this problem :

• Fix bug, send us pull request and we can integrate to DROID.
• Run DROID in separate JVM.
• Use Low level DROID API.
• Use NANITE
  

With Low level API, you can create your own instance AsynchDroid and then submit identification task for processing. SubmissionGateway as one of AsynchDroid still create new thread, but you will be able to better control live-cycle of this instance.

NANITE(https://github.com/openpreserve/nanite). It's project from British Library. I don't have any experience with this, but what I know, they provide more "friendly API" for DROID.

Radek, The National Archives

--
You received this message because you are subscribed to the Google Groups "droid-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to droid-list+unsubscribe@googlegroups.com.
To post to this group, send email to droid...@googlegroups.com.
Visit this group at https://groups.google.com/group/droid-list.
For more options, visit https://groups.google.com/d/optout.

[Matt Palmer]

You could also look at the command line project in droid. This doesn't use profiles or the same multithreaded approach. It would be simpler to extract the logic in a reusable way, rather than treating DROID like an embedded server.

On the other hand, I think there are some differences in how the command line identifies files from the guide version which have crept in over time. This was not the original intention!

I do think there needs to be a library version of Droid, with a well defined API. It was on the original TODO list, bit wasn't the priority at the time...

Regards

Matt