Tomcat Stuck Thread Issue Resolution

[Peter Matthew Eichman]

Hello all,

Some good news from the UMD corner of the Fedora community: We have solved our Tomcat stuck thread problem!

We finally tracked the issue down to the interaction of the following:

1. We were creating hundreds of resources at a time (the majority of which were web annotations to support our OCR searching capabilities)
2. We were running the internal audit module [1]

Given that for every content write to the repository, the audit module required 2 writes to the repository, we believe that the problem stemmed from filling up the Modeshape event buffer way too rapidly for it to deal with.

To confirm that, we rebuilt our umd-fcrepo-webapp without the internal audit module (an extremely simple change [2]), and have been conducting additional batch load tests on it. So far, we have not had any stuck thread issues whatsoever, and we are closing in on completing our second batch of 500 newspaper issues.

Going forward, we are dropping the internal audit module, and are working on setting up the fcrepo-audit-triplestore [3] Camel route. We had been interested in doing this anyway; the stability and performance issue thus served as a kick in the pants to get it done! :-)

I hope our trials and tribulations of these past weeks can serve as guidance to other folks adopting Fedora 4, so that they can avoid those same pitfalls.

-Peter

[1] https://github.com/fcrepo4-exts/fcrepo-audit

[2] https://github.com/umd-lib/umd-fcrepo-webapp/commit/2f5fc462cf5926cbc8508d9102b040ea97369e52

[3] https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/tree/master/fcrepo-audit-triplestore

--

Peter Eichman
Senior Software Developer
University of Maryland Libraries

peic...@umd.edu

[Peter Matthew Eichman]

Per discussion on today's tech call, I have created this ticket to keep track of this issue, and for folks to continue investigation into whether there is a bug in the fcrepo-audit module.

https://jira.duraspace.org/browse/FCREPO-2648

-Peter

[Andrew Woods]

Hello Peter,

Thank you for the persistence in uncovering this elusive issue!

Regards,

Andrew

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.