Java Deadlocks with Coldbox 4.2

14 views
Skip to first unread message

Kevin Anderson

unread,
Aug 22, 2016, 5:33:02 PM8/22/16
to col...@googlegroups.com

We have just recently upgraded a Coldbox 3.8 application to CB 4.2. Everything is mostly working fine; however, there is an issue which is causing the server to crash consistently. We’ve not yet been able to put a finger on the exact issue; however, monitoring with the Java Mission Control (5.5) shows java deadlocks occurring. This was NOT happening with 3.8. We did have some code that was using threads, but that was refactored to be single-threaded, in order to eliminate that as an issue.

 

Attached is a stacktrace from the Mission Control application. It appears the problem is happening within the Coldbox framework (i.e. funcONREQUESTSTART -> funcRELOADCHECKS -> funcGETSETTING -> funcSETTINGEXISTS)

 

Has anyone else ran into this issue? We are (literally) getting hundreds of ajp threads blocked. Any ideas would be appreciated.

 

Thanks!

 



Kevin S. Anderson

CIDNE/INDURE Mobile - Software Engineer

Intelligent Software Solutions Inc.

Analysis and Information Management Solutions

Phone: 719.452.7456 | Cell: 719.930.2599 

 

 

API-Block.txt

br...@bradwood.com

unread,
Aug 22, 2016, 5:46:14 PM8/22/16
to col...@googlegroups.com
Hi Ken, to be clear that stack trace does not show a deadlock, it shows a blocked thread.  A deadlock is a fatal scenario in which 2 threads have exclusive access to resources that the other thread needs.  A blocked thread means that one thread is simply waiting on another to release a shared resource.  The mere existence of a waiting thread doesn't signal a problem, but if you're seeing tons of them, something might be awry.
 
I'd need to see a full stack trace of the entire JVM to tell what's going on.   The trace you attached doesn't show the actual resource being waited on, nor the thread that owns it.
 
Also, what version of Adobe CF are you running?

Thanks!

~Brad

ColdBox/CommandBox Developer Advocate
Ortus Solutions, Corp

E-mail: br...@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com 
 
 
--------- Original Message ---------
--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.
To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/09EED81E-1B80-4657-80B5-221785AFA910%40issinc.com.
For more options, visit https://groups.google.com/d/optout.

Kevin Anderson

unread,
Aug 22, 2016, 5:49:46 PM8/22/16
to col...@googlegroups.com

Hi, Brad:

 

Server is running OS Windows 2008 with SQL Server 2008 R2 and Adobe ColdFusion 11 (Update 9).

 

Attached is a thread trace. It appears the problem is in /coldbox/system/web/Controller.cfc, but I’m not sure why that would be.

Block-in-Controller.txt

br...@bradwood.com

unread,
Aug 22, 2016, 5:55:45 PM8/22/16
to col...@googlegroups.com
Thanks Ken, but I need a stack trace of the ENTIRE JVM.  Basically, every single thread that's running.  If you let me know how your getting the stack traces, I can help you.  (Fusion-reactor, built in server monitor, command line, etc)  
 
This stack gives me a bit more information about the object that the thread is waiting to access (a synchronized struct) but doesn't show me what thread is currently owning the lock on the struct.  
 
Also, how is this issue manifesting itself?  Are you getting exceptions thrown?  Are you seeing threads waiting a very long time to complete that look like this?
 
I would fully expect one thread to wait if another thread was concurrently modifying a share struct (in this case, the ColdBox settings) however I wouldn't expect the wait to be more than a few nanoseconds.
--------- Original Message ---------

Matt Quackenbush

unread,
Aug 22, 2016, 6:02:58 PM8/22/16
to col...@googlegroups.com

It's Kevin. ;-)


--------- Original Message ---------

--------- Original Message ---------

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+unsubscribe@googlegroups.com.


To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/09EED81E-1B80-4657-80B5-221785AFA910%40issinc.com.
For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+unsubscribe@googlegroups.com.

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+unsubscribe@googlegroups.com.

To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/F556C7C6-934C-44C1-A553-C85D41937E25%40issinc.com.
For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+unsubscribe@googlegroups.com.

To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/20160822145539.a7243acce6934569d3743d1abfb94fec.717e54a836.mailapi%40email01.godaddy.com.

br...@bradwood.com

unread,
Aug 22, 2016, 6:07:04 PM8/22/16
to col...@googlegroups.com
Dang, I did it twice.  Sorry Kevin!  Dunno how I got Ken stuck in my head :)
--------- Original Message ---------
--------- Original Message ---------

--------- Original Message ---------

 

 

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.


To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/09EED81E-1B80-4657-80B5-221785AFA910%40issinc.com.
For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.

To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/F556C7C6-934C-44C1-A553-C85D41937E25%40issinc.com.
For more options, visit https://groups.google.com/d/optout.

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.

 

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.

To post to this group, send email to col...@googlegroups.com.

Kevin Anderson

unread,
Aug 23, 2016, 8:37:51 AM8/23/16
to col...@googlegroups.com

Thanks, Matt, for defending the “honor” of my name! LOL

 

Brad: You can call me just about anything. Many do and not near as nice as “Ken!”

 

I’ll get the full stacktrace from the system engineering folks this morning and send it your way.

Kevin Anderson

unread,
Aug 23, 2016, 8:53:17 AM8/23/16
to col...@googlegroups.com

I think the attached is what you are looking for, Brad.

Thread-Print.txt

br...@bradwood.com

unread,
Aug 23, 2016, 11:48:51 AM8/23/16
to col...@googlegroups.com
Thanks KEVIN! Can you show me what's on line 791 of D:\APPROOT\CIDNEMobile\wwwroot\models\SearchService.cfc?
 
There appears to be some code that is  performing a very expensive duplicate() on the ColdBox setting structure that is locking it so no other thread can touch it.
 
This is in thread ajp-bio-8014-exec-10 BTW.
--------- Original Message ---------

Kevin Anderson

unread,
Aug 23, 2016, 12:07:24 PM8/23/16
to col...@googlegroups.com

Line 791 is a comment. We aren’t doing any duplicate() anywhere of the Coldbox settings structure.

 

We “had” been using threads (component invocation of CFThread) for the searches but pulled that out. There is a duplicate() function used for the results from the search to avoid the issue caused by structs being passed by reference. I can pull that out if it is indeed an issue.

 

Below is the relevant code from the SearchService that may be what you are referencing:

 

// Execute query for specified report in single-threaded mode
try {
   var data = threadData;
   var thread_reportType = data.module & "." & data.reportType & "." & data.entity;
   var thread_aReports = data.QueryHelper.SearchReports(theaterFilter = data.theaterFilter,
                                                        module = data.module,
                                                        reportType = data.reportType,
                                                        entity = data.entity,
                                                        startDate = data.startDate,
                                                        endDate = data.endDate,
                                                        userGroupKey = data.userGroupKey,
                                                        reportConfig = data.reportConfig,
                                                        startRDTS = data.startRDTS,
                                                        maxrows = data.maxrows,
                                                        latitude = data.latitude,
                                                        longitude = data.longitude,
                                                        distance = data.distance,
                                                        refresh = data.refresh,
                                                        customParams = data.customParams);

   if( arraylen(thread_aReports) ) {
      for( var j=1; j <= arraylen(thread_aReports); j++ ) {
         arrayAppend(unsortedReports, duplicate(thread_aReports[j]));
      }
   }
}
catch( Any e ) {
   if( isDefined("thread_reportType") ) {
      var reportCountIndex = replace(thread_reportType, ".", "", "all");
      if( structKeyExists(reportCounts, reportCountIndex) ) {
         structUpdate(reportCounts[reportCountIndex], "status", "#threadData.reportType# terminated abnormally. Message: #e.message#, Detail: #e.detail#");
      }
   }
   LOGGER.error("Error in search. Message=#e.message#, Detail=#e.detail#");
}

Reply all
Reply to author
Forward
0 new messages