Clean shutdown of Tomcat with Cargo not working?

805 views
Skip to first unread message

Erik Godding Boye

unread,
Aug 21, 2015, 4:22:29 AM8/21/15
to Codehaus Cargo
Hi,

I have been struggeling trying to figure out why the following log statement is logged (twice?) when stopping (or restarting) Tomcat 8.x using Cargo:
WARN  [org.apache.catalina.core.StandardServer] [] StandardServer.await: Invalid command '' received

Investigating the Tomcat source code, my current theory is that Cargo issues an invalid command ('') to the Tomcat shutdown port when attempting to shut down Tomcat cleanly.
The Tomcat shutdown service (that reports this warning), expects the shutdown command to be 'SHUTDOWN', and reports this warning when it receives the '' command.
The shutdown command is configured in Tomcat server.xml (//Server@shutdown), and the default value is 'SHUTDOWN'.

I suspect this to be a bug in the Cargo Tomcat container, but I am not sure.

Regards,
Erik

S. Ali Tokmen

unread,
Aug 21, 2015, 9:28:52 AM8/21/15
to codehau...@googlegroups.com
Hi Eril

Interesting catch! I checked and here's the command sent by Cargo:
java -Dcatalina.home=[...] -Dcatalina.base=[...] -Djava.io.tmpdir=[...] -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=[...]/catalina-base/conf/logging.properties -classpath [...]/bin/tomcat-juli.jar:[...]/bin/bootstrap.jar:[...]/tools.jar org.apache.catalina.startup.Bootstrap stop
... and the answer from Tomcat:
Aug 21, 2015 3:24:10 PM org.apache.catalina.core.StandardServer await
WARNING: StandardServer.await: Invalid command '' received
Should it then be "shutdown" instead of "stop"? Or is it something else?

Please advise

S. Ali Tokmen
http://ali.tokmen.com/
http://contact.ali.tokmen.com/
--
You received this message because you are subscribed to the Google Groups "Codehaus Cargo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codehaus-carg...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codehaus-cargo/cce494bd-30b2-498c-8877-f66d3039ac18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Erik Godding Boye

unread,
Aug 25, 2015, 12:33:08 PM8/25/15
to Codehaus Cargo

Hi,

 

Thank you for your feedback. I have absolutely no idea what is going on here. 

 

I tested with a vanilla download of Tomcat 8.x (tested both 8.0.24 and 8.0.26), and the command sent by Cargo is more or less the same as the command resulting from running shutdown.bat/shutdown.sh in Tomcat. There are some minor differences, but I cannot understand how these can explain this very strange behavior:

1) The classpath is a bit different: [..]\bin\bootstrap.jar:[..]\bin\tomcat-juli.jar (with Tomcat). Different order of the jar-files, and tools.jar is not included.

2) Tomcat supplies an extra system property: '-Djava.endorsed.dirs="[..]\endorsed', but in the vanilla download this folder does not even exist, so I am quite sure this does not matter

 

When you invoke shutdown.bat/shutdown.sh Tomcat replies (in the server log):

25-Aug-2015 18:01:42.427 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.

and after that Tomcat shuts down cleanly.

 

To answer your question, it seems like the command sent by Cargo is (more or less) correct. There must be something strange going on....

 

To dig more into this, I think we have to debug Tomcat when Cargo tries to shut it down. But I do not how to do that.

Any ideas?

 

Regards,
Erik

S. Ali Tokmen

unread,
Aug 25, 2015, 3:40:19 PM8/25/15
to codehau...@googlegroups.com
Hi Erik

The instructions for debugging are on https://codehaus-cargo.github.io/cargo/Debugging.html

Hope this helps

Erik Godding Boye

unread,
Sep 3, 2015, 8:04:40 AM9/3/15
to Codehaus Cargo

Hi,

 

Sorry for taking so long to get back to you. I finally managed to dig deeper into this strange issue.

 

First, the problem is not as serious as I first assumed. Cargo shuts down Tomcat cleanly. The only issue is annoying log messages generated by Tomcat when shut down by Cargo.

 

When I managed to debug the right stuff (the JVM where Cargo is running; not the container/Tomcat JVM), I found out that these annoying log messages (StandardServer.await: Invalid command '' received) originates from code in the method AbstractLocalContainer#waitForCompletion. In particular the following, and rather naive (but presumably harmless), line: https://github.com/codehaus-cargo/cargo/blob/master/core/api/container/src/main/java/org/codehaus/cargo/container/spi/AbstractLocalContainer.java#L361

 

The Tomcat shutdown port is in fact a property ending with ".port" (cargo.rmi.port), which makes Cargo issue a request to the shutdown port with no "command" (it just try to connect to port). This happens, even several times, before the correct shutdown command (issued by Cargo) is received and processed by Tomcat - which results in these annoying log messages.

 

What do you think? How hard would it be eventually fix this "bug"?

 

Regards,

Erik

S. Ali Tokmen

unread,
Sep 4, 2015, 2:18:29 PM9/4/15
to codehau...@googlegroups.com
Hi Erik

Thanks for the investigations - Patch is not done: https://codehaus-cargo.atlassian.net/browse/CARGO-1337

If you want to try it out now, use the SNAPSHOT version as described on https://codehaus-cargo.github.io/cargo/Maven2+Plugin+Installation.html#Maven2PluginInstallation-snapshots

Regards

Erik Godding Boye

unread,
Sep 7, 2015, 3:22:34 AM9/7/15
to Codehaus Cargo
Hi,

I just tried the latest snapshot version, and can confirm that the problem is now gone. Thanks a lot!

Any chance for a release including this fix (Cargo 1.4.16)?

We are about to roll out the Cargo Daemon to a bunch of servers - to facilitate remote application deployment using Cargo. 
I do not know when we will have a chance to upgrade the Cargo Daemon, so a release would be greatly appreciated!

Regards,
Erik

S. Ali Tokmen

unread,
Sep 7, 2015, 2:27:46 PM9/7/15
to codehau...@googlegroups.com
Hi Erik

Sounds good :) If you actually only need the Cargo Daemon, why not use the SNAPSHOT WAR file while waiting for the full release?

Check out: https://oss.sonatype.org/content/repositories/snapshots/org/codehaus/cargo/cargo-daemon-webapp/1.4.16-SNAPSHOT/

Erik Godding Boye

unread,
Sep 7, 2015, 4:40:00 PM9/7/15
to Codehaus Cargo
Hi,

There is no chance that our operational guys will accept a non-released version of anything in production. And even in test environments, they will not accept non-released versions of 3. party dependencies.
No problem - this is a minor issue for us. But we look forward to a released version that includes this fix. :-)

Erik
Reply all
Reply to author
Forward
0 new messages