unresponsive aterm.jar

[Scott Wood]

Just reposting a conversation with Brian at arcitecta support.  Here's my question: 

Hey folks,

Our MediaFlux server has not responded for an hour or so.  At this point, we're just looking to retstart it, but the /etc/init.d/mediaflux script tries to shut it down by sending it a "terminate" via aterm, which just locks up.  Interactive aters just stall at the blue screen after credentials are entered, and no logs have been written to for an hour.

Are we safe to just issue a "kill [PID of java -jar /QCMG/mediaflux/mediaflux/bin/aserver.jar]"?  As always, thanks for your help.

Regards,

And his response:

Yes its "probably" ok to kill the process, but the question is what else is happening? 

Are there any logs in the OS that point to any problems... running out of memory or disk errors etc?
You should also get a Java stack trace.... now I haven't done this is a while but on Linux systems this is done with jstack
jstack -l <pid>
or

jstack -F -l <pid> if the jstack hangs as well.

Post them to the list.
Note that there are usually two aserver.jar processes. The master aserver.jar process and the actual aserver.jar instance.
In a "ps -ef"
$ ps -ef | grep aserv
  501  7627  7127   0 Wed01pm ??         0:47.53 java -jar /Users/brian/mf/bin/aserver.jar
  501  7628  7627   0 Wed01pm ??         1:40.26 java -d64 -Xmx512m -XX:ErrorFile=/Users/brian/mf/volatile/logs/jvm/jvm_crash_20140129_1318_pid%p.log -cp /Users/brian/mf/bin/aserver.jar ..etc etc
7628 is the pid of the instance, 7627 is the master
When you kill -HUP 7628 the master should restart the instance, if it dies but does not restart then kill 7627 as well.
Start with  HUP then a TERM
You should not have to "kill -KILL"

[Scott Wood]

Hey Brian,

We don't have much in the way of access to the server itself, as it is run by another admin in another buildong, but they don't expect there are any system issues that could be contributing to the issue, and the system seems to have lots of resources free, at a glance.

Jstack (even with a -F) seems to hang on the server instance so no joy there.

As for the kill -HUP <master PID>, again, no results.  A kill -TERM <master PID> kills the master but leaves the instance running.  I tried resorting to the kill -TERM of the instance and even that has no effect.

Shall we resort to the ever dreaded "kill -9"?

Thanks for your guidance.

Regards,

Scott

[Scott Wood]

Sorry to leave a question unresolved here, but rather than leave you all hanging (if you'll pardon the choice of words), I figured I'd follow up.  The server that mediaflux was running on was rebooted before we could troubleshoot any further.  All seems to have come back up happily, but we did not get a chance to sort out what had happened.

Regards,
Scott

[Brian OConnor]

Hi glad you back and running.

Hmm if the *box* was becoming unresponsive it sounds to me like a server issue!  I would be on guard  for further issues.

you might also like to run "xodb.analyze" to verify that the database is good.

I would push to get the admins to check the server.

[Jason Lohrey]

If jstack -F will not respond, then there is something quite awry. 

What version of the JRE/JVM are you running?

Jason

--
You received this message because you are subscribed to the Google Groups "mediaflux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mediaflux+...@googlegroups.com.
To post to this group, send email to medi...@googlegroups.com.
Visit this group at http://groups.google.com/group/mediaflux.
For more options, visit https://groups.google.com/groups/opt_out.

JASON LOHREY

Chief Technical Officer