commands failing with java.nio.channels.OverlappingFileLockException

[Dan Nemec]

We're running 3.4.8 and just migrated our ctier VM from a slow box to
a fast box. Some commands will work, but others are giving the below
error. Once that error starts, no command will run as they will all
fail with the same error. The error is on the ctier server who is
dispatching the commands to a site. You can see that the command was
dispatched and ran successfully on the node, but when the results were
returned, the server gave the error. The commands run fine if run
directly on the node.

Why does this error happen, and how do I clean up and stop it?

--snipped from verbose output--

number of nodes to dispatch to: 1, (1 threads)
preparing for sequential execution...
returning RemoteCommand object for command: Status
SSHExec vapp11.atls1 -> "ctl -p Stage1 -t SPTomcatServiceModule -o
sso.app.vapp11.atls1 -c Status"
dispatching to proxy on node: vapp11.atls1
Connecting to vapp11.atls1:22
cmd : ctl -p Stage1 -t SPTomcatServiceModule -o sso.app.vapp11.atls1 -
c Status
[a...@vapp11.atls1 Stage1.SPTomcatServiceModule.sso.app.vapp11.atls1
Status][INFO] begin workflow command (1/1) -> "assertServiceIsUp " ...

[a...@vapp11.atls1 Stage1.SPTomcatServiceModule.sso.app.vapp11.atls1
assertServiceIsUp][WARN] OUTPUT SUCCESS Service listening on
"vapp11.atls1:19101", sso.app is up.

[a...@vapp11.atls1 Stage1.SPTomcatServiceModule.sso.app.vapp11.atls1
Status][INFO] end workflow command (1/1) -> "assertServiceIsUp "

Overriding previous definition of property "result"

-post-execute:

execute:
Exiting /opt/ctier/ctl/depots/Stage1/modules/Site/commands/Status.xml.
Overriding previous definition of property "result"
Overriding previous definition of property "result"

Command failed:
/opt/ctier/ctl/depots/Stage1/modules/Site/commands/Status.xml:29:
java.nio.channels.OverlappingFileLockException
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:116)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run
(Parallel.java:428)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList
(FileChannelImpl.java:1170)
at sun.nio.ch.FileChannelImpl$SharedFileLockTable.add
(FileChannelImpl.java:1072)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:834)
at java.nio.channels.FileChannel.lock(FileChannel.java:860)
at com.controltier.ctl.cli.ctldeploy.GetServerFile.execute
(GetServerFile.java:133)
at
com.controltier.ctl.cli.ctldeploy.DeploymentUtils.updateDeploymentsProperties
(DeploymentUtils.java:111)
at
com.controltier.ctl.tasks.controller.node.NodeDispatchAction.perform
(NodeDispatchAction.java:54)
at com.controltier.ctl.types.controller.ExecuteAction.perform
(ExecuteAction.java:136)
at
com.controltier.ctl.tasks.controller.Controller.performExecuteAction
(Controller.java:353)
at com.controltier.ctl.tasks.controller.Controller.execute
(Controller.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
... 3 more
--- Nested Exception ---
java.nio.channels.OverlappingFileLockException
at sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList
(FileChannelImpl.java:1170)
at sun.nio.ch.FileChannelImpl$SharedFileLockTable.add
(FileChannelImpl.java:1072)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:834)
at java.nio.channels.FileChannel.lock(FileChannel.java:860)
at com.controltier.ctl.cli.ctldeploy.GetServerFile.execute
(GetServerFile.java:133)
at
com.controltier.ctl.cli.ctldeploy.DeploymentUtils.updateDeploymentsProperties
(DeploymentUtils.java:111)
at
com.controltier.ctl.tasks.controller.node.NodeDispatchAction.perform
(NodeDispatchAction.java:54)
at com.controltier.ctl.types.controller.ExecuteAction.perform
(ExecuteAction.java:136)
at
com.controltier.ctl.tasks.controller.Controller.performExecuteAction
(Controller.java:353)
at com.controltier.ctl.tasks.controller.Controller.execute
(Controller.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run
(Parallel.java:428)
at java.lang.Thread.run(Thread.java:619)

[Anthony Shortland]

Hi Dan ... what version of Java are you using exactly?

[Dan Nemec]

We're using

jdk1.6.0_16

Dan

On Oct 21, 11:22 am, Anthony Shortland <anth...@controltier.com>
wrote:

[Anthony Shortland]

Hmm ... well Greg better jump in on this one ... could you try running ControlTier under Java 1.5 Dan to confirm whether this is 1.6 related? We've seen a couple of compatibility issues since we added support for the latest JDK.

By the way; are you using Sun's JDK or OpenJDK?

I have:

[anthony@centos53 ~]$ which java
/usr/bin/java
[anthony@centos53 ~]$ java -version
java version "1.6.0"
OpenJDK  Runtime Environment (build 1.6.0-b09)
OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)

... on 64-bit CentOS 5.3 for example. And what OS? e.g.:

[anthony@centos53 ~]$ uname -a
Linux centos53 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
[anthony@centos53 ~]$ cat /etc/redhat-release 
CentOS release 5.3 (Final)

Thanks?

Anthony.

Anthony Shortland

[ Developer | ControlTier Open Source Project | mobile: 650.215.3117 aim: anthony....@me.com yahoo: anthony.shortland irc.freenode.net: #controltier skype: anthony.shortland ]

ControlTier Software Inc. is now DTO Solutions Inc.

[Dan Nemec]

Still fails with 1.5.0_19. I tried just the ctl command on 1.5 and
then restarting ctl server as well with 1.5. Both still give error.

app@vctier01 /opt/ctier/bin $ which java
/usr/bin/which: no java in (/opt/ctier/pkgs/ctl-1.4.8/bin:/opt/ctier/
pkgs/jetty-6.1.14/bin:/opt/ctier/pkgs/ctl-1.4.8/bin:/opt/ctier/pkgs/
jetty-6.1.14/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin)
app@vctier01 /opt/ctier/bin $ $JAVA_HOME/bin/java -version
java version "1.5.0_19"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_19-b02)
Java HotSpot(TM) Server VM (build 1.5.0_19-b02, mixed mode)
app@vctier01 /opt/ctier/bin $ uname -a
Linux vctier01.atlnp1 2.6.18-128.1.14.el5 #1 SMP Wed Jun 17 06:38:05
EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
app@vctier01 /opt/ctier/bin $ cat /etc/redhat-release
CentOS release 5.3 (Final)


On Oct 22, 12:50 am, Anthony Shortland <anth...@controltier.com>

> aim: anthony.shortl...@me.com yahoo: anthony.shortland  

[Anthony Shortland]

OK ... I think that covers the groundwork ... perhaps Greg or Alex can go back to the stack trace below and take a look? Looks like a very low level race condition to me.

Anthony Shortland

[ Developer | ControlTier Open Source Project | mobile: 650.215.3117 aim: anthony....@me.com yahoo: anthony.shortland irc.freenode.net: #controltier skype: anthony.shortland ]

[Anthony Shortland]

Oh ... you are running on a standard local file system are you? Not a networked file system or similar? Here's something relevant by the way: http://forums.sun.com/thread.jspa?threadID=5324314

[Greg Schueler]

Hi Dan,

make sure that the JAVA_HOME value is set in $CTL_BASE/etc/profile.

this file is sourced from the ctl/ctl-exec script and may override your shell environment variable when you run the commands or start the jetty server.

[Greg Schueler]

er, sorry that was incorrect about starting jetty using jetty.sh, but it is true when executing ctl/ctl-exec