Trouble running Java JMX RMI in pod

[Pål Kristensen]

Hi

I have setup a docker image that contains a 3.party app running on Java. I have successfully got the image running on minikube., But when trying to deploy the image on our AWS Kubernetes cluster the the apps will not bound to the default RMI port 1098. RMI then tries any other higher port and finally runs out out bounds.

Any help will be appreciated.

Deployment definitions
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: geomapadmin
spec:
  revisionHistoryLimit: 0
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 30%
  template:
    metadata:
      labels:
        app: GeomapAdmin
    spec:
      containers:
 
      - name: agsserver
        image: XXXXXXXXXXXXXXXX
        tty: true
        stdin: true
 #       securityContext:
 #         privileged: true 
        command: ["sh", "-c"]
        args:
          - '/bin/sh -c "/arcgis/server/startserver.sh && trap : TERM INT; sleep infinity & wait"'       
        ports:
          - name: agshttp
            protocol: TCP
            containerPort: 6080          
          - name: agshttps
            protocol: TCP
            containerPort: 6443              
          - name: openejb
            protocol: TCP
            containerPort: 4000            
          - name: jmx
            protocol: TCP
            containerPort: 4001           
          - name: naming
            protocol: TCP
            containerPort: 4002          
          - name: derby
            protocol: TCP
            containerPort: 4003          
          - name: tcpcluster
            protocol: TCP
            containerPort: 4004         
          - name: websockets
            protocol: TCP
            containerPort: 6180
          - name: rmi
            protocol: TCP
            containerPort: 1098


logs from the container on Kubernetes
[arcgis@agsserver-3005693441-b95wf /]$ cat /arcgis/server/framework/etc/service.log | more
...
Start invoked
Wed Mar 29 12:35:27 UTC 2017:Start NodeAgent.start().
Wed Mar 29 12:35:28 UTC 2017:Loading all observers.
arcgis_cloud_platform=null
arcgis_cloud_platform=null
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'LogServiceObserver'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'ServerUninstallObserver'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'ServerConfigObserver'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'Server upgrader'.
Wed Mar 29 12:35:29 UTC 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.
Wed Mar 29 12:35:29 UTC 2017:Regular startup of Server.
Wed Mar 29 12:35:29 UTC 2017:Starting RMI connector for NodeAgent.
Wed Mar 29 12:35:30 UTC 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exce
ption is:
java.net.ConnectException: Connection refused]284:8
Wed Mar 29 12:35:30 UTC 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exce
ption is:
java.net.ConnectException: Connection refused]284:10
Wed Mar 29 12:35:38 UTC 2017:ERROR: Unable to start the RMI connector for NodeAgent

----

[arcgis@agsserver-3005693441-b95wf /]$ cat /arcgis/server/framework/etc/service_error.log | more
Wed Mar 29 12:35:38 UTC 2017:java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:49)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:415)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:134)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:56)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:299)
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:668)
at com.esri.arcgis.discovery.util.JMXServerController.a(JMXServerController.java:213)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:38)
... 4 more
Exception in thread "main" com.esri.arcgis.discovery.nodeagent.NodeAgentException: Could not start RMI connector. An instance of NodeAgent may be running or the JMX-RMI port is in use by another process.
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:137)
at com.esri.arcgis.discovery.nodeagent.impl.Main.start(Main.java:56)
at com.esri.arcgis.discovery.nodeagent.impl.Main.main(Main.java:299)
Caused by: java.lang.RuntimeException: Failed to start JMX Server
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:49)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.startJMXServer(NodeAgent.java:415)
at com.esri.arcgis.discovery.nodeagent.impl.NodeAgent.start(NodeAgent.java:134)
... 2 more
Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
at java.net.ServerSocket.<init>(ServerSocket.java:232)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at com.esri.arcgis.discovery.util.IPUtil.isPortAvailable(IPUtil.java:668)
at com.esri.arcgis.discovery.util.JMXServerController.a(JMXServerController.java:213)
at com.esri.arcgis.discovery.util.JMXServerController.start(JMXServerController.java:38)
... 4 more


Log from Minikube
Start invoked
Fri Mar 31 13:10:52 UTC 2017:Start NodeAgent.start().
Fri Mar 31 13:10:52 UTC 2017:Loading all observers.
arcgis_cloud_platform=null
arcgis_cloud_platform=null
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'LogServiceObserver'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'ServerUninstallObserver'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'ServerConfigObserver'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'AutomaticMachineRenameObserver'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'VerifyMachineRegistrationObserver'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'Server upgrader'.
Fri Mar 31 13:10:52 UTC 2017:NodeAgent loaded observer 'PlatformServicesPluginObserver'.
Fri Mar 31 13:10:52 UTC 2017:Regular startup of Server.
Fri Mar 31 13:10:52 UTC 2017:Starting RMI connector for NodeAgent.
Fri Mar 31 13:10:53 UTC 2017:Invoking beforeStart() for all observers.
Fri Mar 31 13:10:53 UTC 2017 JMXServerContoller.getJMXConnector() failed to return connection - Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi281:8
Fri Mar 31 13:10:53 UTC 2017:Registering and starting log service on AGSSERVER-1117655299-83XTH.
Fri Mar 31 13:10:54 UTC 2017:Registering NodeAgent as JMX bean.
Fri Mar 31 13:10:54 UTC 2017:Loading all plugins.
arcgis_cloud_platform=null
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'ProcessMonitor'.
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'DirectoryCleaner'.
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'UlimitMonitor'.
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'LogFileLimiter'.
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'ConfigSynchronizer'.
Fri Mar 31 13:10:54 UTC 2017:Loading plugin.. com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Fri Mar 31 13:10:54 UTC 2017:Props {PeriodicInterval=1, NodeAgentCustomPlugin=false}
Fri Mar 31 13:10:54 UTC 2017:Loaded plugin com.esri.arcgis.discovery.admin.plugins.ServiceInstanceMonitor
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'ServiceInstanceMonitor'.
Fri Mar 31 13:10:54 UTC 2017:NodeAgent loaded plugin 'PlatformServicesPluginObserver'.
Fri Mar 31 13:10:54 UTC 2017:Starting all plugins.arcgis
StartRMID - arg: -J-Djava.security.policy=file:////arcgis/server/framework/etc/rmid.policy
StartRMID - arg: -J-Dsun.rmi.activation.execPolicy=com.esri.arcgis.discovery.servicelib.impl.RMIDPolicyChecker
StartRMID - arg: -J-Djava.class.path=/arcgis/server/framework/lib/server/arcgis-servicelib.jar:/arcgis/server/framework/lib/server/arcgis-common.jar
StartRMID - arg: -J-Djava.rmi.server.useCodebaseOnly=true
StartRMID - arg: -port
StartRMID - arg: 1098

[Marcin Owsiany]

You might get a better answer if you try https://kubernetes.io/docs/troubleshooting/

My personal approach would be to run under "strace -f -tt" to find out what error the process is getting when setting up the socket to listen() on 1098.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/9f749acc-5ac1-41f1-a5af-c81acdf36877%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcin Owsiany  --  Software Engineer

Google Poland z.o.o., ul. Emilii Plater no. 53, 00-113, Warsaw

Sąd Rejonowy dla m. st. Warszawy w Warszawie,

XII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000240611.

NIP: 525-23-44-078, Kapitał zakładowy Spółki 400 000, 00 zł.