Problem creating remote SSH node

[Vincent Lee]

Hi all,

I am trying to create a remote instance on a separate physical server so that I can use dedicated resources for the 2nd instance but still administer it from the 1st server, where the DAS sits. I have followed the instructions from https://blog.payara.fish/creating-a-simple-deployment-group but I am stuck at the point of creating the 2nd Instance.

The remote node was created successfully - I've SSHed into the 2nd server and checked that the directories are created, but one anomaly is that the Node Directory is never created:

I have left it empty here in the screenshot, but I have tried it with different physical locations but it is never created on the 1st or 2nd server for some reason.

The node itself was created successfully, and I could ping it successfully as well. Next I created a separate config and create a new instance on the 2nd node pointing to the new config. It takes a long time and the the screen just goes blank:

The log shows the following entry:

[2019-12-28T05:41:27.528+0000] [Payara 5.191] [SEVERE] [] [javax.enterprise.config.api] [tid: _ThreadID=42 _ThreadName=admin-thread-pool::admin-listener(1)] [timeMillis: 1577511687528] [levelValue: 1000] [[
  Configuration not added. java.lang.InterruptedException]]

[2019-12-28T05:41:27.532+0000] [Payara 5.191] [SEVERE] [] [org.glassfish.admingui] [tid: _ThreadID=225 _ThreadName=admin-thread-pool::admin-listener(8)] [timeMillis: 1577511687532] [levelValue: 1000] [[
  RestResponse.getResponse() gives FAILURE.  endpoint = 'https://localhost:4848/management/domain/create-instance'; attrs = '{node=SegmailUAT2, name=segmail-uat-2, config=segmail-uat-config, deploymentgroup=null}']]

[2019-12-28T05:41:27.600+0000] [Payara 5.191] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=225 _ThreadName=admin-thread-pool::admin-listener(8)] [timeMillis: 1577511687600] [levelValue: 900] [[
  StandardWrapperValve[FacesServlet]: Servlet.service() for servlet FacesServlet threw exception
java.lang.IllegalStateException
	at org.apache.catalina.connector.ResponseFacade.setBufferSize(ResponseFacade.java:287)
	at com.sun.faces.context.ExternalContextImpl.setResponseBufferSize(ExternalContextImpl.java:974)
	at com.sun.faces.context.PartialViewContextImpl$DelayedInitPartialResponseWriter.<init>(PartialViewContextImpl.java:664)
	at com.sun.faces.context.PartialViewContextImpl.getPartialResponseWriter(PartialViewContextImpl.java:343)
	at com.sun.faces.context.AjaxExceptionHandlerImpl.handlePartialResponseError(AjaxExceptionHandlerImpl.java:177)
	at com.sun.faces.context.AjaxExceptionHandlerImpl.handle(AjaxExceptionHandlerImpl.java:101)
	at javax.faces.context.ExceptionHandlerWrapper.handle(ExceptionHandlerWrapper.java:93)
	at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:94)
	at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
	at javax.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:708)
	at javax.faces.webapp.FacesServlet.service(FacesServlet.java:451)
	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1628)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:339)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
	at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:251)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:755)
	at org.apache.catalina.core.StandardPipeline.doChainInvoke(StandardPipeline.java:579)
	at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:755)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:575)
	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:368)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:208)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:208)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$0(WorkerThreadIOStrategy.java:90)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.lang.Thread.run(Thread.java:748)
]]

The instance is created in the DAS but stopped, and if I try to start it:

If I try to create the node/instance directory manually on the 2nd server and start the instance, I get this error:

Has anyone encountered such an issue or if I am doing something wrong here? Thanks!