Version Blue 4.1.2 173 not starting after upgrade

[Peter Ondruška]

Dear all,

Is there anyone else having problem starting Blue 4.1.2 173 (on IBM JDK 8) after upgrading from 4.1 172?

I do not have log file available right now but it was related to health check monitoring. After manually removing its configuration from domain config everything is back to normal.

I will also send the log later. Shall I create an issue in GitHub?

Peter
--
kaibo, s.r.o., ID 28435036, registered with the commercial register
administered by the Municipal Court in Prague, section C, insert 141269.
Registered office and postal address: kaibo, s.r.o., Kališnická 379/10,
Prague 3, 130 00, Czech Republic.
https://www.kaibo.eu

[Peter Ondruška]

Relevant part of the log file:

[2017-08-20T19:42:24.697+0200] [Payara 4.1] [SEVERE] [NCLS-CORE-00014] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944697] [levelValue: 1000] [[
  Shutdown required
MultiException stack 1 of 1
MultiException stack 1 of 2
java.lang.NoClassDefFoundError: org.jvnet.hk2.config.types.Property
    at com.sun.proxy.$Proxy151.<clinit>(Unknown Source)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:437)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:750)
    at org.glassfish.config.support.TranslatedConfigView.getProxy(TranslatedConfigView.java:191)
    at org.glassfish.config.support.GlassFishConfigBean.createProxy(GlassFishConfigBean.java:98)
    at org.jvnet.hk2.config.Dom.createProxy(Dom.java:992)
    at org.jvnet.hk2.config.ConfigModel$CollectionNode$1.get(ConfigModel.java:455)
    at org.jvnet.hk2.config.ConfigBean$2.get(ConfigBean.java:207)
    at java.util.AbstractList$Itr.next(AbstractList.java:369)
    at fish.payara.nucleus.healthcheck.configuration.HealthCheckServiceConfiguration$Duck.getCheckerByType(HealthCheckServiceConfiguration.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:508)
    at org.jvnet.hk2.config.Dom.invokeDuckMethod(Dom.java:1129)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:1080)
    at org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:170)
    at com.sun.proxy.$Proxy60.getCheckerByType(Unknown Source)
    at fish.payara.nucleus.healthcheck.preliminary.BaseHealthCheck.postConstruct(BaseHealthCheck.java:99)
    at fish.payara.nucleus.healthcheck.stuck.StuckThreadsHealthCheck.postConstruct(StuckThreadsHealthCheck.java:78)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:508)
    at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287)
    at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:333)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:786)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:785)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: post construct on fish.payara.nucleus.healthcheck.stuck.StuckThreadsHealthCheck
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:393)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:786)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:785)
]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
  Failed to start, exiting]]

[2017-08-20T19:42:24.700+0200] [Payara 4.1] [INFO] [NCLS-CORE-00092] [javax.enterprise.system.core] [tid: _ThreadID=224 _ThreadName=Thread-48] [timeMillis: 1503250944700] [levelValue: 800] [[
  Server shutdown initiated]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
  Exception in thread "main" ]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
  java.lang.reflect.InvocationTargetException]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)]]

[2017-08-20T19:42:24.698+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944698] [levelValue: 1000] [[
      at java.lang.reflect.Method.invoke(Method.java:508)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:102)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
  Caused by: ]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
  java.lang.RuntimeException: org.glassfish.embeddable.GlassFishException: Server failed to start]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:225)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:81)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl.start(EmbeddedOSGiGlassFishImpl.java:75)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71)]]

[2017-08-20T19:42:24.699+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944699] [levelValue: 1000] [[
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:123)]]

[2017-08-20T19:42:24.700+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944700] [levelValue: 1000] [[
      ... 6 more]]

[2017-08-20T19:42:24.700+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944700] [levelValue: 1000] [[
  Caused by: ]]

[2017-08-20T19:42:24.700+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944700] [levelValue: 1000] [[
  org.glassfish.embeddable.GlassFishException: Server failed to start]]

[2017-08-20T19:42:24.701+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944701] [levelValue: 1000] [[
      at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:237)]]

[2017-08-20T19:42:24.701+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944701] [levelValue: 1000] [[
      at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:223)]]

[2017-08-20T19:42:24.701+0200] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1503250944701] [levelValue: 1000] [[
      ... 12 more]]

On 22 August 2017 at 11:21, Peter Ondruška <peter.o...@kaibo.eu> wrote:

Dear all,

Is there anyone else having problem starting Blue 4.1.2 173 (on IBM JDK 8) after upgrading from 4.1 172?

I do not have log file available right now but it was related to health check monitoring. After manually removing its configuration from domain config everything is back to normal.

I will also send the log later. Shall I create an issue in GitHub?

Peter

--

Peter Ondruška

[Mike Croft]

Hi Peter,

How did you do the upgrade? Did you try starting an existing domain with healthcheck configuration using 173, or did you reconfigure a fresh domain?

Mike

[Michaël COINDEAU]

Hi,

We do have the same problem on this upgrade. Payara Full 172 to 173, same stacktrace as Peter and domain not starting.

Our domain dir is outside from the appserver dir, and the server is a simlink (live) as suggested in Payara's upgrade guide.

We stopped the server, changed the link and tried to restart without success.

Peter, did you found a way to start your 173 ?

Michaël

[Peter Ondruška]

Dear Michael,

I did manually remove <health-check-service-configuration> from config/domain.xml. This should help, have you done this? You better make backup copy of config/comain.xml before you do this. Not sure if booting with 172 and disabling health check service works as 173 on blue does not start even with checkers set to disabled.

p.

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/c0fc4abe-2d40-4a4a-a406-1790fc1b3b28%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Peter Ondruška

[Michaël COINDEAU]

I tried to set the enable flag @ false but it did'nt worked.

Thanks about the tip, removing the whole configuration tag worked.

I'll try to create a ticket on Payara's Github.

Le lundi 28 août 2017 12:26:30 UTC+2, Peter Ondruška a écrit :

Dear Michael,

I did manually remove <health-check-service-configuration> from config/domain.xml. This should help, have you done this? You better make backup copy of config/comain.xml before you do this. Not sure if booting with 172 and disabling health check service works as 173 on blue does not start even with checkers set to disabled.

p.

On 28 August 2017 at 11:26, Michaël COINDEAU <m.coi...@gmail.com> wrote:

Hi,

We do have the same problem on this upgrade. Payara Full 172 to 173, same stacktrace as Peter and domain not starting.

Our domain dir is outside from the appserver dir, and the server is a simlink (live) as suggested in Payara's upgrade guide.

We stopped the server, changed the link and tried to restart without success.

Peter, did you found a way to start your 173 ?

Michaël

Le mercredi 23 août 2017 10:17:32 UTC+2, Mike Croft a écrit :

Hi Peter,

How did you do the upgrade? Did you try starting an existing domain with healthcheck configuration using 173, or did you reconfigure a fresh domain?

Mike

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/c0fc4abe-2d40-4a4a-a406-1790fc1b3b28%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Peter Ondruška

[Mike Croft]

Hi Michael/Peter - did you already have some healthcheck configuration before the upgrade (I assume so) and what was configured?

[Michaël COINDEAU]

Yes, it was active. Mine was configured with those parameters :

     <health-check-service-configuration historical-trace-enabled="true" enabled="true">

        <cpu-usage-checker time="1" enabled="true">

          <property name="threshold-critical" value="80"></property>

          <property name="threshold-warning" value="50"></property>

          <property name="threshold-good" value="0"></property>

        </cpu-usage-checker>

        <connection-pool-checker time="1" enabled="true">

          <property name="threshold-critical" value="80"></property>

          <property name="threshold-warning" value="50"></property>

          <property name="threshold-good" value="0"></property>

        </connection-pool-checker>

        <heap-memory-usage-checker time="1" enabled="true">

          <property name="threshold-critical" value="90"></property>

          <property name="threshold-warning" value="65"></property>

          <property name="threshold-good" value="0"></property>

        </heap-memory-usage-checker>

        <machine-memory-usage-checker time="1" enabled="true">

          <property name="threshold-critical" value="90"></property>

          <property name="threshold-warning" value="80"></property>

          <property name="threshold-good" value="0"></property>

        </machine-memory-usage-checker>

        <hogging-threads-checker time="1" enabled="true"></hogging-threads-checker>

        <garbage-collector-checker time="1" enabled="true"></garbage-collector-checker>

        <log-notifier enabled="true"></log-notifier>

        <xmpp-notifier></xmpp-notifier>

        <datadog-notifier></datadog-notifier>

        <email-notifier></email-notifier>

        <slack-notifier></slack-notifier>

        <eventbus-notifier></eventbus-notifier>

        <hipchat-notifier></hipchat-notifier>

        <new-relic-notifier></new-relic-notifier>

        <snmp-notifier></snmp-notifier>

        <jms-notifier></jms-notifier>

      </health-check-service-configuration>

[Peter Ondruška]

Dear Mike,

I "think" I had configured these:

      <health-check-service-configuration>
        <cpu-usage-checker unit="DAYS" time="1" enabled="false">

          <property name="threshold-critical" value="80"></property>
          <property name="threshold-warning" value="50"></property>
          <property name="threshold-good" value="0"></property>
        </cpu-usage-checker>

        <connection-pool-checker unit="HOURS" time="1" enabled="false">

          <property name="threshold-critical" value="80"></property>
          <property name="threshold-warning" value="50"></property>
          <property name="threshold-good" value="0"></property>
        </connection-pool-checker>

        <heap-memory-usage-checker unit="HOURS" time="4" enabled="false">

          <property name="threshold-critical" value="80"></property>
          <property name="threshold-warning" value="50"></property>
          <property name="threshold-good" value="0"></property>

        </heap-memory-usage-checker>
        <hogging-threads-checker unit="HOURS" time="1" enabled="false"></hogging-threads-checker>
        <garbage-collector-checker unit="DAYS" time="1" enabled="false"></garbage-collector-checker>
        <machine-memory-usage-checker unit="HOURS" time="1" enabled="false">

          <property name="threshold-critical" value="80"></property>
          <property name="threshold-warning" value="50"></property>
          <property name="threshold-good" value="0"></property>

        </machine-memory-usage-checker>
        <log-notifier></log-notifier>
        <email-notifier enabled="true"></email-notifier>
        <xmpp-notifier></xmpp-notifier>
        <datadog-notifier></datadog-notifier>

        <slack-notifier></slack-notifier>
        <eventbus-notifier></eventbus-notifier>
        <hipchat-notifier></hipchat-notifier>
        <new-relic-notifier></new-relic-notifier>
        <snmp-notifier></snmp-notifier>
        <jms-notifier></jms-notifier>
      </health-check-service-configuration>

enabled is false, because this is my 172 domain soon to be upgraded to 173. Except hogging-threads-checked that I think never worked on blue--most probably due to JVM not reporting required information. I will have a look once I have more time ;).

p.

On 28 August 2017 at 14:18, Mike Croft <backtoth...@gmail.com> wrote:

Hi Michael/Peter - did you already have some healthcheck configuration before the upgrade (I assume so) and what was configured?

--

You received this message because you are subscribed to the Google Groups "Payara Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/7280438d-7b44-4bb1-9222-b17d723650c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Peter Ondruška

[Mike Croft]

Hi Peter/Michael,

I investigated this a bit myself. The cause is the ConnectionPool checker - removing only this checker means the domain can start. I created an internal issue PAYARA-2010 to get a fix for this issue. Keep an eye on our GitHub PRs to see the fix for that issue. We haven't long released 173 and this seems like an important issue to fix, so it is likely that a fix will be in 174 (though, of course, I can offer no guarantees :-) )

Mike

[Michaël COINDEAU]

Thanks for considering it and for your feedback.

[Steve Millidge]

Internal issue number is PAYARA-2010.

Workaround is to fixup the manifest file of the jar; payara41/glassfish/modules/healthcheck-cpool.jar

In the Import-Package: section of the Manifest add the import

,org.jvnet.hk2.config.types;ver sion="[2.5,3)"