JDK update causing Cassandra not to start

[Prashant Hegde]

Hi, 

One of our server got updated JDK 8 u 162 ( from JDK 8 u 151 ) today. After, that, we are hitting an issue where Cassandra is refusing to start. In cassandra, debug.log we are seeing an exception which points to this issue ( https://issues.apache.org/jira/browse/CASSANDRA-14173 )

== Exception in cassandra/logs/debug.log == 

java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;

        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150) ~[na:1.8.0_162]

        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135) ~[na:1.8.0_162]

        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405) ~[na:1.8.0_162]

        at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104) ~[apache-cassandra-3.11.0.jar:3.11.0]

        at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143) [apache-cassandra-3.11.0.jar:3.11.0]

        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) [apache-cassandra-3.11.0.jar:3.11.0]

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]

        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]

===== 

After that we manually installed JDK 8 u151. We updated the environment to point to the older Java. But, nothing is helping.

How can we fix this issue?

We also tried changing cassandra-env.sh to point to the right JDK. But if we do that, there is no activity at all in cassandra/logs/debug.log. 

Thanks

Prashant

[Chris Donati]

Upgrading to the latest version of Cassandra should fix the issue. We merged a commit to master yesterday with that change: https://github.com/AppScale/appscale/pull/2703

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Prashant Hegde]

Hi Chris,

Any idea how do we get this onto a production server? 

Should we do a appscale upgrade?

Prashant

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.
To post to this group, send email to appscale_community@googlegroups.com.

Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.
To post to this group, send email to appscale_community@googlegroups.com.

[Chris Donati]

Unfortunately, an `appscale upgrade` won't fix the issue since the current released version of AppScale does not have the commit that fixes it.

If you'd like to just upgrade Cassandra manually, you should:

1. Copy your Cassandra configuration files (cassandra-env.sh and cassandra.yaml) from /opt/cassandra/cassandra/conf to a backup location. (this is important because AppScale modifies the configuration files from the released version)
2. Remove the old Cassandra directory (/opt/cassandra/cassandra) (DO NOT remove /opt/appscale/cassandra since that is the data directory)
   
3. Extract the new apache-cassandra-3.11.2-bin.tar.gz to /opt/cassandra and rename the directory to "cassandra" so that you have /opt/cassandra/cassandra/bin, etc. instead of /opt/cassandra/apache-cassandra-3.11.2/bin, etc.
4. Copy your backed up configuration files back to /opt/cassandra/cassandra/conf
5. Run `chown -R cassandra /opt/cassandra/cassandra`
6. Cassandra should now be able to start.
   

If you instead want to upgrade to current master, you could:

1. Stop appscale with `appscale down`
2. Update the repo in /root/appscale to the latest commit on master (git checkout master && git pull origin master)
   
3. Run a build with `bash debian/appscale_build.sh`
4. Start appscale again with `appscale up`
   

[Prashant Hegde]

Chris,

Thanks so much for the detailed instructions.

We did "git pull orgin master" and "appscale_build" and server is up and running !!

This issue came from nowhere, and can't thank you enough for your timely help.

Prashant

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsubscribe@googlegroups.com.

To post to this group, send email to appscale_community@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsubscribe@googlegroups.com.

To post to this group, send email to appscale_community@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsubscribe@googlegroups.com.

To post to this group, send email to appscale_community@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.