How to integrate AppScale 1.8 with Cassandra 3.2.1

[Carolina Simões Gomes]

Hello,

My team and I have been using AppScale 1.8 (old version I know) for a research project, which is compatible with Cassandra 2.3.1. We want to replace that Cassandra with Cassandra 3.2.1, as we have made changes to Cassandra 3.2.1's Java driver for our research project purposes. The goal is to have Java applications running in AppScale that can utilize the extra features we have developed for the Cassandra driver directly, without going through the AppDB Datastore/Memcache infrastructure. 

With that goal in mind, we have made a few changes to AppScale scripts in order to integrate Cassandra 3.2.1:

1. Changing the command used to start Cassandra in cassandra_helper.rb, which doesn't require "cassandra start" anymore

2. Replacing the cassandra-env.sh and cassandra.in.sh templates with cassandra-env.sh and cassandra.in.sh from Cassandra 3.2.1, *plus* adding the same JVM_OPTS from the template cassandra.in.sh (e.g. setting up the JMX port configuration to APPSCALE-JMX-PORT).

3. Using the cassandra.yaml from Cassandra 3.2.1, except the fields "initial_token", "seeds" and "local_address", which were replaced by "APPSCALE*" definitions, similarly to the template cassandra.yaml.

4. Changed the default Java version to Java 8, as that's required by Cassandra 3.2.1, but still keeping the Java 7 used by AppScale 1.8 (so in theory I don't think that should be a problem, as from briefly looking the scripts, they access the Java 7 folder path directly and not using JAVA_HOME).

After doing the above, Cassandra is started and shutdown normally through the appscale up and down commands, and the logs from /var/log/appscale show it is working fine. But then when I actually try to do "appscale up", the initialization of AppScale gets hung up on trying to start memcached. That's what I believe, at least, because it never completes, never shows any failure messages, and when I type "appscale status", the status is "Starting up memcache". When I check whether the memcached process is running, it is not. The logs in /var/log/appscale, however, say everything has been initialized, including memcached in port 11211.

Any ideas on where to even begin investigating this? I am at a loss.

[Carolina Simões Gomes]

And sorry, I actually mean the version of AppScale we are using is 2.3.1 and Cassandra version was originally 2.0.7 but we changed to 3.2.1.

Template cassandra.in.sh: https://github.com/AppScale/appscale/blob/2.3.1/AppDB/cassandra/templates/cassandra.in.sh

So I added the JVM_OPTS from the template above to Cassandra 3.2.1's cassandra.in.sh:

# Arguments to pass to the JVM
	JVM_OPTS=" \
	-ea \
	-Xms128M \
	-Xmx1G \
	-XX:TargetSurvivorRatio=90 \
	-XX:+AggressiveOpts \
	-XX:+UseParNewGC \
	-XX:+UseConcMarkSweepGC \
	-XX:+CMSParallelRemarkEnabled \
	-XX:+HeapDumpOnOutOfMemoryError \
	-XX:SurvivorRatio=128 \
	-XX:MaxTenuringThreshold=0 \
	-Dcom.sun.management.jmxremote.port=APPSCALE-JMX-PORT \
	-Dcom.sun.management.jmxremote.port=6666 \
	-Dcom.sun.management.jmxremote.ssl=false \
	-Dcom.sun.management.jmxremote.authenticate=false"

[chris....@appscale.com]

Hi Carolina,

That sounds like an interesting research project!

The first place I usually look when debugging an issue during an 'appscale up' is usually the controller log (/var/log/appscale/controller-17443.log). Unfortunately, we didn't have very helpful stacktraces for that service back in AppScale 2.3.1, but it may help to see what the last line was before the service crashed (if it did crash). Sometimes the controller starts services in parallel, so it may be tricky to track down exactly what thread is problematic.

The output of 'monit summary' is also usually helpful for determining which service or services are having difficulty starting. You may want to check /var/log/monit.log to see if monit is having difficulty starting any particular services.

If you would like to, you can give me a link to a branch with your changes, and I'll spin it up and try to see what the issue is.

-Chris

[Carolina Gomes]

Hello,

I have more details about this issue. Although the status when doing "appscale status" is "Starting up memcache", in fact the memcache process is started and the logs don't point anything wrong. What is actually wrong is that Cassandra is started on port 6666 (as per the JVM_OPTS above, which I got from the cassandra.in.sh file of AppScale 2.3.1 for Cassandra 2.0.7) and then, memcache tries to start the same Cassandra also on port 6666, thus complaining of "listen address already in use" and resulting in "Does not exist" for the cassandra-9999 process when I do "monit summary".

So the question is why, even though Cassandra is being started and shutdown when I do "appscale up" and "appscale down" (meaning monit can control it to some extent I guess), monit is somehow still trying to start another one on the same port. I know that cassandra is started in the script cassandra_helper.rb of AppDB/cassandra with the line that gets fed into monit:

start_cmd=${CASSANDRA_EXECUTABLE} start -p ${CASSANDRA_PID_FILE}

Because I am running as root and Cassandra 3.2.1 complains otherwise, and also says it cannot recognize the "start" command, I changed that to the command below (the -R means to force Cassandra to start even user is root; this is the command I would usually use to start Cassandra 3.2.1 on my own). Clearly the command below is causing the issue with monit.

start_cmd=${CASSANDRA_EXECUTABLE} -R -p ${CASSANDRA_PID_FILE}

Any ideas?

Sent from my iPhone

--
You received this message because you are subscribed to a topic in the Google Groups "AppScale Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/appscale_community/8YHBbWH7Oq8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[chris....@appscale.com]

I was able to replicate the issue that you are seeing. Since the format of cassandra.in.sh changed from 2.0.7 to 3.2.1 (and a jvm.options file was added), it's probably best to use the default file and not override the JVM_OPTS in that way.

I was able to get things working by telling AppScale to use the default JMX port instead of changing the cassandra.in.sh script. Here are the steps I took (starting from the state where you were seeing the problem):

• Make sure AppScale is stopped (`appscale down`)
  
• Copy the default bin/cassandra.in.sh from 3.2.1 to ~/appscale/AppDB/cassandra/cassandra/bin/cassandra.in.sh. (You could overwrite the one in the 'templates' folder as well, though that is only used during a build.)
• Edit ~/appscale/AppController/terminate.rb, changing `-p 7070` on line 54 to `-p 7199`.
• Edit ~/appscale/AppDB/cassandra/cassandra_helper.rb, removing the line that substitutes the JMX port (line 106).
• Edit ~/appscale/AppDB/cassandra/templates/cassandra.yaml, making the following changes (AppScale checks the thrift port before continuing.)
  
• start_rpc: true
• rpc_address: 0.0.0.0
• broadcast_rpc_address: APPSCALE-LOCAL

When I did an `appscale up` after making those changes, it worked. I didn't do any further testing, though. Let me know if that works for you or if you run into any other issues.

To unsubscribe from this group and all its topics, send an email to appscale_community+unsub...@googlegroups.com.

[Carolina Gomes]

I didn't have access to email today so ended up making it work on my own, the solution was essentially the same!

Another thing too is that I had to change the JAVA_HOME of cassandra.yaml to Java 8 since AppScale 2.3.1 uses 7. So Java apps work (we are not using Datastore/Memcache APIs so I think those are not working due to the issue I describe below).

Other than that, another problem is that AppScaleDashboard doesn't work because pycassa (used by cassandra_interface.py) is not compatible with the newer versions of Cassandra. I am going to start working on replacing that with the new Cassandra python driver by Datastax soon.

Is there a reason why the Appscale team chose to use pycassa instead of the python driver (maybe not available at the time the project started)? Would porting the interface to the driver be a good contribution to the project? :-)

Sent from my iPhone

To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.

[chris....@appscale.com]

Nice. I'm glad to hear that you got it working.

Yes. It is my understanding that the Python driver was not available when the datastore was first implemented.

The good news is that we already have a working branch for using the Python driver with the datastore. Our plan is to merge it into master within the next few weeks to replace the thrift interface.

Here's the branch if you'd like to try it out: https://github.com/AppScale/appscale/pull/2036

[Carolina Simões Gomes]

That's great! I will pull that and test it out :-)

[Carolina Gomes]

PhD Student

University of Toronto
+1 (416) 931-4774

To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "AppScale Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/appscale_community/8YHBbWH7Oq8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "AppScale Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/appscale_community/8YHBbWH7Oq8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.