Support for the Raspberry Pi 2 with Ubuntu Trusty is broken for AppScale 3.0.1

Christian Baun

unread,

Sep 5, 2016, 9:32:37 AM9/5/16

to AppScale Community

Hi all,

I try to install the latest AppScale on a Raspberry Pi 2 with Ubuntu Trusty, following the tutorial

https://github.com/AppScale/appscale/wiki/Installing-AppScale-on-Raspberry-Pi-2

but executing the bootstrap.sh results in...

==========================

# bash bootstrap.sh --tag dev

...

make[1]: *** [libzkmt_la-mt_adaptor.lo] Error 1

make[1]: Leaving directory `/var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/zkc-3.4.5/c'

make: *** [all] Error 2

*** extconf.rb failed ***

Could not create Makefile due to some reason, probably lack of

necessary libraries and/or headers. Check the mkmf.log file for more

details. You may need configuration options.

Provided configuration options:

--with-opt-dir

--without-opt-dir

--with-opt-include

--without-opt-include=${opt-dir}/include

--with-opt-lib

--without-opt-lib=${opt-dir}/lib

--with-make-prog

--without-make-prog

--srcdir=.

--curdir

--ruby=/usr/bin/ruby1.9.1

extconf.rb:51:in `safe_sh': command failed! make 2>&1 (RuntimeError)

from extconf.rb:76:in `block (2 levels) in <main>'

from extconf.rb:71:in `chdir'

from extconf.rb:71:in `block in <main>'

from extconf.rb:55:in `chdir'

from extconf.rb:55:in `<main>'

Gem files will remain installed in /var/lib/gems/1.9.1/gems/zookeeper-1.4.11 for inspection.

Results logged to /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/gem_make.out

failed!

==========================

I can compile zookeeper manually by patching zookeeper...

# cd /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/zkc-3.4.5/c/src/

# wget https://raw.githubusercontent.com/serverdensity/zookeeper-debian-packaging/master/build/debian/patches/fixes/ZOOKEEPER-705.patch

# patch mt_adaptor.c < ZOOKEEPER-705.patch

# cd ..

# ./configure

# make

But what do I have to do next?

Best Regards and thanks for any help. :-)

Christian

BTW: I also tried it with AppScale 2.7.1 because Raspberry Pi 2 support was a feature of this revision and when I execute ~/appscale-2.7.1/debian/appscale_build.sh the script fails with this output:

Fetching: json-2.0.2.gem (100%)

ERROR:  Error installing json:

json requires Ruby version ~> 2.0.

Ubuntu Trusty contains ruby1.9.1

Christian Baun

unread,

Sep 6, 2016, 11:33:23 AM9/6/16

to AppScale Community

Hi All,

Now, I was able to install AppScale 2.7.1 on a Raspberry Pi 2 with Ubuntu Trusty. There were two issues.

First, Ruby > 2.0 was required to install json. This was not so simple because I found no armhf repository with Ruby 2.1.x for Trusty. To solve this, I compiled and installed Ruby 2.1.5 via these commands.

# \curl -L https://get.rvm.io | bash
# source /usr/local/rvm/scripts/rvm
# rvm get stable
# rvm install ruby-2.1.5
# update-alternatives --install /usr/bin/ruby ruby /usr/local/rvm/rubies/ruby-2.1.5/bin/ruby 400 \
--slave /usr/bin/rake rake /usr/local/rvm/rubies/ruby-2.1.5/bin/rake \
--slave /usr/bin/ri ri /usr/local/rvm/rubies/ruby-2.1.5/bin/ri \
--slave /usr/bin/rdoc rdoc /usr/local/rvm/rubies/ruby-2.1.5/bin/rdoc \
--slave /usr/bin/gem gem /usr/local/rvm/rubies/ruby-2.1.5/bin/gem \
--slave /usr/bin/irb irb /usr/local/rvm/rubies/ruby-2.1.5/bin/irb

# ruby --version
ruby 2.1.5p273 (2014-11-13 revision 48405) [armv7l-linux-eabihf]

# gem --version

2.4.8

Google does not provide the appengine-java-sdk-1.8.4.zip any more and the installation process was not able to fetch it. To solve this, I modfied inside the file ~/appscale-2.7.1/AppServer_Java/build.xml the value of gae_url. This worked for me:

Now, after I installed appscale and the appscale-tools-2.7.1 and after I created a (single node) cluster, I wait for the AppController to finish the pre-processing tasks and I there is no progress.

# appscale up

Starting AppScale 2.7.1 over a virtualized cluster.

Log in to your head node: ssh -i /root/.appscale/appscale1eef490f81304ca0a5db86ff4c04a65b.key ro...@10.0.2.100

Head node successfully initialized at 10.0.2.100.

Copying over deployment credentials

Starting AppController at 10.0.2.100

Please wait for the AppController to finish pre-processing tasks.

I there anything I can do to find out why it hangs?

Best Regards
Christian

chris....@appscale.com

unread,

Sep 6, 2016, 3:07:10 PM9/6/16

to AppScale Community

Hi Christian,

@lmandres removed the reference to his custom zookeeper gem repo in https://github.com/AppScale/appscale/pull/1953. I would try starting from master, applying the changes from that pull request in reverse and building with `bash debian/appscale_build.sh`. There's a pull request on the gem's repo to address the issue, but it looks like it needs some additional work.

There are a few issues with building from source on 2.7.1. The first, as you discovered, is that the json gem removed support for Ruby 1.9. In current master, we are using version 1.8.3 of the gem rather than a later version of Ruby. We haven't tested with later versions of Ruby, so there may be issues if you try with 2.1.

I believe the latest version of httparty is also problematic. Current master uses version 0.13.7.

Lastly, as you mentioned, it seems that google has removed the 1.8.4 Java SDK. We have it hosted in our S3 build repo, and current master uses that.

To answer your question about debugging it after it hangs at that point, I would take a look at the end of /var/log/appscale/controller-17443.log to see if there are any errors or stack traces.

-Chris

Christian Baun

unread,

Sep 7, 2016, 4:50:56 AM9/7/16

to AppScale Community

Hi Chris,

thanks for your reply. :-)

When i start AppScale 2.7.1, it is stalled infinitely. The "appscale up" process does still exist, but nothing happens for several hours.

# ps aux | grep appscale

root 1281 0.1 2.1 24132 20228 pts/1 S+ 10:14 0:02 /usr/bin/python /usr/local/bin/appscale up

No controller*.log is created. /var/log/appscale/ only contains an (empty) directory "cassandra".

# ls -la /var/log/appscale/

total 12

drwxrwxr-x 3 root adm 4096 Sep 6 16:06 .

drwxrwxr-x 13 root syslog 4096 Jan 1 1970 ..

drwxr-xr-x 2 root root 4096 Sep 6 16:06 cassandra

As next step, I try to apply lmandres' zookeeper gem into AppScale 2.9 or 3.*.

Best Regards

Christian

Christian Baun

unread,

Sep 7, 2016, 9:23:13 AM9/7/16

to AppScale Community

Hi Chris,

I tried now the installation of AppScale 2.9 and by using the lines from the lmandres' patch and the custom zookeeper gem, it worked. When I try to start AppScale, it still hangs at the "Please wait for the AppController to finish pre-processing tasks" point but now I have a file /var/log/appscale/controller-17443.log which says repetitively:

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- soap/rpc/httpserver (LoadError)

from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

from /root/appscale/AppController/djinnServer.rb:5:in `<main>'

Obviously, Ruby 1.9.1 was used here, but I have Ruby 2.1.5 installed:

# ruby --version

ruby 2.1.5p273 (2014-11-13 revision 48405) [armv7l-linux-eabihf]

# gem --version

2.4.8

Do you have any idea what I can try to fix this?

Best Regards

Christian

Am Dienstag, 6. September 2016 21:07:10 UTC+2 schrieb chris....@appscale.com:

chris....@appscale.com

unread,

Sep 7, 2016, 8:30:04 PM9/7/16

to AppScale Community

My guess is that the gems (soap4r in particular from that error message) were installed for your ruby 2.1 installation. I would change your default ruby (and gem) version back. For example, I get '1.8.23' as the output from `gem --version`. After doing that, you can run the build again. It may be enough to just run the installgems function in debian/appscale_install_functions.sh after changing your default gem binary.

I submitted a new pull request to fix the gem build for ARM. Until there's a new release that includes the patch, I'll look into hosting a custom gem for building AppScale on ARM.

Christian Baun

unread,

Sep 8, 2016, 2:44:11 PM9/8/16

to AppScale Community

With Ruby < 2.0 I cannot install the json gem. Therefore I uncommented the line

gem install json ${GEMOPT}

inside appscale_install_functions.sh, switched back to Ruby 1.9.3...

# ruby --version

ruby 1.9.3p484 (2013-11-22 revision 43786) [arm-linux-eabihf]

# gem --version

1.8.23

executed successfully appscale_build.sh...

# gem list

*** LOCAL GEMS ***

bundler-unload (1.0.2)

docile (1.1.5)

executable-hooks (1.3.2)

gem-wrappers (1.2.7)

httparty (0.14.0)

httpclient (2.8.2.3)

json (2.0.2)

multi_xml (0.5.5)

rake (11.2.2)

rubygems-bundler (1.4.4)

rvm (1.11.3.9)

simplecov (0.12.0)

simplecov-html (0.10.0)

soap4r-ruby1.9 (2.0.5)

zookeeper (1.4.11)

and then executed "appscale up" but the situation did not improve. AppScale hangs at the "Please wait for the AppController to finish pre-processing tasks" point: and /var/log/appscale/controller-17443.log gets filled with these entries:

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- soap/rpc/httpserver (LoadError)

from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

from /root/appscale/AppController/djinnServer.rb:5:in `<main>'

chris....@appscale.com

unread,

Sep 8, 2016, 4:21:05 PM9/8/16

to AppScale Community

I'm having trouble reproducing the issue that you are seeing. Your gem list surprises me. I'm not sure how you managed to get version 2.0.2 of the json gem installed for your Ruby 1.9 environment. If I try `gem install json -v 2.0.2`, I get an an error message: "json requires Ruby version ~> 2.0." I can change ruby versions with rvm and install that json 2.0.2, but when I switch back to the system's ruby, it doesn't show up in the gem list.

In any case, I would install version 1.8.3 of the json gem and version 0.13.7 of the httparty gem. Those are the versions installed on current master.

What's the output if you open an irb shell and run `RUBY_VERSION` and then `require 'soap/rpc/httpserver'`?

Christian Baun

unread,

Sep 19, 2016, 5:42:13 AM9/19/16

to AppScale Community

After doing some tests I found out that version 1.8.3 of the json gem and version 0.13.7 of the httparty gem installs fine when Ruby 1.9.3 is installed.

For this reason and because my image was in a state which made it unable to reproduce all the steps I did, I tried a fresh installation.

I used again the Ubuntu Trusty image and this time I installed Ruby 1.9.3 via rvm before installing AppScale.

# rvm list

rvm rubies

=* ruby-1.9.3-p551 [ armv7l ]

# => - current

# =* - current && default

# * - default

# ruby --version

ruby 1.9.3p551 (2014-11-13 revision 48407) [armv7l-linux-eabi]

# gem --version

2.4.8

At this point in time, there was no other ruby version installed on the system.

Next, I installed version 1.8.3 of the json gem and version 0.13.7 of the httparty gem (which both worked) and installed the latest AppScale release and the corresponding appscale tools.

https://github.com/AppScale/appscale/archive/3.1.0.tar.gz

https://github.com/AppScale/appscale-tools/archive/3.1.0.zip

(I tried it via the compressed archives because I wanted to get something which is reproducible for some time.)

The compilation via /root/appscale-3.1.0/debian/appscale_build.sh worked and all gems have been installed...

# gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)

bundler-unload (1.0.2)

docile (1.1.5)

executable-hooks (1.3.2)

gem-wrappers (1.2.7)

httparty (0.13.7)

httpclient (2.8.2.4)

io-console (0.3)

json (1.8.3, 1.5.5)

minitest (2.5.1)

multi_xml (0.5.5)

rake (11.2.2, 0.9.2.2)

rdoc (3.9.5)

rubygems-bundler (1.4.4)

rvm (1.11.3.9)

simplecov (0.12.0)

simplecov-html (0.10.0)

soap4r-ruby1.9 (2.0.5)

zookeeper (1.4.11)

... but when I tried to start AppScale, the starting procedure did not pass a point, when the log "/var/log/appscale/controller-17443.log" got filled with this message:

/usr/bin/ruby: No such file or directory -- /root/appscale/AppController/djinnServer.rb (LoadError)

Obviously, installing AppScale from the directory "/root/appscale-3.1.0/" was not the best idea. Creating a symbolic link "/root/appscale/" to "/root/appscale-3.1.0/" did not help much.

Next, I tried to install AppScale 3.1.0 as described here ...

https://github.com/AppScale/appscale/wiki/Installing-AppScale-from-source

... via "bash bootstrap.sh" and this caused the installation of another ruby 1.9.3

# ruby --version

ruby 1.9.3p484 (2013-11-22 revision 43786) [arm-linux-eabihf]

# gem --version

1.8.23

# gem list

*** LOCAL GEMS ***

bundler-unload (1.0.2)

docile (1.1.5)

httparty (0.13.7)

httpclient (2.8.2.4)

json (1.8.3)

multi_xml (0.5.5)

rake (11.2.2)

simplecov (0.12.0)

simplecov-html (0.10.0)

soap4r-ruby1.9 (2.0.5)

zookeeper (1.4.11)

The installation procedure did also install the required gems and after it finished, I again tried to start AppScale and now I get this result:

# head -n150 /var/log/appscale/controller-17443.log

I, [2016-09-19T11:08:14.631387 #15774] INFO -- : ==== Starting AppController (pid: 15774) ====

E, [2016-09-19T11:08:15.820336 #15774] ERROR -- DjinnServer: OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=unknown state

/usr/lib/ruby/1.9.1/openssl/ssl-internal.rb:172:in `accept'

I, [2016-09-19T11:08:16.125938 #15774] INFO -- : Converting 'verbose' with value 'False'.

I, [2016-09-19T11:08:16.126343 #15774] INFO -- : Converting 'alter_etc_resolv' with value 'False'.

I, [2016-09-19T11:08:16.126551 #15774] INFO -- : Converting 'ips' with value '[]'.

I, [2016-09-19T11:08:16.126747 #15774] INFO -- : Converting 'table' with value 'cassandra'.

I, [2016-09-19T11:08:16.127024 #15774] INFO -- : Converting 'user_commands' with value '[]'.

I, [2016-09-19T11:08:16.127267 #15774] INFO -- : Converting 'autoscale' with value 'True'.

I, [2016-09-19T11:08:16.127459 #15774] INFO -- : Converting 'replication' with value '1'.

I, [2016-09-19T11:08:16.127659 #15774] INFO -- : Converting 'hostname' with value '10.0.2.102'.

I, [2016-09-19T11:08:16.127833 #15774] INFO -- : Converting 'flower_password' with value 'appscale'.

I, [2016-09-19T11:08:16.128005 #15774] INFO -- : Converting 'max_memory' with value '400'.

I, [2016-09-19T11:08:16.128193 #15774] INFO -- : Converting 'clear_datastore' with value 'False'.

I, [2016-09-19T11:08:16.128415 #15774] INFO -- : Converting 'keyname' with value 'appscale790533e13a9f4294b2dfdab2fcc0bd3a'.

I, [2016-09-19T11:08:16.128605 #15774] INFO -- : Converting 'appengine' with value '1'.

W, [2016-09-19T11:08:16.129035 #15774] WARN -- : max_images is less than the number of nodes!

I, [2016-09-19T11:08:16.668074 #15774] INFO -- : Starting iaas_manager on port 17444 with start command [/usr/bin/python2 /root/appscale/InfrastructureManager/infrastructure_manager_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/InfrastructureManager/infrastructure_manager_service.py /usr/bin/python2]

I, [2016-09-19T11:08:16.796947 #15774] INFO -- : Started InfrastructureManager successfully!

I, [2016-09-19T11:08:16.797497 #15774] INFO -- : /etc/appscale/zookeeper_locations.json doesn't exist: not restoring data.

I, [2016-09-19T11:08:16.846542 #15774] INFO -- : Reloading nginx service.

[ALERT] 262/110817 (15872) : config : no <listen> line. Nothing to do !

[ALERT] 262/110817 (15872) : Fatal errors found in configuration.

I, [2016-09-19T11:08:17.286290 #15774] INFO -- : Spawning/setting up other nodes.

I, [2016-09-19T11:08:17.287166 #15774] INFO -- : Nodes info after starting remotes: .

I, [2016-09-19T11:08:17.564989 #15774] INFO -- : Done initializing nodes.

I, [2016-09-19T11:08:19.779658 #15774] INFO -- : Starting haproxy on port 9999 with start command [/usr/sbin/service haproxy start] and stop command [/usr/sbin/service haproxy stop]

I, [2016-09-19T11:08:19.892680 #15774] INFO -- : HAProxy configured and started.

I, [2016-09-19T11:08:23.255697 #15774] INFO -- : Starting nginx on port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop]

monit: action failed -- There is no service by that name

I, [2016-09-19T11:08:23.426713 #15774] INFO -- : Nginx configured and started.

cat: /var/run/haproxy.pid: No such file or directory

I, [2016-09-19T11:08:23.592029 #15774] INFO -- : Reloading nginx service.

I, [2016-09-19T11:08:23.815880 #15774] INFO -- : Reloading nginx service.

I, [2016-09-19T11:08:23.984620 #15774] INFO -- : Only saw one machine, therefore my node is also a slave node

I, [2016-09-19T11:08:27.509696 #15774] INFO -- : Starting API Services.

I, [2016-09-19T11:08:27.511124 #15774] INFO -- : Waiting for DB services ...

I, [2016-09-19T11:08:27.519983 #15774] INFO -- : Starting database services.

I, [2016-09-19T11:08:27.520638 #15774] INFO -- : Starting up Cassandra seed node

I, [2016-09-19T11:08:27.629082 #15774] INFO -- : Starting zookeeper.

[ ? ] console-setup

[ ? ] dns-clean

[ ? ] ejabberd

[ ? ] irqbalance

[ ? ] killprocs

[ ? ] kmod

[ ? ] networking

[ ? ] ondemand

[ ? ] pppd-dns

I, [2016-09-19T11:08:29.674158 #15774] INFO -- : Starting cassandra on port 9999 with start command [/opt/cassandra/cassandra/bin/cassandra start -p /var/appscale/appscale-cassandra.pid] and stop command [/bin/bash -c 'kill -s SIGTERM `cat /var/appscale/appscale-cassandra.pid`']

[ ? ] rc.local

[ ? ] sendsigs

[ ? ] umountfs

[ ? ] umountnfs.sh

[ ? ] umountroot

[ - ] zookeeper

I, [2016-09-19T11:08:34.559338 #15774] INFO -- : Initializing ZooKeeper.

stop: Unknown instance:

I, [2016-09-19T11:08:36.702109 #15774] INFO -- : Starting zookeeper on port 2181 with start command [/usr/sbin/service zookeeper start] and stop command [/usr/sbin/service zookeeper stop]

monit: action failed -- There is no service by that name

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.

I, [2016-09-19T11:08:36.823621 #15774] INFO -- : Done configuring zookeeper.

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.

Error: The node does not have system_traces yet, probably still bootstrapping

error: -1

-- StackTrace --

java.lang.ArrayIndexOutOfBoundsException: -1

at java.util.ArrayList.elementData(ArrayList.java:400)

at java.util.ArrayList.get(ArrayList.java:413)

at org.apache.cassandra.dht.ByteOrderedPartitioner.describeOwnership(ByteOrderedPartitioner.java:255)

at org.apache.cassandra.service.StorageService.effectiveOwnership(StorageService.java:4017)

at org.apache.cassandra.service.StorageService.effectiveOwnership(StorageService.java:87)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)

at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)

at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)

at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)

at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)

at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)

at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)

at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)

at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1488)

at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)

at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1329)

at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1421)

at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:849)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)

at sun.rmi.transport.Transport$2.run(Transport.java:202)

at sun.rmi.transport.Transport$2.run(Transport.java:199)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:198)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack

UN 10.0.2.102 100.7 KB 256 100.0% 61c19e35-94e7-4d0e-94e9-7856c30e3c4c rack1

2016-09-19 11:13:47,189 INFO policies.py:283 Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '10.0.2.102'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes

... Now, the file gets constantly filled with these error messages ...

I think this is the showstopper

"nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'."

But there is some service running at port 7199

# telnet 127.0.0.1 7199

Trying 127.0.0.1...

Connected to 127.0.0.1.

Escape character is '^]'.

What can I try/do to fix this?

Best Regards

chris....@appscale.com

unread,

Sep 19, 2016, 1:19:01 PM9/19/16

to AppScale Community

The log message you posted is actually normal behavior. The "Failed to connect" message is the output of `nodetool status`, which the controller is continually running until Cassandra comes up. The block starting with "Datacenter: datacenter1" and ending with "UN 10.0.2.102 ..." indicates that Cassandra is in an "Up Normal" state. I agree that this is confusing, though. I'll look into replacing that output with something like "Waiting for Cassandra to start."

It's likely that the controller crashed after this point, though. I would look at the end of the controller log for something like "Failed to prime Cassandra", which would indicate that it finally crashed.

If you are running AppScale on just a single raspberry pi, keep in mind that memory will be extremely tight and operations will be very slow. We typically recommend reserving at least 4GB for Cassandra + whatever memory your app needs to run, so the 1GB available on the pi is very limiting. In addition, some of the default timeouts on the Cassandra driver are too small for the speed of the pi.

I've put together a branch containing the changes I needed to make in order to get it started. It's mostly tweaks to reduce memory consumption and remove driver timeouts. I would also suggest decreasing `max_memory` in your AppScalefile to something like 100. Let me know if that helps.

Christian Baun

unread,

Sep 19, 2016, 3:28:53 PM9/19/16

to AppScale Community

It works! AppScale starts after I applied your tweaks and after I reduced the GPU share of the main memory to the minimum value which is 16 MB. Per dafault it is 64 MB. The second part was mandatory too. I reduced the GPU share by adding this line into the file /boot/config.txt

gpu_mem=16

You need to do a reboot to apply this. Whithout this step, AppScale again did not start because of some memory problems but the start process went much much further compared with the situation before I applied you tweaks.

Initially, after ubuntu has been started (and the GPU share is reduced to 16 MB), this much memory is available:

# free -h

total used free shared buffers cached

Mem: 971M 98M 873M 308K 15M 40M

-/+ buffers/cache: 41M 929M

Swap: 0B 0B 0B

And after AppScale 3.1.0 has been started with your memory tweaks, this much main memory is available:

# free -h

total used free shared buffers cached

Mem: 971M 942M 29M 380K 11M 71M

-/+ buffers/cache: 859M 112M

Swap: 0B 0B 0B

Tomorrow I will test the functionality.

Thanks a lot for your kind help!

Reply all

Reply to author

Forward