appscale-run-instances is frozen

[Florian]

Hi,

my Appscale installation is running as Xen-VM on an old Desktop-
System, so there is only a single Appscale-Instance with 2GB of RAM
and two cores. I configured the ips.yaml like described in
http://code.google.com/p/appscale/wiki/Single_Node_AppScale_Deployment.

After the execution of the following commands as a non root user at
dom0, the system hangs:
$ appscale-add-keypair --ips ips.yaml
$ appscale-run-instances --file guestbook.tar.gz --ips ips.yaml --
table cassandra -v

The last reaction is:
Enter your new password:
Enter again to verify:
Please wait for AppScale to prepare your machines for use.

The following modifications were made by myself on the Appscale VM:

1.) echo 0 > /proc/sys/net/ipv4/tcp_syncookies
In order to get rid of the following kernel-messages while "appscale-
run-instances":
"Dec 10 07:19:26 appscale-image kernel: [ 1010.844088] possible SYN
flooding on port 17443. Sending cookies."

2.) sysctl net.netfilter.nf_conntrack_acct=1

I appended the log, that was stored under /tmp/ of the Appscale
instance.

Regards
Florian

The logfile /tmp/192.168.10.10.log :

/usr/lib/ruby/1.8/webrick/ssl.rb:86: warning: method redefined;
discarding old listen
/root/appscale/AppController/djinn.rb:26: warning: `*' interpreted as
argument prefix
kill: No such process
[Thu Dec 10 11:33:38 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:33:38 -0800 2009] Waiting for data from the load
balancer or cmdline tools
[Thu Dec 10 11:34:04 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:34:04 -0800 2009] Public IP address is 192.168.10.10,
Private IP address is 192.168.10.10
[Thu Dec 10 11:34:04 -0800 2009] Djinn locations: Node at pub
192.168.10.10, priv 192.168.10.10 is currently load_balancer,
cassandra_master, appengine
[Thu Dec 10 11:34:04 -0800 2009] DB Credentials:
{"keyname"=>"appscale", "ips"=>"", "replication"=>"1",
"table"=>"cassandra", "keypath"=>"appscale.key",
"hostname"=>"192.168.10.10"}
[Thu Dec 10 11:34:04 -0800 2009] Apps to load: guestbook
[Thu Dec 10 11:34:04 -0800 2009] My index = 0
[Thu Dec 10 11:34:29 -0800 2009] Got data from another node! DLoc =
[#<DjinnJobData:0x7f74753647a0 @jobs=["load_balancer",
"cassandra_master", "appengine"], @private_ip="192.168.10.10",
@public_ip="192.168.10.10">], {"keyname"=>"appscale", "ips"=>"",
"replication"=>"1", "table"=>"cassandra", "keypath"=>"appscale.key",
"hostname"=>"192.168.10.10"}, AppsToLoad = guestbook
[Thu Dec 10 11:34:29 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:34:34 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:34:34 -0800 2009] pre-loop: Node at pub 192.168.10.10,
priv 192.168.10.10 is currently load_balancer, cassandra_master,
appengine
[Thu Dec 10 11:34:34 -0800 2009] Master is at 192.168.10.10, slaves
are at
[Thu Dec 10 11:34:39 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:34:49 -0800 2009] About to run [start_load_balancer()]
[Thu Dec 10 11:34:49 -0800 2009] Starting up Load Balancer
Sending 'start' command

The following watches were affected:
appscale-mongrel-8000
appscale-mongrel-8001
appscale-mongrel-8002
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
^M100 4948 100 4948 0 0 302k 0 --:--:-- --:--:--
--:--:-- 302k^M100 4948 100 4948 0 0 280k 0 --:--:--
--:--:-- --:--:-- 0
% Total % Received % Xferd Average Speed Time Time
Time Current Dload Upload Total Spent Left Speed
100 4948 100 4948 0 0 1832k 0 --:--:-- --:--:--
--:--:-- 1832k
100 4948 100 4948 0 0 1227k 0 --:--:-- --:--:--
--:--:-- 0
E, [2009-12-10T11:35:55.293105 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
^M100 4948 100 4948 0 0 1659k 0 --:--:-- --:--:--
--:--:-- 1659k^M100 4948 100 4948 0 0 1165k 0 --:--:--
--:--:-- --:--:-- 0
E, [2009-12-10T11:36:30.717079 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
[Thu Dec 10 11:34:54 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:35:04 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:35:09 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:35:19 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:35:24 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:35:35 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:35:50 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:35:55 -0800 2009] About to run
[CassandraHelper.start_master(self)]
[Thu Dec 10 11:35:55 -0800 2009] Starting up Cassandra as master
[Thu Dec 10 11:36:00 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:36:00 -0800 2009]
[Thu Dec 10 11:36:05 -0800 2009]
[Thu Dec 10 11:36:10 -0800 2009] 0
Starting Cassandra: Using cassandra for the datastore
rm -f /var/cassandra/*/*
/root/appscale/AppDB/cassandra/cassandra//bin/cassandra start
DONE SETTING UP CASSANDRA. REMEMBER TO PRIME THE DB
Listening for transport dt_socket at address: 8888
DEBUG - Loading settings from /root/appscale/AppDB/cassandra/
cassandra//bin/../conf/storage-conf.xml
DEBUG - Starting to listen on 192.168.10.10:7001
[Thu Dec 10 11:36:15 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:36:15 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:36:20 -0800 2009] Waiting on 192.168.10.10:9160 to be
open (currently closed).
[Thu Dec 10 11:36:30 -0800 2009] prime cassandra database
get_keys exception eval() arg 1 must be a string or code object
get_keys exception eval() arg 1 must be a string or code object
get_keys exception eval() arg 1 must be a string or code object
CREATE TABLE SUCCESS FOR USER AND APPS
['CA_ERROR:', 'email', 'pw', 'date_creation', 'date_change',
'date_last_login', 'applications', 'appdrop_rem_token',
'appdrop_rem_token_exp', 'visit_cnt', 'cookie', 'cookie_ip',
'cookie_exp', 'cksum', 'enabled']
['CA_ERROR:', 'name', 'version', 'owner', 'admins_list', 'host',
'port', 'creation_date', 'last_time_updated_date', 'yaml_file',
'cksum', 'num_entries', 'tar_ball', 'enabled', 'classes']
/var/lib/python-support/python2.6/MySQLdb/__init__.py:34:
DeprecationWarning: the sets module is deprecated
from sets import ImmutableSet
[Thu Dec 10 11:36:30 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:36:40 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
/var/lib/python-support/python2.6/MySQLdb/__init__.py:34:
DeprecationWarning: the sets module is deprecated
from sets import ImmutableSet
E, [2009-12-10T11:40:03.058217 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:40:03.078932 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:40:23.322121 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:40:33.431861 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
[Thu Dec 10 11:36:40 -0800 2009]

UAServer is at [192.168.10.10]

[Thu Dec 10 11:36:40 -0800 2009] About to run [start_appengine()]
[Thu Dec 10 11:36:40 -0800 2009] starting appengine - pbserver is at
[192.168.10.10]
[Thu Dec 10 11:36:45 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:36:45 -0800 2009] Get app data for guestbook said
[Error: CA_ERROR:Not found]
[Thu Dec 10 11:36:51 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: CA_ERROR:Not found
[Thu Dec 10 11:37:01 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:37:11 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:37:16 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:38:47 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: CA_ERROR:Not found
[Thu Dec 10 11:38:52 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:38:57 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:39:07 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:39:17 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:39:27 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:39:42 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:39:47 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: CA_ERROR:Not found
[Thu Dec 10 11:39:47 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:40:03 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:40:13 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:40:13 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:40:28 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:40:33 -0800 2009] Did
[CrzuQAptEoHkfMIXfGYMESqlbPeaH2Wm] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [false]
[Thu Dec 10 11:40:48 -0800 2009] WaitinE, [2009-12-10T11:41:44.314746
#4962] ERROR -- DjinnServer: OpenSSL::SSL::SSLError: SSL_accept
SYSCALL returned=5 errno=0 state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:42:34.975801 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:43:25.573126 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:43:30.628132 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:43:50.838640 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:43:50.859398 #4962] ERROR -- DjinnServer:
OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0
state=SSLv3 read client certificate A
/usr/lib/ruby/1.8/openssl/ssl.rb:166:in `accept'
E, [2009-12-10T11:44:21.311411 #4962] ERROR -- DjinnServer: syslog
already open (RuntimeError)
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
…
/root/appscale/AppController/djinn.rb:28:in `new'
/root/appscale/AppController/djinn.rb:28:in `open'
/root/appscale/AppController/djinn.rb:28:in `initialize'
/usr/lib/ruby/1.8/openssl/ssl.rb:56:in `peeraddr'
/usr/lib/ruby/1.8/openssl/ssl.rb:56:in `peeraddr'
/usr/lib/ruby/1.8/webrick/server.rb:166:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/1.8/soap/rpc/httpserver.rb:123:in `run'
/usr/lib/ruby/1.8/logger.rb:659:in `start'
/root/appscale/AppController/djinnServer.rb:119
/root/appscale/AppController/djinnServer.rb:119:in `initialize'
/root/appscale/AppController/djinnServer.rb:119:in `new'
/root/appscale/AppController/djinnServer.rb:119
g for app data to have instance info for app named guestbook: Error:
CA_ERROR:Not found
[Thu Dec 10 11:40:53 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:03 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:13 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:24 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:44 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:44 -0800 2009] Did
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR] match known secret
[nXBzm8rlOWNOXA2ZTD1NOR3C19LDqikR]? [true]
[Thu Dec 10 11:41:49 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: CA_ERROR:Not found
...

[Chris Bunch]

Hi Florian,
Are you using AppScale 1.2? If so, you can modify two methods in the
file appscale/AppController/djinn.rb to fix the problem:

class Exception
alias real_init initialize
def initialize(*args)
real_init *args

File.open("/root/appscale/.appscale/exception.log", "w+") { |file|
file.write(message) }
begin
Syslog.open("appscale") { |s| s.debug(message) }
rescue RuntimeError
# this can occur if we write to syslog at the same time
# as one of the python components
end
end
end


And:

def self.log_debug(msg)
puts "[#{Time.now}] #{msg}"
begin
Syslog.open("appscale") { |s| s.debug(msg) }
rescue RuntimeError
# can also occur if two threads are in the appcontroller
# and both attempt to write to syslog at the same time
# used to guard this with a lock, but since it didn't
# resolve the python/ruby problem, decided against it
# in favor of this
end
end

This issue is fixed in AppScale 1.3 Beta as well, and comes up
infrequently whenever the Ruby AppController and the Python Protocol
Buffer Server try to access syslog at the same time (or other scenarios
described in the comments above).

Let me know how that works for you.

Thanks,

> --
>
> You received this message because you are subscribed to the Google Groups "AppScale Community" group.
> To post to this group, send email to appscale_...@googlegroups.com.
> To unsubscribe from this group, send email to appscale_commun...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/appscale_community?hl=en.
>
>

[Florian]

Hi Chris,

Yes, Appscale is running in version 1.2. When I updated the file
according to your description, there were no significant changes.
So I executed the script "appscale_scratch_install.sh" in the Appscale-
VM. Now I assume, that I have the latest version running on my server.

The "appscale-run-instances" application is still frozen, but I now
get constantly the same error:

"Waiting for app data to have instance info for app named guestbook:
Error: CA_ERROR:Not found"

I hope this was a step into the correct direction.

Do you have any further ideas?

Thanks,
Florian

Execution of "appscale-run-instances --file guestbook.tar.gz --ips
ips.yaml -v"
New secret key is iyTrGsRSVUhzmGtbEF11fQiCvOJoGG2E

{"keyname"=>"appscale", "ips"=>"", "replication"=>"1",
"table"=>"cassandra", "keypath"=>"appscale.key",
"hostname"=>"192.168.10.10"}

Head node successfully created at 192.168.10.10. It is now starting up
cassandra via the command line arguments given.
scp ...
Killing and starting server at 192.168.10.10
Please wait for the controller to finish pre-processing tasks.

This AppScale instance is linked to an e-mail address giving it
administrator privileges.
Enter your desired administrator e-mail address:

The new administrator password must be at least six characters long
and can include non-alphanumeric characters.

Enter your new password:
Enter again to verify:
Please wait for AppScale to prepare your machines for use.

After 10 minutes the following error was displayed and did not
disappear after about one hour.
The corresponding log on the server (tail /tmp/192.168.10.10.log):

UAServer is at [192.168.10.10]

[Sun Dec 13 01:24:31 -0800 2009] About to run [start_appengine()]
[Sun Dec 13 01:24:31 -0800 2009] starting appengine - pbserver is at
[192.168.10.10]
[Sun Dec 13 01:24:31 -0800 2009] Connection refused by 192.168.10.10
[Sun Dec 13 01:24:31 -0800 2009] DB not currently running at
192.168.10.10. Going into recovery mode and waiting for DB to come
online.
[Sun Dec 13 01:24:36 -0800 2009] Get app data for guestbook said
[Error: CA_ERROR:Not found]
[Sun Dec 13 01:24:37 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:24:42 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:24:47 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:24:52 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:24:57 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:25:02 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:25:07 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Sun Dec 13 01:25:12 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found
...

> ...
>
> Erfahren Sie mehr »

[Ross]

Florian <f.weisp...@googlemail.com> wrote:

[Chris Bunch]

Hi Florian,
I wouldn't recommend running the scratch install script in your
already built image. Theoretically, it should give you the latest
version of everything, but we've never tried building a new system from
inside a system that already had everything built in it (only brand new
Jaunty images).

What I would recommend is to instead download the new AppScale 1.3
(non-beta) and use that VM instead. It has a major fix for Cassandra
that may be related to what your log is showing is a problem below.

Let me know how that works for you.

Thanks,

[Florian]

Hi Chris,

I have downloaded the new stable 1.3 release and tested all databases
(Voldemort, MemcacheDB, MongoDB, and Cassandra), which are described
under "Single_Node_AppScale_Deployment". But none of the configuration
worked for me, as you could see in the log. [After each test, I
rebooted the VM.]
On the Appscale-VM, the only database-stuff I found is: cat /root/
appscale/AppController/config.yaml
---
adapter: hypertable
hostname: http://hornets.cs.ucsb.edu:8080
database: appscale
username: root
password: fakepwd
But this is still the original one.

Is there a big fault, that I am making, in building the following test
system:
Xen is running on "Linux server 2.6.26-2-xen-amd64 #1 SMP Sun Jun 21
08:09:04 UTC 2009 x86_64 GNU/Linux" with the debian-package "xen-
hypervisor-3.4".
The physical system (dom0) has an additional virtual interface with
the IP: 192.168.10.1
The Appscale-VM has the IP 192.168.10.10 (and could communicate with
the internet).
On dom0 the following ips.yaml is used:
---
:controller: 192.168.10.10
The deployment is started from dom0 as a none-root user.

Thanks
Florian

1.)
appscale-run-instances --table mongodb --file guestbook.tar.gz --ips
ips.yaml -v
...

Please wait for AppScale to prepare your machines for use.

Exception seen: #<Class:0x7f14a52cdc68>


[Mon Dec 14 15:20:55 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: MO_ERROR:data not found for
particular row and column for table name: APPS__ on row key:
guestbook, with columns: ['name', 'language', 'version', 'owner',

'admins_list', 'host', 'port', 'creation_date',
'last_time_updated_date', 'yaml_file', 'cksum', 'num_entries',

'tar_ball', 'enabled', 'classes', 'indexes'] and list: ['MO_ERROR:']

2.)
appscale-run-instances --table voldemort --file guestbook.tar.gz --ips
ips.yaml -v
…
Exception seen: #<Class:?????>

[Tue Dec 15 07:19:58 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: VO_ERROR:Not found
[Tue Dec 15 07:20:08 -08INFO - Thrift: GET called store:appscale,
key:__table_APPS___guestbook
INFO - Thrift: GET called store:appscale, key:__table_APPS___guestbook
INFO - Thrift: GET called store:appscale, key:__table_APPS___guestbook

3.)
appscale-run-instances --table memcachedb --file guestbook.tar.gz --
ips ips.yaml -v
…

Please wait for AppScale to prepare your machines for use.

[Tue Dec 15 10:09:33 -0800 2009] DB not currently running at

192.168.10.10. Going into recovery mode and waiting for DB to come
online.

[Tue Dec 15 10:09:38 -0800 2009] Get app data for guestbook said
[Error: MC_ERROR:Not found]
[Tue Dec 15 10:09:39 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: MC_ERROR:Not found
[Tue Dec 15 10:09:44 -0800 2009] Waiting for app data to have instance
info for app named guestbook: Error: MC_ERROR:Not found

4.)
appscale-run-instances --table cassandra --file guestbook.tar.gz --ips
ips.yaml -v
…

Please wait for AppScale to prepare your machines for use.

[Tue Dec 15 11:20:22 -0800 2009] DB not currently running at

192.168.10.10. Going into recovery mode and waiting for DB to come
online.

[Tue Dec 15 11:20:28 -0800 2009] Get app data for guestbook said
[Error: CA_ERROR:Not found]
[Tue Dec 15 11:20:28 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Tue Dec 15 11:20:33 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

[Tue Dec 15 11:20:38 -0800 2009] Waiting for app data to have instance

info for app named guestbook: Error: CA_ERROR:Not found

> ...
>
> Erfahren Sie mehr »

[Chris Bunch]

Hi Florian,
The big problem that I'm seeing is that regardless of which database
you choose, the tools get stuck here:

> Please wait for AppScale to prepare your machines for use.
> Exception seen: #<Class:0x7f14a52cdc68>

Whereas regular usage of the tools is much different (here's one example):

> Please wait for AppScale to prepare your machines for use.

> AppController just started
> Copying over needed files and starting the AppController on the other VMs
> Done starting up AppScale, now in heartbeat mode

An earlier e-mail you sent out showed a lot of SSL exceptions being
thrown, so I have two ideas for you:

1) Can you download the tools onto your domU (your only VM) and run them
from there? That may get around the SSL exceptions that are being
thrown.

2) Can you run it over any database and once the tools die from that
exception, can you paste the contents
of /root/appscale/.appscale/exception.log ? That file contains the last
exception thrown before the AppController dies, so knowing what the
problem is would definitely help.

Thanks,

[Florian]

Hi Chris,

thanks it is running :-)
The following steps were made:
1.) Created a new user on the Appscale-VM (possible optional)
2.) Logged in, as this user.
3.) downloaded the "appscale-tools"-Archiv and extracted it
4.) appended it into the PATH
5.) Created the ips.yaml
6.) Executed "appscale-add-keypair"
7.) Executed "appscale-run-instances"
8.) Saw no erros and an application in the Web-UI

One thing I should mention, it only worked for me with the external ip
of the VM in the ips.yaml file.

FYI: With 127.0.0.1 the following output was created:
cat /root/appscale/.appscale/exception.log
no implicit conversion from nil to integer

Thanks once again,
Florian

> ...
>
> Erfahren Sie mehr »