Appscale autoscaling testing

[kash...@plumgrid.com]

Hi guys,

This is my first post here. I am playing with appscale these days and actually testing autoscaling. I have installed appscale from source on a VM and it went great. After that I created a GAE (hello world) app with a 5 seconds time delay before returning the "hello world" response. My view looks like:

from django import http
import time

def home(request):
    time.sleep(5)
    return http.HttpResponse('Hello World!')

I deployed this app successfully by following the appscale documentation. I can hit the URL and get the "hello world" response! great. Now i wanted to send a flood of requests to see appserver autoscales or not. I sent 50 requests in parallel using Python multiprocessing.Process module and i can see appserver's count increased from 1 to 3 in appscale status output. I increased the number of requests to 100 and now I am receiving the following error after certain number of processes:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "autoscale.py", line 8, in hello
    response = urllib2.urlopen('http://192.168.33.10:8080')
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 502: Bad Gateway

It looks like appscale is using dev_appserver to serve the requests rather than any web server? Do i missed any configuration regarding this? Please suggest what will be the fix. Please find the output of the command "appscale status" given below if its helpful:

vagrant@appscale-image0:~$ appscale status
Status of node at 192.168.33.10:
    Currently using 7.0 Percent CPU and 73.90 Percent Memory
    Hard disk is 6 Percent full
    Is currently: load_balancer, taskqueue_master, zookeeper, db_master, taskqueue, memcache, shadow, login, appengine
    Database is at 192.168.33.10
    Is in cloud: cloud1
    Current State: Preparing to run AppEngine apps if needed
    Hosting the following apps: hello-django-app-engine
    The number of AppServers for app hello-django-app-engine is: 1

View status information about your AppScale deployment at http://192.168.33.10:1080/status
vagrant@appscale-image0:~$

Thanks

[Navraj Chohan]

Hi,

If you set "verbose: True" in your AppScalefile you'll see autoscaling information in the controller log (/var/log/appscale/controller-17443.log). If the machine does not have enough resources, it will not scale up the number of application servers. Optionally, you can set a set number of instances in your AppScalefile by setting the "appengine: 3" option. That will provide a set number of instances, which is fine in most cases where the resources and workload are static.

AppScale also scales up doing VM scaling, adding more nodes which run the application servers. This requires running it in a cloud deployment on one of the supported infrastructures (EC2, Euca, OpenStack, GCE). 

Thanks,

Raj

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

[kash...@plumgrid.com]

Hi Raj,

Thanks for your response. I set the verbose: True to see more details in the log. But i could not find any proper information specifically regarding this HTTP Error. I have observed two kind of errors:
502, Bad request
503, Service not available

I observed these errors whenever i run 100 process in a loop to put request on the deployed appengine app. Do you think these errors are related to the resources (RAM, CPU) etc?


Please see the following logs that i captured where ever errors were thrown:

---------------------

E, [2014-10-02T06:16:02.253660 #7451] ERROR -- DjinnServer: SSL_read:: decryption failed or bad record mac (OpenSSL::SSL::SSLError)
/usr/lib/ruby/1.8/openssl/buffering.rb:36:in `sysread'
/usr/lib/ruby/1.8/openssl/buffering.rb:36:in `fill_rbuff'
/usr/lib/ruby/1.8/openssl/buffering.rb:159:in `eof?'
/usr/lib/ruby/1.8/webrick/httpserver.rb:55:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/1.8/soap/rpc/httpserver.rb:123:in `run'
/usr/lib/ruby/1.8/logger.rb:658:in `start'
/root/appscale/AppController/djinnServer.rb:147
/root/appscale/AppController/djinnServer.rb:147:in `initialize'
/root/appscale/AppController/djinnServer.rb:147:in `new'
/root/appscale/AppController/djinnServer.rb:147
E, [2014-10-02T06:16:02.491001 #7451] ERROR -- : An exception of type SOAP::FaultError was thrown: Method Failed.
I, [2014-10-02T06:16:25.496218 #7451]  INFO -- : Removing a new AppServer from 192.168.33.10 for hello-django-app-engine

---------------------


Thanks,
Kashif

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Navraj Chohan]

Hi,

You can ignore that SSL error, although the SOAP error is of concern.

Do you see the machine being maxed out in terms of CPU or memory?

Take a look at the nginx logs (/var/log/nginx/) or application logs (/var/log/appscale/) to see why you were getting 502s or 503s.

You can use the command "appscale logs <dir_name>" to gather them. Feel free to send them directly to me if you want me to take a look

(https://github.com/AppScale/appscale/wiki/Managing-Apps,-Machines,-and-Logs#getting-logs-from-your-appscale-deployment).

Thanks,

Raj

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.