Re: AppScale crash

[Navraj S. Chohan]

Hi Gwyn,
You want to check to make sure you didn't run out of disk space or memory. Also on your appscale node run 'monit summary' and make sure all processes are running.
See the troubleshooting guide in the wiki:
https://github.com/AppScale/appscale/wiki/Troubleshooting
As well as the FAQ:
https://github.com/AppScale/appscale/wiki/FAQs

If you still cannot resolve the issue please manually tar up the logs on your instance and sent them my way.
Thanks,
Raj

On Oct 21, 2014 4:53 AM, "Gwyn Howell" <gho...@ancoris.com> wrote:

Our AppScale instance has crashed.

When I do 'appscale status' I get:

Traceback (most recent call last):

  File "/usr/local/appscale-tools/bin/appscale", line 79, in <module>

    LocalState.generate_crash_log(e, traceback.format_exc())

  File "/usr/local/appscale-tools/bin/../lib/local_state.py", line 936, in generate_crash_log

    with open(crash_log_filename, 'w') as file_handle:

IOError: [Errno 30] Read-only file system: '/root/.appscale/log-33623a33-fd6d-473f-8aca-a8c12c18ee31'

When I do 'appscale logs logs' I get the same thing.

This is the third time appscale has crashed on us in a just a few weeks. Last time 'appscale destroy' sorted it out, this time not even that works, I get the following stacktrace:

Terminating instances in a virtualized cluster with keyname appscale5f3a1cb92b8f4615907bfa2f0fe0e081

shell> ssh -i /root/.appscale/appscale5f3a1cb92b8f4615907bfa2f0fe0e081.key -o LogLevel=quiet -o NumberOfPasswordPrompts=0 -o StrictHostkeyChecking=no -o UserKnownHostsFile=/dev/null [email] 

Exception in thread Thread-1:

Traceback (most recent call last):

  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

    self.run()

  File "/usr/lib/python2.7/threading.py", line 504, in run

    self.__target(*self.__args, **self.__kwargs)

  File "/usr/local/appscale-tools/bin/../lib/remote_helper.py", line 931, in stop_remote_appcontroller

    cls.ssh(host, keyname, 'service appscale-controller stop', is_verbose)

  File "/usr/local/appscale-tools/bin/../lib/remote_helper.py", line 347, in ssh

    cls.SSH_OPTIONS, user, host), is_verbose, num_retries,stdin=command)

  File "/usr/local/appscale-tools/bin/../lib/local_state.py", line 719, in shell

    the_temp_file = tempfile.NamedTemporaryFile()

  File "/usr/lib/python2.7/tempfile.py", line 442, in NamedTemporaryFile

    dir = gettempdir()

  File "/usr/lib/python2.7/tempfile.py", line 261, in gettempdir

    tempdir = _get_default_tempdir()

  File "/usr/lib/python2.7/tempfile.py", line 208, in _get_default_tempdir

    ("No usable temporary directory found in %s" % dirlist))

IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/root']

Even so, 'appscale destroy' is not a viable solution in a live environment.

We don't have anything particularly special going on here. Just a vanilla appscale env running on a ubuntu vm, with 1 app on it.

I need to get this stable, or at the very least, recoverable when it dies. 

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Gwyn Howell]

Hi Raj,

Thanks for the reply.

I apologise - it was a problem on our vm stack - nothing to do with App Scale.

Case closed!

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.