Unexpected error causing appscale not to start

[Prashant Hegde]

Hi Team

We had a misbehaving task running in a loop which we had to bring it down. While the task was running we stopped appscale. After this, when we restarted appscale, the app is not coming up.

When we examined log, one thing caught our eye. In the transaction groomer log, we are seeing this:

========

2020-03-09 01:17:46,579 ERROR transaction_groomer.py:196 Unexpected error while grooming localapp
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/appscale/datastore/scripts/transaction_groomer.py", line 192, in start
    yield self._groom_project()
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/appscale/datastore/scripts/transaction_groomer.py", line 256, in _groom_project
    oldest_valid_tx_time = yield self._fetch_and_clean(index, worker_count)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/appscale/datastore/scripts/transaction_groomer.py", line 381, in _fetch_and_clean
    new_children = yield self._tornado_zk.get_children(container_path)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result

======

Can you let us know what could be going wrong? Any help is greatly appreciated.

How do we recover from this error?

[Prashant Hegde]

OK, we figured out the issue. After we cleaned the version-2 directory of zookeeper, and restarted, it worked.

[Graziano Obertelli]

Great, thanks for following up.

> packages/appscale/datastore/scripts/transaction_groomer.py", line
> 381, in _fetch_and_clean
> new_children = yield
> self._tornado_zk.get_children(container_path)
> File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line
> 870, in run
> value = future.result()
> File "/usr/local/lib/python2.7/dist-
> packages/tornado/concurrent.py", line 215, in result
> ======
>
> Can you let us know what could be going wrong? Any help is greatly
> appreciated.
> How do we recover from this error?
>
>
>
>
>

--
You received this message because you are subscribed to the Google
Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to appscale_commun...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/appscale_community/CAHXAtA9XvFv9o%2BscH_Y2BVc6J-awoSuiahDzfh2apKooE4e3UQ%40mail.gmail.com
.
--
Graziano Obertelli | VP Operations
805.570.1647 graz...@appscale.com
www.appscale.com

[Prashant Hegde]

Dear Graziano,

This is version 3.8.1 appscale. What is the documented way to backup the datastore? Earlier versions had an appscale-backup script. This does not seem to be there in this version.

Can you please advise?

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/5efd64784b9d6757d3aca5a555c061398243566a.camel%40appscale.com.

[Prashant Hegde]

Team, we are in a difficult situation where we need to backup the data and restore on another system. Can you please give some pointers on how to do it in version 3.8.0 ? 

Please help.

[Chris Donati]

Though the bulkloader is known for being on the slow side, it should get the job done: https://github.com/AppScale/gts/wiki/Migration-into-AppScale,-Backup-&-Recovery#using-the-bulkloader

If it's too slow for your purposes, there are other options like the datastore_admin scheduler, which can kick off a mapreduce job to split the work up between application instances and push the resulting files to an object store. However, this is _much_ tricker to set up, and I would not recommend trying it unless the bulkloader won't work for you.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/CAHXAtA8ZJZT_fDuVseGF0uZR%2Bi9RjMcocs1yKov3DKqEaAyV2w%40mail.gmail.com.

[Prashant Hegde]

Thanks so much Chris, for your response.

We will try the bulkloader first, and see if it works. 

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/CAAHK8%2BeFU972gG%2BfcrU%3DyBrbboU5Qi1-xg4h6hv-zU4ww7p0jg%40mail.gmail.com.