Backup / Restore Feature

[Anthony Shapley]

Hi,

We recently ran a full app back up, with code, then did a bunch of package updates, then updated appscale, cleaned the whole environment and upon running the restore feature, we're getting a whole bunch of errors, really could do with some help:

The full output from SSH is:

2016-04-06 21:27:47,135 INFO connection.py:575 Connecting to ***:2181

2016-04-06 21:27:47,137 INFO client.py:444 Zookeeper connection established, state: CONNECTED

2016-04-06 21:27:47,138 INFO zktransaction.py:173 Starting GC thread

2016-04-06 21:27:47,142 INFO datastore_server.py:326 Starting DatastoreDistributed

2016-04-06 21:27:47,144 INFO restore_data.py:80 Got the restore lock.

2016-04-06 21:27:47,144 INFO restore_data.py:176 Restore started

2016-04-06 21:27:47,145 INFO restore_data.py:183 Restoring "gsc-replica" data from: /opt/appscale/backups/gsc-replica-20160405-131317/gsc-replica-20160405-131317-34.backup

2016-04-06 21:27:47,170 INFO restore_data.py:162 Storing a batch of 100 entities...

2016-04-06 21:27:55,783 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:55,872 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:55,981 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:56,134 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:56,378 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:56,783 INFO pool_logger.py:71 Connection 140510543215312 (***:9160) in pool 140510550178448 failed: [Errno 104] Connection reset by peer

2016-04-06 21:27:56,797 ERROR cassandra_interface.py:147 Retried 6 times. Last failure was error: [Errno 104] Connection reset by peer

Traceback (most recent call last):

  File "/root/appscale/AppDB/cassandra/cassandra_interface.py", line 145, in batch_put_entity

    cf.batch_insert(multi_map, write_consistency_level=CONSISTENCY_QUORUM)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/columnfamily.py", line 1046, in batch_insert

    allow_retries=self._allow_retries)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 577, in execute

    return getattr(conn, f)(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f

    return new_f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f

    return new_f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f

    return new_f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f

    return new_f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f

    return new_f(self, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 148, in new_f

    (self._retry_count, exc.__class__.__name__, exc))

MaximumRetryException: Retried 6 times. Last failure was error: [Errno 104] Connection reset by peer

2016-04-06 21:27:57,047 INFO zktransaction.py:197 Closing ZK connection

2016-04-06 21:27:57,047 INFO zktransaction.py:185 Stopping GC thread

2016-04-06 21:27:57,075 INFO zktransaction.py:191 GC is done

2016-04-06 21:27:57,076 INFO connection.py:542 Closing connection to ***:2181

2016-04-06 21:27:57,076 INFO client.py:448 Zookeeper session lost, state: CLOSED

Traceback (most recent call last):

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 295, in <module>

    main()

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 290, in main

    ds_restore.run()

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 81, in run

    self.run_restore()

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 184, in run_restore

    self.read_from_file_and_restore(backup_file)

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 163, in read_from_file_and_restore

    self.store_entity_batch(entities_to_store)

  File "/root/appscale/AppDB/backup/scripts/restore_data.py", line 133, in store_entity_batch

    self.ds_distributed.dynamic_put(self.app_id, put_request, put_response)

  File "/root/appscale/AppDB/datastore_server.py", line 1466, in dynamic_put

    raise dbce

dbconstants.AppScaleDBConnectionError: 'Exception on batch_insert: Retried 6 times. Last failure was error: [Errno 104] Connection reset by peer'

Things I have tried:

• Removing all firewall / iptables rules on the server running Datastore/Cassandra
• Relocating the whole backup onto the server running Datastore/Cassandra

I don't know enough about the system try anything else, so I was hoping if someone could point us in the right direction or confirm whether or not this is a bug? 

The command we're running is:

 python $APPSCALE_HOME/AppDB/backup/scripts/restore_data.py --app-id gsc-replica --backup-dir /opt/appscale/backups/gsc-replica-20160405-131317/ -c

 

Any advice greatly appreciated. We're running the latest version of AppScale on Ubuntu 14.04.4.

Thanks

Ant

[Meni Vaitsi]

Hi Anthony,

From the output you sent it seems that Cassandra might be down or inaccessible.

I have a few questions about your environment:

• Is this a multinode deployment? If so, how many database nodes does the deployment have?
  
• What steps did you follow to update AppScale?
• Was AppScale running when you did the restore?

Thanks

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems, Inc.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Anthony Shapley]

Hi Meni

• Is this a multinode deployment? If so, how many database nodes does the deployment have?
  

• 3 Servers, only 1 database server.
  
  

• What steps did you follow to update AppScale?

• appscale down
• Full Ubuntu Release update
• Grabbed the latest release from Github
• Ran the update
• Followed the instructions on Github line for line pretty much.
  
  

• Was AppScale running when you did the restore?

• Yes, AppScale is running when doing the restore. It is running now, and an empty version of the app is talking to datastore, on the database server.
• AppScale also asked we deployed the code (hence how I know the above).

Thanks

Ant

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Anthony Shapley]

Hi Meni

One thing I did also notice and forgot to mention is its starting with backup file 35. When I attempted to run the restore from the database server/node - it started on file 25. Should it not start with file "-1" initially?

Ant

On Wednesday, April 6, 2016 at 10:04:25 PM UTC+1, Meni Vaitsi wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Meni Vaitsi]

Hi Anthony,

 

One thing I did also notice and forgot to mention is its starting with backup file 35. When I attempted to run the restore from the database server/node - it started on file 25. Should it not start with file "-1" initially?

The order in which the backup files are processed is arbitrary. As long as all the partial backup files are in the original backup directory this should not worry you.

Did you run the appscale/debian/appscale_build.sh script after the upgrade?

Are you able to store new data from the empty application right now? If so, then the datastore is not the problem most likely.

Answering these questions will help determine next steps. :)

-Meni

 

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

[Anthony Shapley]

Hi Meni

Yes - we ran the build scripts. This was done across all 3 servers.

Yes - we're able to store data from the empty version of the app. I.e. if we re register its storing our new users credentials etc.

Interested to hear whats next =p its starting to sound serious. Also good to know regarding the ordering.

Thanks

Ant

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Meni Vaitsi]

Ok, good to hear. Seems like you're doing all the right things!

Here's a couple things you can try:

• Restart AppScale with: appscale down; appscale up
• Run the restore script like so:

python $APPSCALE_HOME/AppDB/backup/scripts/restore_data.py --app-id gsc-replica --backup-dir /opt/appscale/backups/gsc-replica-20160405-131317/ -c --debug

This should print more logs during restore and give us more information.

• If you still see the Connection reset by peer  errors, I suggest you decrease the batch size of entities that are restored at a time and run the restore script again. You can do that by editing the appscale/AppDB/backup/scripts/restore_data.py file and change this line from 100 to something like 10.

Let me know if any of the above helps.

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems, Inc.

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

[Anthony Shapley]

Hi Meni,

I have followed all of your instructions, including running this with --debug and with the reduction from 100 to 10 and the data has now imported, however it isn't imported properly from what I can tell. The application is kind of half working. 

Do we need to do anything with Cassandra or otherwise to rebuild indexes, or anything along those lines?

Really appreciate your help so far.

Thanks

Anthony

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Meni Vaitsi]

Hi Anthony,

Glad to hear the import worked.

The restore operation should be creating the indexes while storing the entities in the datastore.

Can you elaborate a bit on the weird behavior you're observing? Are there specific API calls failing or not returning entities?

It would also be helpful if you can send me the AppScale logs to see if there are any obvious errors. Instructions on how to collect AppScale logs can be found here.

Thanks

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems, Inc.

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

[Anthony Shapley]

Hi Meni,

I think I sent this privately yesterday - an you let me know if you didn't get the logs.

Ant

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Meni Vaitsi]

Hi Anthony,

Thanks for sending the logs over. I'll take a look.

In the meantime, can you give an example of API calls that are not working properly in your app?

Also, are you on AppScale 2.7.1 or current repo master?

Thanks

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems, Inc.

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

[Anthony Shapley]

Hi Meni

I have a programmer looking into what API calls we're using exactly. As some do seem to be working, some don't.

We are using the current repo master I think, I followed the upgrade instructions which pulls the repo master? If there is a way I can more deffinitively answer this let me know and I'll gladly do it. We also seem to now be having issues doing app deployments. Have done appscale up/down with no success.

Thanks

Ant

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Meni Vaitsi]

Hi Anthony,

The latest code is stable so that should not be a problem.

What issues are you experiencing with deploying an app?

Can you send over the command you're running and the output?

Thanks

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems, Inc.

Hi Anthony,

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.