migrating a Google Apps hosted instance to appspot.com?

[Adam Sah]

My company's codereview instance was created on Google Apps years ago, and it's (finally) time to migrate it to appspot.com - what's the recommended way to do this (ideally, preserviing the issue numbers, so email links can be forwarded).

The lab settings page https://admin.google.com/<MYDOMAIN>.com/AdminHome?appId=codereview&chromeless=1#OGX:LabServiceSettings  doesn't have anything useful.

thx!

adam

[Vincent Zanotti]

Hey Adam,

I added an export servlet some time in the past, and didn't get time to document it yet, but if you're familiar with the AppEngine export tool, you should be able to use it quite easily. Here are the command lines I used.

# Export the data from the old instance.

appcfg.py -e <your email address> download_data \

    --url http://<your Google Apps rietveld domain>/remote_api \

    --filename /tmp/codereview.data \

    --log_file /tmp/codereview.log \

    --batch_size 100

# Import the data into your new instance.

appcfg.py -e <your email address> upload_data \

    --url http://<your AppSpot rietveld domain>/_ah/remote_api \

    --filename /tmp/codereview.data --batch_size 100

A couple things that may help:

• appcfg.py comes from the AppEngine SDK, which I imagine you had to download anyway to set up a new instance of Rietveld;
  
• it works better if the target instance of Rietveld is empty; I actually haven't tried with one that has been used before, but it will likely fail;
• don't forget to secure your target instance before importing the data (eg. requiring login; by default Rietveld is open to all);
• if you end up serving on your own domain name (by attaching your new Rietveld instance to your Google Apps domain), you may need to do some work for https to work;
• finally, note the very subtle difference between the two URLs (/remote_api, vs. /_ah/remote_api); this is not a typo :)

Please ask if you have any questions, or if you have feedback. This would help me writing that piece of documentation :)

Hope that helps,

- Vincent

--
You received this message because you are subscribed to the Google Groups "codereview-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codereview-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[adam sah]

omg it worked !!!  that's *awesome*.

FYI I've installed rietveld-on-appengine for two startups the past week, and it's being used as the basis for a discussion-heavy text CMS as well.

ᐧ

[Vincent Zanotti]

Awesome, glad it worked for you, and that you remain a Rietveld fan :)

[Perry Ismangil]

With the impending shutdown of hosted version, I'm moving to our own instance.

Export worked fine, but Import resulted in 404. I've triple checked the URL, including the /_ah/ bit etc.

In fact I can see in the logs in dev console the request came in.

What's next to debug this?

To unsubscribe from this group and stop receiving emails from it, send an email to codereview-discuss+unsub...@googlegroups.com.

[Guido van Rossum]

IIRC the hosted version was using a custom data model?

To unsubscribe from this group and stop receiving emails from it, send an email to codereview-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "codereview-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to codereview-disc...@googlegroups.com.

[Perry Ismangil]

Well, the migration doc seems to suggest it's the same, so you can export import:

https://docs.google.com/document/d/1GP5uTwiLRAVeY-mXxfhqMPEud3ieB16A8xEbbBDfE6M/pub

--
Perry Ismangil
me.ismangil.com

[Vincent Zanotti]

Thanks Perry for the feedback, I have updated the doc. The data model is the same (or more precisely the data model exposed by the export mechanism is the same), however the target Rietveld instance needs the remote_api to be enabled.

Basically, it should be a matter of adding the following to your app.yaml, Perry:

builtins:

  - remote_api: on

Keep me posted!

Thanks,

- Vincent

[Perry Ismangil]

Great that solved that one.

Another point after that caught me out is to enable billing first, because the upload is causing over quota for the free plan in write operations.

Now I am stuck on a worker thread error, it then keeps backing off.

[INFO    ] [WorkerThread-1] Backing off due to errors: 2048.0 seconds

--

Perry Ismangil

[Vincent Zanotti]

Thanks for the update on billing, I added that to the caveat list. Were you able to finish the import despite the initial errors/backoff?

Thanks,

- Vincent

[Perry Ismangil]

No, the import is still incomplete. About 1000 entities left to import.

We decided to live with it. The past reviews are nice to have, but not essential.

--

Perry Ismangil

[Vincent Zanotti]

Ah, sorry to hear that, but glad you found a reasonable way forward!

[Ernesto Andrade]

Has anyone exported a large # of entities?

Our instance has over 1.5M entities...I've been running into issues after about 5 days of running the export.

I tried larger batch sizes to help with the situation, but it seems anything > 1000 causes issues. 
I've only had one successful run (a month ago) with batch size set to 500.

Thanks,
-Ernie

[Vincent Zanotti]

Hey Ernie,

Do you know what kind of troubles you are getting with large batch sizes? It's possible I may be able to help increase your batch size (which would also help with the total runtime). Also, what kind of issues are you running into after 5 days? I do see a handful of errors in the logs, and they unfortunately all seem to be authentication errors :/

Cheers,

- Vincent

[Ernesto Andrade]

Vincent has been awesome and helping me in an offline thread, thought I'd share here in case anyone else runs into a similar problem...

I have attached the log file from the recently failed export.   It doesn't have any indication on what the error is, but it appears to be a situation where one of the worker threads goes into an endless backoff loop while the rest of the threads fail to make progress.

Ok, this definitely smells like an authentication issue (after 5 days the auth token has likely expired; it also matches what I see in my logs). Looking at the top of the file, it seems the exporter is using fairly low throttling limits (in particular on the bandwidth front); feel free to increase that, to see if that help.

Regarding the larger batch sizes, I don't have a log file, but if I remember correctly, it fails to spawn multiple threads and eventually times out.

Ok. Anything beyond 1k is going to be problematic anyway, since the AppEngine datastore is doing throttling of its own\.

 

I currently have an export running with a batch size set to 40...

It has transferred 365K entities so far after about 6 hours.  If I'm lucky, it will finish without error in 24 hours.

Keep me posted! And if this fails, maybe try a larger batch size with a much increased bandwidth limit (actually you can also try to increase the other throttling settings, don't worry about hammering us too hard).

[Ernesto Andrade]

Hi Vincent,

It looks like my export has stalled:
[DEBUG 2015-07-24 02:02:58,813 bulkloader.py] Entities transferred total: 1551041
[DEBUG 2015-07-24 02:02:59,918 bulkloader.py] Entities transferred total: 1551081
[DEBUG 2015-07-24 02:03:02,956 bulkloader.py] Entities transferred total: 1551121
[DEBUG 2015-07-24 02:03:03,638 bulkloader.py] Entities transferred total: 1551121
[DEBUG 2015-07-24 02:03:06,870 bulkloader.py] Entities transferred total: 1551161
[DEBUG 2015-07-24 02:03:07,599 bulkloader.py] Entities transferred total: 1551161
[DEBUG 2015-07-24 02:03:10,854 bulkloader.py] Entities transferred total: 1551161
[DEBUG 2015-07-24 02:03:11,872 bulkloader.py] Entities transferred total: 1551201
[DEBUG 2015-07-24 02:03:14,824 bulkloader.py] Entities transferred total: 1551201
[DEBUG 2015-07-24 02:03:15,672 bulkloader.py] Entities transferred total: 1551241
[DEBUG 2015-07-24 02:03:18,968 bulkloader.py] Entities transferred total: 1551281
[DEBUG 2015-07-24 02:03:19,563 bulkloader.py] Entities transferred total: 1551281
[DEBUG 2015-07-24 02:03:22,885 bulkloader.py] Entities transferred total: 1551281
[DEBUG 2015-07-24 02:03:23,681 bulkloader.py] Entities transferred total: 1551321
[DEBUG 2015-07-24 02:03:26,880 bulkloader.py] Entities transferred total: 1551321
[DEBUG 2015-07-24 02:03:27,732 bulkloader.py] Entities transferred total: 1551361
[DEBUG 2015-07-24 02:03:30,865 bulkloader.py] Entities transferred total: 1551361
[DEBUG 2015-07-24 02:03:31,766 bulkloader.py] Entities transferred total: 1551401
[DEBUG 2015-07-24 02:03:34,806 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:35,418 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:37,563 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:39,670 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:41,711 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:43,731 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:46,071 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:48,079 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:50,152 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:52,324 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:54,647 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:56,958 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:03:58,927 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:04:00,968 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:04:02,902 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:04:04,856 bulkloader.py] Entities transferred total: 1551408
[DEBUG 2015-07-24 02:04:06,945 bulkloader.py] Entities transferred total: 1551408

Can you help identify what the issue is here?  It's been running since 22-Jul @ 6PM (~36 hours).
I'd hate to have to restart the process...  I looked through the doc for appcfg.py, and there doesn't appear to be support for resuming partial downloads.

Thanks,
-Ernie

[Lance Erickson]

As an aside, I was able to re-start a download I had to kill. The exporter creates a small sqlite progress db and a results db, named like 'bulkloader-progress-20150731.111111.sql3' and 'bulkloader-result-....sql3'. If there are existing files, they can be specified as options to the exporter and it will resume the download, via --result_db_filename and --db_filename ( for the progress db). I believe they were created in the working directory of wherever I ran the script initially. Maybe that is useful for someone.