Automated way to switch from M/S to HR?

[Will Reiher]

Since I have been having a number of issues with Deadlines on the M/S I would like to move to HR but creating a new app and migrating data is really not an option. Is the app engine team working on a click of a button type upgrade that doesn't require duplication and/or migrating to a new application-id etc?

[Ikai Lan (Google)]

We're working on a migration tool that'll be better than the one that exists now. You will be able to preserve your current application ID. Unfortunately, there will still be a copy step, so this will likely result in a read-only period for your app.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

On Wed, Aug 3, 2011 at 12:53 PM, Will Reiher <wrel...@gmail.com> wrote:

Since I have been having a number of issues with Deadlines on the M/S I would like to move to HR but creating a new app and migrating data is really not an option. Is the app engine team working on a click of a button type upgrade that doesn't require duplication and/or migrating to a new application-id etc?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ymI6tGec5fYJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Will Reiher]

Ikai,

That's great news! I'm not so worried about a temporary read-only period it's the the idea of switching the application id that was scary. I've already done it once, and while it was fairly painless, it did affect our https requests point to the app-id appspot url.

Thanks for the update - I'm sure many other people will be making the switch.

[David]

I've just about given up on the thought that AppEngine is a good
service. It's good in theory, but I get about 25% errors in bursts
and it's generally timeouts on the datastore. Those error bursts are
frequent. Lately is been almost daily. But pretty much every upgrade
they do has a couple weeks before it starts to feel stable again. The
bad part is we have no control over the servers so we just have to eat
the errors as they come since they are coming from the datastore. I
need a reliable/stable environment where I feel I can do something
about problems when they come up. So I'm probably making the switch,
just off of AppEngine.

[johnP]

Amen - temporary read-only is fine. The big problem is that my app
makes use of a ton of keys, and in many cases I need to store the keys
as strings (or store keys in pickled blobs). The real chore is to
write scripts to rebuild all the keys after the migration.

I'm hoping that preserving the app ID will also preserve these
keys...

please???

johnP

On Aug 3, 1:02 pm, Will Reiher <wrele...@gmail.com> wrote:

[Robert Kluin]

I've not heard of the ability to allow you preserve the app id as far
as the datastore is concerned, only the ability to create an alias so
it acts like the same app. I wonder if an alias is what Ikai was
referring too or if he meant some way to really preserve the app id.

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Carter Maslan]

I felt your pain on MS; but after moving to HRD Tuesday, our app has not experienced a single timeout error in 24 hours. So don't give up.

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Tim Hoffman]

Preserving appid is what I am waiting on.  Readonly is fine.

I hope Ikai is correct saying we can preserve them.

T

[Ikai Lan (Google)]

I was referring to an alias. The tool should understand keys and String keys (I think ... ), but it's going to break if you do something like SHA1 hash the keys and store those values in a List property.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Ikai Lan (Google)]

Double checked with Greg Darke, turns out it will NOT work on String Keys. You need to persist re-read the Entities and persist them as keys.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Tim Hoffman]

Hi Ikai

So from what you have said will it definately handle conversion of ListProperty(db.Key)  ?

Thanks

Tim  

[Ikai Lan (Google)]

Yes. That should convert correctly. When it's serialized to a protocol buffer the type is preserved, and the migration tools will be aware that the Key needs to be decoded and rewritten.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/nVMNJJy440QJ.

[johnP]

can you clarify: "You need to persist re-read the Entities and persist
them as keys. "

On Aug 4, 4:45 pm, "Ikai Lan (Google)" <ika...@google.com> wrote:
> Double checked with Greg Darke, turns out it will NOT work on String Keys.
> You need to persist re-read the Entities and persist them as keys.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
> On Thu, Aug 4, 2011 at 9:53 AM, Ikai Lan (Google) <ika...@google.com> wrote:
>
>
>
>
>
>
>
> > I was referring to an alias. The tool should understand keys and String
> > keys (I think ... ), but it's going to break if you do something like SHA1
> > hash the keys and store those values in a List property.
>
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App Engine
> > plus.ikailan.com | twitter.com/ikai
>

> > On Wed, Aug 3, 2011 at 8:13 PM, Robert Kluin <robert.kl...@gmail.com>wrote:
>
> >> I've not heard of the ability to allow you preserve the app id as far
> >> as the datastore is concerned, only the ability to create an alias so
> >> it acts like the same app.  I wonder if an alias is what Ikai was
> >> referring too or if he meant some way to really preserve the app id.
>
> >> Robert
>

[Greg Darke (Google)]

On 5 August 2011 10:13, johnP <jo...@thinkwave.com> wrote:
> can you clarify: "You need to persist re-read the Entities and persist
> them as keys. "

If your entities are stored as a String, or a list of strings, you
will need to read the entities, and change their type to be either a
Key (or reference property) or a list of Keys and then perform a Put
to store them back in your datastore.

If datastore keys are stored as strings, the migration tool will not
be able to update them during the migration.

[johnP]

I'm storing the keys as a string (not the actual entities themselves).

Am I understanding you correctly with the following?

old_keys = [str(key1), str(key2),str(key3)]
new_keys = [str(db.Key(i)) for i in old_keys]





On Aug 4, 5:32 pm, "Greg Darke (Google)" <darke+goo...@google.com>
wrote:

[Tim Hoffman]

Hi John

I have a db.ListProperty(db.String) or db.StringListProperty and I have stored keys in it, which of course will be in the str(key) form.

The conversion tools doesn't convert this sort of thing mainly because unless there is an additional mechanism for 

describing what this field holds the conversion tools can't know.

So you would need to something like

Convert all the strings back to keys

keys = [db.Key(i) for i in obj.some_list_property]

then stick them in some property that is defined as db.ListProperty(db.Key)

Now the trick will be working out the order of events and possibly performing multiple stages 

so you can get you data converted and make the sites work before and after transfer/conversion

Haven't really worked out the order of events I have to go through.

Rgds

Tim 

[Tim Hoffman]

Hi John

You examples are the same 

old_keys = [str(key1), str(key2),str(key3)] 
new_keys = [str(db.Key(i)) for i in old_keys] 

new_keys would be 

new_keys = [db.Key(i) for i in old_keys] 

assuming new_keys has been declared as db.ListProperty(db.Key)

Rgds

Tim

[Greg Darke]

Yes, this is the correct way to perform this operation. I would
suggest adding a new property to the model to store the keys and
update your code to use this new property. This will allow you to
continue using your application while you are performing your schema
migration.

For example:

# Old model definition
class MyModel(db.Model):
links = db.StringListProperty()

# New model definition
class MyModel(db.Model):
links = db.StringListProperty()
link_keys = db.ListProperty(db.Key) # You may also want to set
indexed = False

def apply_schema_change(self):
if not links:
# There is nothing to update
return
self.link_keys = [db.Key(str_key) for str_key in self.links]
self.links = [] # We don't use this anymore, so delete the data

Then you would call the apply_schema_change method after reading the
model in from datastore, ensuring that the data is now in the new
format.

Then I would use the mapper framework to apply the new schema to all
of the entities in your datastore.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/8_fgPFgpABAJ.

> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

--
Greg Darke

[Robert Kluin]

So here are my thoughts regarding this: if I have to map over all of
my data re-writing it to prepare it for the tool to convert, why not
just migrate to the other app while I'm doing that? Otherwise I'm
going to pay to read and write it to fix it, then pay again to read
it, transfer it, and rewrite it. This adds up fast if you've got much
data.

Aside from the cost factor, the migration to HR is an opportunity to
"fix" or improve your schemas. Things like long field names, unused
indexes, entity groups, etc.... All of those items that require
rewriting all of your data to adjust -- the migration is a great time
to do it. On a migration I've recently been working on, I estimate I
can reduce our storage use by about 50% with very minor changes (field
names and indexes!).

I've already migrated a couple apps, each time I've setup a job to map
over my data, maybe do some conversions, put it in some suitable
format (like csv or json) then send batches to the new app. On the
new app side I write a simple 'receiver' handler that does any needed
conversions and then writes the entities (in batches or one-by-one
transactionally, depending on the case).

Writing the sender and receiver handlers has usually been pretty quick
for me, even with very complex data. The other advantage is that you
may not need to move all of your data. Some items like stats that are
computed from your original data, may be able to be recomputed by the
exact same code as it comes in.

Would love to hear other people's thoughts on this.

Robert

[johnP]

My question was unclearly formulated. I intended to ask, would it be
possible to do what I described *after* the migration? In other
words, would calling db.Key() on the str(old_key) 'convert' it to the
new key? My plan is to do a simple brute-force copy, then convert
keys on-demand in a lazy fashion after migration.

I previously saw a 'rebuild key' script:
http://blog.shnap.com/migrating-to-the-google-app-engine-high-repl I
assume this can be called 'after migration'. But my question is based
on (wishful?) thinking that perhaps, simply calling db.Key on a legacy
MS string key would be recognized and converted to the appropriate
format for the aliased MS datastore.

I also understand that I can do a schema change before the migration.
And do appreciate Robert's suggestion to make a virtue out of
necessity during the conversion. But my philosophy is to keep things
simple and not try to do too much at one time:)

But

[johnP]

Sorry - I got the link wrong earlier. It is:

http://blog.shnap.com/migrating-to-the-google-app-engine-high-repli


johnP

[Joops]

Hi all,

Just to through my migration hat in the ring.

I migrated my (non-production app) from MS to HR and it worked fine.
(or at least it seems to have worked fine)
I had used the Key ID's instead of Keys to handle references between
entities.

Am i correct in saying that the only benefit of using key's over id's
is that you get the nice syntactic sugar way of fetching things?


Thanks very much,

J.

> > Yes, this is the correctwayto perform this operation. I would

> > suggest adding a new property to the model to store the keys and
> > update your code to use this new property. This will allow you to
> > continue using your application while you are performing your schema
> > migration.
>
> > For example:
>
> > # Old model definition
> > class MyModel(db.Model):
> >  links = db.StringListProperty()
>
> > # New model definition
> > class MyModel(db.Model):
> >  links = db.StringListProperty()
> >  link_keys = db.ListProperty(db.Key)  # You may also want to set
> > indexed = False
>
> >  def apply_schema_change(self):
> >    if not links:
> >      # There is nothing to update
> >      return
> >    self.link_keys = [db.Key(str_key) for str_key in self.links]
> >    self.links = []  # We don't use this anymore, so delete the data
>
> > Then you would call the apply_schema_change method after reading the
> > model in from datastore, ensuring that the data is now in the new
> > format.
>
> > Then I would use the mapper framework to apply the new schema to all
> > of the entities in your datastore.
>

> > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Robert Kluin]

On Fri, Aug 5, 2011 at 07:17, Joops <john...@gmail.com> wrote:
> Hi all,
>
> Just to through my migration hat in the ring.
>
> I migrated my (non-production app) from MS to HR and it worked fine.
> (or at least it seems to have worked fine)
> I had used the Key ID's instead of Keys to handle references between
> entities.
>
> Am i correct in saying that the only benefit of using key's over id's
> is that you get the nice syntactic sugar way of fetching things?

Keys can also encode the namespace and any partent-child relationship
data; if those two things are not a concern then there is effectively
no difference.

Storing the ids directly is also not-bad because you will not
accidentally dereference it leading to lots of inadvertent datastore
calls. Your code will be nice and explicit.

Robert

[Ikai Lan (Google)]

If you guys haven't seen Johan's community update, here's a link to sign up for early access to the migration tool:

https://docs.google.com/a/google.com/spreadsheet/viewform?authkey=CLXR0LMN&formkey=dERMcDZuMnlycHoyZDd4Vy1PNXlhWlE6MQ#gid=0

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Will Reiher]

I ended up just moving it all manually to a new HR app. Is it still possible to create an alias from the old app to the new app - rather than shutting it down? Right now I've turned off datastore writes and I have it redirecting all secure traffic to the new appspot url.

[Rajkumar Radhakrishnan]

Hi,

One of my app is a multi-tenant, metadata-driven online database app. I have used the key().id() values extensively within my app for entity-names, attribute-names and as values (including for custom namespace-id fields).

This app instance has been facing recurring DeadlineExceededErrors in instance loading and the suggestion has been to move to HR, from my current MS datastore.

My only apprehension in going ahead with the migration is related to the key-id values I have used. Will the HR migration tool retain the key-ids when the data is migrated ?

If this is not the case, I can keep the old app running and ask customers to move to a new HRD deployment manually (export & import app and then their data).

Thanks & Regards,

Raj

[Joshua Smith]

ids do not change when you migrate. The full key changes, because it includes the app name. But the id() part is preserved. You should be fine.

[Rajkumar Radhakrishnan]

Thanks for the info, Joshua. I was almost convinced that moving to HR would solve the issues, until I see this mail making me wonder if this is not just about MS/HR..

http://groups.google.com/group/google-appengine/browse_thread/thread/89058dc033bc8d83?pli=1 

Regards, 

Raj