Deploy drops data

[Phil McDonnell]

Twice now a deploy has cleared several of the data tables in my deployed datastore.  Does anyone know what causes this?  The symptoms both times are consistent.  I do a deploy and the datastore drops several of the tables.  Strangely, one very old (and generally unused) table is not dropped and my blob data is still there.  Is this dropping of data/tables  on deploy a known "feature" that I'm somehow not understanding?  Note, this does not happen on every deploy.

We are still pre-launch, but I'm extremely concerned that this could happen when we are live.  If we drop our data when we are live it would cost us a lot of money.

Thanks so much,

Phil

[Simon Knott]

Hi,

Are you talking about deploying into Production GAE, or into the development server?  Deploying into Production GAE will never clear down any data, as far as I'm aware.

Cheers,

Simon

[Barry Hunter]

That doesn't even make sense

The Datastore has no concept of 'table'.

All Entities - regardless of their kind- are stored in one 'bigtable'
- they are all intermingled. In fact all Entities from everyone's apps
are stored in the same bigtable.

So to delete a 'table' - which is really just all entities of a given
kind for a given app - would need to loop though and delete each
individual entity. Once there are no entities of a given kind/model
left, the 'kind' has effectively been dropped.

I would guess need to check you aren't somehow using the application
version in the Kind name or even the key?. Then when the application
is updated, the version changes and therefore the old entities become
inaccessible.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[Ronoaldo José de Lana Pereira]

Hi.

We have a live app and deploy new versions very often. This never happened. Some thoughts:

Since your are talking in terms of tables, are your using the fresh new Google Cloud SQL stuff? If so, you may be interested on checkout on the SQL Discuss group here.

Are you doing something to "cleanup" your data when your instance is warming up? Maybe there is something there... Or maybe you are using the Datastore Admin to delete something after the deploy? This is not automatic, through. If any data is deleted via admin console, it will apper in the Admin Logs section.

Are you using the namespaces feature? This may break things if you are using some namespace name that changes between deploys. Then, your data will not be visible unless you use the same namespace.

Hope this helps,

-Ronoaldo

[Ikai Lan (Google)]

Thanks for the explanation Barry. Phil, it sounds like you are talking about the local datastore. Can you clarify?

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Phil McDonnell]

Hi guys,

Apologies for using unclear terminology on this.  I understand the details of how the data is stored on BT. This is not local - this is on the production instance.

This idea of something happening that makes the data inaccessible sounds possible. I'm running on version 2 of my app.  Could that be an issue?  I'm not very familiar with the namespace feature, but I can read up on this.  Are there other ways that the data might disappear?

Ikai, in case it's useful, my app id is coupflipsite.

Thanks so much for all the thoughts here.  Getting to the bottom of this is extremely important for us and I very much appreciate your wisdom on this.

Thanks again,

Phil

[Phil McDonnell]

I investigated the namespace idea and I don't believe that's being used here at all. The one data class that is still accessible is a remnant from some earlier work and no longer used which may explain why it doesn't become inaccessible/disappear.

Any more ideas on how entire datastore classes might become inaccessible directly after a deploy?  It does not happen every time I deploy, but when it does it makes all of our data inaccessible, with the exception of the one data class that we don't use anymore.

Thanks for your help and input,
Phil

[Phil McDonnell]

Also, minor possibility, but I noticed "Datastore Admin" was enabled in my settings.  Perhaps there is a bug associated with that and deploys?

Thanks,

Phil

[Barry Hunter]

Well I was just going to suggest you use the datasotre admin
http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html

to view your datastore, to see if the data is still there.

On Mon, Oct 31, 2011 at 2:32 PM, Phil McDonnell

[Phil McDonnell]

Sadly, the source app is java so I can't run the datastore admin.  Enabling the feature it was a mistake.

[Barry Hunter]

You can
http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html

See the "A Note for Java Developers" section.

On Tue, Nov 1, 2011 at 4:45 PM, Phil McDonnell

[Phil McDonnell]

My understanding is that only python services can be backed up using the datastore admin.  The wording on the "A Note for Java Developers" is only for the "target" application.  I'm trying to back up a java app which means my java app is the "source" instead of the "target".

[Barry Hunter]

Oh, right sorry. I wasnt thinking about using it as backup. Just a
easy way to inspect the Kinds you have in the relevent application. To
see if the kinds you think have been "dropped" are actully still
there. A easy way to view the Datastore stats.

I though you would have to use the same trick to make a python version
of the main (ie source in that documents terms) app too, to view the
stats in the datastore.

On Fri, Nov 4, 2011 at 8:21 PM, Phil McDonnell