data change for large number of namespaces

[Rajesh Gupta]

Anyone using namespaces for real projects?

Cloud Dataflow was my latest attempt to do the data changes for entities in large number of namespaces.  The dataflow pipelines also does not help here

Anyone has any framework developed to do migration/process data for large number of namespaces.

--

Rajesh

www.VeersoftSolutions.com

www.AccountingGuru.net

Accounting/Inventory/Orders on Google Cloud Platform and Mobile

[Nick (Cloud Platform Support)]

Hey Rajesh,

It seems that your question is a bit vague here. What kind of migration or data transformation are you looking to perform, how are you using namespaces, how would you like to use Dataflow, etc.? 

[Rajesh Gupta]

Hello Nick,

Our data is organized by a tenant/company.  Each tenant/company's data is in a separate namespace.

So, we have many namespaces.

The web app is continuously evolving, and hence we have to do many times schema migration of the datastore entities.

Following Objectify link, tells more details on the kind of migrations we do.

https://github.com/objectify/objectify/wiki/SchemaMigration

Dataflow pipelines are created by specifying one namespace for a  pipeline. The namespace issue is here

https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/83

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/078f3c3d-b119-4f92-aa8b-15733afc8093%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Schnitzer]

First: I think the namespace feature is a horrible idea and should never be used (and I say this as someone that runs a whitelabeled, multitenant system).

That said, doing migrations with namespaces shouldn't be all that different from doing migrations without namespaces, you just have to add one more layer of mapping (enqueue one rewrite task per namespace). Unless you're doing something really exotic, data migration is really just load/save entities and figuring out how to 'map' a task onto the task queue for every entity in the system.

Rather than the dataflow system (which seems far more complex than necessary for this simple process), here's some code that may help:

https://gist.github.com/stickfigure/1a345f9a8a56897da6a0

It just uses the scatter property to enqueue tasks per shard that themselves enqueue tasks per entity. Just like the map/reduce framework. The only twist is that you need to start with one task per namespace and run the scatter query on each namespace separately.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijY0jg7SAHkVea%3DDazTe8_4cSGBH8Z68QCr_LSSfp0dTbg%40mail.gmail.com.

[husayt]

This is one of the rare occasions, where I would disagree with Jeff here. For us namespaces are one of the best things in appengine. Obviously, this is just an abstraction and can be achieved manually. But having this implemented is a big timesaver and also is an important safety feature. We have implemented a CRM for small and medium businesses and, it helps to be able to isolate business data by namespace. The same way as DataStore by its design pushes developers to write scalable applications,  namespaces help to enforce proper sandboxing of business data. 

Obviously, it might not be the right tool for many other problems, but for us it works very well. I even wish, there was a namespace support for BigQuery and other services.

Anyway, It's there, you can use it or leave. Having choice is great. But again, it would probably not be the right solution for most of the projects, so only use them if you understand the implications.

As for migration, Jeff, thanks for gist. That was very useful