Whats best practice to separate environments for a Django App on Google App Engine?

[Silvan Spross]

I'm used to deploy to managed servers through some Fabric scripts (or CI's using them). I also run a project with Docker, where Django, Celery, Redis, RabbitMQ, Postgres and Ngnix are in separate containers.

However, now I want to give Google App Engine a try. I don't want to use Google Container Engine or Google App Engine Managed VM for now. I consider them as another option if using plain Google App Engine does not work out.

My goals are:

1. Separated Environments
   Development, Stage and Production (where stage and production can be just different app engine projects and so can be considered as "the same").
   
   
2. Using Task Queue (instead of Celery)
   Used for async tasks and scheduled ones (cronjob).
   
   
3. Using Memcache (instead of Redis)
   For caching keys up to caching templates etc.
   
   
4. Using Cloud Storage (Buckets) as Storage backend
   For media files.
   
   
5. Using CloudSQL (instead of Postgres)
   I don't care about nonrel for the moment. Using Datastore should be a future goal here.

My subject is all about how to setup and separate environments (1). But if someone has feedback or input about another goal, why e.g. not to use it, I'm very happy to hear! Btw. my test project is much alike https://github.com/GoogleCloudPlatform/appengine-django-skeleton for now.

What bugs me at the moment:

• Load different settings e.g. database based on 
  os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')
  
  Is this best practice?
  
  
• I run my application local with:
  source env/bin/activate

  dev_appserver.py app.yaml
  to have python libraries that use C extensions loaded locally (like mysql) and that
  google.appengine
  
  is available. But how can I run django management commands locally? Whats best practice here?
  
  
• And the biggest one: Whats best practice to migrate production database? I heard stuff about interacting with the database through a separate GCE instance running scripts. Or is there no best practice because with Datastore "you don't need" migrations ;)

I would be very happy about some tipps pointing me to the right direction. Thank you!

Silvan

[Silvan Spross]

Btw. at the moment I ended up with

ON_APP_ENGINE = os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')

if ON_APP_ENGINE:

   DATABASES = {

       'default': {

           'ENGINE': 'django.db.backends.mysql',

           'HOST': '/cloudsql/supple-century-124208:supple-century-124208',

           'NAME': 'appengine_django_skkeleton',

           'USER': 'root',

       }

   }

else:

   DATABASES = {

       'default': {

           'ENGINE': 'django.db.backends.mysql',

           'NAME': 'appengine_django_skkeleton',

           'USER': 'root',

           'PASSWORD': '1234',

           'HOST': '127.0.0.1',

           'PORT': '3306',

       },

       'prod': {

           'ENGINE': 'django.db.backends.mysql',

           'NAME': '<prod-database-name>',

           'USER': '<prod-database-user>',

           'PASSWORD': '<prod-database-password>',

           'HOST': '<prod-database-ip>',

           'PORT': '3306',

       }

   }

and than running management task through the remote api shell

remote_api_shell.py -s APP-ID.appspot.com

import os

from google.appengine.ext import vendor

vendor.add('lib')

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")

import django

django.setup()

from django.core import management

management.call_command('makemigrations', 'polls', interactive=True)

management.call_command('migrate', interactive=False)

management.call_command('migrate', database='prod', interactive=False)

management.call_command('createsuperuser', database='prod', interactive=True)

./manage.py was getting unusable after I started using things from `from google.appengine.api`. Is there an easier way to use management commands with the GAE stuff loaded then djangae is doing it? https://github.com/potatolondon/djangae/blob/master/djangae/sandbox.py

[Nick (Cloud Platform Support)]

Hey Silvan,

These are a whole lot of excellent questions and details. I'll try to offer my feedback on several points and address your questions. 

Without knowing what your system is, it's hard to tell beyond the abstract notion of "separating environments" what your goals should be in development. However, you seem to have a strong knowledge of many tools and it seems that you'll be able to select tools and services which meet the needs you have.

As far as the question of os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'), I can say that while this is not exactly a "universal best practice", there's nothing wrong with it, either. It's just a particularity of how to detect the production environment on GAE python. Pretty much every web framework in every language will have some idiomatic means of determining whether we're in production or development, and choosing appropriate database and API connection params / code paths, etc. accordingly.

As far as running django management commands locally, there are tons of toolchains out there for managing development commands. I just learned about Fabric based on your mention of it, and it seems like an excellent tool for python developers to do exactly what you need. As for more specifics of exactly what kinds of questions can arise in regard to "local management of Django", we enter a situation where we need more specifics to spur the development of more specific answers. For example, does your "local Django" (really the development environment version of your code which uses the Django framework) connect to a remote DB? Or a local copy of the DB?

Now, as to the question of best practices in migrating production databases, this depends largely on two factors: the database technology itself and the environment it's deployed to. A MySQL server running on a GCE instance is different from Cloud SQL, which is different from Datastore, which is different from a MongoDB server, etc. Finally, the system which interacts with the database is also critical for determining the specifics of a migration strategy: will the app require a period of read-only activity? Can the migration happen incrementally? Etc.

One strategy for management of databases which you mentioned, used especially when you have multiple database instances (as in the case where you want to have multiple separate environments on the same infrastructure), would be to have an admin instance from which scripts are run. One benefit of this is that both control and logging are centralized. Logging separately into each database instance to manage it isn't necessarily scalable, and is almost impossible or an anti-pattern when dealing with primary-replica DB systems, sharding, etc.

With Datastore, often you see schema migrations, in which a user's "schema" for the schemaless datastore needs to update, in which case the relevant elements need to either be incrementally-migrated as they're updated / retrieved or all at once, through simple iteration with cursors, a Map-Reduce job, etc. In addition to this sense of the word "migration", often people will backup their Datastore to Cloud Storage which basically means a JSON / binary serialization of entities for insertion into another app's Datastore (or in more exotic cases, for translation to an external RDBMS, etc.).

As to your final post, manage.py from Django and remote_api from App Engine can work well together. If you find complexity is increasing, something like Fabric, again, is a very valuable tool. You could run Fabric off of your "admin instance" in the cloud, while keeping a local copy of all scripts and credentials in the rare case that your admin instance needs redundancy (of course you can save images of the instance and merely spin these up in such a case). Another advantage of having a cloud-based admin instance which centralizes your administrative scripts is the more consistent uptime than simply running these off your laptop, or a special box in your workspace, etc.

I hope some of these thoughts have been helpful in responding to your very interesting post. You seem well-placed to develop any solutions required as the need for them comes along. A commitment to experimentation and reading documentation is indispensable in our era where a proliferation of potential solutions for each of the classic problems (and new problems) of web systems development, all with great selling-points and specific peculiar features, confronts us as developers.

Feel free to use this forum for general, high level discussion as you've done here, and feel free to ask questions about any services in particular. Thanks for contributing your thoughts and use-case details!

Sincerely,

Nick
Cloud Platform Community Support

[Silvan Spross]

Hi Nick

Thank you very much for your reply and thoughts. I appreciate your time! I tend to use GAE managed vm instead of "vanilla" GAE At the moment. I know it's beta right now, but I hope you keep it up and I think this way we are not too much tied to GAE stuff. This also solves some of my "problems" I described above too.

I'm in a hurry right now, but I'll, if you don't mind, try to ask more specific and better explained questions later about my current problems.

Regards,

Silvan

[Nick (Cloud Platform Support)]

No worries Silvan, glad to be of help! And yes, Managed VMs are awesome. I wouldn't worry too much about them going away any time soon based on how much attention they're getting and how much the community is enjoying them!