Persistent storage for Managed VMs, please?

[Avanish Raju]

Hi AppEngine/Managed VM gurus,

Can we please have a way to mount a persistent disk on to our managed VMs?

Now, I'm aware that Managed VMs are intended to be for "stateless use". But here's my use case, the problem, and the solutions I've tried:

Use Case: I need secure web access from my Google AppEngine Java standard module to an Apache Solr/SolrCloud instance/cluster.

Problem:

Apache Solr uses a custom DB which stores its index on files. Now, if Datastore, Blobstore or GCS, or some other Google Cloud Platform technology could somehow emulate the file system, that would work great. But none does. These files are the only "state" for Solr, just like "Datastore" would be "the state" for other AppEngine runtimes.

Options tried:

1) Barebones Google Compute Engine instance: I setup Solr there, and opened up the firewall to access the instance from AppEngine. This has two problems:

a) There's no way to setup firewall rules between GAE and GCE. So I have to secure my traffic myself.

b) Apparently, HTTPS client requests from GAE to GCE are blocked. (Tried using Sockets API from GAE; requests to other services work though)

2) Managed VMs: With a Dockerfile that can start up, fetch the Solr release, start up. All good. But... the disk is ephemeral, of course. So how do I back up data? Problem:

No Google Cloud Storage read/write scope. No way to provide extra metadata/scopes. Otherwise I could have "saved the data" to GCS, and pulled it up and set it through a Dockerfile when bringing up the instance.

3) Managed VMs with an attached disk: I was able to successfully write up a script so that when my Docker container started up, it would get my AppEngine java module(which has Compute API scope by default) to setup and "attach" a disk to the Managed VM, if it wasn't already there! But... problem here:

Docker doesn't permit "mount" to be run - this needs "privileged-mode docker" to be running (otherwise we get a "mount: Permission Denied"), and i couldn't find a way to pass on "privileged mode" to the Docker on Managed VM.

So, any solutions? Can I not setup GAE <-> Apache Solr securely on Google Cloud Platform? Must I move over my GAE Java stack off to pure GCE or other IAAS offerings, or are there any solutions possible here?

Thanks for reading!

(And hope you have a solution!)

Avanish