dev_appserver now compatible with Cloud Datastore Emulator.

[Steren Giannini]

Dear App Engine users,

We are pleased to announce that the App Engine local development server (dev_appserver) is now compatible with the Cloud Datastore Emulator.

You can opt-in to this feature by running:

 dev_appserver.py --support_datastore_emulator

By using the --support_datastore_emulator flag, dev_appserver will use Cloud Datastore Emulator for local datastore emulation. You can read more about the migration process here.

The development server and Cloud Datastore Emulator support the interoperation of ndb and Cloud Datastore client libraries. 

Please help us test this feature. Our goal is to enable it by default in the future.

Please share your feedback by replying directly to this thread. We are looking forward to it!

Thanks,

Steren

Product Manager

Google Cloud

[Vitaly Bogomolov]

Hi, Steren

Our goal is to enable it by default in the future.

Good news.

What about google.appengine.ext.testbed.init_datastore_v3_stub() ? Is it also will be touched by these changes?

WBR, Vitaly

[Kai Wang]

Hi Viataly,

This change alone does not affect testbed.

But we do plan to allow testbed to use datastore emulator too. Please stay tuned :)

[Yuri Sergiichuk]

Hi Steren,

Could you please tell me if the feature is generally available for any language and not only Python?

I'm using Java and Gradle AppEngine Plugin to start the local dev server, is it possible to start the emulator at the same time?

Thanks in advance for the answer.

Regards,

Yuri

[Giuliano Ribeiro]

Really nice, but how to put it to work ?

I'm using with Go, even setting up to true, my app still saving data in the cloud.

Thank you, 

[Kamran (Google Cloud Support)]

Hello Yuri,

dev_appserver.py is a command to start the local development server for Python, PHP and Go. If you want to use local Datastore emulator feature for Java, for instructions visit this article.

[Kamran (Google Cloud Support)]

Hello Giuliano,

As described in this article the Datastore Emulator is still a beta release. If you notice any defects please feel free to open issues in Google issue tracker with all details which needed to reproduce the issues and we'll be glad to investigate.

[Kai Wang]

Hi Glullano,

What go libraries are you using? Is it google.golang.org/appengine/datastore (link) ?

Did it store locally before?

You can email  k...@google.com for further details :)

[Giuliano Ribeiro]

Hi Kai Wang, I'm using "cloud.google.com/go/datastore"

As I read in the documentation this datastore package support the emulator, I was thinking it should be easy to use. But even enabling the emulator in the dev_appserver, my app is reading and writing directly in the cloud, ignoring the emulator. 

Thanks,

[Kai Wang]

According to the documentation, you need to have the env var "DATASTORE_EMUALTOR_HOST" set. 

Also, FYI, this "cloud.google.com/go/datastore" (officially, it is called Google Cloud Client Library), should be able to connect to the datatore emulator, no matter the code runs inside or outside of the dev_appserver.

Please check the env var and lmk if they work :)

[Giuliano Ribeiro]

When I setup the local env, see the log:

WARNING  2018-06-01 13:21:25,741 api_server.py:581] Detected environment variable DATASTORE_EMULATOR_HOST=localhost:9999, dev_appserver will speak to the Cloud Datastore emulator running on this address. The datastore_path /var/folders/n4/vq69gqrn3470jyqdp1sxp2ww0000gn/T/appengine.None.gribeiro/datastore.db will be neglected.

If you want datastore to store on /var/folders/n4/vq69gqrn3470jyqdp1sxp2ww0000gn/T/appengine.None.gribeiro/datastore.db, remove DATASTORE_EMULATOR_HOST from environment variables and restart dev_appserver

And my app still writing/reading from the cloud.

[Kai Wang]

Hi Giuliano,

Note that, for app engine standard runtime go, the recommended datastore library is

"google.golang.org/appengine/datastore".  I've verified on my side this library works with dev_appserver + datastore emulator.

As for "cloud.google.com/go/datastore". I am not every sure how to make it work in App Engine standard app.  I do have verified this cloud library can speak to the datastore emulator when it is outside of App Engine dev_appserver. Unfortunately I've not been able to make this library work inside an app engine golang app. Could you kindly attach some  code snippet, such that run a simple GAE go app with the cloud datastore library?

Detected environment variable DATASTORE_EMULATOR_HOST=localhost:9999, 

FYI, are you sure the emulator ran on localhost:9999 ?

Thank you so much for trying this feature.

I am all ears to follow up :)

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/vyzCiWKOngc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/edfbf328-bab1-47ce-9e39-9f5734229582%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kai Wang]

Actually, "cloud.google.com/go/datastore" should also work. However, at this moment, you have to pass the DATASTORE_EMULATOR_HOST with the '--env_var' flag to dev_appserver:

 --env_var DATASTORE_EMULATOR_HOST=localhost:8088

Please try this flag, hopefully it should work :)

Dev_appserver currently prunes shell env vars from local golang runtime process. The '--env_var' flag allows you to pass environment variable to the local go runtime process.

[Cosmin Poieană]

Yes, it works! Because the emulator starts, sets the required env vars, requirements are installed and finally the app is served, but with stripped env vars, then additional ones are taken from app.yaml or manually provided by the dev_appserver.py (as above). Don't forget to mention its port also with "--datastore_emulator_port 8088" for example.

miercuri, 20 iunie 2018, 23:40:04 UTC+3, Kai Wang a scris:

Actually, "cloud.google.com/go/datastore" should also work. However, at this moment, you have to pass the DATASTORE_EMULATOR_HOST with the '--env_var' flag to dev_appserver:

 --env_var DATASTORE_EMULATOR_HOST=localhost:8088

Please try this flag, hopefully it should work :)

Dev_appserver currently prunes shell env vars from local golang runtime process. The '--env_var' flag allows you to pass environment variable to the local go runtime process.

On Tue, Jun 19, 2018 at 5:54 PM Kai Wang <k...@google.com> wrote:

Hi Giuliano,

Note that, for app engine standard runtime go, the recommended datastore library is

"google.golang.org/appengine/datastore".  I've verified on my side this library works with dev_appserver + datastore emulator.

As for "cloud.google.com/go/datastore". I am not every sure how to make it work in App Engine standard app.  I do have verified this cloud library can speak to the datastore emulator when it is outside of App Engine dev_appserver. Unfortunately I've not been able to make this library work inside an app engine golang app. Could you kindly attach some  code snippet, such that run a simple GAE go app with the cloud datastore library?

Detected environment variable DATASTORE_EMULATOR_HOST=localhost:9999, 

FYI, are you sure the emulator ran on localhost:9999 ?

Thank you so much for trying this feature.

I am all ears to follow up :)

To unsubscribe from this group and all its topics, send an email to google-a...@googlegroups.com.

[Alessandro Bollini]

On Friday, 25 May 2018 20:34:53 UTC+2, Kamran (Google Cloud Support) wrote:

Hello Yuri,

dev_appserver.py is a command to start the local development server for Python, PHP and Go. If you want to use local Datastore emulator feature for Java, for instructions visit this article.

Still unable to connect to the Datastore Emulator managed by the Java 8 Local Development Server using Google Client Library: I unsuccessfully tried the following variations on what suggested by the linked article:

1) java_dev_appserver.sh target/war/

2) DATASTORE_EMULATOR_HOST=localhost:8080 java_dev_appserver.sh target/war/

The first one throws the following exception after trying to connect to the actual cloud datastore

Caused by: com.google.datastore.v1.client.DatastoreException: Unauthenticated., code=UNAUTHENTICATED
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:226)
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:267)

The second one stalls for some time before throwing

Caused by:
com.google.datastore.v1.client.DatastoreException: Non-protobuf error: <html>|<head>|<meta http-equiv="Content-Type" …

My maven dependencies are

<dependencyManagement>
   <dependencies>
   
       <dependency>
           <groupId>com.google.cloud</groupId>
           <artifactId>google-cloud-bom</artifactId>
           <version>0.110.0-alpha</version>
           <type>pom</type>
           <scope>import</scope>
       </dependency>
   
   </dependencies>
</dependencyManagement>

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-datastore</artifactId>

</dependency>

Hope someone is able to suggest the proper setup…

[Jose Antonio Soni Solchaga]

Hello Alessandro,

The error message you are experiencing is because of wrong authentication.

Here you can see a Stack Overflow question where this was answered.

[Alessandro Bollini]

Agreed, but a little debugging shows that Google Client Library is trying to connect to the actual cloud endpoint of the project: supplying the credentials through `gcloud auth application-default login` or `gcloud iam service-accounts keys create` results in the test code connecting to the production datastore…

According to https://cloud.google.com/docs/authentication/production#finding_credentials_automatically, if no credentials are provided the hosting environment should automatically provide connection hints for the local resources: I’d expect Java 8 Local Development Server to automatically connect Google Client Library to the local Datastore Emulator, replicating the application behaviour when deployed in cloud, but that’s apparently not the case.

I’m currently working around the issue by testing locally using both the Local Development Server and the standalone Datastore Emulator as:

gcloud beta emulators datastore start \
—project=project-id \
--host-port=localhost:8081 \
--data-dir=target/war

DATASTORE_EMULATOR_HOST=localhost:8081 java_dev_appserver.sh \
--port=8080 \
target/war

However, the process is quite cumbersome and difficult to automate: what I’m looking for is a way to automatically connect Google Client Library to the Datastore Emulator managed by the Java 8 Local Development Server when launching the app with something like https://github.com/GoogleCloudPlatform/app-maven-plugin, e.g. `mvn appengine:run`.

-----------------------------------------------------------------------------------------------------------------------------------------
alessandro bollini ph.d. - tech lead - linkedin.com/in/alessandrobollini