Re: Dockerized GAE Java 11 app and Datatore Emulator connection issue

[H Singh]

Finally! I got it to work. All I had to do was to add the protocol "http://" to the DATASTORE_EMULATOR_HOST and use "docker.for.mac.localhost" instead of 0.0.0.0 or localhost.

DATASTORE_EMULATOR_HOST=http://docker.for.mac.localhost:8081

I have no idea why adding the protocol makes the difference and would be great to know the reason. I've not seen it in any docs.

Now I'm combined both containers into the same docker-compose and changed the host to http://datastore:8081 by using the service name. This is what my final setup looks like:

version: '3'

services:

  datastore:

    image: my-datastore-image-name

    volumes:

      - ${PWD}/gcloud:/root/.config/gcloud

    ports:

      - 8081:8081

    command: gcloud beta emulators datastore start --host-port 0.0.0.0:8081 --project my-project-id --store-on-disk

  app:

    image: my-app-image-name

    working_dir: /app

    volumes:

      - ${PWD}:/app

      - ${PWD}/.m2:/root/.m2/

    ports:

      - 8080:8080

    depends_on:

      - datastore

    command: java -Xmx64m -jar /app/target/my-app.jar

    environment:

      - DATASTORE_EMULATOR_HOST=http://datastore:8081

      - DATASTORE_PROJECT_ID=my-project-id

I spent about 8-10 hours to get this working. Hope it helps others.

On Wednesday, July 20, 2022 at 9:15:43 PM UTC+8 H Singh wrote:

I got help on this forum last week to get my GAE Java 11 app to connect to cloud datastore emulator on my local environment on Mac natively (without Docker). This is working well.

Now, I'm trying to run my app (GAE Java 11 runtime) and the cloud datastore emulator inside two separate docker containers. I was able to get my app running in cloud-sdk-docker image-based container to connect to the real Firestore cloud datastore. But  I can't get the connection working with datastore emulator, which is running inside a container.

Quick Summary first:

1. App on local -> Datastore emulator on local : WORKS

2. App on local -> Cloud datastore : WORKS

3. App inside Container -> Cloud datastore : WORKS

4. App inside Container -> Datastore emulator inside Container : DOESN'T WORK

5. App inside Container -> Datastore emulator on local : DOESN'T WORK

In cases 4 and 5, inside the container, the app fails at ObjectifyService.init() call. My classes do not get registered.

static {

  logger.info("Executing ModelDao init ...");

  ObjectifyService.init(); // For Objectify v6

  logger.info("After ModelDao init.");

  ObjectifyService.register(Model.class);

  ObjectifyService.register(Usr.class);

  // Register other classes ...

}

I get the following message on the console:

Executing ModelDao init ...

com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine

Failed to detect whether we are running on Google Compute Engine.

The "After ModelDao init" message doesn't appear and none of the classes get registered. 

I'm using Objectify v6. Here's my Maven dependency:

<dependency>

  <groupId>com.googlecode.objectify</groupId>

  <artifactId>objectify</artifactId>

  <version>6.0.7</version>

<dependency>

My datastore emulator starts up and I see the following environment variables in .config/gcloud/emulators/datastore/env.yaml file:

---

DATASTORE_DATASET: my-project-id

DATASTORE_EMULATOR_HOST: 0.0.0.0:8081

DATASTORE_EMULATOR_HOST_PATH: 0.0.0.0:8081/datastore

DATASTORE_HOST: http://0.0.0.0:8081

DATASTORE_PROJECT_ID: my-project-id

Here's the docker-compose.yml for the datastore:

version: '3'

services:

  datastore:

    image: my-datastore-image-name

    volumes:

      - ${PWD}/gcloud:/root/.config/gcloud

    ports:

      - 8081:8081

    command: gcloud beta emulators datastore start --host-port 0.0.0.0:8081 --project my-project-id --store-on-disk

I can the message "Ok" when I access 0.0.0.0:8081 from the browser.

I'm setting the env vars in the GAE app container using docker-compose.yml file as shown below, and I've verified that the variables are set correctly by entering the container manually.

version: '3'

services:

  app:

    image: my-app-image-name

    working_dir: /app

    volumes:

      - ${PWD}:/app

      - ${PWD}/.m2:/root/.m2/

    ports:

      - 8080:8080

    command: java -Xmx64m -jar /app/target/my-app.jar

    environment:

      - DATASTORE_DATASET=my-project-id

      - DATASTORE_EMULATOR_HOST=0.0.0.0:8081

      - DATASTORE_EMULATOR_HOST_PATH=0.0.0.0:8081/datastore

      - DATASTORE_HOST=http://0.0.0.0:8081

      - DATASTORE_PROJECT_ID=my-project-id

The issue is around the HOST variables. 

1. If I comment out DATASTORE_EMULATOR_HOST,  DATASTORE_EMULATOR_HOST_PATH, and DATASTORE_HOST above, then ObjectifyService.init() works and classes get registered, but at the time of connecting to the datastore, I get an I/O error (503).

2. If I use localhost instead of 0.0.0.0, ObjectifyService.init() will work but I'll get the 503 error at the time of making a DB call.

3. I've also tried using "docker.for.mac.localhost" instead of 0.0.0.0, but that has the same problem as with #1. 

4. If I run "curl 0.0.0.0:8081" from inside the app container, it doesn't work. But if I run "curl docker.for.mac.localhost:8081", it returns "Ok".

5. I've also tried defining both app and datastore services in the same docker compose file, and reference "datastore:8081" but it again causes the same ObjectifyService.init() issue as with #1.

I've tried so many other things, but just cannot get it to work.

I'll appreciate any help on this. Thanks!