Custom runtime - using search API / AppEngine services

[Adam Evans]

I'm trialing AppEngine and have deployed an Haskell based application using a Docker image and custom runtime support. This was reasonably trivial.

I now want to interact with the services provided by AppEngine. The first service I want to experiment with is the Search / Searchable Document and Indexes https://cloud.google.com/appengine/docs/java/search/ .

Unfortunately no Haskell SDK exists. Also no public rest API's seem to be documented for Search. I see no documentation on the custom runtime pages of how outside of the provided SDK's custom runtimes should interact with services in AppEngine.

Investigating I found that

• Custom Runtimes run some sidecars on the VM to work within AppEngine https://github.com/GoogleCloudPlatform/appengine-sidecars-docker
• One interesting side car looks to be an API Proxy running on port 10001 https://github.com/GoogleCloudPlatform/appengine-sidecars-docker/blob/master/api_proxy/proxy.go
• I found a external github project adding search support to PHP. This looks to use the API sidecar over HTTP with protobuffs https://github.com/tomwalder/php-appengine-search
• The appengine-php-sdk looks to make use of the API sidecar  for interacting with AppEngine services https://github.com/GoogleCloudPlatform/appengine-php-sdk/blob/master/google/appengine/runtime/VmApiProxy.php

Is there any official information on how CustomRuntimes should interact with AppEngine services?

Is there any information on the API Proxy sidecar, using it, the RPC methods, ProtoBuff files?

Is there any simple HTTP/REST endpoints for the search service I can just call?

[Tom Walder]

I guess you could deploy a PHP/Python/Standard runtime app to just expose a RESTish API to the Search service.

I've done this before.

[Adam Evans]

Yes.

That looks to be the simplest / obvious solution, take the off the shelf Java / Python AppEngine SDK and create a RESTish wrapper around the search features. Deploying in AppEngine I'd then need to look at service to service auth to restrict access so my index is not publicly available.

The primary reason for investigating AppEngine is cheap computing on the the micro instances + search vs AWS EC2 with ElasticSearch. I would prefer not have an instance specifically as a search sidecar to keep the cost down.

I'll try and investigate the ProtoBuf api reverse engineered by the PHP project. It would be useful if Google opened up the spec if providing custom runtimes so 3rd parties can implement the libraries, it'd also make AppEngine a no brainer as a PaaS with custom runtime support.

[Thomas Koch]

Hi Adam,

have you had success trying Haskell on AppEngine? I'd love to try it out myself once I've time for this and would love to learn from your experience or help you if you got stuck.

Regards,

Thomas

[Adam Evans]

I have had some trivial toy Haskell projects running on AppEngine just testing it out.

Using the AppEngine custom runtime it's trivial to deploy an application.

I use OSX as my development machine which complicates it a little but all you need to do is create a statically linked binary and a Dockerfile to start it. The only interesting bit in the Docker file is my app looks for a port environment variable which i hardcore and I use a optimised base image meaning the docker container is around 20Mb in total. https://github.com/fpco/haskell-scratch

FROM fpco/haskell-scratch:integer-gmp

COPY ./.stack-work/install/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/lts-7.0/8.0.1/bin/myapp-exe /usr/bin/
CMD PORT=8080 myapp-exe

I'm using stack to build my projects. As I'm on OSX I need it to tell stack to build the app in a linux docker container to run in the 'fpco/haskell-scratch' container above.

in the stack.yml I add the below

image:
  container:
    base: "fpco/haskell-scratch:integer-gmp"

docker:
  enable: true

Then to build I run the below which generates the executable used in the Dockerfile

stack build

If on Linux you probably won't need to build in a docker container and just use: stack build --ghc-options='-optl-static -optl-pthread' --force-dirty.

Interacting with the services in AppEngine where they have a public API shouldn't be an issue, hit the documented HTTP endpoint as they specify.

It's the API's which don't have a public Google Cloud equivalent that are problematic.

Search being one.

AppEngine give the tools for custom runtimes but don't give docs for interacting with things such as the search API so you can write your own API using your own runtime. I've not had anytime yet to reverse engineer the protocol buffer sidecar they have on the instances....If i wasn't playing with a toy project and on a deadline I'd write a separate service using Scala + the Java sdk's to deploy as a thin REST service for search / mail then call that from Haskell.

With the custom runtimes AppEngine should publish the sidecar API so people can create open source SDK's which would only help grow AppEngine.