Building using Docker images with WAVM WASM runtime

[Frank Taylor]

Hi,

I am trying to build an optimised envoy with the WAVM WASM runtime. I'd like to use the Docker build image.

So far I've had to hack the do_ci.sh script to add the "--define wasm=wavm" and the resulting builds have crashed shortly after starting to run WASM code.

What's the correct incantation to make this work?

Thanks,

Frank.

[Piotr Sikora]

I'm not sure if using "ci/do_ci.sh" is the recommended way to build a Docker image, but "--define wasm=wavm" should be enough to build Envoy that uses WAVM instead of V8 for executing Proxy-Wasm plugins (why that choice, btw?).

The tests with WAVM are running fine, I just verified that locally with Envoy@2a9565da52 in a build with "-c opt", so it should work. Perhaps there is a bug, but you need to provide more details about the crash (and ideally, whether the same Proxy-Wasm plugins works fine with the default V8 engine). Filling a bug report on GitHub might be the best way to do that.

Best regards,

Piotr Sikora

--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/cad4ee0b-885e-423d-99d5-130d085c8890n%40googlegroups.com.

[Frank Taylor]

Piotr,

thanks for the quick response. 

I'm not sure if using "ci/do_ci.sh" is the recommended way to build a Docker image, but "--define wasm=wavm" should be enough to build Envoy that uses WAVM instead of V8 for executing Proxy-Wasm plugins (why that choice, btw?).

To clarify, I am using the Docker image to perform the build (as documented here: https://github.com/envoyproxy/envoy/blob/main/bazel/README.md#building-envoy-with-the-ci-docker-image) to avoid needing to install the whole toolchain.

As to why... I've built an authorization extension (PoC) which works in v8, but the performance is not much better than a traditional external authz plugin. Given that v8 is interpreted, I wanted to see how the compiled WAVM version compared.

The tests with WAVM are running fine, I just verified that locally with Envoy@2a9565da52 in a build with "-c opt", so it should work. Perhaps there is a bug, but you need to provide more details about the crash (and ideally, whether the same Proxy-Wasm plugins works fine with the default V8 engine). Filling a bug report on GitHub might be the best way to do that.

I've separately raised an issue on the GitHub project to check the current status of the WAVM runtime and how to collect debug information to troubleshoot this. (https://github.com/envoyproxy/envoy/issues/20134). The extension works on v8.

The purpose of this question was to confirm/ask how to build envoy with WAVM correctly to rule out issues with the build process. Since asking this, I've tried various approaches to building envoy and all have had the same issue executing my extension. More details are in the issue linked above.

 

Best regards,

Piotr Sikora

Thanks,

Frank.

[Piotr Sikora]

Let's continue the WAVM discussion on GitHub, thanks for filing the issue.

> As to why... I've built an authorization extension (PoC) which works in v8, but the performance is not much better than a traditional external authz plugin. Given that v8 is interpreted, I wanted to see how the compiled WAVM version compared.

V8 is JIT'ed, not interpreted, but I'm surprised by your findings (could share the numbers?). Even though the performance was never a selling point for Wasm (in non-browser environments), it should be better than going to an external system (even on localhost).

Best regards,

Piotr Sikora

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/184c0e53-6c2d-4b49-86e7-a7d0bb7b1123n%40googlegroups.com.

[Frank Taylor]

Ah... I had understood it was some level of bytecode interpretation (see:
https://www.solo.io/blog/the-state-of-webassembly-in-envoy-proxy/), hence
wanting to explore WAVM to ensure that we are getting the best out of our
extension. If this is a false assumption, then it may not be worth pushing to
get the WAVM version working. I am still happy to help work this through if it
would help.

As to results. This is a PoC of an authorization service we want to build. I
am exploring different approaches before building out the solution. The
extension needs to authorize HTTP requests, but authz is based on external
state which needs to be captured and maintained in the extension. Because of
the lack of state sharing between WASM VMs, I need two extensions: one on the
HTTP worker threads to handle requests, and one running as a global service
extension. They talk to each other over message queues.

I want to compare this design against a traditional external authz sidecar
(which is much easier to build!) to determine if the performance gains are
worth the extra development cost and complexity.

The setup is comparing the traditional external authz extension with my WASM
extension.

* External authz is configured to call a trivial localhost service (in a
  different container on the same host) which always returns a 200 status...
  so allows the request through.
* The WASM is composed of two parts:
  - The core (service) extension which accepts messages on a message queue.
    For now, this service simply responds to messages by sending a response on
    the designated response queue (nominated in the request).
  - An http worker extension which intercepts HTTP requests and sends a
    message to the core service and awaits a response before continuing (or
    failing) the HTTP request. On startup this extension negotiates a unique
    queue name (using the shared key/value store) so that the core service
    knows where (which HTTP worker thread) to send its responses.

This is the bare bones of the communication flow and should enable us to
determine the potential upside of the extension vs external authz.

I saw the following performance (running in Rancher Desktop on my laptop,
using docker image envoyproxy/envoy:v1.19.1, v8 WASM runtime, the WASM is
authored in AssemblyScript, load generated using Apache JMeter running
locally, 100 threads x100 requests = 10k in total):

    - ext_authz: 620 req/s
    - WASM queue: 660 req/s +6%
    - no authz: 1040 req/s +70%

So, not very promising.

By staying "in-process" I was hoping to see results closer to the no authz
numbers, e.g. pushing 800 req/s.