AMQP support for Envoy

[aco...@redhat.com]

I am starting a proof-of-concept for an AMQP-HTTP bridge implemented as an Envoy filter (or maybe a pair of them).
I'll be using the AMQP library at http://qpid.apache.org/releases/qpid-proton-0.22.0/proton/cpp/api/index.html
Depending on how well it goes, it may become a serious project. No progress yet, just throwing it  out there in case anyone has thoughts.

Newbie question: can I generate API docs for the C++ classes needed to build a filter or should I grit my teeth and read the headers?
Are there any guides to  building envoy filters around? I am already hard at work deciphering https://github.com/envoyproxy/envoy-filter-example

[Matt Klein]

We don't have any guides or easy ways to generate doxygen docs currently. The headers are well documented.

The repo was recently reorged in a way that should make it much easier to learn from existing filters: https://github.com/envoyproxy/envoy/tree/master/source/extensions

--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+unsubscribe@googlegroups.com.
To post to this group, send email to envo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/a7ac989c-8dde-4ba2-9bd8-2e0cbade1d19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Alan Conway]

On Tue, Apr 17, 2018 at 6:56 PM, Matt Klein <mkl...@lyft.com> wrote:

We don't have any guides or easy ways to generate doxygen docs currently. The headers are well documented.

The repo was recently reorged in a way that should make it much easier to learn from existing filters: https://github.com/envoyproxy/envoy/tree/master/source/extensions

Thanks - one further question: can envoy load plugins from shared libraries or is it required to build a custom envoy binary to add new filter types? Is dynamic loading planned, or would it be of interest as a contribution?
 

[Matt Klein]

Adding loadable module support is something that I would like to see happen. It is tracked here: https://github.com/envoyproxy/envoy/issues/2053

Given how the static registration process works I don't think this will be terribly difficult to do. We just need someone to dig in. If you are interested please follow up in the GH issue.

[aco...@redhat.com]

On Tue, Apr 17, 2018 at 3:10 PM, <aco...@redhat.com> wrote:

I am starting a proof-of-concept for an AMQP-HTTP bridge implemented as an Envoy filter (or maybe a pair of them).
I'll be using the AMQP library at http://qpid.apache.org/releases/qpid-proton-0.22.0/proton/cpp/api/index.html
Depending on how well it goes, it may become a serious project. No progress yet, just throwing it  out there in case anyone has thoughts.

I've writing a Network::Filter that can be configured between a Listener for AMQP connections and the HTTP connection manager. It intercepts AMQP messages from downstream in onData() and replaces the Buffer::Instance content with HTTP requests before continuing up the filter chain. The idea being to do the reverse in onWrite() and correlate HTTP responses as AMQP response messages.

It looks doable, but it feels like I'm doing it in the wrong place, as I don't have access to Envoy's HTTP parsing facilities in a Network::Filter so I'm doing some trivial mapping by hand. What would be a better approach:

1. Can I make use of the HTTP:: streams and codec from a Network::Filter directly? I know the HTTP connection manager itself does this, but I can't (yet) see how to pull out what I would need.
2. Can I write a Network::Filter and HTTP::Filter that collaborate as a unit somehow?
3. Is there a more efficient way to pass pre-parsed or not-yet-encoded HTTP objects like HeaderMaps between filters, instead of always encoding/decoding http in raw data buffers?

In some ways it seems like my filter needs to duplicate some behavior of the HTTP connection manager, but I want to keep duplication to a minimum and us standard Envoy features as much as possible. 

Any hints much appreciated!

Cheers,
Alan.

[Matt Klein]

I think what you are doing is probably the best way forward. You should be able to directly use the HTTP1 or HTTP2 encoder/decoder (codec) to convert between HTTP messages and bytes.

--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+unsubscribe@googlegroups.com.
To post to this group, send email to envo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/ad54a167-2be1-4961-a30e-3bbc432fe58b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alan Conway]

On Tue, Apr 24, 2018 at 6:32 PM, Matt Klein <mkl...@lyft.com> wrote:

I think what you are doing is probably the best way forward. You should be able to directly use the HTTP1 or HTTP2 encoder/decoder (codec) to convert between HTTP messages and bytes.

Looking at HttpConnectionManagerConfig::createCodec()  I can see how I would create my own codec, but there's quite a bit of configuration involved. Ideally I would like to use the same configuration as the upstream connection manager and not duplicate it on my filter. Is there any way to navigate to the connection manager's config from my Network::Filter?

I realise it's not ideal to have a  dependency between filters, but having possibly conflicting duplicate config on the AmqpHttpFilter and HttpConnectionManager seems worse.

[Alan Conway]

On Thu, Apr 26, 2018 at 11:52 AM, Alan Conway <aco...@redhat.com> wrote:

On Tue, Apr 24, 2018 at 6:32 PM, Matt Klein <mkl...@lyft.com> wrote:

I think what you are doing is probably the best way forward. You should be able to directly use the HTTP1 or HTTP2 encoder/decoder (codec) to convert between HTTP messages and bytes.

I am having trouble understanding how to use the HTTP codec to encode/decode bytes independently of a Network::Connection. All the constructors for impls of HTTP::Stream/Encoder/Decoder seem to require a Connection. Is there a way to construct the codec classes separately, or should I be trying to create a fake Network::Connection that is being fed Buffers by my filter and not by a socket?

[Matt Klein]

Yeah that current code is structured such that it assumes a connection.

I think unfortunately your options are either to attempt to refactor somehow to avoid requiring a Network::Connection or probably simpler to do what you suggest which is to provide some type of fake Network::Connection which only implements the required methods.

[Alan Conway]

On Fri, Apr 27, 2018 at 6:17 PM, Matt Klein <mkl...@lyft.com> wrote:

Yeah that current code is structured such that it assumes a connection.

I think unfortunately your options are either to attempt to refactor somehow to avoid requiring a Network::Connection or probably simpler to do what you suggest which is to provide some type of fake Network::Connection which only implements the required methods.

Thanks! Another option is to use the nodejs API directly. I'll see what's easiest.
 

Another question for you:  I'm intercepting and replacing the downstream data in onWrite() - works fine. However sometimes I generate data in onRead() that needs to go downstream.  I can't call  Connection::write(data) like the echo example, because that goes through the filter chain, so my generated data gets mixed up with data from upstream.

My solution is to have onRead() save the generated data in a member buffer and call Connection::write() with an *empty* buffer. That generates an empty call to onWrite() so I can send the generated data. It works but could empty writes be optimized away someday? If so is there a way to write to the connection but only via my downstream filters?

[Matt Klein]

My solution is to have onRead() save the generated data in a member buffer and call Connection::write() with an *empty* buffer. That generates an empty call to onWrite() so I can send the generated data. It works but could empty writes be optimized away someday? If so is there a way to write to the connection but only via my downstream filters?

Yeah I can't guarantee this will never change and I agree it's kind of hacky. I would probably just do it for now and add liberal comments. The better solution is probably to have a filter specific write call that only invokes filter downstream of the calling filter. If you want to work on that it would be nice addition to the filter API. 

[Alan Conway]

On Wed, May 2, 2018 at 3:34 AM, Matt Klein <mkl...@lyft.com> wrote:

My solution is to have onRead() save the generated data in a member buffer and call Connection::write() with an *empty* buffer. That generates an empty call to onWrite() so I can send the generated data. It works but could empty writes be optimized away someday? If so is there a way to write to the connection but only via my downstream filters?

Yeah I can't guarantee this will never change and I agree it's kind of hacky. I would probably just do it for now and add liberal comments. The better solution is probably to have a filter specific write call that only invokes filter downstream of the calling filter. If you want to work on that it would be nice addition to the filter API. 

+1

This is starting to come together, hopefully will have something preliminary for review by next week.
 

[Alan Conway]

On Wed, May 2, 2018 at 10:26 AM, Alan Conway <aco...@redhat.com> wrote:

On Wed, May 2, 2018 at 3:34 AM, Matt Klein <mkl...@lyft.com> wrote:

My solution is to have onRead() save the generated data in a member buffer and call Connection::write() with an *empty* buffer. That generates an empty call to onWrite() so I can send the generated data. It works but could empty writes be optimized away someday? If so is there a way to write to the connection but only via my downstream filters?

Yeah I can't guarantee this will never change and I agree it's kind of hacky. I would probably just do it for now and add liberal comments. The better solution is probably to have a filter specific write call that only invokes filter downstream of the calling filter. If you want to work on that it would be nice addition to the filter API. 

Just checked the doc for the existing Network::Connection::write() -

  /**
   * Write data to the connection. Will iterate through downstream filters with the buffer if any
   * are installed.
   * @param data Supplies the data to write to the connection.
   * @param end_stream If true, this indicates that this is the last write to the connection. If
   *        end_stream is true, the connection is half-closed. This may only be set to true if
   *        enableHalfClose(true) has been set on this connection.
   */
  virtual void write(Buffer::Instance& data, bool end_stream) PURE;
 

So it sounds like write() was intended to do what I want.

My observation is that write() called from onData() results in a call to this->onWrite() - I think "downstream" should exclude the current filter.

If changing the existing write() might break things, I'll add writeDownstream()

[Alan Conway]

By the way - I just did my first sucessful AMQP request/response translated via envoy to a GET on httpd, and got the HTML page back in an AMQP message :)

I'm quite impressed by envoy's flexibility. In particular writing all this as a NetworkFilter makes a completely clean break - the downstream side of the connection is pure AMQP, upstream is pure HTTP. No modifications needed to the existing envoy codebase (apart from the minor issues we've discussed) Since it looks like a regular HTTP connection to envoy, all of envoy's HTTP routing and other features apply.

Next the other half - allow envoy to be configured to make AMQP connections and use them as sinks for HTTP requests.
I need to learn more about envoy's outbound route/cluster side, but based on my experience so far I'm optimistic it will be up to the task.

[Matt Klein]

If changing the existing write() might break things, I'll add writeDownstream()

The correct solution here is to add functionality to the network filter callbacks to perform writes. You can model this after how HTTP filters work when encoding headers, data, etc.

I.e., in https://github.com/envoyproxy/envoy/blob/master/include/envoy/network/filter.h you will need to add write() to ReadFilterCallbacks, and you might need to add WriteFilterCallbacks, with a write function. Optimally, the Connection accessor would be made constant, so any modifiers to the connection need to operate through the callbacks which can handle filter iteration. This will be a bit of work, but not too terrible.

 I need to learn more about envoy's outbound route/cluster side, but based on my experience so far I'm optimistic it will be up to the task.

Unfortunately, this side of things has not yet had an investment in extensibility, although this has come up before. I think for your use case, the easiest way of accomplishing what you need to do is going to be to make the HTTP connection pool concept statically pluggable and configurable: https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/cluster_manager_impl.cc#L922. I think this could bed one relatively easily using the standard static registration systems that we normally do. Once this is done, I think it would be pretty straightforward to build an HTTP -> AMQP connection pool. 

[zyf...@gmail.com]

I recently is also do turn http to tcp, how are your progress?, can refer to your code.

在 2018年4月18日星期三 UTC+8上午6:10:37，aco...@redhat.com写道：

[Alan Conway]

First cut of the AMQP-HTTP bridge is on github at https://github.com/alanconway/envoy-amqp/

Not at all complete yet but any feedback is welcome. The README describes the current status.

--
You received this message because you are subscribed to a topic in the Google Groups "envoy-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/envoy-dev/0S8zMhLPfgE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to envoy-dev+unsubscribe@googlegroups.com.

To post to this group, send email to envo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/bbf369dc-e067-4bbd-a65c-e7f1259167e5%40googlegroups.com.

[Alan Conway]

On Sat, May 5, 2018 at 12:19 PM, Matt Klein <mkl...@lyft.com> wrote:

If changing the existing write() might break things, I'll add writeDownstream()

The correct solution here is to add functionality to the network filter callbacks to perform writes. You can model this after how HTTP filters work when encoding headers, data, etc.

I.e., in https://github.com/envoyproxy/envoy/blob/master/include/envoy/network/filter.h you will need to add write() to ReadFilterCallbacks, and you might need to add WriteFilterCallbacks, with a write function. Optimally, the Connection accessor would be made constant, so any modifiers to the connection need to operate through the callbacks which can handle filter iteration. This will be a bit of work, but not too terrible.

On my todo list, the write(emtpy) hack is working well meantime.
 

 I need to learn more about envoy's outbound route/cluster side, but based on my experience so far I'm optimistic it will be up to the task.

Unfortunately, this side of things has not yet had an investment in extensibility, although this has come up before. I think for your use case, the easiest way of accomplishing what you need to do is going to be to make the HTTP connection pool concept statically pluggable and configurable: https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/cluster_manager_impl.cc#L922. I think this could bed one relatively easily using the standard static registration systems that we normally do. Once this is done, I think it would be pretty straightforward to build an HTTP -> AMQP connection pool. 

I like the extension & configuration architecture you have, I'd like to stay within it as much as possible: The doc says " cluster manager exposes APIs to the filter stack that allow filters to obtain a L3/L4 connection " which is all I need from a cluster. Instead of pluggable cluster implementations (and the new config that implies) would it make sense to modify the existing cluster config to allow a filter chain per cluster? That feels like a smaller effort with a bigger payoff - I can re-use much of my existing filter code (just flip onData and onWrite),  users won't have to learn new cluster configuration - clusters will just have a familiar filter chain, and  outbound extensions can be done with the existing filter framework (less doc. to write). There may be pitfalls I'm not aware of.

[Alan Conway]

Looking at this I think what I'm considering is adding a simplified version of listener filter-chain configuration on the cluster configuration. I think there's no need for filter matching logic in this case, since we know where the connection is going - just a simple list of filters. It looks like I could have the cluster manager pass the configured filters when creating a  HostImpl, which would create the filter chain on each client connection.

No code written yet so this might all be a horrible idea, but it seems plausible.

[Matt Klein]

In general, yes, I think there would be a lot of value in allowing upstream filters. We can start with L3/L4 filters that are attached to TCP connections created on behalf of a cluster, which is what you propose. 

The downside of this for your use case is that you are going to have to re-parse HTTP bytes to translate them to AMQP. The pluggable connection pool implementation I mentioned would allow you to deal with HTTP message concepts.

If you want to tackle either of these and contribute them I think it would be great. This area is loosely tracked in https://github.com/envoyproxy/envoy/issues/173. If you want to tackle it I think we should continue the conversation there with a more complete design proposal.