Help on using REST Proxy behind ELB

[Stuart Wong]

Hey all,

I'm hoping someone can provide some assistance on using the REST Proxy behind AWS ELB. We have put the Kafka REST Proxy (RP) behind an AWS ELB (proxy mode enabled) which is fronted by an API gateway so only secured connections (i.e. client must obtain token and such) are allowed. We provide a user friendly DNS for the ELB DNS which is given to the API gateway and simple test connections are successful. However, the base_uri returned is always the user friendly DNS. We tried using just the ELB DNS but that only resulted in the ELB DNS being returned as the base_uri. As you can imagine, given the RP is stateful (this is so wrong) clients need to be routed to the same RP instance as returned in the base_uri, but we can't see how this going to work regardless.

We're wondering how others are using the RP. If behind an ELB, how is that setup accomplished? Is some client code being used to get around having round-robin connections to RP instances? Is HAProxy or nginx used instead of ELB? Should we instead be doing VPC peering, which seems to defeat the purpose of our microservice based architecture?

I appreciate any feedback and thoughts.

Thanks,

Stuart.

[Ewen Cheslack-Postava]

Hi Stuart,

ELB supports sticky sessions which allow you to route requests to the same proxy instance if you cannot configure your instances to give out a correct base_uri for the instance or if such a base_uri would not be directly routable from the clients using the proxy. Here are the general instructions for configuring stickiness: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-sticky-sessions.html You can assign a cookie that you use in subsequent requests to get the requests to stick to the same instance. Note that you only need to use these for consumer requests, and, if something starts failing, you can start up a new consumer instance elsewhere and allow the old one to expire (if you are unable to explicitly delete it).

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5026a15d-1cc7-45fe-99ea-cff58a39cf39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Stuart Wong]

Thanks Ewen.

We did try ELB sticky sessions but it did not work. Our thinking is perhaps the API gateway which sits between the client and ELB has something to do with it not working, or maybe our client needs to do more work. We're still investigating.

Do you have any thoughts or experience around how yourself or others are securing access to the RP?

Appreciate your reply.

- Stuart

On Tuesday, November 3, 2015 at 12:23:33 PM UTC-6, Ewen Cheslack-Postava wrote:

Hi Stuart,

ELB supports sticky sessions which allow you to route requests to the same proxy instance if you cannot configure your instances to give out a correct base_uri for the instance or if such a base_uri would not be directly routable from the clients using the proxy. Here are the general instructions for configuring stickiness: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-sticky-sessions.html You can assign a cookie that you use in subsequent requests to get the requests to stick to the same instance. Note that you only need to use these for consumer requests, and, if something starts failing, you can start up a new consumer instance elsewhere and allow the old one to expire (if you are unable to explicitly delete it).

-Ewen

On Mon, Nov 2, 2015 at 6:24 AM, Stuart Wong <cgs....@gmail.com> wrote:

Hey all,

I'm hoping someone can provide some assistance on using the REST Proxy behind AWS ELB. We have put the Kafka REST Proxy (RP) behind an AWS ELB (proxy mode enabled) which is fronted by an API gateway so only secured connections (i.e. client must obtain token and such) are allowed. We provide a user friendly DNS for the ELB DNS which is given to the API gateway and simple test connections are successful. However, the base_uri returned is always the user friendly DNS. We tried using just the ELB DNS but that only resulted in the ELB DNS being returned as the base_uri. As you can imagine, given the RP is stateful (this is so wrong) clients need to be routed to the same RP instance as returned in the base_uri, but we can't see how this going to work regardless.

We're wondering how others are using the RP. If behind an ELB, how is that setup accomplished? Is some client code being used to get around having round-robin connections to RP instances? Is HAProxy or nginx used instead of ELB? Should we instead be doing VPC peering, which seems to defeat the purpose of our microservice based architecture?

I appreciate any feedback and thoughts.

Thanks,

Stuart.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5026a15d-1cc7-45fe-99ea-cff58a39cf39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen