Guide to deploying on AWS?

[Avi Flax]

Hello all,

I need to deploy a subset of the platform (Kafka (so also ZK) + Schema Registry, for now) to AWS for a production system, and I’m finding myself struggling to synthesize the various hints, tips, and examples I’ve found around the Web.

So… is there a written guide somewhere that concisely walks through deploying the platform to AWS in a production-appropriate configuration?

I’m having trouble deciding between “raw” EC2 or maybe using Elastic Beanstalk for some components… I’m extremely motivated to reduce the cognitive load involved in ops for my company’s systems, so I’d prefer to use something as high-level as possible. I’m hoping to use Terraform to define and provision our AWS infrastructure but that doesn’t actually help with any decisions. I’m open to using Docker containers with either Elastic Beanstalk or EC2 Container Service, but the Confluent repo containing dockerfiles says that they are not production-ready. I’d love to use a predefined CloudFormation stack (or templates?) even though I’ve been underwhelmed by CF in the past but I haven’t found one.

Thanks!

Avi

[Ben Davison]

Hi Avi,

We just went through this - some quick notes before I leave for the day.

1) Don't use elastic beanstalk for kafka and zookeeper, elastic beanstalk is mainly used for stateless services (and we also ran into an issue where we wanted to use d2.xlarge for space reasons and it didn't exist)

We use opsworks, which I feel is a nice level of just running a raw Ec2 instance and having something inflexible like EB.

We also use cloudformation(using troposphere) to define the opsworks stack, which makes it really easy to move it about.

Thanks,

Ben

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/1dbf6399-a158-43c6-ba1e-932d04dde3b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Avi Flax]

On Wednesday, April 20, 2016 at 1:22:29 PM UTC-4, Ben Davison wrote:

 

We just went through this - some quick notes before I leave for the day.

Thanks Ben! I really appreciate it!

 

1) Don't use elastic beanstalk for kafka and zookeeper, elastic beanstalk is mainly used for stateless services (and we also ran into an issue where we wanted to use d2.xlarge for space reasons and it didn't exist)

That makes sense. I think that’s actually what I had in mind — to use Beanstalk just for the Registry — but now that I think about it, it’s too much cognitive load to learn Beanstalk just for that one component (i.e. insufficient uniformity).

 

We use opsworks, which I feel is a nice level of just running a raw Ec2 instance and having something inflexible like EB.

Cool, I’ll check it out. I haven’t been thrilled with Chef in the past, but maybe Opsworks smooths over the rough parts.

We also use cloudformation(using troposphere) to define the opsworks stack, which makes it really easy to move it about.

Very interesting, I’ll check that out too.

Thanks so much!

Avi 

[Liquan Pei]

Hi Avi,

There is a blog post on this http://www.confluent.io/blog/deploying-apache-kafka-on-aws-elastic-block-store-ebs

Thanks,

Liquan

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4ec574a9-ace0-42fb-9e6c-9bc311b71400%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Liquan Pei | Software Engineer | Confluent | +1 413.230.6855

Download Apache Kafka and Confluent Platform: www.confluent.io/download

[Avi Flax]

On 4/22/16, 00:53, "confluent...@googlegroups.com on behalf of Liquan Pei" <confluent...@googlegroups.com on behalf of liq...@confluent.io> wrote:

>There is a blog post on this http://www.confluent.io/blog/deploying-apache-kafka-on-aws-elastic-block-store-ebs

Thanks Liquan. I’ve read that and it’s definitely interesting and relevant. That said, it’s far from a guide, and that’s what I’m hoping to find.

Maybe I’ll have more luck asking about this on the Kafka mailing list — for me, ZK and Kafka are the most challenging components to deploy; the Schema Registry is a style of app that I’m very familiar with.

Thanks,
Avi

[Hemant Puri]

Any luck in finding a Cloudformation scripts for running Kafka cluster either on EC2 instance or as containers managed by ECS?

[Avi Flax]

> On Sep 17, 2016, at 14:32, Hemant Puri <heman...@gmail.com> wrote:
>
> Any luck in finding a Cloudformation scripts for running Kafka cluster either on EC2 instance or as containers managed by ECS?

Nope. I eventually wrote my own Terraform modules for this. They’re working fairly well for me. (Terraform is fantastic, but it doesn’t hold your hand, and it doesn’t cover configuration management.) I’d like to try to release them as open source if I can find the time.

Good luck!

————
Software Architect @ Park Assist
We’re hiring!

[Guillaume Simard]

Did you ever get to open source it Avi Flax?

We could use a starting point for Terraforming Kafka in AWS.

Thanks!

[Robin Moffatt]

You've seen this, right? https://aws.amazon.com/quickstart/architecture/confluent-platform/

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/7c5fe27d-43e5-44be-ab51-bbdf9ba271a4%40googlegroups.com.

[Avi Flax]

> On Jul 18, 2017, at 13:18, Guillaume Simard <gui...@gmail.com> wrote:
>
> Did you ever get to open source it Avi Flax?
> We could use a starting point for Terraforming Kafka in AWS.

Sorry, never got around to it.

And honestly, I think it’s probably a good thing. The Kafka cluster I’ve provisioned with Terraform is up and running and working for my firm, but it’s messy — it’s not modularized, it doesn’t use auto-scaling groups, the schema registry is not in a HA configuration, configuration is via brittle inline shell-script provisioning, etc.

At this point, personally I’m very much looking forward to Confluent’s cloud offering. I signed up a few minutes after it was announced (I was in the room) but not so much as a peep yet 🙁.

Apart from that, if I needed to launch a cluster on AWS right now, I’d probably either use the pre-packaged “quickstart” that uses CloudFormation, as Robin linked to, or I’d look into running the Docker images[1] on ECS or Kubernetes.

Good luck!

Avi

[1] https://github.com/confluentinc/cp-docker-images