Google Groups

Re: [kafka-dev] Re: Run kafka accross multiple datacenters


eonnen Apr 6, 2011 9:49 AM
Posted in group: Kafka
I looked at this as an option. The biggest hurdle seemed to be
integrating the SSL handshake and SSLEngine into the lower-level NIO
layers. Netty makes that 100% easier but I figured adding Netty to the
stack wasn't an option.

On Wed, Apr 6, 2011 at 9:11 AM, Jay Kreps <jay....@gmail.com> wrote:
> (good) patches accepted, of course.
>
> -Jay
>
> On Wed, Apr 6, 2011 at 8:54 AM, Jun Rao <jun...@gmail.com> wrote:
>> Chaker,
>> Currently, SSL support is not in our near term roadmap. For sensitive data,
>> we recommend encryption at the application layer.
>> However, if there are enough interests in SSL support, we can look into that
>> in the future.
>> Jun
>>
>> On Wed, Apr 6, 2011 at 12:27 AM, Chaker Nakhli
>> <Chaker...@javageneration.com> wrote:
>>>
>>> Yes, SSL tunneling would solve the problem. Imho, this is a key
>>> feature as many companies have their infrastructure split over
>>> multiple data centers.
>>>
>>> Do you have an idea on when this feature will be shipped?
>>>
>>> Thank you
>>> --
>>> Chaker Nakhli
>>>
>>> On Apr 6, 12:32 am, Erik Onnen <eon...@gmail.com> wrote:
>>> > I'm actively looking at how best to do this. The leading candidate is
>>> > to have a consumer in one facility push to the second Kafka server
>>> > over an STunnel channel although I'm open to other suggestions.
>>> >
>>> > On Tue, Apr 5, 2011 at 1:30 PM, Chaker Nakhli
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > <chaker.nak...@javageneration.com> wrote:
>>> > > Thank you for your quick answer. This sounds like a good idea.
>>> >
>>> > > I'm still worried about security and data privacy as both consumer and
>>> > > producers will be placed in dmzs and will be establishing connections
>>> > > over
>>> > > the internet. Is it possible to use a secure protocol (eg https)
>>> > > natively?
>>> > > Or as a tunnel?
>>> >
>>> > > Thanks,
>>> > > --
>>> > > Chaker Nakhli
>>> >
>>> > > Jun Rao <jun...@gmail.com> wrote:
>>> >
>>> > >> One approach is to set up 1 kafka instance per producer site and have
>>> > >> another kafka instance on the consumer site. The latter will be
>>> > >> pulling all
>>> > >> messages from kafka instances from the producer sites. The consumer
>>> > >> can then
>>> > >> consume from the kafka instance on the consumer site. Currently, our
>>> > >> kafka
>>> > >> server supports emdedded consumers pulling data from a remote kafka
>>> > >> instance. It's possible to extend that to support pulling from
>>> > >> multiple
>>> > >> remote kafka instances.
>>> >
>>> > >> Thanks,
>>> >
>>> > >> Jun
>>> >
>>> > >> On Tue, Apr 5, 2011 at 10:20 AM, Chaker Nakhli
>>> > >> <Chaker.Nak...@javageneration.com> wrote:
>>> >
>>> > >>> Hi all,
>>> >
>>> > >>> I'd like to deploy kafka with several producers spread accross
>>> > >>> different datacenters and all consumers in one central datacenter. I
>>> > >>> am mainly interested in live continuous processing.
>>> >
>>> > >>> What is the best architecture to achieve this?
>>> > >>> I suppose I should put the kafka server in a machine in my consumers
>>> > >>> network dmz. But I am worried about security for both internet/dmz
>>> > >>> and
>>> > >>> dmz/lan connections.
>>> >
>>> > >>> Your suggestions welcome!
>>> >
>>> > >>> Cheers,
>>> >
>>> > >>> Chaker
>>
>