Selling kafka managed instance

[Nicolas Verhaert]

Hi everyone, I am new to kafka, and I am defending a green field microservice architecture against a reseller of azure. They are pushing us to ESB, and app services. Which is not suited/ future proof in my opinion. My customer develops software and has a very heavy monolithic architecture that we are trying to make as agile as possible. I am hoping that you can help me in providing solid resources to proove why kafka is becomming the defacto standard. Can you please help me to sell kafka as the most future proof choice of technology?

[Alex Nogueira]

Hi,

Can you send more details about the situation?

All depends on as you are at this moment.

One application? Many integrations? How is the architecture distributed today? 

Thanks,

Em qua., 7 de abr. de 2021 às 09:03, Nicolas Verhaert <nic...@lminds.com> escreveu:

Hi everyone, I am new to kafka, and I am defending a green field microservice architecture against a reseller of azure. They are pushing us to ESB, and app services. Which is not suited/ future proof in my opinion. My customer develops software and has a very heavy monolithic architecture that we are trying to make as agile as possible. I am hoping that you can help me in providing solid resources to proove why kafka is becomming the defacto standard. Can you please help me to sell kafka as the most future proof choice of technology?

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/1c6f4a17-174f-47d1-9c90-1e910253c569n%40googlegroups.com.

--

Alex Nogueira
34 9661 1093

[Nicolas Verhaert]

Alex,

First of all, thank you for your reply...

I'll try to explain best as possible. We are handling sensitive data about users, payment info, so gdpr compiancy will be needed in the near future. The main focus of the company is software delivery. They have around 400 employees. Their architecture hasn't been changed since 20 years. So no contrainerization, custom release pipelines (very heavily scripted), ... .

AS-IS:

On-premise @Customers:

Sharded architecture with on premise cobol installations (no mainframe), fronted by java api's that trigger cobol files in the on premise installation (api also on prem for every installation, no centralization here at this point).

Software has several fronts (different products, but all written in java with the one cobol as a template parser)

Centralized: other aquired softwares (different languages and topologies, mostly ionic app, php web application, with mysql datastore). Have their own security implementations. Integrations with kafka connectors would fit these perfectly I think.

Issues:

Org limited to 4 releases / year

Lots of analysis required for small changes (even something trivial as making a field visible to certain roles)

Not fault tolerant

No CI/CD (release flow very cumbersome)

Hard coded roles (changing permissions, or even sending additional email is a new release)

Very heavy to integrate with external parties, like planning softwares, sap, ...

Very hard server management (there is no containerization used, this affects deploys (15 days of work for 1 release), for every new library needed from software, technology needs to install this on the server, which makes the servers hard to upgrade)

Tightly coupled between infrastructure/technology and software delivery

No observability at application level

Own data center overallocated

Employees overworked

TO-BE:

We were contacted to build a mobile application on a green field whilst building a microservice architecture that would fit the organisation. In this architecture we focussed on loosly coupling. To achieve this we worked out a definition of a microservice that we think achieves this goal and strongly limits complexities around security, scaling and eventing. This definition is Kafka centric. The goal is developing a SDK so the focus when developing microservices is (almost) all business requirements oriented.

In this definition we use kafka for the following elements:

     - Kafka is used for in memory tables for example: online users, it will load this from a stream and/or snapshot if that is available

     - Business can configure roles and permissions. Kafka is used to propagate changes to permissions/roles to the respective microservices.

    - SDK provides abstraction for all CRUD in the organisation. All models are dynamically provided with swagger, api's and security based on roles and permissions, these abstraction create topics for every object in the architecture and events crud related events to these topics.

    - Background tasks like writing last seen dates to the userstore are controlled by a process that limits this to one instance when scaled (to prevent duplicate calls to db, and prevent buggy jobs to consume more than one instance). Kafka is used here for this.This is also used to determine which instance will create the snapshots (shared disk)

    - We use kafka admin api, to enable microservices to create their own topics when needed

All this functionality is abstracted into the SDK with very little overhead to developers. It should provide loose coupling so the organisation can benefit from the microservice architecture:

- Less analysis

- Less development time

- Loose coupling between infrastructure and delivery

- Less releases

- Better time to market

- Less complexity for developers

- Highly observable application layer

- Better reuse-ability of services

Problems with technolgy provided by the external party:

- App services is limited (I dont think it can run background threads at all? I dont want to do eventing via http triggers, It seems very logical that a threaded consumer should be possible)

- No possibility to connect a client of any sorts (kafka / esb) this is achieved by http triggers on an api level

- Cost (ESB is very expensive, if all microservices use multiple subscribers, and all is evented from the application would add up to 5x the cost (if I use the azure calculator correctly), development cost would be much higher if we not achieve loose coupling / autonomy on software level)

- Loose coupling: The inner working of an ESB is not suited for this kind of intens eventing (microservices subscribing to same topics)

- Vendor lock in (I want to guard that my client will have control about their own releases/incidents without having to go past an external party)

- Kubernetes / Docker vs App services:  provides a better platform which enables them to lift and shift dockerized applications to prevent overallocation of own datacenter, it also provides the multithreading we need to build reliable/scalable microservices

- Future proofing: Kafka seems the best and most flexible way forward since design patterns for ESB are possible in kafka also. It provides easy integration with external parties, because of kafka connectors we can even integrate without changing code or planning releases which would be beneficial for all parties.

Long term:

- Transactionality with eventstore

- Bulkheading

- Circuit breakers

Arguments against:

- We use kafka for something it is not meant to do (?!)

- ESB is the leading standard

I want to push for a managed instance with confluence, to eliminate any worries about running kafka in production or is event hub from Azure enough? But I need to convince them to sign of on the architecture in the first place. All help is appreciated.

Kind regards

Nicolas

  

Op woensdag 7 april 2021 om 14:38:05 UTC+2 schreef alex.eustaq...@gmail.com: