Client, Protocol and new API prototype

[Tim Fox]

Hello All,

As mentioned in https://groups.google.com/forum/#!topic/ksql-dev/yxcRlsOsNmo I have been working on a prototype for a new client, protocol and network API for ksqlDB. This experiment has been driven by attempting to write an application. The idea is that only by writing a real(ish) app can we know where the pain points are when writing apps for ksqlDB. 

I started by writing a simple app using an event sourcing approach - a shopping application where you can view items, add them to your basket, place orders etc. Event sourcing (and DDD) is probably the most common way of modelling business applications using events. If it’s not easy to write such an application using ksqlDB then i think we have a problem.

# The client

I’ve created a new client - the idea is that an application should get everything it needs from this client. It should:

• Support both direct and async interaction styles.
• Support reactive streams / flow API (well.. A copy of flow as we don’t support Java 9 :( )
• Support streaming of pull and push queries.
• Support inserting of rows.
• Support execution of DDL (not implemented yet)
• Support other ops such as list/describe etc (not implemented yet)
• Runs over Websockets and TCP (only Websockets implemented currently)
• Can be used from a browser
• Deliberately simple API

You can see the client classes here:

https://github.com/purplefox/ksql/tree/api_client_proto/ksql-api/src/main/java/io/confluent/ksql/api/client

You can see some usage examples of the client here:

https://github.com/purplefox/ksql/blob/api_client_proto/ksql-api/src/test/java/io/confluent/ksql/api/Examples.java

# The protocol

I’ve created a new simple binary protocol for communication between client and server.

• Multiplexed protocol over Websockets or TCP
• Designed for high performance streaming of data
• Also supports request/response semantics
• Supports flow control / back pressure
• Deliberately super simple

You can see the protocol here:

https://github.com/purplefox/ksql/blob/api_client_proto/ksql-api/src/main/java/io/confluent/ksql/api/protocol/WireProtocol.java

The basic idea is there are multiple channels multiplexed over each connection. Each channel represents a stream of data (either client to server or server to client). Each query is streamed over its own channel. Each request/response interaction also occurs over its own channel. Channels use a TCP like window based flow control approach.

# The server side

I’ve created a new API server implemented using Vert.x - this serves protocol connections and plugs in to existing server side functionality.

https://github.com/purplefox/ksql/tree/api_client_proto/ksql-api/src/main/java/io/confluent/ksql/api/server

Currently only pull/push queries and inserts are implemented, but we can support everything in the current “REST” API, at which point we can turn that off.  For simple request/response interactions (e.g. describe table) we can also allow clients to interact over HTTP if they prefer, this would be quite easy to do. 

The integration with current server side functionality is here:

https://github.com/purplefox/ksql/tree/api_client_proto/ksql-rest-app/src/main/java/io/confluent/ksql/rest/server/apiactions

# The application

I’ve created a simple example application, using Vert.x:

https://github.com/purplefox/ksql/blob/api_client_proto/ksql-real-app/src/main/java/io/confluent/retail/RetailApp.java

To run it, run the Starter class in your IDE: https://github.com/purplefox/ksql/blob/api_client_proto/ksql-real-app/src/main/java/io/confluent/server/Starter.java then open your browser to http://localhost:8080.

It’s a shopping app, using event sourcing principles. It uses a few different topics and a couple of aggregations. The application is a single class written using Vert.x. The application uses push queries, pull queries and inserts. You can see how simple usage of the client is. The app is small but does quite a lot:

• Serves and renders a catalogue page showing list of items to buy - this uses a pull query.
• Displays the users current shopping basket using a push query which pushes all the way out to the browser using websockets where it is rendered.
• Allows user to add items to basket  - causes insert into a topic.
• Allows user to place an order - does a pull query to get current basket and inserts into topic.
• Displays count of current orders (push query to browser using websockets).

The example currently works against a Fake ksqlDB backend as it requires a pull query with more than just a point lookup, which we don’t currently support.

Everything is currently displayed in a current page using frames for simplicity (the goal of this is not to demonstrate good web design! ;) )

https://github.com/purplefox/ksql/pull/11 (this is not meant to be merged, just to show the files)

Enjoy!

[Tim Fox]

• Can be used from a browser

This was a bit unclear - we should aim to write a JS client at some point, which could be used from browser as it can use WebSockets.