Streaming ETL using Kafka for Network Rail data

[UEA Network Rail Study]

https://github.com/confluentinc/demo-scene/tree/master/rail-data-streaming-pipeline

Have not tried this out yet, but looks pretty useful

[Christopher Stafford]

We're using Kafka internally as an initial dump for the incoming data streams before they are handled by the various services, and once you've got round some of the more tedious configuration hurdles it makes things flow more easily and makes development a lot easier. 

Our setup is a lot less ambitious than this one though. Definitely worth a look

[Robin Moffatt]

Thanks, I was just about to come and post the links myself :) 

I presented this at a meetup in Germany this week. You can find the slides here: http://rmoff.dev/kafka-trains-slides-01. The code is as linked to below. 

Thanks to people on this list who helped me out with my initial ActiveMQ problems, it was worth it. 

If anyone has any questions about what I've built I am happy to answer them. 

On Friday, July 12, 2019 at 12:03:46 PM UTC+1, UEA Network Rail Study wrote:

[Christopher Stafford]

That's a really interesting and impressive set of slides. Definitely worth looking at if you're starting a project from scratch.

[Chris Bailiss]

Hi Robin

Interesting slides, thanks for sharing.  There is a fair amount of technology in there!

May I ask what the objective of the work was?  (Apologies if I missed this in the slides).  Is it running permanently online? (publicly?).  Or was it a demonstrator / built to meet the requirements of a customer?

Thanks

Chris

[Robin Moffatt]

Hi Chris, 

It is running permanently, on my home server. 

I built it as a proof of concept really, to see what could be done around Apache Kafka and streaming data. 

thanks, Robin. 

--
You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/aJ6n509OLO8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/2db959f7-de92-4169-bc80-b4d580a34a9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.