How can we connect Amazon Kinesis Streams to PostgreSQL?

[Yasho Vardhan]

Hi All,

We are working on an implementation of Snowplow Analytics with Scala Stream Collector and are facing some issues in storing the data in PostgreSQL database.

 

We have set up the Amazon Kinesis pipeline. The Collector and Enricher are both running. How we can put this data in PostgreSQL database?

https://github.com/snowplow/snowplow/wiki/Setting-up-PostgreSQL

The article above talks mainly about loading data from EMRETL Runner. Is there any documentation we can refer to for Scala Kinesis Enrich?

Thanks.

[Fred Blundun]

Hi Yasho,

We don't currently support writing to Postgres from Kinesis. We do support writing to Elasticsearch using the Elasticsearch Sink. We also have a PR for writing to Redshift from Kinesis.

Any contributions in this area would be very welcome!

Regards,

Fred

[Alex Dean]

Just to add to Fred's email -

To write to Redshift or Postgres currently from the Kinesis pipeline, you setup a lambda architecture, whereby the Kinesis S3 component writes the raw events (from the Scala Stream Collector) to S3, and then the regular Snowplow batch pipeline processes them through to Redshift or Postgres.

As Fred says, we want to support real-time dripfeeding of events into Redshift (and Postgres and other relational storage targets), but there is a lot of upfront work to do first on Iglu (our schema repository system) to prepare for this.

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean