Kinesis Enrich - Caught exception when initializing LeaseCoordinator

[Daniel Zohar]

Hi guys,

I'm trying to run the Kinesis Enrichment JAR with basic configuration and the following command (the JAR is from your hosted assets) :

./snowplow-kinesis-enrich-0.5.0 --config enrich.conf --resolver resolver.json

The output of the program:

[main] INFO com.snowplowanalytics.snowplow.enrich.kinesis.sinks.KinesisSink - Stream snowplow_enriched exists and is active

[main] INFO com.snowplowanalytics.snowplow.enrich.kinesis.sinks.KinesisSink - Stream snowplow_bad exists and is active

[main] INFO com.snowplowanalytics.snowplow.enrich.kinesis.sources.KinesisSource - Using workerId: localhost:09368057-fa23-49b8-b8a6-257258b7fce7

[main] INFO com.snowplowanalytics.snowplow.enrich.kinesis.sources.KinesisSource - Running: XXXXX.

[main] INFO com.snowplowanalytics.snowplow.enrich.kinesis.sources.KinesisSource - Processing raw input stream: snowplow_raw

[main] INFO com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - With failover time 10000ms and epsilon 25ms, LeaseCoordinator will renew leases every 3308ms and take leases every 20050ms

[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for both region name as us-east-1, and Amazon Kinesis endpoint as https://kinesis.us-east-1.amazonaws.com. Amazon Kinesis endpoint will overwrite region name.

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 1

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator

[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator....
....

Seems like the errors are coming from the AWS Kinesis library. Here is where I suspect it's coming from (code on Github).

It continues to retry until it fails after 20 attempts. Any idea what's wrong? 

As an aside, it would have been very helpful if the app used Slf4jLogger like the Collector so the actual errors would have been visible and would also allow to see debug messages.

Thanks!

[Fred Blundun]

Hi Daniel,

It is actually possible to configure logging for Scala Kinesis Enrich by running the jar file like this:

java -jar -Dorg.slf4j.simpleLogger.defaultLogLevel=debug snowplow-kinesis-enrich-0.5.0 --config enrich.conf --resolver resolver.json

This should print all debug messages from the Kinesis Client Library. (Watch out because the output will become very verbose.) Could you try rerunning with this change to logging? Hopefully that will provide more clues about what's going wrong.

Regards,

Fred

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Zohar]

Thanks Fred! 

Apparently I was missing DynamoDB access in the IAM policy. Something worth adding to the docs :)

Cheers

[Fred Blundun]

Hi Daniel,

Glad you worked out what was going wrong! I have guidance on configuring logging to the Kinesis Enrich wiki page (https://github.com/snowplow/snowplow/wiki/Run-Scala-Kinesis-Enrich#configuring-the-log-level).

Regards,

Fred

[Yuval Herziger]

I've been experiencing the same issue, thanks for the info Daniel.

-Yuval