Issue: Could only be replicated to 0 nodes instead of minReplication (=1)

[Peer Jakobsen]

Hi,

We just started to try to set up Snowplow Analytics using the documentation

When we run the EmrEtlRunner in 80% of the cases the job "Elasticity S3DistCp Step: Raw S3 -> HDFS" is going to fail because it tries to copy S3 files to HDFS before there are any available data nodes. 

Here is part of the error message

File /tmp/4ce15ab9-a5f3-42d0-91fa-af584d8b8509/files/1 could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

Here is the parameters we run the emr-etl-runner with:
./deploy/snowplow-emr-etl-runner --config config/config.yml --resolver config/iglu_resolver.json --enrichments config/enrichments

We are using m1.xlarge instances and we are using AMI version 3.11.0 because AWS support told us that we would never be able to get 3.7.0 working with our VPC setup.

Any hints to what we might be doing wrong? It's intermittent and actually does work 20% of the runs.

Thanks.

[Peer Jakobsen]

Btw: Here is AWS telling us that we can't use AMI version 3.7.0

https://forums.aws.amazon.com/message.jspa?messageID=695983#695983

[Alex Dean]

Hi Peer,

Thanks for raising. We have been stuck on AMI 3.7.0 for a while due to an extremely difficult-to-track down bug in the JRE version used by AMI 3.8.0+ (and indeed all of the AMI 4.0.0 series). You can read more about the bug here (it's basically a java.net.URL serialisation bug):

http://stackoverflow.com/questions/34717583/why-does-java-net-url-tostring-throw-a-nullpointerexception-on-emr-ami-3-8-0

Now that we have fixed this, we are preparing a release, R77 Great Auk, which will move the Snowplow pipeline up to the AMI 4.0.0 series. This release should be out within the next fortnight.

https://github.com/snowplow/snowplow/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+77+[CLI]+Great+Auk%22

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean

[Peer Jakobsen]

Hi Alex,

Thanks for the update. I am looking forward to trying out R77 then. 

Is that release then going to fix the issue with the "Elasticity S3DistCp Step: Raw S3 -> HDFS" job failing?

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "Snowplow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/snowplow-user/Pb-gOrL0hos/unsubscribe.
To unsubscribe from this group and all its topics, send an email to snowplow-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kind Regards

Peer Jakobsen

CTO

 

MovieStarPlanet

Amagerfælledvej 56, 1 (2nd floor)

2300 Copenhagen, Denmark

Phone: +45 51915526

Skype: pj_moviestarplanet

Web:  http://corporate.MovieStarPlanet.com

[Alex Dean]

I can't say for sure because we have never seen that issue (EMR job flow steps starting before the cluster has provisioned), but R77 is certainly necessary for running Snowplow on a 4.x.x series AMI...

A