An weird exception happened to my camus job

[Jerry MJ]

Hi 

I've been using CAMUS for a while and it was running very smoothly.  Recently, we have a weird issue.  From time to time, my avro file download fails due to noroutetohostexception.  My job ended up with the following problem.  If anyone has any idea, please let me know.

topic:prod-eg_v3_2-big_data_v0.1.321-2c83c partition:1 beginOffset:14153364 estimatedLastOffset:14226894
[KafkaReader] - bufferSize=1048576
[KafkaReader] - timeout=30000
[KafkaReader] - Connected to leader tcp://71.74.46.26:9092 beginning reading at offset 14153364 latest offset=14226894
[KafkaReader] - Exception generated during fetch
[KafkaReader] - Exception generated during fetch

......

java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Unknown Source)
	at sun.nio.ch.Net.connect(Unknown Source)
	at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
	at kafka.network.BlockingChannel.connect(Unknown Source)
	at kafka.consumer.SimpleConsumer.connect(Unknown Source)
	at kafka.consumer.SimpleConsumer.getOrMakeConnection(Unknown Source)
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source)
	at kafka.metrics.KafkaTimer.time(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source)
	at kafka.metrics.KafkaTimer.time(Unknown Source)
	at kafka.consumer.SimpleConsumer.fetch(Unknown Source)
	at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source)
	at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source)
	at com.linkedin.camus.etl.kafka.common.KafkaReader.fetch(KafkaReader.java:167)
	at com.linkedin.camus.etl.kafka.common.KafkaReader.<init>(KafkaReader.java:81)
	at com.linkedin.camus.etl.kafka.mapred.EtlRecordReader.nextKeyValue(EtlRecordReader.java:216)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:512)
	at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:351)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:282)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
	at org.apache.hadoop.mapred.Child.main(Child.java:271)

[yash.r...@lookout.com]

Looks like a network problem to me.