Connection reset with Hive

[sp]

Hi,

I see the following exception when running my Hive query on an almost
empty table running on a 6 node cluster. It works fine on a single
node cluster though.

java.io.IOException: Could not get input splits
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:
157)
at
org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat.getSplits(HiveCassandraStandardColumnInputFormat.java:
320)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:
286)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient.access
$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:
838)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:
791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1059)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:
791)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:
765)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:657)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:
130)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:
1063)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:
456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:
153)
... 27 more
Caused by: java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:
227)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access
$200(ColumnFamilyInputFormat.java:70)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat
$SplitCallable.call(ColumnFamilyInputFormat.java:190)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat
$SplitCallable.call(ColumnFamilyInputFormat.java:175)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:
129)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:
129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:
101)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:
378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:
297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:
204)
at org.apache.cassandra.thrift.Cassandra
$Client.recv_set_keyspace(Cassandra.java:471)
at org.apache.cassandra.thrift.Cassandra
$Client.set_keyspace(Cassandra.java:456)
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:
218)
... 8 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:
127)
... 18 more
Job Submission failed with exception 'java.io.IOException(Could not
get input splits)'
ERROR 17:07:09,270 Job Submission failed with exception
'java.io.IOException(Could not get input splits)'
java.io.IOException: Could not get input splits
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:
157)
at
org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat.getSplits(HiveCassandraStandardColumnInputFormat.java:
320)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:
286)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient.access
$500(JobClient.java:170)
at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient.access
$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:
838)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:
791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1059)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:
791)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:
765)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:657)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:
130)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:
1063)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:
456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:
153)
... 27 more
Caused by: java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:
227)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access
$200(ColumnFamilyInputFormat.java:70)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat
$SplitCallable.call(ColumnFamilyInputFormat.java:190)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat
$SplitCallable.call(ColumnFamilyInputFormat.java:175)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:
129)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:
129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:
101)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:
378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:
297)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:
127)
... 18 more

My cassandra cluster is healthy and I have a replication factor of 1.
Anyone come across this problem?

Many thanks.

[Cathy Daw]

This issue was introduced with the upgrade from C* 0.8.2 which we shipped with Brisk to C* 0.8.4.

There are numerous references to this exception which we believed will be fixed by upgrading to C* 0.8.5 and changing rpc_address to listen_address.  Some users still see the issue with this remedy so we will continue testing this today and update you with findings.

[sp]

THanks for the update. I am currently using 8.2. Also, I see the
error when I map external tabel to a column family. It works fine if
I just create a Hive table with no mapping to column family.