cannot find field sitecode

32 views

Skip to first unread message

Josh Bronson

unread,

Apr 29, 2013, 7:33:43 PM4/29/13

to elephant...@googlegroups.com

I'm using Hive 0.7.1+42.27-2~maverick-cdh3 and Hadoop 0.20.2+923.142-1~maverick-cdh3 from Cloudera's CDH3u2 distribution.

I'm running a query against a Hive table created with the following command:

CREATE TABLE IF NOT EXISTS `weblogs`

PARTITIONED BY (day string)

ROW FORMAT SERDE "com.twitter.elephantbird.hive.serde.ProtobufDeserializer"

WITH SERDEPROPERTIES ("serialization.class"="<snip>.weblog.WebLogRecordProtobuf$WebLogRecord")

STORED AS

INPUTFORMAT "com.twitter.elephantbird.mapred.input.DeprecatedRawMultiInputFormat"

OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"

LOCATION "/data/rebin/final";

I populated the table by creating partitions

ALTER TABLE `weblogs` ADD PARTITION (DT = '201X-XX-XX') location '201XXXXX';

and copying files into those subdirectories manually.

Here's the query:

SELECT count(SiteCode) AS cnt FROM weblogs WHERE day="2013-04-22" SORT BY cnt DESC LIMIT 1;

Things worked fine when the table was not partitioned, but now I'm seeing this:

2013-04-29 23:16:23,729 WARN org.apache.hadoop.mapred.Child: Error running child

java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)

at org.apache.hadoop.mapred.Child.main(Child.java:264)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 9 more

Caused by: java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)

... 14 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 17 more

Caused by: java.lang.RuntimeException: Map operator initialization failed

at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)

... 22 more

Caused by: java.lang.RuntimeException: cannot find field sitecode from [org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@79f03d7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@58b51c29, org.apache.hado

at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:321)

at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldRef(UnionStructObjectInspector.java:96)

at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904)

at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)

at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)

at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:73)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)

at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)

at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)

... 22 more

2013-04-29 23:16:23,733 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Any ideas?

Thanks,

Josh Bronson

unread,

Apr 30, 2013, 12:45:37 PM4/30/13

to elephant...@googlegroups.com

Oof! My apologies, I should have dug further before I posted this. It looks like I ran into the same issue mentioned here: https://groups.google.com/forum/#!topic/elephantbird-dev/iVoI55fJIhI

Changing all of the column names to lowercase in my protobuf model appears to have resolved the issue.

Sorry,

Josh Bronson

--
You received this message because you are subscribed to a topic in the Google Groups "elephantbird-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elephantbird-dev/eGIQPrckjIs/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to elephantbird-d...@googlegroups.com.
To post to this group, send email to elephant...@googlegroups.com.
Visit this group at http://groups.google.com/group/elephantbird-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Reply all

Reply to author

Forward

0 new messages