I have an avro file that have the following definition.
public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"UserProfileFlatAggregate\",\"namespace\":\"
com.bluekai.analytics.jobs\",\"fields\":[{\"name\":\"uuid\",\"type\":\"string\"},{\"name\":\"profile\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"SiteCatNsTuple\",\"fields\":[{\"name\":\"site\",\"type\":\"int\"},{\"name\":\"ns\",\"type\":\"int\"},{\"name\":\"cat\",\"type\":\"int\"}]}},\"aliases\":[\"bag\"]}]}");
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
@Deprecated public java.lang.CharSequence uuid;
@Deprecated public java.util.List<com.bluekai.analytics.jobs.SiteCatNsTuple> profile;
in my code I am trying to flatten the array and generate uuid, cat pairs.
avroPipe = UnpackedAvroSource[UserProfileFlatAggregate](args("input")).read
.flatMapTo('profile -> ('uuid, 'cat)){
userProfile: UserProfileFlatAggregate =>
val uuid = userProfile.getUuid
val profile = userProfile.getProfile
for(t <- profile.asScala) yield (uuid, t.getCat)
}
this gives the following exception.
Caused by: cascading.pipe.OperatorException: [com.twitter.scalding.a...][com.twitter.scalding.RichPipe.eachTo(RichPipe.scala:480)] operator Each failed executing operation
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
at cascading.flow.stream.SourceStage.call(SourceStage.java:53)
at cascading.flow.stream.SourceStage.call(SourceStage.java:38)
... 4 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to com.bluekai.analytics.jobs.UserProfileFlatAggregate
at com.bluekai.ds.Categories$$anonfun$4.apply(CategorisJob.scala:36)
at com.twitter.scalding.FlatMapFunction.operate(Operations.scala:39)
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
... 8 more
I am wondering what is the proper way to flatten that array.
thanks,