I am trying to write a part file compactor in Cascading which reads in thousands of avro files and reduce the number of files to 50. Output has the same schema as input. Avro schema contains a map field. I am getting the following error when I run the compaction:
"error":"Error: cascading.CascadingException: unable to compare stream elements in position: 0
cascading.tuple.hadoop.util.DeserializerComparator.compareUnTypedTuples(DeserializerComparator.java:239)
cascading.tuple.hadoop.util.DeserializerComparator.compareTuples(DeserializerComparator.java:194)
cascading.tuple.hadoop.util.BaseTupleComparator$StreamComparison.compare(BaseTupleComparator.java:97)
cascading.tuple.hadoop.util.BaseTupleComparator.compare(BaseTupleComparator.java:74)
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1265)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:126)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:129)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:129)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:126)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:126)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:126)
org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:126)
org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1593)
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1635)
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160)
Caused by: shaded.org.apache.avro.AvroRuntimeException: Can't compare maps!
shaded.org.apache.avro.generic.GenericData.compare(GenericData.java:984)
shaded.org.apache.avro.specific.SpecificData.compare(SpecificData.java:333)
shaded.org.apache.avro.generic.GenericData.compare(GenericData.java:988)
shaded.org.apache.avro.specific.SpecificData.compare(SpecificData.java:333)
shaded.org.apache.avro.generic.GenericData.compare(GenericData.java:961)
shaded.org.apache.avro.specific.SpecificData.compare(SpecificData.java:333)
shaded.org.apache.avro.generic.GenericData.compare(GenericData.java:946)
shaded.org.apache.avro.specific.SpecificRecordBase.compareTo(SpecificRecordBase.java:81)
shaded.org.apache.avro.specific.SpecificRecordBase.compareTo(SpecificRecordBase.java:30)
cascading.tuple.hadoop.util.TupleElementComparator$1.compare(TupleElementComparator.java:49)
cascading.tuple.hadoop.util.TupleElementComparator$1.compare(TupleElementComparator.java:36)
cascading.tuple.hadoop.util.TupleElementComparator.compare(TupleElementComparator.java:87)