[scalding] error in serialization

[Ittay Dror]

java.lang.IllegalAccessException: Class com.twitter.chill.Instantiators$$anonfun$normalJava$1 can not access a member of class org.apache.avro.Schema$BytesSchema with modifiers "public"

My job class contains:
  override def ioSerializations =
    super.ioSerializations :+ classOf[cascading.avro.serialization.AvroSpecificRecordSerialization[GenericRecord]]

This appeared to have happened after mapping some values, which were ByteWritable (which is the values you get reading a byte array field) into String. When the pipe carried these values as is, the error did not manifest.

Regards,
Ittay

[Christopher Severs]

Hi Ittay,

Can you post a stripped down example where this happens so I can have a look at what is going on? I suspect the scalding-avro implicits are trying to serialize the byteswritable stuff when it shouldn't be.

Thanks,
Chris

[Ittay Dror]

i tried, but as i wrote, it manifested itself after an unrelated change, so once i changed things it went away.

the culprit was an inner object in my job that wrapped UnpackedAvroSource. Once I moved it to its own file, serialization worked

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/gFZ2KTSXLSU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/4f611bde-c8fe-4ab6-ac0b-af8c5f298e98%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[sourabh chaki]

I am also getting similar error, but not at org.apache.avro.Schema$BytesSchema.Here it is trying to access sun.nio.cs.UTF_8.

Caused by: java.lang.IllegalAccessException: Class com.twitter.chill.Instantiators$$anonfun$normalJava$1 can not access a member of class sun.nio.cs.UTF_8 with modifiers "public"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:261)
	at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:253)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:517)
	at com.twitter.chill.Instantiators$$anonfun$normalJava$1.apply(KryoBase.scala:161)
	at com.twitter.chill.Instantiators$$anon$1.newInstance(KryoBase.scala:122)

When I execute the scalding job using hadoop jar in hdfs mode , it works. But if I use oozie to execute the job using java action, it throws this error.

Any pointers will be helpful.

Thanks

Sourabh

[Oscar Boykin]

We discussed this just yesterday on IRC:

https://www.dropbox.com/s/uu4b7mz10vsaera/Screenshot%202014-09-10%2007.57.32.png?dl=0

you should put the avro serialization first, not last. Kryo is in front of that and will try to serialize anything it gets, which will break this.

Also note:

https://github.com/twitter/chill/

chill-avro.

We probably should make it easier to use avro in nested structures, so for that you want to add Avro to your default Kryo serializations.

Added this issue:

https://github.com/twitter/scalding/issues/1043

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAP972tHjVQsP_m6nxFwhF%2B0VZnTOKupBo3-QaMfn3Y6gRK2wgw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[sourabh chaki]

Hi Oscar,

Thanks for your quick response. I am not using Kyro serialization directly from my code instead I am using using AvroPackedSource.

My code:

class MyJob(args: Args) extends Job(args) {

 val views = PackedAvroSource[AvroEntity](args(INPUT)).read

    .somemapper

    .groupby_on_some_field

    .write(Tsv(OUTPUT))

}

Is there any quick fix available to make this work?

Thanks

Sourabh

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CANX%3DQ2rHJ801XsnDAE9EOqcpjaHPd6V70Qru5cpDxWtqjXDMDA%40mail.gmail.com.

[Simone Franzini]

I am experiencing the same issues with Avro/Kryo and Scalding 0.11.0.

I cannot resolve the null pointer exception issue even by adding an Avro serializer to ioSerializations, as suggested in some other posts.

Regarding the posted IRC chat, could you clarify how chill-avro was wired in?

What is the AvroFriendlyKryo class?

In chill-avro I can only find an object but no class.

Thanks.