scoobi for Scala 2.10

[Debasish Ghosh]

(posting on dev list as per suggestion from Eric)

Hi -

We are planning to use Scoobi in a project. It will be Scala 2.10 - in fact one version of the application runs with native MR jobs in Scala. Now the following issues are of immediate concern :-

1. when will scala 2.10 port of Scoobi be available ? I have taken the branch that was forked (scoobi-on-2.10) and managed to compile after some hiccups. I have also managed to run the WordCount example from the examples folder.
2. Our main requirement is Avro. I could not find avro-plugin for Scala 2.10 and sbt 0.12. I looked at https://github.com/radlab/SCADS/wiki/Avro-Plugin which is an older version of the plugin. Is it available ?
3. One requirement that we have is to handle Avro GenericRecord. We need to read the schema and do map/reduce generically .. e.g currently we do getFields from the schema and do stuff depending on the data type of the field. I saw a pull request on handling GenericRecord (https://github.com/NICTA/scoobi/pull/150), but possibly it's not there in this branch. Would like to have some more info on whether this will be possible with the new pull request.

Any information on the above would be extremely helpful.

Thanks.

- Debasish

[Eric Torreborre]

Hi,

Quick answer on the first point.

The master branch has now been moved to Scala 2.10 and there is a 0.7.0-SNAPSHOT for Scala 2.10 (and cdh4).

I hope that the Avro experts in the group can comment on n.2 and I need to dig through the history to understand what's going on with 3.

E.

[Debasish Ghosh]

Thanks Eric for the quick response. Regarding #3, I think the pull request was merged in cdh3 branch but not in others. I am hoping that pull request will help me implement generic stuff with Avro GenericRecord.

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "scoobi-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scoobi-dev/sEB34GmI-jI/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to scoobi-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Debasish Ghosh
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

[Christopher Severs]

The SCADs plugin authors are active on the Avro mailing list I think. It would be good to see if they have plans to support 2.10.

Outside of that though you can always use the Java version of Avro inside Scala without any issues and I bet the sbt plugin to generate classes from schemas at compile time will work fine.

[Alex Cozzi]

Hi Eric, 

I did try out release 0.7.0-SNAPSHOT with scala 2.10 on our chd3 on our hortonworks cluster and it seems to be working fine. I tried only one program, but a non-trivial one, so I think I will start to work with this version now and give it a good shakedown.