scalding.avro examples?

[Mason]

Anyone have example code they can share about using Scalding to read and
process Avros? I've seen the examples in the scalding.avro repo, but
more reference code would help

[Mansoor Ashraf]

 This is how I am using avro with Scalding 

Lets say your Schema looks some what like this

{"namespace": "com.kohls.bigdata.model",

    "type": "record",

    "name": "Recommendations",

    "fields": [

        {

              some fields here ......

          }]

}

Once you compile the schema, you will get a top level Recommendation Object. Then you can read and write avro encoded file as follow

 PackedAvroSource[Recommendations](args("input"))

    .read

    .map('Recommendations -> 'sum) {

    rec: Recommendations => // you have your avro pojo, go nuts

  }

    .write(Csv(args("output"), fields = 'sum))

hope this helps

[Mason]

Thanks for your response Mansoor. Are you manually specifying your Avro schema?

A primary reason for me to use Avro is packaging the schema with the data, so I don't have to hardcode my schemas all over the place.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mansoor Ashraf]

Hi Mason,

It really depends on the use case. All our data for batch processing is encoded with the Schema. However for more real time use cases such as high volume data being process through Kafka/Storm or data sent over the wire in a form of a service we do not encode schema with the data just to increase the throughput. Decoupling data and schema also come in handy when you have a back compatible schema and then different client can use different version of the schema independent of each other. I believe LinkedIn uses a ZooKeeper based schema registry to lookup schema on the fly and does not encode both Schema and data together as well. There is a presentation from them somewhere which talks in great detail about this very topic.

[jeffo]

We put together an example project skeleton (with two examples) that uses the scalding.avro project at https://github.com/Tapad/scalding-avro-integration

It's pretty trivial, but I think it's helpful if you're just getting started working with avro from scalding.

[Christopher Severs]

That's awesome, thanks!

If anyone is interested in helping out with the actual scalding.avro project by the way I'm more than happy to add contributors to the repo.

[Mansoor Ashraf]

How are you guys unit testing your avro/scalding job since AvroScheme currently does not support local mode?

[Oscar Boykin]

Etsy contributed a wrapper to use hadoop taps in local mode:

https://github.com/Cascading/maple/blob/develop/src/jvm/com/etsy/cascading/tap/local/LocalTap.java

maple is a dependency of scalding.  Might be useful for the avro case (maybe).

--

You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[jeffo]

Currently we only do integration testing against our local file system. We still pass the --hdfs flag, but we restrict our input and output data to smaller samples set of data that exist on the local file system.

[Christopher Severs]

You can also use JobTest and .runHadoop then just create an in memory list of Avro records. However, I would like to have a local mode for scalding.avro, which means having a local mode in cascading.avro, which I don't think is too much work.

[Marius Soutier]

Hi,

I tried to use that but I'm running into an IndexOutOfBoundsException:

Caused by: java.lang.IndexOutOfBoundsException: Index: 12, Size: 1

at java.util.ArrayList.rangeCheck(ArrayList.java:635)

at java.util.ArrayList.get(ArrayList.java:411)

at cascading.tuple.Tuple.getObject(Tuple.java:236)

at cascading.tuple.util.NarrowTupleList.get(NarrowTupleList.java:71)

at cascading.tuple.Tuple.getObject(Tuple.java:236)

at cascading.tuple.Tuple.getDouble(Tuple.java:280)

My code looks like this:

      JobTest(classOf[<JobClass>].getName)

        .arg("input", "inputFile")

        .arg("output", "outputFile")

        .source[<AvroClass>](UnpackedAvroSource("inputFile", <AvroClass>.SCHEMA$), List(<avroObj>))

        .sink[Double](TextLine("outputFile")) { outputBuffer =>

          ...

        }

        .runHadoop

        .finish

[Oscar Boykin]

Link to the job?

Looks like there is mismatch in the data in a pipe, and what is being pulled out.

--

You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.

Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/6a191b9b-3e7d-465d-a87e-5e8b3daf4ef6%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Marius Soutier]

The job is as simple as it gets, and also works perfectly fine on the server:

  UnpackedAvroSource(args("input"), MyAvroClass.SCHEMA$).read

    .groupBy('id) { _.size }

    .write(Tsv(args("output")))

[Marius Soutier]

And the test also works fine when using PackedAvroSource.

[Christopher Severs]

Can you post the Schema (or something that has the same structure with different names if you prefer) ?

[Christopher Severs]

Oh sorry, also, are you using scalding.avro (which is old and unsupported now) or the newer official scalding-avro module that is part of Scalding 0.9.x ?

On Friday, February 14, 2014 8:40:16 PM UTC+2, Marius Soutier wrote:

[Marius Soutier]

Uglified schema:

@namespace("avro.test.namespace")

protocol MyAvroProtocol {

  record MyAvroRecord {

    long id = null;

    string otherId;

    string date = "";

    int result = 1;

    string a = "";

    string b = "";

    string c = "";

    string d = "";

    string e = "";

    string f = "";

    int g = 4;

    string h = "FTI";

    double i = 0.00;

    double j = 0.00;

    string k = "";

    double l = 0.0;

    int m = 0;

    int n = -1;

    int o = 0;

    string p = "";

    int q = 2;

    int r = 0;

    int s = 0;

    int t = 0;

    int u = 0;

    int v = 0;

    double w = 0.00;

    double x = 0.00;

    double y = 0.00;

    double z = 0.00;

    double aa = 0.00;

    double bb = 0.00;

    double cc = 0.00;

    int dd = 0;

    int ee = 0;

    int ff = 0;

    int gg = 0;

    int hh = 0;

    int ii = 0;

  }

}

I’m using com.twitter.scalding-avro_2.10 0.9.0rc4. 

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2462c42a-476a-4b84-9184-90bd6194ee46%40googlegroups.com.

[Oscar Boykin]

Can you post the code to one that fails? And even better: make a pull request against scalding with a failing unit test.

We can get it green and add it to the repo. 

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/0e7b69ec-20e0-46bb-8da4-d028ec70172e%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Marius Soutier]

Done:

https://github.com/twitter/scalding/pull/790

Cheers

- Marius

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CANX%3DQ2pOL233HiBerk8OJ-HkWRxJR5d-T766a7SkHOmiOv-2PA%40mail.gmail.com.