Skip header line (Scalding)

[amitmor]

This question is about Scalding:

I couldn't find out how to tell Scalding TextLine("filename")#read  to skip the header in an input pipe. Any ideas ?

Thanks,
Amit

[Oscar Boykin]

We don't use header lines at Twitter, so we haven't tested that.

You could look up the cascading docs and implement a source that does this (see TextLine source in scalding) or point me to how to do it in cascading and I might be able to get it into the next jar we publish (today or tomorrow).

--
Oscar Boykin
@posco http://twitter.com/posco

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/7VELeBvEunoJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

[Sam Ritchie]

The TextDelimited scheme implements this in Cascading 2, if you want some reference:

https://github.com/cwensel/cascading/blob/wip-2.0/src/hadoop/cascading/scheme/hadoop/TextDelimited.java#L226

--

Sam Ritchie, Twitter Inc

703.662.1337

@sritchie09

(Too brief? Here's why! http://emailcharter.org)

[amitmor]

Thanks you Oscar and Sam !

I have written a little thing: https://gist.github.com/2781595
Do with it whatever you wish to, never mind if you don't !
Apperantly, only TextDelimited can accept the headers flags, so that's what I have implemented.

Amit

On Tuesday, May 22, 2012 8:49:32 PM UTC+3, Sam Ritchie wrote:

The TextDelimited scheme implements this in Cascading 2, if you want some reference:

https://github.com/cwensel/cascading/blob/wip-2.0/src/hadoop/cascading/scheme/hadoop/TextDelimited.java#L226

On Tue, May 22, 2012 at 9:33 AM, Oscar Boykin <os...@twitter.com> wrote:

We don't use header lines at Twitter, so we haven't tested that.

You could look up the cascading docs and implement a source that does this (see TextLine source in scalding) or point me to how to do it in cascading and I might be able to get it into the next jar we publish (today or tomorrow).

--
Oscar Boykin
@posco http://twitter.com/posco

On Tuesday, May 22, 2012 at 3:53, amitmor wrote:

This question is about Scalding:

I couldn't find out how to tell Scalding TextLine("filename")#read  to skip the header in an input pipe. Any ideas ?

Thanks,
Amit

--

[Oscar Boykin]

Thanks Amit (and Sam).

If you make a pull request for this to scalding/develop (change to DelimitedScheme and Tsv) I would pull it.

Don't you want to be a contributor?  :)

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.

To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/snN_yOqSJLMJ.

To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--
Oscar Boykin :: @posco :: https://twitter.com/intent/user?screen_name=posco

[amit.m...@gmail.com]

Sent from my HTClu.o