taking field names as argument

3 views
Skip to first unread message

Xiaolin Li

unread,
Aug 26, 2016, 4:58:34 PM8/26/16
to Scalding Development
Hi,

I am trying to write a trait function in scalding to apply a mapping function on part of the fields in the pipe, and I want the function to take the list of fields as argument. Could anyone give me some hints on that? (sth like someMappingFun(self: Pipe, cols: Fields) : Pipe = {...}

Thank you! 




P. Oscar Boykin

unread,
Aug 26, 2016, 5:03:31 PM8/26/16
to Xiaolin Li, Scalding Development
that is how people do that in the deprecated Fields API. That is not really an issue with the TypedPipe API.

--
You received this message because you are subscribed to the Google Groups "Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Xiaolin Li

unread,
Aug 26, 2016, 5:28:05 PM8/26/16
to Scalding Development, xiaoli...@u.northwestern.edu
Thank you! Is it possible to mix the TypedPipe API with Field API together? My current functions are written in Fields APIs and migrating them to Field API may take some time.

Oscar Boykin

unread,
Aug 26, 2016, 6:52:30 PM8/26/16
to Xiaolin Li, Scalding Development
yes, you can move over bit, by bit.

To go from a `Pipe` to `TypedPipe` you need to use `TypedPipe.from[(Int, String, Long)](somePipe, ('som, 'b, 'other))` for instance.

Then to go back, use `TypedPipe.toPipe(('result, 'somethingelse))` for instance.
Reply all
Reply to author
Forward
0 new messages