Keep original tuple when using built-in operations such as DateParser?

[Nate Murray]

Greetings,

I feel I am missing something very basic in using Cascading.

Say I have a tuple with fields ["ts", "some_id", "some_id2"]. What I
would like to do is use a DateParser over Each() Tuple and convert the
`ts` String into a `long` and retain the original fields "some_id" and
"some_id2" (in the output Tuple).

However when I use DateParser like so:

pipe = new Each(previousPipe, new DateParser("yyyy-MM-dd
HH:mm:ss"));

I get a list of Tuples that are *only* the `long` of milliseconds
since epoch. There is no id present in the Tuple so I cannot do a
CoGroup to merge tuple streams. Trying to specify Fields.ALL seems to
result in either "cascading.tuple.TupleException: operation added the
wrong number of fields" or "cascading.pipe.OperatorException: unable
to resolve selector using incoming".

While it is easy enough to write my own BaseOperation to do this, I
wanted to double check and make sure I am not missing something
obvious. Any thoughts on how to use built-in operations but still
retain access to the original Tuple?

Thanks,

Nate

[Chris K Wensel]

you need to use the ctor for DateParser that declares a different
fieldname. By default DateParser returns "ts". this collides with the
"ts" field you already have.

http://bit.ly/23Y8y

OR,

use wip-1.1 and use Fields.REPLACE instead of Fields.ALL.

ckw

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

[Nate Murray]

Thanks Chris, that worked. For posterity here's my solution:


// original fields
Fields logFields = new Fields("ts", "some_id", "some_id2");
// tmp fields to avoid conflict
Fields tmpFields = logFields.append(new Fields
("timestamp")).subtract(new Fields("ts"));
// rename `ts` -> `timestamp`
importPipe = new Each(importPipe, new Fields("ts"), new Identity
(new Fields("timestamp")), tmpFields);
// parse our `timestamp` field, return the original logFields
(this means we are not keeping our `timestamp` field, just grabbing
`ts` which was emitted from DateParser)
importPipe = new Each(importPipe, new Fields("timestamp"), new
DateParser("yyyy-MM-dd HH:mm:ss"), logFields);

Now String `timestamp` is converted to long `ts` and the rest of the
fields are intact.

Nate