Subsets of data using the PartitionTap

[Elliot West]

I've been wondering how people are currently using PartitionTap to read subsets of a partitioned dataset? It seems to me that currently one can currently only read all partitions with this Tap. I'd expect that the most common use cases would be to read in subsets of partitions. Data segmentation by a key is one of the main virtues of partitioning after all.

We always require a small subset of partitions in our jobs (last hour, last day, last month, etc.) and provide a subclass of Hfs as a parent tap to the PartitionTap. This subclass provides some path filtering in an overridden getChildIdentifiers() method. Is there a way to achieve this without stepping outside the Cascading SDK, and if not, is it on a roadmap?

Thanks - Elliot.

[Chris K Wensel]

we want to provide a predicate interface that limits the returned partitions, but haven’t gotten to it.

feel free to spec something out, happy to work with you to see if we can get a contribution.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAC3gpCb8GkbjyXPiDqB%3DM%2BRTRnyGhVtX-BoQcyhtoUanfSofyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

[Dave Maughan]

Hi Chris,

I've done some work on this feature, utilizing the existing cascading.operation.Filter interface to limit the returned partitions. It would be great to get some feedback on the PR here: https://github.com/Cascading/cascading/pull/31

Thanks - Dave.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/F55922A6-13F2-4DE7-AE5F-26F631252FBC%40wensel.net.

[Chris K Wensel]

awesome! I’ll take a look as soon as I can. 

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAEGAa51YcqR3raN-oieHFQn2-xRYbFeAErD5wQp%3DWkn48cytSg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net