Question about the Constraints.toEntityPredicate and Constraints.toKeyPredicate.

[RD SR]

Hi CDK users,

  Could someone help me understand the code in the Constraints class better?

Here's my doubt.

I see that Constraints.toEntityPredicate is used to filter records. Partition level filtering is applied before at a file level when constructing the Reader. So as I see it, every record read from the reader would have already satisfied the partition predicates if any, so why apply the partition predicates again when filtering records?

Best,

R.

[Ryan Blue]

The entity predicate applies a set of constraints to an entity, without
making assumptions about what other predicates have already been applied.

You're right that while reading we can simplify a set of predicates and
remove the ones that are satisfied by the partitioning scheme. To do
that, we have another method: minimizeFor(StorageKey). That way, you can
filter storage keys and for each one get a new set of residual
constraints. Then using that new set, you can get a predicate for the
entities in that partition.

Does that make sense?

rb


--
Ryan Blue
Software Engineer
Cloudera, Inc.

[RD]

Yes it does. Thanks Ryan!