How to split DList to multiple location ?

[Jeff Zhang]

Hi,

I am trying to split a DList by one key, and store each group into different location. Is there API for this usage ? I know that using filter is a workaround, but it is not so elegant. Thanks for any help.

[Eric Torreborre]

Do you mean something like:

def splitByKey(list: DList[(K, Iterable[V])]): Seq[(K, DList[V])] = {

  val keys = list.map(_._1).materialise.run

  keys map { k => (k, list.filter(_._1 == k)) }

}

That doesn't seem very optimal to me, I'll think if we can do better.

E.

[Eric Torreborre]

Hi Jeff,

I thought about that and I think that you could attempt to write a new DataSink with a specific OutputFormat.

If you look into the existing TextOutputFormat you will see that a "LineWriter" is created once and for all for all the key-values. 

In your case you could instantiate a new LineWriter, using a different DataOutputStream, when you encounter a new key.

If you don't have time to do it, please open an issue on github and we'll try to do something. OTOH, if you have time to do it, please consider making it a pull request :-).

Cheers,

Eric.