Emit multiple keys from Mappers

[Horia]

What is the highest performance way to emit multiple keys from Mappers?
So far, I have the following code but am concerned that the flatten method will be too slow.

val keyList = fromTextFile(filePaths)

keyList.map( key => {
    val valList = getValList(key)
    val kvPairs = new Array[(String, String)](valList.length)
    valList.foldLeft(0)( (idx, val) => {
        kvPairs(idx) = (key, val)
        idx + 1
    })
    kvPairs
} ).flatten


Any and all comments greatly appreciated. Thanks!

[Matthew Rathbone]

I think that should be fine, we do that all the time on pretty large datasets. You shouldn't need the additional array though, can do it in one line:

keyList.flatMap{key =>

  getValList(key).map{v => (key, v)}

}

:-)

Matthew Rathbone
==
matthew....@gmail.com

--
You received this message because you are subscribed to the Google Groups "scoobi-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scoobi-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.