Is that possible to do exact "LATERAL VIEW EXPLODE" in scalding?

[Jing Lu]

Dear Scalding community,

In HIVE, "LATERAL VIEW EXPLODE" is frequently used if the value is a list. Is there any way in Scalding, we can do something similar?

Thanks

[Oscar Boykin]

I don't really know lateral view explode, but from what I can read it looks like flatMap + joining/aggregation. Do you want to give an example of the problem you have in mind and we can give a more concrete answer?

--
You received this message because you are subscribed to the Google Groups "Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalding-dev/e8c7bff9-1257-452c-8ced-353643cd7727%40googlegroups.com.

[Jing Lu]

Hi Oscar,

Thanks for helping.

For example, if I have a table like this:

pageid	adid_list
contact_page	[3, 4, 5]
front_page	[1, 2, 3]

I want to create another table like this:

pageid (string)	adid (int)
"front_page"	1
"front_page"	2
"front_page"	3
"contact_page"	3
"contact_page"	4
"contact_page"	5

Thanks,

Jing

[Oscar Boykin]

that's flatMap.

val rows: TypedPipe[(PageId, List[AdId])] = ...

rows.flatMap { case (page, ads) =>

  val pageIdPairs: List[(PageId, AdId)] = ads.map { ad => (page, ad) }

  pageIdPairs

}

or more succinctly:

rows.flatMap { case (page, ads) => ads.map((page, _)) }

the name comes map + flatten, so you first do a map, then you flatten (remove one layer of nesting).

Does that make sense?

To view this discussion on the web visit https://groups.google.com/d/msgid/scalding-dev/CAJVaavFMS7RuGfKvFC0f%2B0Fmr9fXjdDG13O-JLhZBPFRL7FWJQ%40mail.gmail.com.

[Jing Lu]

Yes, it does. Thanks for the explanation! 

Best