Nested 'Group-By's

[Punit Naik]

I have some data in which I want to perform multiple 'group-by's and they are nested. Can anyone please provide me with a one-liner or anything else for nested groups-by?

[Matt Bossenbroek]

That's a pretty vague request. Do you have an example (maybe in regular clojure) as to what you'd like it to do?

-Matt

On Friday, March 18, 2016 at 2:48 AM, Punit Naik wrote:

I have some data in which I want to perform multiple 'group-by's and they are nested. Can anyone please provide me with a one-liner or anything else for nested groups-by?

--
You received this message because you are subscribed to the Google Groups "PigPen Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pigpen-suppor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Punit Naik]

Okay. So I did the first group-by on userType like this:

(def data (pig/load-json "/path"))

(def grouped-by-type (pig/group-by :ga_userType data))

And then inside it I wanted to do another group-by so I did this:

(def grouped-by-type-id (pig/map (fn [[type d]] (->> d (pig/group-by :ga_userId))) grouped-by-type))

But when I dumped 'grouped-by-type-id', it gave me this error:

AssertionError Assert failed: (map? relation)  pigpen.raw/eval3191/command--3192/fn--3195 (raw.clj:42)

So how do I do this?

[Matt Bossenbroek]

When you’re inside any user function, it’s just normal clojure. The `pig/` versions are only for the top level script generation.

Just drop the pig/ inside the pig/map fn to use clojure.core/group-by:

(def grouped-by-type-id (pig/map (fn [[type d]] (->> d (group-by :ga_userId))) grouped-by-type))

-Matt

[Punit Naik]

Thanks a lot Matt.

[Punit Naik]

Hi Matt

So your fix worked. But since I am using group-bys inside group-bys, my final output is only a single line which is a list of lists like this:

[[{"userType":"F","userId":"902785","modId":"9","total_time_spent":16}],[{"userType":"F","userId":"1179688","modId":"9","total_time_spent":207}]]

I wanted the program to write a single map per line.

How do I do this?

[Matt Bossenbroek]

You either need to combine those maps using some aggregation, or use mapcat to flatten the results. If you have a regular clojure example to work with, that would help a lot.

-Matt

[Punit Naik]

Thanks again Matt. As my data structure was a list of lists, I did:

->>(

(pig/mapcat concat)

(pig/map #(into {} %)))

And it properly denormalised it.