How to run aggregations on data?

[Punit Naik]

I have a json linke this:

{:pagePath "/" :timeSpent 23}
{:pagePath "/w" :timeSpent 3}
{:pagePath "/" :timeSpent 211}
{:pagePath "/" :timeSpent 86}
{:pagePath "/w" :timeSpent 2}

And I want total time spent on a page. So I want this output:

{:pagePath "/" :totalTimeSpent 320}
{:pagePath "/" :totalTimeSpent 5}

I know I have to do a "group-by" on "pagePath" but I am a bit confused. Can anyone show me a PigPen solution for this?

[Matt Bossenbroek]

There are two options:

=> (->>

     (pig/return [{:pagePath "/" :timeSpent 23}

                  {:pagePath "/w" :timeSpent 3}

                  {:pagePath "/" :timeSpent 211}

                  {:pagePath "/" :timeSpent 86}

                  {:pagePath "/w" :timeSpent 2}])

     (pig/group-by :pagePath)

     (pig/map (fn [[pagePath vs]]

                {:pagePath pagePath

                 :timeSpent (->> vs

                              (map :timeSpent)

                              (reduce +))}))

     (pig/dump))

({:pagePath "/", :timeSpent 320} {:pagePath "/w", :timeSpent 5})

Or with a fold:

=> (->>

     (pig/return [{:pagePath "/" :timeSpent 23}

                  {:pagePath "/w" :timeSpent 3}

                  {:pagePath "/" :timeSpent 211}

                  {:pagePath "/" :timeSpent 86}

                  {:pagePath "/w" :timeSpent 2}])

     (pig/group-by :pagePath {:fold (->>

                                      (fold/map :timeSpent)

                                      (fold/sum))})

     (pig/map (fn [[pagePath timeSpent]]

                {:pagePath pagePath

                 :timeSpent timeSpent}))

     (pig/dump))

({:pagePath "/", :timeSpent 320} {:pagePath "/w", :timeSpent 5})

Read more about folding here: https://github.com/Netflix/PigPen/wiki/Folding-Data

-Matt

--
You received this message because you are subscribed to the Google Groups "PigPen Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pigpen-suppor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.