How to pass aggregated result to another function?

[Punit Naik]

Hi

I used a "conj" aggregation in a window after "grouping by" on a particular key and passed the result onto another function. The function received the original segments rather than receiving "conjed" list of lists from the first window.

How should I do this?

[Punit Naik]

I do realise that I can use the "trigger/sync"  functionality but I want to perform more aggregations (more windows) after the first aggregation.

[Mike Drogalis]

Onyx's windows different from Spark - they don't chain in succession. Each task has a function, zero or more windows, flow conditions, lifecycles, and so forth. Segment proceed from task to task and have their function applied.

If you're looking to have two tasks, A and B, both windowed where B's window contents are populated by A's window contents, you want to use trigger/emit. trigger/emit sends contents of one window to another task. Then in your onyx/fn, you want to return empty vector so that you're original segment doesn't proceed downstream.

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/595ca6f6-14b3-426c-ad7b-d05cbea9b39a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Punit Naik]

Thanks Michael. Could you please add some examples on triggers though? Especially for trigger/emit?

On Thursday, May 4, 2017 at 8:11:06 PM UTC+5:30, Mike Drogalis wrote:

Onyx's windows different from Spark - they don't chain in succession. Each task has a function, zero or more windows, flow conditions, lifecycles, and so forth. Segment proceed from task to task and have their function applied.

If you're looking to have two tasks, A and B, both windowed where B's window contents are populated by A's window contents, you want to use trigger/emit. trigger/emit sends contents of one window to another task. Then in your onyx/fn, you want to return empty vector so that you're original segment doesn't proceed downstream.

On Thu, May 4, 2017 at 3:40 AM, Punit Naik <naik.p...@gmail.com> wrote:

I do realise that I can use the "trigger/sync"  functionality but I want to perform more aggregations (more windows) after the first aggregation.

On Thursday, May 4, 2017 at 4:08:41 PM UTC+5:30, Punit Naik wrote:

Hi

I used a "conj" aggregation in a window after "grouping by" on a particular key and passed the result onto another function. The function received the original segments rather than receiving "conjed" list of lists from the first window.

How should I do this?

--
You received this message because you are subscribed to the Google Groups "Onyx" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.

To post to this group, send email to onyx...@googlegroups.com.

[Mike Drogalis]

All of the windowed examples use triggers in onyx-examples. There's also an example of using trigger/emit in the tests.

To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.

To post to this group, send email to onyx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/ee62e69c-2ddf-42a7-8903-93f4b93d127f%40googlegroups.com.

[Punit Naik]

Hi Mike

I just looked at both the examples and the test. There is no example of windowing where you use trigger/emit in the onyx-examples repo and in the test case there is emit example for only one window which does not really help me in understanding the flow for multiple windows.

Can you please add a basic example for chaining multiple windows?

On Tuesday, May 9, 2017 at 3:54:38 AM UTC+5:30, Mike Drogalis wrote:

All of the windowed examples use triggers in onyx-examples. There's also an example of using trigger/emit in the tests.

On Sun, May 7, 2017 at 11:57 PM, Punit Naik <naik.p...@gmail.com> wrote:

Thanks Michael. Could you please add some examples on triggers though? Especially for trigger/emit?

On Thursday, May 4, 2017 at 8:11:06 PM UTC+5:30, Mike Drogalis wrote:

Onyx's windows different from Spark - they don't chain in succession. Each task has a function, zero or more windows, flow conditions, lifecycles, and so forth. Segment proceed from task to task and have their function applied.

If you're looking to have two tasks, A and B, both windowed where B's window contents are populated by A's window contents, you want to use trigger/emit. trigger/emit sends contents of one window to another task. Then in your onyx/fn, you want to return empty vector so that you're original segment doesn't proceed downstream.

On Thu, May 4, 2017 at 3:40 AM, Punit Naik <naik.p...@gmail.com> wrote:

I do realise that I can use the "trigger/sync"  functionality but I want to perform more aggregations (more windows) after the first aggregation.

On Thursday, May 4, 2017 at 4:08:41 PM UTC+5:30, Punit Naik wrote:

Hi

I used a "conj" aggregation in a window after "grouping by" on a particular key and passed the result onto another function. The function received the original segments rather than receiving "conjed" list of lists from the first window.

How should I do this?

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/595ca6f6-14b3-426c-ad7b-d05cbea9b39a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.

[Anuj Kumar]

Punit,

You can see this example. I have chained couple of fixed windows- https://github.com/anujsrc/play/blob/master/clojure/onyx-stream/src/clj/play/onyx_stream/twitter/chain_windows.clj

You can run using lein run as mentioned here- https://github.com/anujsrc/play/tree/master/clojure/onyx-stream#usage

Thanks,

Anuj

To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.

To post to this group, send email to onyx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/a4241082-f247-41c4-8d01-d7771e6e9e97%40googlegroups.com.

[Punit Naik]

Thanks Anuj.