Sliding Window Durability and Optimising Memory Usage

[b...@respond-software.com]

When a segment comes in, I'd like to know some details about the previous 30 days of segments. So I'd like to have a 30 day sliding window, sliding every day.

I have two questions related to this:

1. If I stop/deploy/start the job to do a monthly release, does the aggregation window reset or do we still have 30 days of data in the window when the upgrade boots up? I know the window is backed by rocksdb, just wondering if that backing covers job upgrades (the window id would stay the same).

2. I really just want to see the data for the past 30 days. So I want a 30 day sliding window that slides every midnight, but I only care about the extent where tonight is the upper bound. Is there a way to inform Onyx to discard the data in the other 29 extents and just keep this one?

I'm just trying to see the tradeoffs between using a sliding window, versus querying a database each time I process a segment.  We expect fairly high throughput, and every segment needs to both read from the aggregation and update the aggregation.  I wrote a custom aggregation function, but it's not much different than collect-by-key.

Thanks in advance!

[Mike Drogalis]

Hi Ben,

On Tue, Jul 26, 2016 at 9:40 AM, <b...@respond-software.com> wrote:

When a segment comes in, I'd like to know some details about the previous 30 days of segments. So I'd like to have a 30 day sliding window, sliding every day.

I have two questions related to this:

1. If I stop/deploy/start the job to do a monthly release, does the aggregation window reset or do we still have 30 days of data in the window when the upgrade boots up? I know the window is backed by rocksdb, just wondering if that backing covers job upgrades (the window id would stay the same).

As of release 0.9.9 (the current release), window contents cannot be recovered across jobs. We have an open ticket for this, and are addressing it in the next major release. The window deltas are captured by BookKeeper. The window isn't technically backed by RocksDB. The window itself sits in memory as a data structure. RocksDB is used as a local datastore to track segment IDs that have passed through this peer, ensures that segments aren't applied more than once to the window aggregate.

We're hoping to ship the next major release within 2 months to address cross-job window recovery.

2. I really just want to see the data for the past 30 days. So I want a 30 day sliding window that slides every midnight, but I only care about the extent where tonight is the upper bound. Is there a way to inform Onyx to discard the data in the other 29 extents and just keep this one?

Onyx's windowing model has rich time support, and covers both wall clock time and event time. What you're describing in the paragraph above is wall clock time ("tonight", "right now"). Wall clock time is relative to the view of a particular machine. Event time is a timestamp, or otherwise totally ordered value, that is defined by the data record itself, and is agnostic to any particular machine. Onyx is able to gracefully handle out of order message processing (e.g. a segment for an event that happened last October) because by default, it operates using event time.

Users are often a little confused about how sliding windows work because they're presumed to work on wall clock time, when in fact they work on event time. At first glance, it might appear as if the Onyx peer is using a timer, and when the timer elapses the slide value, it creates a new partition of the window. (midnight, per day, in your example). It's actually pretty different (and more efficient) under the hood. Onyx uses an implementation of the Window-ID algorithm to lazily create "panes" of a window when a new value has been encountered that would make the window "slide". The full algorithm, and notes, are listed here.

Back to your question, though. You're asking to do something kind of interesting here. You want to retain 30 days of data. Is that event time, or wall clock time? It doesn't actually matter, but if you want to handle out of order messages, you'll want to use timestamps on the data records themselves (event time). On top of that, you only care about the last 30 days ("now", or "tonight midnight"), as viewed by the wall clock of a particular machine, and otherwise want to discard the window contents. You can set up a 24 hour timer trigger with a discarding refinement mode to blow away window contents for windows that you know for certain have gone out of range.

I'm just trying to see the tradeoffs between using a sliding window, versus querying a database each time I process a segment.  We expect fairly high throughput, and every segment needs to both read from the aggregation and update the aggregation.  I wrote a custom aggregation function, but it's not much different than collect-by-key.

If you're doing something that's fairly high throughput, I'd recommend using the windowing feature since you'll get locality of data on the box. You can easily cripple a database, and as a result your processing throughput, by running out a datastore every time during an aggregation.

Happy to answer anything else if that was unclear. 

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/afd0c175-bbb2-4bcc-a300-9eb78961d7dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Heilers]

Yes thanks, we are setting the :window/window-key to a timestamp in the segment, that came from the source system.

Thanks for already working on the first question, and for the advice on the second one.  Just to clarify, I think what I really want is not to use :onyx.refinements/discarding directly, but to write something similar that sometimes returns `state` and sometimes returns nil, based on the upper bound of the window.

[Mike Drogalis]

You could definitely do that, yep. You can write a custom refinement mode that conditionally discards window contents, and use a timer trigger to periodically clean up. If there's something preventing such an implementation, I'd be open to modifying Onyx to support that.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/a6241412-1902-438a-9c3c-9ed6e416478a%40googlegroups.com.