Recommended way to handle late arriving events.

[Mark Griffin]

We have a use case where events are sent from mobile devices, often in batches. We want to use the timestamp of when the event was created, however this is often way past the window period for the realtime node's plumber, so by the time the event arrives it's game over for that event. We could extend this window period, but that that is still a fixed interval where we would risk losing events. We would also to prefer not the carry the memory burden that a long window would incur.

So we need some kind of process where late arriving events (not a majority of our traffic, but a significant part of it) can be persisted to historical nodes so the data is available in the query result set -- ideally as "real time" as it can be, ie, once a day batch jobs to incorporate the stragglers probably won't meet the biz requirements.

I found some comments on previous posts that seem to imply there are some hooks or recommended methods for this:

"Most of the segments in a Druid cluster are immutable "historical" segments. To update data that already exists in a historical segment, you have to build a new segment for that interval of data with an increased version identifier. Druid uses MVCC to surface data from segments with higher version identifiers and obsolete outdated data from segments with older version identifiers. This process isn't really well documented but we will try in the near future to add more information on the wiki."

This sounds like a reasonable solution, but also sounds like it's potentially expensive and should only be run as often as one would run a batch job.

"Our realtime ingestion always has an accompanying batch process. The batch process and realtime processes deal with different data ranges and do not clobber each other. For example, over the course of a day, our realtime nodes collect data and periodically hand that data off to the rest of the Druid cluster. At the end of the day, our batch process runs and builds a daily segment for the previous day's data. This segment enters the cluster and obsoletes segments that were built by realtime ingestion for the previous day. At this point, the realtime node is ingesting and working with data that is more recent than the previous day."

Again, very similar and probably exactly what we need, would love to learn more about the implementation details if it's a good fit.

Any guidance or suggestions are greatly appreciated.

- Mark

[Eric Tschetter]

Mark,

"Teeing" off the events into some long-term storage (like hdfs or s3) and then re-processing in batch is the general method of handling this.  You can re-process from batch as often as you like (i.e. it can be faster than once a day), but it's still a batch process that is just run more often.

Btw, do you know what percentage of your events are late arriving and by how much?  Metamarkets has dealt with events published from mobile devices and a 10 minute window generally did a good job.

Technically, it would be possible to setup something that could take the events and merge them into the old segments and publish a new one, but that code doesn't yet exist and the mechanism to make it easier to implement (the indexing service) is being documented to the point where it could actually be useful with the next version (0.6).

So, depending on all sorts of variables it could be possible to start with it one way and then move into the other over time.

--Eric

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/8d82d154-4a07-4363-a609-118ebcce7899%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mark Griffin]

Yes, I was considering sending all events that are outside of the current window period to another queue (we are using rabbit) where they can either be persisted to S3 for batch processing, or potentially become another firehose for the druid cluster, if that's possible.

I would say maybe 25% of our data will be outside of the current window. One of the reasons is that our mobile apps often hold back (some of) their events until the next user session, which is the next time the app is run. So the events from the last session are not sent until the beginning of the next session.  There is also another use case where the app may be collecting data in offline mode, then syncing that data up to the server, so retaining the sample data timestamp is important for later analysis.

- Mark

[Eric Tschetter]

Mark,

That makes a lot of sense.  Teeing the data set and adding it in batch is gonna be the best method of updating your data sets right now.  This is definitely a limitation that I hope we can shore up sooner rather than later though.

--Eric

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/ea4be245-60a7-410c-a637-cf42126728e2%40googlegroups.com.