Equivalent feature to Flink's allowed lateness

[Jean Rossier]

Hello,

is there anything similar to Flink's allowed lateness in siddhi ?

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#allowed-lateness

Thanks

Jean

[Chiran Fernando]

Hi Jean,

At the moment, only the Siddhi Session window supports configuring allowed latency for the window. We support externalTime and  externalTimeBatch windows which act based on external time.

Below is a sample Siddhi app using the external timestamp for your reference.

@App:name("ExpireOnExternalTT")
@App:description("Description of the plan")
define stream LoginEvents (timestamp long, ip string);
define window LoginWindow (timestamp long, ip string) externalTimeBatch(timestamp, 1 sec) output all events;
@info(name = 'query0')
from LoginEvents
insert into LoginWindow; 
@info(name = 'query1') 
from LoginWindow 
select timestamp, ip, count() as total  
insert all events into uniqueIps;

Furthermore, we support reordering out of order events using siddhi-execution-reorder extension based on the K-Slack and alpha K-Stack algorithms.

Using this extension you can pre-order the events and pass the events to the window thereafter. 

Please elaborate more on your use-case, so that we can provide you with a comprehensive solution. 

Also, we can consider adding this support to our roadmap, if we cannot achieve the requirement from the provided approaches.

[Jean Rossier]

Hi Chiran,

I have a use case where I need to do some calculation over events that occurred in the last 7 days. For that I'm using an externalTime window.

In my use case, some messages could be delivered lately, meaning that a message for a date in the past (let's say 2 days ago) could be delivered today, and this message should still be included in the calculation, at the right place (according to its timestamp).

The siddhi-execution-reorder extension will probably help for that. I will look at it.

Thanks

Jean