Window function, detecting non-occurring events, performance

23 views
Skip to first unread message

Nikola Veber

unread,
May 5, 2019, 1:14:17 PM5/5/19
to siddhi-user
Hi everyone,

I have enjoyed playing with Siddhi in the last couple of days. I would have a question regarding the inner-workings of window function.


I am testing the following way:
- submit events at limited rate (ca 1200/sec)
- for every second event, submit the confirmation event immediately
- window size is set to 60 minutes

For the first 350k events, the troughput on the sender side (Java, over TCP) looks like this:
10000 1211 msg / sec
...
100000 649 msg / sec
...
250000 235 msg / sec
...
350000 165 msg / sec

Do you have any suggestions on how to increase the performance of this task? 

Also, if the state persistence is enabled and set to be executed every minute (resulting in a 4.5 MB dump file after about 300k events), is there a way to ensure that no events are lost in case of failure? 
As far as I understood, if the state is recovered from the file, the last minute of events could be lost. 

Are there any siddhi-specific strategies to ensure no event loss?


Version in use: WSO2 Stream Processor v4.3.0, Siddhi file deployed in a worker node. 
(Machine is a Win7 Desktop, Intel i7)

Thanks!
Best regards
Nikola

Mohanadarshan Vivekanandalingam

unread,
May 6, 2019, 2:05:44 AM5/6/19
to siddh...@googlegroups.com

On Sun, May 5, 2019 at 10:44 PM Nikola Veber <nikola...@gmail.com> wrote:
Hi everyone,

Hi Victor,

Please find my comments below...
 
I have enjoyed playing with Siddhi in the last couple of days. I would have a question regarding the inner-workings of window function.


Actually, above documentation is bit outdated. Now, Siddhi have native support for non-occurring event patterns and you can use that in Stream Processor. 
Please refer below docs,

I am testing the following way:
- submit events at limited rate (ca 1200/sec)
- for every second event, submit the confirmation event immediately
- window size is set to 60 minutes

For the first 350k events, the troughput on the sender side (Java, over TCP) looks like this:
10000 1211 msg / sec
...
100000 649 msg / sec
...
250000 235 msg / sec
...
350000 165 msg / sec

Do you have any suggestions on how to increase the performance of this task? 

If you have a 60 mins window then events will accumulate in the memory and eventually it can slow down the processing; that should be the reason for the performance degradation. 
 

Also, if the state persistence is enabled and set to be executed every minute (resulting in a 4.5 MB dump file after about 300k events), is there a way to ensure that no events are lost in case of failure? 
As far as I understood, if the state is recovered from the file, the last minute of events could be lost. 

Are there any siddhi-specific strategies to ensure no event loss?

Let's assume that you have enabled snapshot persistence in 1 minute interval and there is a failure at 45th seconds then you will be loosing the events that received in 45 seconds and Siddhi can only recover to the earlier state which got persisted. 

If you are looking for a highly reliable and available option then you should bring a messaging layer such as Kafka or NATS. Please refer the below article for better understanding on this. https://bit.ly/2Y90JBj  

Thanks,
Mohan
 

Version in use: WSO2 Stream Processor v4.3.0, Siddhi file deployed in a worker node. 
(Machine is a Win7 Desktop, Intel i7)

Thanks!
Best regards
Nikola

--
Siddhi homepage: http://siddhi.io/
Siddhi Github repo: https://github.com/siddhi-io/siddhi
 
You received this message because you are subscribed to the Google Groups "Siddhi-user" group.
To post to this group, send email to siddh...@googlegroups.com
To unsubscribe from this group, send email to siddhi-user...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/siddhi-user?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "siddhi-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to siddhi-user...@googlegroups.com.
To post to this group, send email to siddh...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
V. Mohanadarshan | Senior Technical Lead | WSO2 Inc. |
(M) 94-771117673 | (E) mo...@wso2.com |
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
Reply all
Reply to author
Forward
0 new messages