Siddhi memory leak

[yahia222]

So i'am trying to run this up with a rate of (1 msg every 150 ms ) this kind of load should be normal however i get always exponentially increasing memory usage until it gets an error and the app craches. 

error :

[2019-05-01_12-42-16_763] ERROR {org.wso2.siddhi.core.util.Scheduler} - java.lang.OutOfMemoryError: Java heap space 
[2019-05-01_12-42-42_182] ERROR {org.wso2.carbon.deployment.engine.internal.SchedulerTask} - Error occurred while scanning deployment repository

java.lang.OutOfMemoryError: GC overhead limit exceeded

script : 

""

 @App:name('position')

@App:description('element position')

@source(type = 'http', receiver.url = "http://0.0.0.0:5005/input",

        @map(type = 'json'))

define stream sensor_event (sensorID string,element string,type string,timeStamp string);

@sink(type='http',publisher.url='http://localhost:3003/output', @map(type='json'))

define stream sensor_alert(element string, sensorID string, timeStamp string);

define function translate[javascript] return string {

  var sensors = {

    '8': 'S',

    '9': 'S',

    '7': 'SG',

    '2': 'S',

    '4': 'NG',

    '6': 'S',

    '5': 'S'

  };

  sensorID=data[0];

  return sensors[sensorID];

   

};

define stream sensor_event_translated (element string, sensorID string,timeStamp string);

from sensor_event

select element, translate(sensorID) as sensorID ,timeStamp 

insert into tracker_event_translated ;

partition with (sensorID of sensor_event_translated)

begin

        @info(name = 'query')

        from sensor_event_translated#window.unique:firstTimeBatch(element,10 sec)

        insert current events into sensor_alert;

end;

"""

[Mohanadarshan Vivekanandalingam]

Hi,

We'll look into this and get back to you. BTW, are you're using a Siddhi engine itself or Stream Processor? Can you please share the version of that?

Have you got a chance to analyze the heap dump? If yes, Have you seen any abnormal amount of objects of some classes?

Thanks,

Mohan

[Sriskandarajah Suhothayan]

Hi 

Just of add to what Mohan has asked. 

Can you give the following information to narrow down the issue. 

What the Siddhi version you are using? 

How many unique sensorIDs you have in your system? 

The reason for these questions is, in Siddhi 4.x and 3.x the created partitions does not get cleaned and therefore when multiple partitions are created based on diverse sensorIDs there is a possibility of the system going out of memory. 

This issue was only fixed in Siddhi 5.x where the system can now clean partitions automatically, and here the user can a purging interval for the partitions. 

I have now added information to the docs here[1].  

[1]https://siddhi-io.github.io/siddhi/documentation/siddhi-5.x/query-guide-5.x/#purge-partition

Regards

Suho

--
Siddhi homepage: http://siddhi.io/
Siddhi Github repo: https://github.com/siddhi-io/siddhi
 
You received this message because you are subscribed to the Google Groups "Siddhi-user" group.
To post to this group, send email to siddh...@googlegroups.com
To unsubscribe from this group, send email to siddhi-user...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/siddhi-user?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "siddhi-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to siddhi-user...@googlegroups.com.
To post to this group, send email to siddh...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.