High memory usage and OOM

[soumya prakash datta]

Hi,

We are testing M3 with the following settings. The issue we are observing is that one node from the cluster is using lot memory than others and eventually going OOM. Then when I restarted the down node, it failed multiple times very quickly reaching OOM and eventually starting up after several restarts. 

We are using a 4 node cluster. All nodes has 256 GB memory, two have 32 core CPU with SSD, other two have 40 core cpu with HDD nodes. ( In our final deployment we will have homogenous hardware ) Data is coming from prometheus remote write and a push pipeline, data rate is about 300k-350k/s .

Namespace config - 

{"registry": {"namespaces": {"default": {"bootstrapEnabled": true,"flushEnabled": true,"writesToCommitLog": true,"cleanupEnabled": true,"repairEnabled": false,"retentionOptions": {"retentionPeriodNanos": "2592000000000000","blockSizeNanos": "43200000000000","bufferFutureNanos": "3600000000000","bufferPastNanos": "32400000000000","blockDataExpiry": true,"blockDataExpiryAfterNotAccessPeriodNanos": "300000000000","futureRetentionPeriodNanos": "0"},"snapshotEnabled": true,"indexOptions": {"enabled": true,"blockSizeNanos": "43200000000000"},"schemaOptions": null,"coldWritesEnabled": false}}}}

We are using replication factor of 3.

Some graphs which seems to be aligning with OOM timeframe from the same server -