Enable soft memory limit for pods

[Andrew Snyder]

Not sure if this has been a considered or not. 

Currently memory limits are hard memory limits that result in OOMKilled, but this is undesirable behavior for workloads that can tolerate errno ENOMEM. 

I have data analytic workloads that are able to tolerate ENOMEM results from memory allocations. By allowing soft limit for memory the workload can proceed with fallback or alternative analytics.  Pre determining which analytics will exceed memory limits for pods is not an exact science since its very much data and algorithm dependent. Also attempting to react to pods with OOMKilled is very cumbersome. 

Currently we have to deploy a kludge with LD_PRELOAD to attempt to track allocations to try and avoid OOMKilled and instead set errno ENOMEM. 

The desired behavior would be to configure the pod to use cgroup memory.soft_limit_in_bytes which would give the desired behavior natively.

I was unable to find current or previous discussion around this, so I am curious of the groups thoughs.