cannot pass --memory-swap Docker parameter with value matching memory setting

[Timo Reimann]

Hi,

I'm trying to disable swapping for containers deployed via Marathon while still maintaining a memory limit. The way to do this on the Docker end is to set --memory and --memory-swap to the same value (both parameters are described here). This works fine when running a Docker container locally:

$ docker run --rm --memory 16M --memory-swap 16m alpine:3.3 echo test
test

Trying to choose a value for --memory-swap lower than --memory will be prevented by the Docker daemon quite expectedly:

$ docker run --rm --memory 16M --memory-swap 15m alpine:3.3 echo test
docker: Error response from daemon: Minimum memoryswap limit should be larger than memory limit, see usage..
See 'docker run --help'.

Now let's do the same via Marathon. Here's a minimalistic app definition:

$ cat alpine.json
{
  "id": "/alpine",
  "cmd": "echo test",
  "instances": 1,
  "cpus": 0.1,
  "mem": 16,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "alpine:3.3",
      "network": "BRIDGE",
      "parameters": [
        {
            "key": "memory-swap",
            "value": "16m"
        }
      ]
    }
  }
}

According to the Marathon documentation, the memory setting is expected to be defined in MB, so from my perspective the memory limit values should match up properly.

Deployment still runs through just fine:

$ curl -X PUT -H "Content-Type: application/json" -d@alpine.json http://mesos-master:8080/v2/apps//alpine
{"version":"2016-02-23T00:25:19.633Z","deploymentId":"675b2a3d-8053-4b16-a55b-c4c43c0b14a3"}

However, the container fails to start: We're getting the same error message as if swap memory was set below memory. From our Mesos log:

docker: Error response from daemon: Minimum memoryswap limit should be larger than memory limit, see usage..
See 'docker run --help'.

Interestingly, it continues to fail when I bump the memory to 20 MBs but succeeds when I specify 32 MB. So it doesn't just seem to be a matter of megabyte vs. mebibyte.

Also tried specifying swap memory limits in bytes instead of megabytes but to no avail. The Marathon logs don't give much either.

Can anyone explain the observed behavior to me? Did I miss something?

Thanks

Timo

[Brenden Matthews]

Strange, I tested it and everything seems to work as expected. Here's my app JSON:

{

  "id":"/sleep",

  "cmd":"sleep 999",

  "instances":1,

  "cpus":1,

  "mem":128,

  "container":{

    "type":"DOCKER",

    "docker":{

      "image":"ubuntu",

      "parameters":[

        {

          "key":"memory-swap",

          "value":"128M"

        }

      ]

    }

  }

}

And here's the corresponding state from the master:

[

  {

    "id":"sleep.35c31235-daf4-11e5-899e-02423d95855a",

    "name":"sleep",

    "framework_id":"29ab0dbb-475f-434d-8abc-2bd99da4f02a-0000",

    "executor_id":"",

    "slave_id":"29ab0dbb-475f-434d-8abc-2bd99da4f02a-S1",

    "state":"TASK_RUNNING",

    "resources":{

      "cpus":1.0,

      "disk":0.0,

      "mem":128.0,

      "ports":"[1960-1960]"

    },

    "statuses":[

      {

        "state":"TASK_RUNNING",

        "timestamp":1456317812.14431,

        "container_status":{

          "network_infos":[

            {

              "ip_address":"10.0.2.218",

              "ip_addresses":[

                {

                  "ip_address":"10.0.2.218"

                }

              ]

            }

          ]

        }

      }

    ],

    "container":{

      "type":"DOCKER",

      "docker":{

        "image":"ubuntu",

        "network":"HOST",

        "privileged":false,

        "parameters":[

          {

            "key":"memory-swap",

            "value":"128M"

          }

        ],

        "force_pull_image":false

      }

    }

  }

]

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pradeep Chhetri]

I feel this has something to do with docker version. I am also able to start container with memory and memory-swap set to same value.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

[Timo Reimann]

Brenden,

thanks for trying to reproduce. Interestingly, I'm not having issues when I specify memory limits as high as 128 MB like you did. Could you do me a favor and try with 16 MB, ideally using the alpine image as well?

FWIW, we are running on Marathon 0.14.

Timo

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

[Pradeep Chhetri]

Hello Timo, 

I tried your marathon app configuration and i am not seeing the docker error you are noticing. I am running Docker 1.9.1. May I know which docker version you are running.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Timo Reimann]

If the problem was strictly Docker-specific, presumably the error would show up via docker run as well which isn't the case for neither Docker 1.8 nor 1.10.

I'm rather thinking it's related to how Marathon passes the memory constraints to Docker, especially since things work fine for me with 128 MB but not 16 MB. Curious to hear how low-memory definitions work for others.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Timo Reimann]

Hey Pradeep,

darn it, that falsifies my hypothesis.

We have a Marathon cluster running on Docker 1.8 and another one on 1.10. With both, I see the error when specifying 16 MBs but not when going with 128 MBs.

Thanks

Timo

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brenden Matthews]

The smallest I can go on the latest Marathon/Mesos combo is 32M, which also works fine.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

[Timo Reimann]

I'd be interested to know more about this restriction. How come there's a lower bound of 32 MB? Is that due to Marathon, Mesos, or both? From what version of Marathon/Mesos is this effective?

If 32 MB is going to be the future minimum, it might just be easier for me to drop my 16 MB endeavor.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

[Brenden Matthews]

It's an arbitrary minimum in Mesos that was introduced at the beginning of time. Here's where it was first added:

https://github.com/apache/mesos/commit/34c69e994dd132a2f6465b889af13ea21e2bd2e4

Here's the last time it was changed (a formatting change):

https://github.com/apache/mesos/commit/5d08c659055f73c0ba61326e6fdfb2cce6447de1

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

[Michael Hausenblas]

I came across this some time ago when writing some Marathon docs examples and IIRC it is indeed inherited from Mesos* (minimum amount of memory per offer) but I'd also like to get it confirmed by someone from the core Mesos team.

*) https://github.com/apache/mesos/blob/master/src/master/constants.cpp#L31


Cheers,
Michael

--
Michael Hausenblas, Developer and Cloud Advocate
Mesosphere Inc., the Datacenter Operating System company
Mobile: +353 86 0215164
http://mesosphere.com | http://mhausenblas.info

> $ curl -X PUT -H "Content-Type: application/json" -...@alpine.json http://mesos-master:8080/v2/apps//alpine

> {"version":"2016-02-23T00:25:19.633Z","deploymentId":"675b2a3d-8053-4b16-a55b-c4c43c0b14a3"}
>
>
> However, the container fails to start: We're getting the same error message as if swap memory was set below memory. From our Mesos log:
>
> docker: Error response from daemon: Minimum memoryswap limit should be larger than memory limit, see usage..
> See 'docker run --help'.
>
>
> Interestingly, it continues to fail when I bump the memory to 20 MBs but succeeds when I specify 32 MB. So it doesn't just seem to be a matter of megabyte vs. mebibyte.
>
> Also tried specifying swap memory limits in bytes instead of megabytes but to no avail. The Marathon logs don't give much either.
>
> Can anyone explain the observed behavior to me? Did I miss something?
>
> Thanks
>
> Timo
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

[Timo Reimann]

Brenden,

Michael,

this would explain why 32 MB work just fine but 31 MB trigger the error. The only part I don't understand is why my 16 MB example works for Pradeep. Pradeep, did you verify that the deployed task actually finishes successfully on Mesos? The final state should be FINISHED as opposed to FAILED.

From my perspective, the Marathon documentation should mention this lower bound somewhere in order to spare future generations the hassle we've been going through. :-) I could provide a PR if you like.

Thanks

Timo

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

[Pradeep Chhetri]

Hello Timo,

Really sorry for the late response. Yes, I can confirm it worked for me. I just double-checked everything:

root@ip-172-31-33-174:~# cat alpine.json

{

  "id": "/alpine",

  "cmd": "echo test",

  "instances": 1,

  "cpus": 0.1,

  "mem": 16,

  "container": {

    "type": "DOCKER",

    "docker": {

      "image": "alpine:3.3",

      "network": "BRIDGE",

      "parameters": [

        {

            "key": "memory-swap",

            "value": "16m"

        }

      ]

    }

  }

}

​

I am using latest stable release of mesos (0.27.1) and marathon (0.15.2). Let me know if i can help in debugging further.

Thank you.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pradeep Chhetri]

I doubt MIN_MEM limit has anything to do with the docker error which Timo is noticing because 

https://github.com/apache/mesos/blob/4d4d7166414f0ebd8d5e40df34070446098a3c91/src/master/validation.cpp#L415-L424

just logs a warning error when memory assigned is less than MIN_MEM and doesn't kill the executor.

I came across this MESOS ticket https://issues.apache.org/jira/browse/MESOS-1807 

Thank you.

[Pradeep Chhetri]

https://issues.apache.org/jira/browse/MESOS-1807  clearly states that the currently mesos just emits a warning and continues to allow executors lower than MIN_MEM (32MB).

I noticed this pull request in marathon-ui actually prevents creating apps lower than 32MB from marathon-ui but you can still create app lower than 32MB using marathon APIs.

​

Thank you.

[Timo Reimann]

Pradeep, that's some interesting findings. Strangely, I can't find the warning log message the code is describing in any of our logs even though the log level should be set appropriately. What I do see though is Mesos admitting 16 MB for the task at one point:

Adding task alpine.3b127edd-dbcb-11e5-99c0-02425a6e465a with resources cpus(*):0.1; mem(*):16; ports(*):[31075-31075] on slave 454f282b-a4b3-4472-92d6-005a2dda67f4-S2

The only difference between your setup and ours is that we are still on Mesos 0.26. While the changelog for 0.27 is huge, I couldn't find anything in particular pertaining to memory behavior. I also temporarily swapped Docker 1.10.1 for Docker 1.9.1 (your version), but to no avail.

I guess I'll have to live with the mystery on why things are the way they are. :-)

Thanks again to Pradeep and everyone else assisting!

Timo

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

[Pradeep Chhetri]

Hello Timo,

Since we both were not able to see that warning log message, I was trying to figure out the reason of that and got the solution finally after getting help in mesos mailing list. Just want to share with you as well.

By default, mesos adds some allowance i.e. extra resources (0.1 CPU + 32 MB memory) for the command executor which leads to small overcommit of resources. Here is the code for reference: https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L3732-L3742

I tested this by starting your alpine docker app with 0.01 CPU and 16 MB memory over marathon and mesos actually gave 0.11 CPU and 48 MB memory (by adding that allowance). Here is screenshot for your reference. 

Left side numbers shows after adding allowance and task details on the right side shows without adding allowance. This answers out question :beers::beers:

Thank you Timo,
​

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Timo Reimann]

Hey Pradeep,

if I understand you correctly, it's the executor that gets some extra CPU and memory resources. Wouldn't that account to the Marathon executor itself only as opposed to the Docker containers managed by it?

Also, if these 32 MB are added in addition to the specified 16 MB of memory, why is it that the error stops occurring when I set --memory-swap to 32 MB already (as opposed to 48)?

I feel like I'm yet missing a piece of the puzzle.

Thanks

Timo

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Pradeep Chhetri
>
> In the world of Linux, who needs Windows and Gates...
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "marathon-framework" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.