Docker experience?

[Shahbaz Chaudhary]

Does anyone on this list know how the use of docker effects latency? I keep hearing about it as if it is the greatest thing since sliced bread but I've heard anecdotal evidence that low latency apps take a hit.

[Brian Caswell]

http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/

---

An Updated Performance Comparison of Virtual Machines and Linux Containers

Isolation and resource control for cloud applications has traditionally been achieve through the use of virtual machines. Deploying applications in a VM results in reduced performance due to the extra levels of abstraction. In a cloud environment, this results in loss efficiency for the infrastructure. Newer advances in container-based virtualization simplifies the deployment of applications while isolating them from one another.

In this paper, we explore the performance of traditional virtual machine deployments, and contrast them with the use of Linux containers. We use a suite of workloads that stress the CPU, memory, storage and networking resources.

Our results show that containers result in equal or better performance than VM in almost all cases. Both VMs and containers require tuning to support I/O-intensive applications. We also discuss the implications of our performance results for future cloud architecture

By: Wes Felter, Alexandre Ferreira, Ram Rajamony, Juan Rubio

On Wed, Nov 11, 2015 at 10:55 AM, Shahbaz Chaudhary <shah...@gmail.com> wrote:

Does anyone on this list know how the use of docker effects latency? I keep hearing about it as if it is the greatest thing since sliced bread but I've heard anecdotal evidence that low latency apps take a hit.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Evans]

If you care about low latency, why on earth would you think that a virtualized OS was a good idea?

There are doubtless good papers on the low latency behaviour of e.g. Solaris zones or FreeBSD jails. That is the yardstick that Docker & its hype should be compared to IMO.

My 2c is that Dockers clumsiness, especially in terms of how it approaches networking probably doom it for low latency, but I'm more than happy to be proved wrong. On the quiet, I quite like Docker & would like it to live up to its spin.

Ben

[Pekka Enberg]

AFAICT, most people use Docker in NAT configuration which indeed adds
network latency. However, when you enable bridged mode, there's pretty
much no difference to a bare metal configuration.

- Pekka

[Shahbaz]

I thought docker wasn't virtualized. In fact, my impression was that docker provides isolation of a virtual machine, without needing to simulate a full cpu (essentially FreeBSD jails for linux + a Golang interface). To clarify, are you saying that freebsd jails maintain good performance but docker doesn't?

You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/8QkiLhHC5-M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mechanical-symp...@googlegroups.com.

[Michael Barker]

My understanding is that when Docker is using its bridge communication mode it uses the Linux software bridge, which significantly limits throughput when compared to native networking or other alternatives, like Mac VLans.

Mike.

[Ben Evans]

Docker isn't virtualized. It's a relatively primitive reimplementation of the equivalent of Solaris zones.

It has a heavily-NATed approach to networking and some other rather unfortunate design decisions.

My overall opinion is: some good ideas, some questionable design decisions, and a community with worrying amounts of NIH.

Ben

[James Cape]

https://access.redhat.com/articles/1407003 (behind a paywall, can't remember the details).

Really depends on the definition of "low latency"...

[Todd Hoff]

This might be of interest:

The impact of Docker containers on the performance of genomic pipelines. https://peerj.com/articles/1273/.

"Docker containers have only a minor impact on the performance of common genomic pipelines, which is negligible when the executed jobs are long in terms of computational time."

[Gil Tene]

Putting aside questions of taste and style, and focusing on the effects on latency (the original question), the analysis from a pure mechanical point of view is pretty simple: Docker uses Linux containers as a mean of execution, with no OS virtualization layer for CPU and memory, and with optional (even if default is on) virtualization layers for i/o. 

CPU and Memory:

From a latency point of view, Docker's (and any other Linux container's) CPU and memory latency characteristics are pretty much indistinguishable from Linux itself. But the same things that apply to latency behavior in Linux apply to Docker. If you want clean & consistent low latency, you'll have to the same things you need to do on non-dockerized and non-containerized Linux for the same levels of consistency. E.g. if you needed to keep the system as a whole under control (no hungry neighbors), you'll have to do that at the host level for Docker as well. If you needed to isolate sockets or cores and choose which processes end up where, expect to do the same for your docker containers and/or the threads within them.  If you were numactl'ing or doing any sort of directed numa-driven memory allocation, the same will apply. Snd some of the stuff you'll need to do may seem counter-style to how some people want to deploy docker, but if you are really interested in consistent low latency, you'll probably need to break out the toolbox and use the various cgroups, tasksets and other cool stuff to assert control over how things are laid out. But if/when you do, you won't be able to tell the difference (in terms of CPU and memory latency behaviors) between a dockeriz'ed process and one that isn't.

I/O:

Disk I/O:

I/O behavior under various confiurations is where most of the latency overhead questions (and answers) usually end up. I don't know enough about disk i/o behaviors and options in docker to talk about it much. I'm pretty sure the answer to anything throughput and latency sensitive for storage will be "bypass the virtualization and volumes stuff, and provide direct device access to disks and mount points". 

Networking:

The networking situation is pretty clear: If you want one of those "land anywhere and NAT/bridge with some auto-generated networking stuff" deployments, you'll probably pay dearly for that behavior in terms of network latency and throughput (compared to bare metal dedicated NICs on normal linux). However, there are options for deploying docker containers (again, may be different from how some people would like to deploy things) that provide either low-overhead or essentially zero-latency-overhead network links for docker. Start with host networking and/or use dedicated IP addresses and NICs, and you'll do much better than the bridged defaults. But you can go to things like Solarflare's NICs (which tend to be common in bare metal low latency environments already), and even do kernel bypass, dedicated spinning-core network stack things that will have a latency behavior no different (on Docker) than if you did the same on bare metal Linux. 

On Wednesday, November 11, 2015 at 7:55:34 AM UTC-8, Shahbaz Chaudhary wrote:

[Shahbaz Chaudhary]

Gil, thanks for the awesomely detailed response. This is exactly what I was looking for.

[Sarunas Vancevicius]

On Thursday, 12 November 2015 18:23:56 UTC+3, James Cape wrote:

https://access.redhat.com/articles/1407003 (behind a paywall, can't remember the details).

Results of overhead (with Onload and without) along with more general information is published under:

http://people.redhat.com/jeder/presentations/summit_2015/virt-vs-containers/summit-2015-containers-and-virt-final.pdf

[Martin Thompson]

On Thursday, 12 November 2015 17:53:17 UTC, Gil Tene wrote:

Networking:

The networking situation is pretty clear: If you want one of those "land anywhere and NAT/bridge with some auto-generated networking stuff" deployments, you'll probably pay dearly for that behavior in terms of network latency and throughput (compared to bare metal dedicated NICs on normal linux). However, there are options for deploying docker containers (again, may be different from how some people would like to deploy things) that provide either low-overhead or essentially zero-latency-overhead network links for docker. Start with host networking and/or use dedicated IP addresses and NICs, and you'll do much better than the bridged defaults. But you can go to things like Solarflare's NICs (which tend to be common in bare metal low latency environments already), and even do kernel bypass, dedicated spinning-core network stack things that will have a latency behavior no different (on Docker) than if you did the same on bare metal Linux. 

We are beginning to see advances on the network side.

http://blog.weave.works/2015/11/13/weave-docker-networking-performance-fast-data-path/

However if you want really fast networking then bypassing the kernel is the best way to go with Solarflare or Mellanox.

 

[ymo]

Can someone explain why they would even use docker in the first place if you want to minimize i/o and compute latency for an app ? 

Normally if you want low latency you would strive to have dedicated cpu for your i/o and compute jobs. To me the selling point of docker was to make a large number of guests that are mostly doing nothing on one host. Whats the point of having more than one hungry guests on the same host ? And if the number of hungry guests is equal to one .. then why even use docker ?

This seems like someone who wants their cake and eat it too ! Meaning i want a large number of collocated guests and i also want low latency !

[Shahbaz]

>Can someone explain why they would even use docker in the first place if you want to minimize i/o and compute latency for an app ? 

Say you have your software installed on a large number of physical machines in different data centers. One way of managing such a system is to package your software and its local dependencies in a docker container, test it locally and deploy the whole thing to remote servers.

As for why use docker when the goal is to minimize latency, that is the reason for the original question. If docker has a huge effect in latency, than this is obviously not a solution. If it does not, then docker could be a fantastic tool.

--

[Marshall Pierce]

Docker is not just about having multiple apps running in relative isolation from each other on the same host. A major benefit is the fact that the people in control of the deliverable app (e.g. a bunch of jars) are now also in control of much more of the final deliverable, namely the userland. Need to update your jvm? Just ship a new image with those bytes with you next app version. Instead of depending on the shared jvm (and libc, and …), you now depend on the kernel, which is typically more satisfactory to more people at the same time than a specific version of the JVM or other userland things. I don’t think Docker has any inherent benefits for low latency specifically, aside from the above which applies to any project as far as I can tell.

> --
> You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Gil Tene]

Docker (which is "userland as a unit") is not about packing lots of thing into a box. Neither is guest-OS-as-a-unit virtualization. Sure, they can both be used for that (and often are), but the biggest benefit they both give is the ability to ship around a consistent, well captured configuration. And the ability to develop, test, and deploy that exact same configuration. This later turns into being able to easily manage deployment and versioning (including roll backs), and being able to do cool things like elastic sizing, etc. There are configuration tools (puppet/chef/...) that can be used to achieve similar results on bare metal as well, of course (assuming they truly control everything in your image), but the ability to pack up your working stuff as a bunch of bits that can "just be turned on" is a very appealing.

I know people who use virtualization even with a single guest-per-host (e.g. an AWS r3.8xlarge instance type is probably that right now). And people who use docker the same way (single container per host). In both cases, it's about configuration control and how things get deployed, and not at all about packing things in a smaller footprint.

The low latency thing then becomes a "does it hurt?" question. And Docker hurts a lot less than hypervisor or KVM based virtualization does when it comes to low latency, and with the right choices for I/O (dedicated NICs, cores, and devices), it becomes truly invisible.