Performance of gvisor

[Al Izi]

Have anybody measured performance impact of gvisor? What is the expected overhead on a system call?

[Yen Ching-Hsuan]

Following chart is what I got from Byte Unix Benchmark and Phoronix Test Suite v7.8.0 last month ( Higher is better)

	syscall	hanoi(recursive)	c-ray (lower is better)	ramspeed	iozone: read/1MB/2GB	iozone: write/1MB/2GB
gvisor	6785	161672	139	16838	3626	100
gvisor-kvm	84995	159497	109	15764	3689	104
kata container	456757	154010	111	16138	70	56
docker runc	123577	161472	106	17572	4495	102

Al Izi於 2018年6月22日星期五 UTC+8上午7時46分07秒寫道：

[Al Izi]

I've done some basic benchmarking myself. As expected GVisor doesn't have any perf impact on CPU and MEM operations. But IO looks pretty bad. With GVisor disk operations become almost 10 times slower. Is there any plans (possibilities) to address perf issue?

[Fabricio Voznika]

Thanks Yen for posting results!

Al, I'm curious to learn more about the benchmarks you ran. On file IO your mileage may vary. Open/Dir operations will be slower because they need to go through the Gofer, so if your workloads opens lots of files, it will see a hit. Read/write operations are performed directly to the host (using donated FDs) and don't see much performance difference.

[Al Izi]

Al, I'm curious to learn more about the benchmarks you ran.

I've used sysbench disk IO.

sysbench --test=fileio --file-total-size=1G prepare

sysbench fileio --file-total-size=1G --file-test-mode=rndrw --time=60 --max-requests=0 run

 

[Al Izi]

BTW, also I've done some basic benchmarking of the network (with iperf). It seems gvisor makes tcp stack around 30 times slower. 

[Al Izi]

I measured the overhead of interception system calls with ptrace. A "blank" system call takes 20 nanoseconds. With ptrace it becomes 7 milliseconds that's is a lot (x 350 slower)! Do you think this could be solved somehow? Also is ptrace the only reason of the slowness? 

[Adin Scannell]

Hi Al,

Can you elaborate on how you’re measuring, and on what? The numbers don’t make much sense to me.

I assume you mean 7 microseconds? And is it 200 nanoseconds? The two CR3 switches on a KPTI kernel will introduce O(100ns) alone. Or maybe it’s an AMD CPU? Still seems fast.

The KVM platform has much smaller raw switching costs, on the order of hundreds of nanoseconds. Yes, ptrace introduces most of the raw switch costs when you’re using the ptrace platform. If it’s useful to you, there are optimizations where we could probably improve by 10-20%. (Though the fundamental cost are the scheduler switches, which are hard to remove if you’re fixed on ptrace.)

-Adin

On Sat, Jul 28, 2018 at 4:36 PM Al Izi <capitan...@gmail.com> wrote:

I measured the overhead of interception system calls with ptrace. A "blank" system call takes 20 nanoseconds. With ptrace it becomes 7 milliseconds that's is a lot (x 350 slower)! Do you think this could be solved somehow? Also is ptrace the only reason of the slowness? 

--
You received this message because you are subscribed to the Google Groups "gVisor Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To post to this group, send email to gvisor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/1f151388-9d6b-4852-a94f-03350330d4ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Al Izi]

I made a simple app that calls a nonexistent system function (in this case the kernel returns immediately) in a loop:

#include <stdio.h>

#include <unistd.h>

int main()

{

printf("<- start ->\n");

for (int i = 0; i < 1000000; i ++) {

syscall(1000);

}

printf("<- end ->\n");

return 0;

}

 
Also there is a simple ptrace hook that does nothing:

#include <stdio.h>

#include <sys/ptrace.h>

#include <sys/types.h>

#include <sys/wait.h>

#include <unistd.h>

#include <sys/user.h>

#include <sys/reg.h>

#include <unistd.h>

#include <sys/syscall.h>

int main()

{

pid_t child;

long orig_eax;

child = fork();

if(child == 0) {

printf("-- start\n");

ptrace(PTRACE_TRACEME, 0, NULL, NULL);

printf("-- test\n");

execl("./app", "app", nullptr);

}

else {

while(true) {

long orig_eax, eax;

long params[3];

int status;

int insyscall = 0;

wait(&status);

if(WIFEXITED(status))

break;

orig_eax = ptrace(PTRACE_PEEKUSER,

child, 8 * ORIG_RAX, NULL);

//printf(">>> syscall %ld\n", orig_eax);

ptrace(PTRACE_SYSCALL, child, NULL, NULL);

}

}

return 0;

}

The app itself takes ~200ms to run:

time ./app

<- start ->

<- end ->

real    0m0.194s

user    0m0.075s

sys     0m0.117s

With ptrace:

time ./trace

-- start

-- test

<- start ->

<- end ->

real    0m25.885s

user    0m0.874s

sys     0m11.179s

[Adin Scannell]

Thanks, this makes sense.

I think your original math may have been off, based on those numbers. It looks like:

Raw cost => 200ns

Ptrace cost => ~25us

Note that you might get up to a ~2x improvement if you implement some of the optimizations in the platform (e.g. the CPU binding). See the time difference between sys+user and real? That’s the scheduler bouncing around.

In regards to your original question, ptrace is indeed fundamentally slow, so I think you would need to consider the KVM platform if you want to reduce the raw syscall cost.

To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/b21c46be-c18c-4a0f-b038-bf1eeafa1bec%40googlegroups.com.

[Shiru Ren]

Hi Adin,

I am currently using gVisor with Docker. I found that compared with docker, gVisor (Ptrace) may introduce a lot of CPU migrations. Therefore, I wire the task goroutines to certain system threads by using runtime.LockOSThread(), and then use SCHED_SETAFFINITY syscall to pin these threads to certain CPUs. Doing so improves performance slightly (about 30% for running sysbench).

 

To further reduce the syscall cost, I have tried the gVisor (KVM platform). However, the performance is still worse than running docker on a VM (especially for system call heavy workloads). I was wondering if there is any optimizations that I can make to further improve the performance of gVisor (KVM platform)? (e.g., modifying the host kernel so that we can forward some syscalls directly to the host kernel?)

Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/CAAHL2QLE6OYnm7JorMN3jp-4gTnEDbS3ehuv1Be1ST06azbCuA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Shiru Ren
Department of Computer Science
School of EECS
Peking University
Beijing, China

[Ian Lewis]

Hi all,

I think it's important to set expectations. gVisor is going to be slower just due to it's architecture when compared to a straight VM (though for containers you still may need proxies and agents). In exchange the benefits of gVisor are in installation & management of the platform itself (it's just a single static binary as opposed to several agents for something like kata), and in the flexibility of running the sandboxes as processes.

That said, some things like network passthrough (https://github.com/google/gvisor#enabling-network-passthrough) might help for network I/O but forwarding individual syscalls to the kernel kind of breaks the notion of a sandbox. Do you have some extra info about your use case? What is the main reason you need the performance? What kind of syscall heavy workloads would you be running? Are you just running sysbench and how are you running it?

To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/CAAay%2B2G4sAJT%3D%3Dwuor1LoFPbdu8UytN%3DCiYHvf5d%3DoH5dswCFQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Ian Lewis |

Developer Advocate |

ianl...@google.com |

+81 (080)3240-8911