stuck with high sys usage when sending 0 sized DefaultFileRegion

[Reynold Xin]

I was testing Spark's new Netty network module with a new workload and suddenly the whole system stopped. Nothing was being transferred and sys usage was ~ 90%. Experimentally, I found that if I skip 0-sized blocks (basically sending a DefaultFileRegion with region length 0), this would happen.

I didn't have time to nail this down, but I thought I'd report this here so others can find it. And maybe someone else can pinpoint the problem. It is either in Netty, or in the JVM, or in the kernel, or in Xen ...

This is what perf shows:

   70.21%            java  [kernel.kallsyms]   [k] pvclock_clocksource_read                                       
                      |
                      --- pvclock_clocksource_read
                         |          
                         |--99.68%-- xen_clocksource_get_cycles
                         |          |          
                         |          |--85.42%-- ktime_get_ts
                         |          |          |          
                         |          |          |--67.81%-- posix_ktime_get_ts
                         |          |          |          SyS_clock_gettime
                         |          |          |          system_call_fastpath
                         |          |          |          |          
                         |          |          |          |--100.00%-- 0x7fffac1fe7d2
                         |          |          |          |          clock_gettime
                         |          |          |          |          |          
                         |          |          |          |          |--24.74%-- 0x7f9f9d4075d0
                         |          |          |          |          |          
                         |          |          |          |          |--24.59%-- 0x7f9f9d2cf6ab
                         |          |          |          |          |          
                         |          |          |          |          |--24.55%-- 0x7f9f9d415901
                         |          |          |          |          |          0x7f9ed9a04188
                         |          |          |          |          |          
                         |          |          |          |          |--24.19%-- 0x7f9f9d2cf803
                         |          |          |          |          |          
                         |          |          |          |          |--0.52%-- 0x7f9f9d42891f
                         |          |          |          |          |          0x7f9ed9561258
                         |          |          |          |           --1.40%-- [...]
                         |          |          |           --0.00%-- [...]

[Reynold Xin]

Sorry I meant if I skip 0-sized blocks, this wouldn't have happened.

[Reynold Xin]

More information about the OS/JDK:

root@ip-10-0-4-108 ~]$ uname -r

3.10.53-56.140.amzn1.x86_64

root@ip-10-0-4-108 ~]$ java -version

java version "1.7.0_65"

OpenJDK Runtime Environment (amzn-2.5.1.2.44.amzn1-x86_64 u65-b17)

OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

[Norman Maurer]

You are using the native transport?

--

---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Reynold Xin]

This is epoll I'm using.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/2gLgE_4q3Pc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.

[Norman Maurer]

Sounds like a bug.... Can you open an issue please?

[Reynold Xin]

Filed. https://github.com/netty/netty/issues/2964

Thanks.

[Norman Maurer]

Thanks!

-- 
Norman Maurer