stuck with high sys usage when sending 0 sized DefaultFileRegion

105 views
Skip to first unread message

Reynold Xin

unread,
Oct 3, 2014, 2:15:27 AM10/3/14
to ne...@googlegroups.com
I was testing Spark's new Netty network module with a new workload and suddenly the whole system stopped. Nothing was being transferred and sys usage was ~ 90%. Experimentally, I found that if I skip 0-sized blocks (basically sending a DefaultFileRegion with region length 0), this would happen.

I didn't have time to nail this down, but I thought I'd report this here so others can find it. And maybe someone else can pinpoint the problem. It is either in Netty, or in the JVM, or in the kernel, or in Xen ...


This is what perf shows:

70.21% java [kernel.kallsyms] [k] pvclock_clocksource_read
|
--- pvclock_clocksource_read
|
|--99.68%-- xen_clocksource_get_cycles
| |
| |--85.42%-- ktime_get_ts
| | |
| | |--67.81%-- posix_ktime_get_ts
| | | SyS_clock_gettime
| | | system_call_fastpath
| | | |
| | | |--100.00%-- 0x7fffac1fe7d2
| | | | clock_gettime
| | | | |
| | | | |--24.74%-- 0x7f9f9d4075d0
| | | | |
| | | | |--24.59%-- 0x7f9f9d2cf6ab
| | | | |
| | | | |--24.55%-- 0x7f9f9d415901
| | | | | 0x7f9ed9a04188
| | | | |
| | | | |--24.19%-- 0x7f9f9d2cf803
| | | | |
| | | | |--0.52%-- 0x7f9f9d42891f
| | | | | 0x7f9ed9561258
| | | | --1.40%-- [...]
| | | --0.00%-- [...]


Reynold Xin

unread,
Oct 3, 2014, 2:15:51 AM10/3/14
to ne...@googlegroups.com
Sorry I meant if I skip 0-sized blocks, this wouldn't have happened.

Reynold Xin

unread,
Oct 3, 2014, 2:18:46 AM10/3/14
to ne...@googlegroups.com
More information about the OS/JDK:

root@ip-10-0-4-108 ~]$ uname -r

3.10.53-56.140.amzn1.x86_64

root@ip-10-0-4-108 ~]$ java -version

java version "1.7.0_65"

OpenJDK Runtime Environment (amzn-2.5.1.2.44.amzn1-x86_64 u65-b17)

OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

Norman Maurer

unread,
Oct 3, 2014, 2:48:40 AM10/3/14
to ne...@googlegroups.com
You are using the native transport?
--

---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reynold Xin

unread,
Oct 3, 2014, 3:01:22 AM10/3/14
to ne...@googlegroups.com
This is epoll I'm using.


--

---
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/2gLgE_4q3Pc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.

Norman Maurer

unread,
Oct 3, 2014, 4:53:13 AM10/3/14
to ne...@googlegroups.com
Sounds like a bug.... Can you open an issue please?

Reynold Xin

unread,
Oct 3, 2014, 4:06:27 PM10/3/14
to ne...@googlegroups.com

Norman Maurer

unread,
Oct 6, 2014, 4:55:49 AM10/6/14
to ne...@googlegroups.com, Reynold Xin
Thanks!

-- 
Norman Maurer
Reply all
Reply to author
Forward
0 new messages