What are New I/O Worker threads?

TY

unread,

Jul 1, 2015, 2:05:07 PM7/1/15

to ve...@googlegroups.com

Hi

I have a file upload verticle application (vertx 2.1.5). The standard verticle accepts incoming file via HTTP and uses vertx httpClient streams to S3. Then, it sends an event via eventBus to a worker verticle and the worker verticle writes data into a database. Note: The worker verticle is set to use mutli-thread. =(

The application runs on a 4 CPU (dual-core). In a thread-dump, I can see 8 vert.x-eventloop-thread-n, so I increased the -instance switch to 8 to match the # of eventloop thread.

1) Is is a good idea to match the # of standard verticle with the # of eventloop thread?

2) After starting up the verticle and I am seeing 16 New I/O boss #n threads and 168 New I/O worker threads, is it normal?

However, most of the New I/O worker threads are in runnable state but does not consume much CPU cycle.

Note: no load to the system when I took the thread dump.

in the thead dump, most of the I/O worker threads are with the stacktrace.

sun.nio.ch.EPollArrayWrapper

sun.nio.ch.EPollSelectorImpl

sun.nio.ch.SelectorImpl

org.jboss.netty.channel.socket.nio.SelectorUtil

org.jboss.netty.channel.socket.nio.AbstractNioSelector

org.jboss.netty.channel.socket.nio.AbstractNioWorker

org.jboss.netty.channel.socket.nio.NioWorker

org.jboss.netty.util.ThreadRenamingRunnable

org.jboss.netty.util.internal.DeadLockProofWorker$1

java.util.concurrent.ThreadPoolExecutor

java.util.concurrent.ThreadPoolExecutor$Worker

java.lang.Thread

3) The name of the threads are duplicate (twice) in the thread dump?

For example:

New I/O boss #53

New I/O worker #1

Thanks

Tony

Tim Fox

unread,

Jul 2, 2015, 6:36:25 AM7/2/15

to ve...@googlegroups.com

On 01/07/15 19:05, TY wrote:

Hi

I have a file upload verticle application (vertx 2.1.5). The standard verticle accepts incoming file via HTTP and uses vertx httpClient streams to S3. Then, it sends an event via eventBus to a worker verticle and the worker verticle writes data into a database. Note: The worker verticle is set to use mutli-thread. =(

A single worker verticle instance? Any particular reason you are using a multi-threaded worker instead of a standard worker?

The application runs on a 4 CPU (dual-core). In a thread-dump, I can see 8 vert.x-eventloop-thread-n, so I increased the -instance switch to 8 to match the # of eventloop thread.

1) Is is a good idea to match the # of standard verticle with the # of eventloop thread?

Yes, if they are doing a lot of work. You should make number of verticles > number of event loops (maybe up to 2 times)

2) After starting up the verticle and I am seeing 16 New I/O boss #n threads and 168 New I/O worker threads, is it normal?

Can you please provide the *full names* of the threads? Any Vert.x thread will be named "vert.x-worker-thread-XXX" or "vert.x-eventloop-thread-XXX".

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

TY

unread,

Jul 2, 2015, 7:12:23 PM7/2/15

to ve...@googlegroups.com

On Thursday, July 2, 2015 at 6:36:25 AM UTC-4, Tim Fox wrote:

On 01/07/15 19:05, TY wrote:

Hi

I have a file upload verticle application (vertx 2.1.5). The standard verticle accepts incoming file via HTTP and uses vertx httpClient streams to S3. Then, it sends an event via eventBus to a worker verticle and the worker verticle writes data into a database. Note: The worker verticle is set to use mutli-thread. =(

A single worker verticle instance? Any particular reason you are using a multi-threaded worker instead of a standard worker?

8 instances of the worker verticles. Long story why we used multi-threaded, Story version is that this application was written like 9-12 months ago and there was misunderstanding about how to scale. We will be changing every vertx applications to NOT use multi-threaded in next release. However, we still need to deal with it for near term.

The application runs on a 4 CPU (dual-core). In a thread-dump, I can see 8 vert.x-eventloop-thread-n, so I increased the -instance switch to 8 to match the # of eventloop thread.

1) Is is a good idea to match the # of standard verticle with the # of eventloop thread?

Yes, if they are doing a lot of work. You should make number of verticles > number of event loops (maybe up to 2 times)

2) After starting up the verticle and I am seeing 16 New I/O boss #n threads and 168 New I/O worker threads, is it normal?

Can you please provide the *full names* of the threads? Any Vert.x thread will be named "vert.x-worker-thread-XXX" or "vert.x-eventloop-thread-XXX".

The name for the New I/O worker threads are

New I/O boss #53

New I/O boss #56

New I/O boss #57

New I/O boss #58

New I/O boss #59

New I/O boss #60

New I/O boss #61

New I/O boss #65

New I/O boss #69

New I/O boss #70

New I/O boss #71

New I/O boss #72

New I/O worker #1

New I/O worker #2

New I/O worker #3

New I/O worker #10

New I/O worker #11

New I/O worker #12

New I/O worker #13

...

..

New I/O worker #70

Yes, I saw

vert.x-eventloop-thread-0

vert.x-eventloop-thread-1

vert.x-eventloop-thread-2

vert.x-eventloop-thread-3

vert.x-eventloop-thread-4

vert.x-eventloop-thread-5

vert.x-eventloop-thread-6

vert.x-eventloop-thread-7

vert.x-worker-thread-0

vert.x-worker-thread-1

vert.x-worker-thread-2

vert.x-worker-thread-3

vert.x-worker-thread-4..

...

vert.x-worker-thread-199

Tim Fox

unread,

Jul 3, 2015, 3:25:59 AM7/3/15

to ve...@googlegroups.com

The threads above aren't Vert.x threads. They look like threads created by Netty directly. Are you using Netty directly in your application (i.e. not Vert.x Vert.x)?

Yes, I saw

vert.x-eventloop-thread-0

vert.x-eventloop-thread-1

vert.x-eventloop-thread-2

vert.x-eventloop-thread-3

vert.x-eventloop-thread-4

vert.x-eventloop-thread-5

vert.x-eventloop-thread-6

vert.x-eventloop-thread-7

You have 8 Vert.x event loop threads. This seems correct seeing that you have 4 cores and Vert.x will choose 2 * number of cores by default.

vert.x-worker-thread-0

vert.x-worker-thread-1

vert.x-worker-thread-2

vert.x-worker-thread-3

vert.x-worker-thread-4..

...

vert.x-worker-thread-199

Did you change the size of the worker pool?

The default max size is 20:

https://github.com/eclipse/vert.x/blob/2.x/vertx-core/src/main/java/org/vertx/java/core/impl/VertxExecutorFactory.java#L39

And it's a fixed thread pool

https://github.com/eclipse/vert.x/blob/2.x/vertx-core/src/main/java/org/vertx/java/core/impl/VertxExecutorFactory.java#L44

so it should not be possible to have more than 20 unless you have provided a system property to increase the pool size

https://github.com/eclipse/vert.x/blob/2.x/vertx-core/src/main/java/org/vertx/java/core/impl/VertxExecutorFactory.java#L43

TY

unread,

Jul 3, 2015, 1:55:41 PM7/3/15

to ve...@googlegroups.com

Yes, we changed the worker thread count to 200.

Tim Fox

unread,

Jul 4, 2015, 3:48:39 AM7/4/15

to ve...@googlegroups.com

On 03/07/15 18:55, TY wrote:
> Yes, we changed the worker thread count to 200.
>

Ok, so all the Vert.x thread look normal to me :)

Reply all

Reply to author

Forward