Daemonic Akka creating non-daemon threads

[Matei Zaharia]

Hi,

I've noticed that in Akka 2.1.x, setting the "daemonic" flag to true no longer ensures that all of Akka's threads are daemons. Instead, if you register any remote actors, there are some leftover threads like this:

"New I/O server boss #18" prio=10 tid=0x00007f83949eb000 nid=0x5fb runnable [0x00007f836d0a8000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)

    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x00000000feae8170> (a sun.nio.ch.Util$2)

    - locked <0x00000000feae8180> (a java.util.Collections$UnmodifiableSet)

    - locked <0x00000000feae8128> (a sun.nio.ch.EPollSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

    at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)

    at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)

    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

    at java.lang.Thread.run(Thread.java:722)

"New I/O worker #17" prio=10 tid=0x00007f83949d9000 nid=0x5fa runnable [0x00007f836d1a9000]

   java.lang.Thread.State: RUNNABLE

    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)

    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)

    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

    - locked <0x00000000feb200e8> (a sun.nio.ch.Util$2)

    - locked <0x00000000feb200f8> (a java.util.Collections$UnmodifiableSet)

    - locked <0x00000000feb200a0> (a sun.nio.ch.EPollSelectorImpl)

    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)

    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)

    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

    at java.lang.Thread.run(Thread.java:722)

Calling ActorSystem.shutdown() does shut these down but, I'd like them to be daemons even if you don't call shutdown(). This seems to have changed from Akka 2.0.x. Is this a known problem or an intended behavior? I've seen one thread related to it here: https://www.assembla.com/spaces/akka/tickets/2733#/activity/ticket: but that seemed to be due to shutdown() not shutting things down, whereas I'd like these threads to be daemons in the first place. Is there any known workaround for it?

[Patrik Nordwall]

Thanks for reporting. We will look into it.

Is it akka 2.1.4? Otherwise, please try with that version.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Patrik Nordwall
Typesafe -  The software stack for applications that scale
Twitter: @patriknw

[Matei Zaharia]

Yes, it was in 2.1.4, and it looks like it's been fixed in https://www.assembla.com/spaces/akka/tickets/3436. I'm curious, will the fix be merged into 2.1.5, or should I upgrade to 2.2?

Matei

[√iktor Ҡlang]

https://github.com/akka/akka/pull/1559/files

Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Matei Zaharia]

Awesome, thanks!

Matei

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/tLBZu5Sr0vU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[√iktor Ҡlang]

I just want to clarify that there is no ship date for a 2.1.5 release at this date.

Cheers,

√

[Jeff Crilly]

We've had some recent "production issues" and it looks like this is the exact issue biting us; (this results in a need to restart our appservers rather periodically.)

The app is on akka 2.1.4.

I scanned the akka 2.1.x to 2.2.x migration guide a while ago (and also just now), and it didn't seem like a "drop in library" change.   (Am I missing something?)

Any other thoughts on how we might mitigate the issue alternatively w/o 2.2 upgrade?   (Thought i'd ask anyhow.)

(Or do we.. ahem.. need to bite the bullet and do the upgrade?  This is an application that is undergoing minor bug fixes, but not major changes, hence rolling a full regression etc is not quite in the plan; we were looking for a narrow fix/solution rather than (afaict) refactoring the akka usage.) 

thx!

[Björn Antonsson]

Hi Jeff,

On Tuesday, 14 January 2014 at 03:54, Jeff Crilly wrote:

We've had some recent "production issues" and it looks like this is the exact issue biting us; (this results in a need to restart our appservers rather periodically.)

The app is on akka 2.1.4.

I’m just curious as to what you mean by periodically. Also, why does you app server outlive the actor systems created inside it? You shouldn’t need to start/stop actor systems frequently.

I scanned the akka 2.1.x to 2.2.x migration guide a while ago (and also just now), and it didn't seem like a "drop in library" change.   (Am I missing something?)

You are right that there are a number of changes between 2.1 and 2.2 that might make upgrading (depending on you application) include some rewriting of you application. The easiest thing is probably to try to build against 2.2.x.

Any other thoughts on how we might mitigate the issue alternatively w/o 2.2 upgrade?   (Thought i'd ask anyhow.)

(Or do we.. ahem.. need to bite the bullet and do the upgrade?  This is an application that is undergoing minor bug fixes, but not major changes, hence rolling a full regression etc is not quite in the plan; we were looking for a narrow fix/solution rather than (afaict) refactoring the akka usage.) 

I think that upgrading to 2.2 is the only available solution for the thread leakage right now, unless you keep the same actor system around in the app server.

B/

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

[Jeff Crilly]

Thx for the feedback!

Re "ouliving"... Afaict, the app just stops getting connections, so its hard to tell there is a problem from the "inside".  (Fwiw, this app also connects to other systems which might be down; hence the app needs to deal with failure and recover those connections -- simply exiting the app is not an option in those scenarios.)

As for "periodically"... good question.. one thing i'm asking for is the "rate of leak".   It is not on the order of "minutes", more like "days", and may actually be "weeks" (I know this sounds odd, but i dont have direct access to the production systems and need to go through a couple layers of IT folks in different geos and timezones.)

Anyhow, I just wanted to check on the options/alternatives that might not be obvious.  I'm a fan of getting on the latest stable (but not bleeding edge).

-jeff

[Björn Antonsson]

Hi Jeff,

On Thursday, 16 January 2014 at 04:34, Jeff Crilly wrote:

Thx for the feedback!

Re "ouliving"... Afaict, the app just stops getting connections, so its hard to tell there is a problem from the "inside".  (Fwiw, this app also connects to other systems which might be down; hence the app needs to deal with failure and recover those connections -- simply exiting the app is not an option in those scenarios.)

So that the app stops getting connections shouldn’t be related to this bug. This bug is about the Netty thread pools not being properly shut down when the Akka actor system is shut down.

Have you, or your ops guys, taken any kind of diagnostics, i.e. thread/heap dump, of the system when it fails?

As for "periodically"... good question.. one thing i'm asking for is the "rate of leak".   It is not on the order of "minutes", more like "days", and may actually be "weeks" (I know this sounds odd, but i dont have direct access to the production systems and need to go through a couple layers of IT folks in different geos and timezones.)

That doesn’t sound odd, just cumbersome and way to common. ;)

B/

[Rohit Gupta]

Hi,

Can someone give me more insight for this bug ? 

We are having a similar issue in our app in production, but I am unable to reproduce the same issue in dev setups and labs. 

A use case or a test to reproduce this would be very helpful. 

Thanks

- Rohit 

[Björn Antonsson]

Hi Rohit,

What problem are you referring to exactly? There where two things discussed in this thread. And they where unrelated.

If you have some problems that you think is a bug, then please describe your problems in a new post.

B/

Thanks

- Rohit 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/

>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html

>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/d/optout.