How to fix : jboss.netty.handler.codec.frame.TooLongFrameException:Adjusted frame length exceeds

[mohammed omar]

I am trying to run my spark on cluster and it is failing at below mentioned stack trace. I had tried to change the config value in reference.conf under lib_managed/jars/akka-remote-2.0.3.jar file by unjaring and jaring method. The setting does not seems to make any difference as i tried to set this to many different values but no luck.

    # (I&O) Increase this if you want to be able to send messages with large payloads
      message-frame-size = 100 MiB



I am using spark-0.6.2 with scala 2.9.3.

Any help or pointers will be highly appreciated.


13/06/17 11:48:03 INFO spark.CacheTracker: CacheTracker successfully removed entries on seeloz-3
13/06/17 11:48:03 INFO spark.CacheTrackerActor: Asked for current cache locations
13/06/17 11:48:05 ERROR actor.ActorSystemImpl: RemoteServerError@akka://sp...@107.215.218.115:50653] Error[org.10485760: 48213060 - discarded
        at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:441)
        at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:407)
        at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:322)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
]
13/06/17 11:48:05 INFO actor.ActorSystemImpl: RemoteClientShutdown@akka://spark@seeloz-3:56324


Exception on the worker node captured through WEB UI of spark:

13/06/17 11:47:58 INFO storage.MemoryStore: Block rdd_16_6 stored as values to memory (estimated size 87.4 MB, free 113.8 MB)
13/06/17 11:48:00 INFO executor.Executor: Serialized size of result for 5 is 44959282
13/06/17 11:48:01 INFO executor.Executor: Finished task ID 5
13/06/17 11:48:01 INFO executor.Executor: Serialized size of result for 4 is 46588602
13/06/17 11:48:03 INFO executor.Executor: Finished task ID 4
13/06/17 11:48:03 INFO actor.ActorSystemImpl: RemoteClientShutdown@akka://sp...@107.215.218.115:50653
13/06/17 11:48:03 WARN actor.ActorSystemImpl: RemoteClientWriteFailed@akka://sp...@107.215.218.115:50653: MessageClass[scala.Tuple3] Error[java.nio.channels.ClosedChannelException:null
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:674)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:642)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:98)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
]
13/06/17 11:48:03 ERROR actor.ActorSystemImpl: RemoteClientError@akka://sp...@107.215.218.115:50653: Error[java.nio.channels.ClosedChannelException:null
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:674)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:642)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:98)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
]
13/06/17 11:48:03 INFO executor.Executor: Serialized size of result for 7 is 48268588
13/06/17 11:48:03 INFO actor.ActorSystemImpl: RemoteClientStarted@akka://sp...@107.215.218.115:50653
13/06/17 11:48:04 INFO executor.Executor: Serialized size of result for 6 is 47977066
13/06/17 11:48:05 INFO executor.Executor: Finished task ID 6
13/06/17 11:48:05 INFO executor.Executor: Finished task ID 7
13/06/17 11:48:05 INFO actor.ActorSystemImpl: RemoteClientShutdown@akka://sp...@107.215.218.115:50653
13/06/17 11:48:05 WARN actor.ActorSystemImpl: RemoteClientWriteFailed@akka://sp...@107.215.218.115:50653: MessageClass[scala.Tuple3] Error[java.nio.channels.ClosedChannelException:null
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:674)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:642)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:98)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
]
13/06/17 11:48:05 ERROR actor.ActorSystemImpl: RemoteClientError@akka://sp...@107.215.218.115:50653: Error[java.nio.channels.ClosedChannelException:null
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:674)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:642)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:98)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
]

[Akka Team]

Hi Mohammed,

replacing a file in a JAR is not the recommended way to configure Akka: try putting only that setting (with its full path, i.e. “akka.remote.netty.message-frame-size=100MiB”) into a file named “reference.conf” on the classpath; unfortunately I do not know how Spark starts its ActorSystem so I cannot promise that this works. Another thing you can try is passing the aforementioned string as a system property to your JVM.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam