Building Spark with Akka 2.4.0

308 views
Skip to first unread message

Rodrigo Boavida

unread,
Dec 1, 2015, 8:35:03 AM12/1/15
to Akka User List
Hello Akka users,

I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0.
I've changed both the scala settings in Spark build files and the main pom.xml file to corresponding akka version - 2.4.0 - and am getting the following exception when starting the master on standalone:

Exception Details:
  Location:
    akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic
  Reason:
    Type top (current frame, locals[9]) is not assignable to 'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9])
  Current Frame:
    bci: @131
    flags: { }
    locals: { 'akka/dispatch/Mailbox', 'java/lang/InterruptedException', 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', 'java/lang/Throwable', 'java/lang/Throwable' }
    stack: { integer }
  Stackmap Frame:
    bci: @152
    flags: { }
    locals: { 'akka/dispatch/Mailbox', 'java/lang/InterruptedException', 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', 'java/lang/Throwable', 'java/lang/Throwable', top, top, 'akka/dispatch/sysmsg/SystemMessage' }
    stack: { }
  Bytecode:
    0x0000000: 014c 2ab2 0132 b601 35b6 0139 4db2 013e
    0x0000010: 2cb6 0142 9900 522a b600 c69a 004b 2c4e
    0x0000020: b201 3e2c b601 454d 2db9 0148 0100 2ab6
    0x0000030: 0052 2db6 014b b801 0999 000e bb00 e759
    0x0000040: 1301 4db7 010f 4cb2 013e 2cb6 0150 99ff
    0x0000050: bf2a b600 c69a ffb8 2ab2 0132 b601 35b6
    0x0000060: 0139 4da7 ffaa 2ab6 0052 b600 56b6 0154
    0x0000070: b601 5a3a 04a7 0091 3a05 1905 3a06 1906
    0x0000080: c100 e799 0015 1906 c000 e73a 0719 074c
    0x0000090: b200 f63a 08a7 0071 b201 5f19 06b6 0163
    0x00000a0: 3a0a 190a b601 6899 0006 1905 bf19 0ab6
    0x00000b0: 016c c000 df3a 0b2a b600 52b6 0170 b601
    0x00000c0: 76bb 000f 5919 0b2a b600 52b6 017a b601
    0x00000d0: 80b6 0186 2ab6 018a bb01 8c59 b701 8e13
    0x00000e0: 0190 b601 9419 09b6 0194 1301 96b6 0194
    0x00000f0: 190b b601 99b6 0194 b601 9ab7 019d b601
    0x0000100: a3b2 00f6 3a08 b201 3e2c b601 4299 0026
    0x0000110: 2c3a 09b2 013e 2cb6 0145 4d19 09b9 0148
    0x0000120: 0100 1904 2ab6 0052 b601 7a19 09b6 01a7
    0x0000130: a7ff d62b c600 09b8 0109 572b bfb1
  Exception Handler Table:
    bci [290, 307] => handler: 120
  Stackmap Table:
    append_frame(@13,Object[#231],Object[#177])
    append_frame(@71,Object[#177])
    chop_frame(@102,1)
    full_frame(@120,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#177]},{Object[#223]})
    full_frame(@152,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#223],Object[#223],Top,Top,Object[#177]},{})
    append_frame(@173,Object[#357])
    full_frame(@262,{Object[#2],Object[#231],Object[#177],Top,Object[#2]},{})
    same_frame(@307)
    same_frame(@317)
   at akka.dispatch.Mailboxes.<init>(Mailboxes.scala:33)
        at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:635)
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:143)
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:120)
        at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121)
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:166)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
        at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55)
        at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:253)
        at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53)
        at org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1074)
        at org.apache.spark.deploy.master.Master$.main(Master.scala:1058)
        at org.apache.spark.deploy.master.Master.main(Master.scala)

---------------

Has anyone encountered this problem before? Seems to be related with a version mismatch at some level with the Akka mailbox. I would very much appreciate any comments or suggestions.

tnks,
Rod

Endre Varga

unread,
Dec 1, 2015, 8:44:44 AM12/1/15
to akka...@googlegroups.com
Hi,

Akka 2.4 is binary compatible with 2.3 (at least should be) so this is either
 - Spark is using internal Akka API which is not guaranteed to be binary compatible (likely)
 - We accidentally broke binary compatibility of user facing API (unlikely)

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Viktor Klang

unread,
Dec 1, 2015, 9:06:59 AM12/1/15
to Akka User List
Which JDK version are you using? Which Scala version?
--
Cheers,
√

Rodrigo Boavida

unread,
Dec 1, 2015, 9:54:47 AM12/1/15
to Akka User List
Hi Endre,

Are you suggesting not to upgrade Akka's dependency in Spark from 2.3.11 to 2.4.0? I will have Spark applications running on top of Akka 2.4.0 and Scala 2,11,7. I've read that 2.4.0 is backwards compatible with 2.3 in binary but by upgrading to 2.4.0 I'm trying to reduce the risk of having Spark overriding any 2.4.0 behavior and potentially adding runtime dependency conflicts. I can see netty for example is more recent in Spark 1.5.2 than in Akka 2.4.0. I would generally like to avoid loading different binary versions of the same library.
Am I going the wrong way here?

I will continue with more details on my answer to Viktor.  

tnks,
Rod

Endre Varga

unread,
Dec 1, 2015, 9:58:01 AM12/1/15
to akka...@googlegroups.com
Hi Rodrigo,

My point is that upgrading to 2.4.0 should work out of the box. If not, it is either because there is some use of Akka internal APIs in the Spark project, or some binary incompatibility in our release process leaked through. I have not suggested anything, I assessed the situation.

Also, Akka 2.4 is built against Java 8, maybe that is your problem?

-Endre

Rodrigo Boavida

unread,
Dec 1, 2015, 9:59:10 AM12/1/15
to Akka User List
Hi Viktor,

I'm using JDK 1.8.0_31

Scala is 2.11.7

Running a 'mvn dependency:tree' I get the following:

[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.5.2: The following artifacts could not be resolved: com.typesafe.akka:akka-remote_2.10:jar:2.4.0, com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0, com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact com.typesafe.akka:akka-remote_2.10:jar:2.4.0 in central (https://repo1.maven.org/maven2) -> [Help 1]

So it seems somehow it's still pulling some 2.10 dependencies. Could this be the cause for the observed problem?

tnks,
Rod

Endre Varga

unread,
Dec 1, 2015, 10:06:06 AM12/1/15
to akka...@googlegroups.com
On Tue, Dec 1, 2015 at 3:59 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:
Hi Viktor,

I'm using JDK 1.8.0_31

Scala is 2.11.7

Running a 'mvn dependency:tree' I get the following:

[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.5.2: The following artifacts could not be resolved: com.typesafe.akka:akka-remote_2.10:jar:2.4.0, com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0, com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact com.typesafe.akka:akka-remote_2.10:jar:2.4.0 in central (https://repo1.maven.org/maven2) -> [Help 1]

So it seems somehow it's still pulling some 2.10 dependencies. Could this be the cause for the observed problem?

Yes, that can be a problem (this is why there are different artifacts). 

-Endre

Rodrigo Boavida

unread,
Dec 1, 2015, 10:10:23 AM12/1/15
to Akka User List
Hi Endre,

This behavior is occurring after building Spark on Java 8. I'm tempted to assume by what I've shown on this thread, there is some incompatibility between Akka 2.3.11 and 2.4.0. 

In the meantime I will still look into the Spark's dependencies to figure out why is it trying to pull 2.10 version.

Any other suggestion is very welcomed.

tnks,
Rod

Rodrigo Boavida

unread,
Dec 1, 2015, 10:32:12 AM12/1/15
to Akka User List
Here is complete dependency resolution. Seems 2.10 dependencies are being pulled from spark-core.

[WARNING] The POM for com.typesafe.akka:akka-remote_2.10:jar:2.4.0 is missing, no dependency information available
[WARNING] The POM for com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0 is missing, no dependency information available
[WARNING] The POM for com.typesafe.akka:akka-testkit_2.10:jar:2.4.0 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 17.154 s]
[INFO] Spark Project Core ................................. FAILURE [ 26.235 s]
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44.339 s
[INFO] Finished at: 2015-12-01T14:24:33+00:00
[INFO] Final Memory: 27M/317M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.5.2: The following artifacts could nolved: com.typesafe.akka:akka-remote_2.10:jar:2.4.0, com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0, com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact com.type:akka-remote_2.10:jar:2.4.0 in central (https://repo1.maven.org/maven2) -> [Help 1]

Endre Varga

unread,
Dec 1, 2015, 10:36:51 AM12/1/15
to akka...@googlegroups.com
2.4.0 is not released against 2.10, so that explains it. You probably cannot update to Akka 2.4.0 until Spark is updated to Scala 2.11.

-Endre

Rodrigo Boavida

unread,
Dec 1, 2015, 10:50:40 AM12/1/15
to Akka User List
Hi Endre,

Spark is supposed to be binary compatible with Scala 2.11. It's actually documented. Here is where they document it, and I'm following the instructions

I actually found that I was missing the -Dscala-2.11 in the maven command when running the dependency, that's why I was getting the previous error.

In attach is the result of running the dependencies.

tnks,
Rod



...
Spark 1.5.2 with Scala 2.11 and akka 2.4.0 dependencies

Rodrigo Boavida

unread,
Dec 1, 2015, 11:37:21 AM12/1/15
to Akka User List
If I revert to Akka 2.3.11 on Scala 2.11 I do not have any problem running Spark.Seems something about the way Spark is calling the API that internally is creating a conflict?

This happens while creating an actorSystem and some mailbox initialization if you look into the log I first posted. 

If there are any hints on where to look I will appreciate it.

Tnks,
Rod
...

Endre Varga

unread,
Dec 1, 2015, 12:03:53 PM12/1/15
to akka...@googlegroups.com
Hi Rodrigo,

On Tue, Dec 1, 2015 at 4:50 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:
Hi Endre,

Spark is supposed to be binary compatible with Scala 2.11. It's actually documented. Here is where they document it, and I'm following the instructions

What you are talking about is source compatibility, not binary compatibility. Being able to *build* Spark on 2.11 without source modification means source compatibility. Being able to use 2.10 jar files with 2.11 would be binary compatibility - which is basically impossible in practice since scala 2.11 is not binary compatible with 2.10. You have misunderstood something here.
 


I actually found that I was missing the -Dscala-2.11 in the maven command when running the dependency, that's why I was getting the previous error. 

In attach is the result of running the dependencies.

The dependencies look correct to me: built for 2.11 and all 2.4.0.

-Endre
 

--

Endre Varga

unread,
Dec 1, 2015, 12:09:39 PM12/1/15
to akka...@googlegroups.com
On Tue, Dec 1, 2015 at 5:37 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:
If I revert to Akka 2.3.11 on Scala 2.11 I do not have any problem running Spark.Seems something about the way Spark is calling the API that internally is creating a conflict?

This happens while creating an actorSystem and some mailbox initialization if you look into the log I first posted. 

If there are any hints on where to look I will appreciate it.

There is another possible explanation. I vaguely recall our Spark team having a similar issue and the cause was the maven-shade plugin. It seems like it modified the contents of the jar ("optimized" it or something) in a way that it break it. 

I'll ask around, maybe someone else can help you here.

-Endre

 

--

Iulian Dragoș

unread,
Dec 1, 2015, 12:28:05 PM12/1/15
to Akka User List


On Tuesday, December 1, 2015 at 6:09:39 PM UTC+1, drewhk wrote:


On Tue, Dec 1, 2015 at 5:37 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:
If I revert to Akka 2.3.11 on Scala 2.11 I do not have any problem running Spark.Seems something about the way Spark is calling the API that internally is creating a conflict?

This happens while creating an actorSystem and some mailbox initialization if you look into the log I first posted. 

If there are any hints on where to look I will appreciate it.

There is another possible explanation. I vaguely recall our Spark team having a similar issue and the cause was the maven-shade plugin. It seems like it modified the contents of the jar ("optimized" it or something) in a way that it break it. 

Indeed. The problem is with the maven shade plugin, it tries to do too many things when building the assembly, and breaks some classes.

I managed to work around it by using `sbt assembly`, which is using another library to build the assembly.

iulian

Rodrigo Boavida

unread,
Dec 1, 2015, 2:52:30 PM12/1/15
to akka...@googlegroups.com
Hi Iulian,

Thanks for the hint. Just to make sure I understood: You actually used sbt assembly to build the spark for server binaries and not just the driver code compilation? Did you also had to change anything in particular in terms of merge conflicts or library exclusions for akka 2.4.0 compatibility?

Tnks.
Rod
From: Iulian Dragoș
Sent: ‎01/‎12/‎2015 17:28
To: Akka User List
Subject: Re: [akka-user] Building Spark with Akka 2.4.0

[The entire original message is not included.]

Iulian Dragoș

unread,
Dec 1, 2015, 4:40:17 PM12/1/15
to Akka User List


On Tuesday, December 1, 2015 at 8:52:30 PM UTC+1, Rodrigo Boavida wrote:
Hi Iulian,

Thanks for the hint. Just to make sure I understood: You actually used sbt assembly to build the spark for server binaries and not just the driver code compilation? Did you also had to change anything in particular in 
terms of merge conflicts or library exclusions for akka 2.4.0 compatibility?

Yes. I remember there were some issues with one or two modules that I had to skip, but I don't have the branch at hand.

Rodrigo Boavida

unread,
Dec 2, 2015, 6:52:07 AM12/2/15
to Akka User List
Hi Iulian,

I'm getting build errors using sbt due to the issues you found in the below thread in July of this year.
https://mail-archives.apache.org/mod_mbox/spark-dev/201507.mbox/%3CCA+3qhFSJGmZToGmBU1=iVY7KR6EB7k8t6D...@mail.gmail.com%3E

Did you get around this meta-notation errors somehow? (also sent you a pvt msg)

tnks,
Rod

Rodrigo Boavida

unread,
Dec 4, 2015, 1:09:09 PM12/4/15
to Akka User List
Just an update on this - since I got the scala notations errors with SBT assembly I went back to the maven build and built with akka 2.3.11 and consumed it on my akka 2.4.1 SBT projects. 

Running on the spark cluster this gave problems as the loading order of the dependencies made it so that akka 2.3.11 classes were loaded upfront and at least one class complained about missing methods when being called by the akka 2.4.1 consumer code. In this case was akka.pattern.AskableActorRef. To work around it I had to include into the Spark node's classpath the akka 2.4.0 jars explicitly.
These kind of conflicts are the reason why I try to avoid any different version of akka (or any other library) running on the same JVM.

For now I've got it running, but's kind of a hack that I'm sure will "hunt me down". I will continue trying to get a running build of Spark with 2.4.0

tnks,
Rod


On Wednesday, December 2, 2015 at 11:52:07 AM UTC, Rodrigo Boavida wrote:
Hi Iulian,

I'm getting build errors using sbt due to the issues you found in the below thread in July of this year.
Reply all
Reply to author
Forward
0 new messages