Connecting spark shell from outside AWS to EC2 master

[Jim Donahue]

This may be a networking issue but ...

I have a master and slave running on EC2 -- the master can see the slave, the slave sees the master and from inside AWS, I can launch a job against the master using its public DNS name (the Spark UI shows the public DNS names for the master and slave).

On my Mac, cruising around the webUI is fine.  However, when I try to run the shell connecting to the same Spark master, everything starts up just fine and then hangs "Connecting to master:" (and no job shows up in the webUI).  Here's the log:

Initializing interpreter...

13/02/20 14:56:48 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:48 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63404 STARTING

Creating SparkContext...

13/02/20 14:56:51 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started

13/02/20 14:56:51 INFO actor.ActorSystemImpl: RemoteServerStarted@akka://sp...@192.168.42.71:63405

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Registered BlockManagerMaster Actor

13/02/20 14:56:51 INFO storage.MemoryStore: MemoryStore started with capacity 1340.7 MB.

13/02/20 14:56:51 INFO storage.DiskStore: Created local directory at /var/folders/q5/q7jx6k2x3_j9y_wq8455lr1r0000gn/T/spark-local-20130220145651-66ad

13/02/20 14:56:51 INFO network.ConnectionManager: Bound socket to port 63406 with id = ConnectionManagerId(jdonahue-MacBookPro1,63406)

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Trying to register BlockManager

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Registered BlockManager

13/02/20 14:56:51 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:51 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63407 STARTING

13/02/20 14:56:51 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.42.71:63407

13/02/20 14:56:51 INFO spark.MapOutputTracker: Registered MapOutputTrackerActor actor

13/02/20 14:56:51 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/q5/q7jx6k2x3_j9y_wq8455lr1r0000gn/T/spark-517a1d1d-f4ef-4d94-938d-b21f98e0cf7a

13/02/20 14:56:51 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:51 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63408 STARTING

13/02/20 14:56:52 INFO io.IoWorker: IoWorker thread 'spray-io-worker-0' started

13/02/20 14:56:52 INFO server.HttpServer: akka://spark/user/BlockManagerHTTPServer started on /0.0.0.0:63409

13/02/20 14:56:52 INFO storage.BlockManagerUI: Started BlockManager web UI at http://jdonahue-MacBookPro1:63409

13/02/20 14:56:52 INFO client.Client$ClientActor: Connecting to master spark://ec2-184-72-88-80.compute-1.amazonaws.com:10081

Spark context available as sc.

Type in expressions to have them evaluated.

Type :help for more information.

scala> 13/02/20 14:56:53 INFO actor.ActorSystemImpl: RemoteClientStarted@akka://spark...@ec2-184-72-88-80.compute-1.amazonaws.com:10081

When I do exactly the same command after logging in to a running EC2 instance, it works just fine and I see the message that the spark shell is connected and the job is running.

I thought it might be a problem being inside our corporate firewall, but I repeated the experiment outside the firewall and got the same behavior.

Any advice much appreciated ...

[Reynold Xin]

Does the security group policy allow port 10081?

--

Reynold Xin, AMPLab, UC Berkeley

http://rxin.org

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jim Donahue]

Yes, I verified that port 10081 is open.  It had to be open for the "inside AWS" connection to be made too, so it's definitely something else.

Jim

On Wednesday, February 20, 2013 3:13:47 PM UTC-8, Jim Donahue wrote:

This may be a networking issue but ...

I have a master and slave running on EC2 -- the master can see the slave, the slave sees the master and from inside AWS, I can launch a job against the master using its public DNS name (the Spark UI shows the public DNS names for the master and slave).

On my Mac, cruising around the webUI is fine.  However, when I try to run the shell connecting to the same Spark master, everything starts up just fine and then hangs "Connecting to master:" (and no job shows up in the webUI).  Here's the log:

Initializing interpreter...

13/02/20 14:56:48 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:48 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63404 STARTING

Creating SparkContext...

13/02/20 14:56:51 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started

13/02/20 14:56:51 INFO actor.ActorSystemImpl: RemoteServerStarted@akka://spar...@192.168.42.71:63405

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Registered BlockManagerMaster Actor

13/02/20 14:56:51 INFO storage.MemoryStore: MemoryStore started with capacity 1340.7 MB.

13/02/20 14:56:51 INFO storage.DiskStore: Created local directory at /var/folders/q5/q7jx6k2x3_j9y_wq8455lr1r0000gn/T/spark-local-20130220145651-66ad

13/02/20 14:56:51 INFO network.ConnectionManager: Bound socket to port 63406 with id = ConnectionManagerId(jdonahue-MacBookPro1,63406)

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Trying to register BlockManager

13/02/20 14:56:51 INFO storage.BlockManagerMaster: Registered BlockManager

13/02/20 14:56:51 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:51 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63407 STARTING

13/02/20 14:56:51 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.42.71:63407

13/02/20 14:56:51 INFO spark.MapOutputTracker: Registered MapOutputTrackerActor actor

13/02/20 14:56:51 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/q5/q7jx6k2x3_j9y_wq8455lr1r0000gn/T/spark-517a1d1d-f4ef-4d94-938d-b21f98e0cf7a

13/02/20 14:56:51 INFO server.Server: jetty-7.5.3.v20111011

13/02/20 14:56:51 INFO server.AbstractConnector: Started SocketC...@0.0.0.0:63408 STARTING

13/02/20 14:56:52 INFO io.IoWorker: IoWorker thread 'spray-io-worker-0' started

13/02/20 14:56:52 INFO server.HttpServer: akka://spark/user/BlockManagerHTTPServer started on /0.0.0.0:63409

13/02/20 14:56:52 INFO storage.BlockManagerUI: Started BlockManager web UI at http://jdonahue-MacBookPro1:63409

13/02/20 14:56:52 INFO client.Client$ClientActor: Connecting to master spark://ec2-184-72-88-80.compute-1.amazonaws.com:10081

Spark context available as sc.

Type in expressions to have them evaluated.

Type :help for more information.

scala> 13/02/20 14:56:53 INFO actor.ActorSystemImpl: RemoteClientStarted@akka://sparkMaster@ec2-184-72-88-80.compute-1.amazonaws.com:10081

[Jim Donahue]

Oh, one thing I should add: the master and slaves get started by running spark.deploy.master.Master and spark.deploy.worker.Worker with the appropriate settings of the webUI port, the listener IP, and the listening port.

Jim

[Patrick Wendell]

Hey Jim,

The spark shell itself launches a few internal servers (inside of SparkEnv/SparkContext) that the slaves make TCP connections to. This will, in general, only work if the shell is being run on a machine with a publicly routable IP address (i.e. not behind a home or office NAT) and the hostname the machine broadcasts is understandable by the slaves.

This is different than the webUI, where your machine is only acting as a TCP client against the ec2 cluster.

- Patrick

--