Re: Failed To Launch spark-shell on EC2
139 views
Skip to first unread message
Patrick Wendell
Dec 23, 2013, 12:14:59 PM12/23/13
to spark...@googlegroups.com
Ya this can be confusing (the logging output gets interpolated with
the shell). Just hitting "Enter" will clear the shell.
On Mon, Dec 23, 2013 at 8:22 AM, Yang JIAO <yangji...@gmail.com> wrote:
> Sorry to have spamed everyone. I just needed to press Enter.
> Please treat it as a joke :)
> thanks
>
>
> On Monday, December 23, 2013 8:18:23 AM UTC-8, Yang JIAO wrote:
>>
>> Hi Community,
>>
>> I'm new to Spark & EC2 and was trying to complete this tutorial:
>> https://github.com/utcompling/applied-nlp/wiki/Spark-AWS-Exercise1
>>
>> I launched a master node and 2 slaves of m1.large instances, then
>> conncected to my master node. When I try to launch spark-shell, the process
>> stuck at a line:
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-86.ec2.internal:56422 with 3.8 GB RAM
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-85.ec2.internal:56106 with 3.8 GB RAM
>>
>> Till now, it has been 30 mins without any furthur messages. Is there
>> someting wrong?
>> Detailed messages are found below
>> Thanks in advance.
>> Yang
>>
>> root@ip-172-31-13-126 spark]$ ./spark-shell
>> Welcome to
>> ____ __
>> / __/__ ___ _____/ /__
>> _\ \/ _ \/ _ `/ __/ '_/
>> /___/ .__/\_,_/_/ /_/\_\ version 0.8.0
>> /_/
>>
>> Using Scala version 2.9.3 (OpenJDK 64-Bit Server VM, Java 1.7.0_45)
>> Initializing interpreter...
>> 13/12/23 15:36:30 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:30 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:37569
>> Creating SparkContext...
>> 13/12/23 15:36:40 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/23 15:36:41 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 13/12/23 15:36:41 INFO storage.MemoryStore: MemoryStore started with
>> capacity 3.8 GB.
>> 13/12/23 15:36:41 INFO storage.DiskStore: Created local directory at
>> /mnt/spark/spark-local-20131223153641-15a1
>> 13/12/23 15:36:41 INFO storage.DiskStore: Created local directory at
>> /mnt2/spark/spark-local-20131223153641-12de
>> 13/12/23 15:36:41 INFO network.ConnectionManager: Bound socket to port
>> 47773 with id = ConnectionManagerId(ip-172-31-13-126.ec2.internal,47773)
>> 13/12/23 15:36:41 INFO storage.BlockManagerMaster: Trying to register
>> BlockManager
>> 13/12/23 15:36:41 INFO storage.BlockManagerMaster: Registered BlockManager
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:44535
>> 13/12/23 15:36:41 INFO broadcast.HttpBroadcast: Broadcast server started
>> at http://172.31.13.126:44535
>> 13/12/23 15:36:41 INFO spark.SparkEnv: Registering MapOutputTracker
>> 13/12/23 15:36:41 INFO spark.HttpFileServer: HTTP File server directory is
>> /tmp/spark-84ff9525-8c17-4ab1-b68c-283326f7b19c
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:50521
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/storage/rdd,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/storage,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages/stage,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages/pool,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/environment,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/executors,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/,null}
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SelectChann...@0.0.0.0:4040
>> 13/12/23 15:36:41 INFO ui.SparkUI: Started Spark Web UI at
>> http://ec2-54-209-23-35.compute-1.amazonaws.com:4040
>> 13/12/23 15:36:41 INFO client.Client$ClientActor: Connecting to master
>> spark://ec2-54-209-23-35.compute-1.amazonaws.com:7077
>> Spark context available as sc.
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Connected to
>> Spark cluster with app ID app-20131223153641-0000
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor added:
>> app-20131223153641-0000/0 on
>> worker-20131223153230-ip-172-31-14-86.ec2.internal-57023
>> (ip-172-31-14-86.ec2.internal:57023) with 2 cores
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Granted
>> executor ID app-20131223153641-0000/0 on hostPort
>> ip-172-31-14-86.ec2.internal:57023 with 2 cores, 6.0 GB RAM
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor added:
>> app-20131223153641-0000/1 on
>> worker-20131223153231-ip-172-31-14-85.ec2.internal-43057
>> (ip-172-31-14-85.ec2.internal:43057) with 2 cores
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Granted
>> executor ID app-20131223153641-0000/1 on hostPort
>> ip-172-31-14-85.ec2.internal:43057 with 2 cores, 6.0 GB RAM
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor updated:
>> app-20131223153641-0000/0 is now RUNNING
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor updated:
>> app-20131223153641-0000/1 is now RUNNING
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>>
>> scala> 13/12/23 15:36:44 INFO cluster.SparkDeploySchedulerBackend:
>> Registered executor:
>> Actor[akka://sparkE...@ip-172-31-14-86.ec2.internal:54098/user/Executor]
>> with ID 0
>> 13/12/23 15:36:44 INFO cluster.SparkDeploySchedulerBackend: Registered
>> executor:
>> Actor[akka://sparkE...@ip-172-31-14-85.ec2.internal:59574/user/Executor]
>> with ID 1
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-86.ec2.internal:56422 with 3.8 GB RAM
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-85.ec2.internal:56106 with 3.8 GB RAM
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
the shell). Just hitting "Enter" will clear the shell.
On Mon, Dec 23, 2013 at 8:22 AM, Yang JIAO <yangji...@gmail.com> wrote:
> Sorry to have spamed everyone. I just needed to press Enter.
> Please treat it as a joke :)
> thanks
>
>
> On Monday, December 23, 2013 8:18:23 AM UTC-8, Yang JIAO wrote:
>>
>> Hi Community,
>>
>> I'm new to Spark & EC2 and was trying to complete this tutorial:
>> https://github.com/utcompling/applied-nlp/wiki/Spark-AWS-Exercise1
>>
>> I launched a master node and 2 slaves of m1.large instances, then
>> conncected to my master node. When I try to launch spark-shell, the process
>> stuck at a line:
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-86.ec2.internal:56422 with 3.8 GB RAM
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-85.ec2.internal:56106 with 3.8 GB RAM
>>
>> Till now, it has been 30 mins without any furthur messages. Is there
>> someting wrong?
>> Detailed messages are found below
>> Thanks in advance.
>> Yang
>>
>> root@ip-172-31-13-126 spark]$ ./spark-shell
>> Welcome to
>> ____ __
>> / __/__ ___ _____/ /__
>> _\ \/ _ \/ _ `/ __/ '_/
>> /___/ .__/\_,_/_/ /_/\_\ version 0.8.0
>> /_/
>>
>> Using Scala version 2.9.3 (OpenJDK 64-Bit Server VM, Java 1.7.0_45)
>> Initializing interpreter...
>> 13/12/23 15:36:30 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:30 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:37569
>> Creating SparkContext...
>> 13/12/23 15:36:40 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/23 15:36:41 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 13/12/23 15:36:41 INFO storage.MemoryStore: MemoryStore started with
>> capacity 3.8 GB.
>> 13/12/23 15:36:41 INFO storage.DiskStore: Created local directory at
>> /mnt/spark/spark-local-20131223153641-15a1
>> 13/12/23 15:36:41 INFO storage.DiskStore: Created local directory at
>> /mnt2/spark/spark-local-20131223153641-12de
>> 13/12/23 15:36:41 INFO network.ConnectionManager: Bound socket to port
>> 47773 with id = ConnectionManagerId(ip-172-31-13-126.ec2.internal,47773)
>> 13/12/23 15:36:41 INFO storage.BlockManagerMaster: Trying to register
>> BlockManager
>> 13/12/23 15:36:41 INFO storage.BlockManagerMaster: Registered BlockManager
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:44535
>> 13/12/23 15:36:41 INFO broadcast.HttpBroadcast: Broadcast server started
>> at http://172.31.13.126:44535
>> 13/12/23 15:36:41 INFO spark.SparkEnv: Registering MapOutputTracker
>> 13/12/23 15:36:41 INFO spark.HttpFileServer: HTTP File server directory is
>> /tmp/spark-84ff9525-8c17-4ab1-b68c-283326f7b19c
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SocketC...@0.0.0.0:50521
>> 13/12/23 15:36:41 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/storage/rdd,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/storage,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages/stage,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages/pool,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/stages,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/environment,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/executors,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/23 15:36:41 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/,null}
>> 13/12/23 15:36:41 INFO server.AbstractConnector: Started
>> SelectChann...@0.0.0.0:4040
>> 13/12/23 15:36:41 INFO ui.SparkUI: Started Spark Web UI at
>> http://ec2-54-209-23-35.compute-1.amazonaws.com:4040
>> 13/12/23 15:36:41 INFO client.Client$ClientActor: Connecting to master
>> spark://ec2-54-209-23-35.compute-1.amazonaws.com:7077
>> Spark context available as sc.
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Connected to
>> Spark cluster with app ID app-20131223153641-0000
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor added:
>> app-20131223153641-0000/0 on
>> worker-20131223153230-ip-172-31-14-86.ec2.internal-57023
>> (ip-172-31-14-86.ec2.internal:57023) with 2 cores
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Granted
>> executor ID app-20131223153641-0000/0 on hostPort
>> ip-172-31-14-86.ec2.internal:57023 with 2 cores, 6.0 GB RAM
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor added:
>> app-20131223153641-0000/1 on
>> worker-20131223153231-ip-172-31-14-85.ec2.internal-43057
>> (ip-172-31-14-85.ec2.internal:43057) with 2 cores
>> 13/12/23 15:36:42 INFO cluster.SparkDeploySchedulerBackend: Granted
>> executor ID app-20131223153641-0000/1 on hostPort
>> ip-172-31-14-85.ec2.internal:43057 with 2 cores, 6.0 GB RAM
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor updated:
>> app-20131223153641-0000/0 is now RUNNING
>> 13/12/23 15:36:42 INFO client.Client$ClientActor: Executor updated:
>> app-20131223153641-0000/1 is now RUNNING
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>>
>> scala> 13/12/23 15:36:44 INFO cluster.SparkDeploySchedulerBackend:
>> Registered executor:
>> Actor[akka://sparkE...@ip-172-31-14-86.ec2.internal:54098/user/Executor]
>> with ID 0
>> 13/12/23 15:36:44 INFO cluster.SparkDeploySchedulerBackend: Registered
>> executor:
>> Actor[akka://sparkE...@ip-172-31-14-85.ec2.internal:59574/user/Executor]
>> with ID 1
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-86.ec2.internal:56422 with 3.8 GB RAM
>> 13/12/23 15:36:44 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> Registering block manager ip-172-31-14-85.ec2.internal:56106 with 3.8 GB RAM
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
Reply all
Reply to author
Forward
0 new messages