Does anybody have an example of how SparkListener is used?

[Bochun Zhang]

I would like to use SparkListener to let me define terminal output when the job is running. Does anybody have an example of how the trait is used?

[Matei Zaharia]

Are you talking about redirecting System.out? That can't be controlled by SparkListener, you'd have to do that outside Java. You can find a few implementations of SparkListener in the Spark codebase. They're used to log what happens in the job and to capture metrics.

Or if you're talking about changing Spark's log messages (all the INFO, WARN, etc messages that come out), that's controlled through log4j.properties; see http://spark.incubator.apache.org/docs/latest/configuration.html#configuring-logging.

Matei

On Sep 26, 2013, at 8:40 PM, Bochun Zhang <zbc...@gmail.com> wrote:

I would like to use SparkListener to let me define terminal output when the job is running. Does anybody have an example of how the trait is used?

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bochun Zhang]

Actually I'm thinking of redirecting log info to System.out. I would also like to modify the messages a slave sends to the master when stage finishes, and output some of these messages to system.out.

 

You received this message because you are subscribed to a topic in the Google Groups "Spark Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-users/FsHfI8rz7ds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spark-users...@googlegroups.com.

[Kay Ousterhout]

Hi Bochun,

If you haven't already, you might want to take a look at the JobLogger (in the scheduler package). It outputs information to a file, so it might be relatively easy to modify for your purposes.  It's not enabled by default -- you need to call addSparkListener(new JobLogger()) on your SparkContext to enable it.

-Kay