Is there a way to suppress Spark Messages during execution?

[angshu rai]

Hi,

I have a few optimization jobs that run for many iterations, as a result I see a lot of Spark "INFO" messages coming up on the screen which has unnecessary IO overhead.

Is there a way to suppress these messages and get them only if an Error occurs? Something like a quiet mode operation?

It would be great to know.

Thanks!

Angshu

[Josh Rosen]

Spark uses log4j for its logging, so you can provide a custom log4j.properties file that changes Spark's logging threshold to WARNING:

http://spark-project.org/docs/latest/configuration.html#configuring-logging

- Josh

[angshu rai]

Thanks Josh, I already have the log4j.properties file in the conf directory which is basically the copy of the template, so it should be showing only warnings but it is not.

Is there a place where I need to mention the path of this properties file so that the mandates are taken from there?

-Angshu

[Matei Zaharia]

Did you actually modify the template to only show warnings? It shows INFO by default. Change the INFO at the top to WARNING.

Matei

[angshu rai]

Ok, had to edit the properties file a bit. Its much cleaner now.

Thanks Matei, Josh!

-Angshu