How can we print some output in the map function

1360 views
Skip to first unread message

Xiaozhe Wang

unread,
Dec 25, 2013, 3:16:01 PM12/25/13
to spark...@googlegroups.com
Hi all,

I am debugging an application on pyspark. When I call 

rdd.map(f).collect()

It will throw some errors. There should be some problems in the function f. I try to print something out and write something to the  file system in the function f, but it cannot work.

Can anybody tell me how to print something out for debugging the application?


Thanks,
Kaden

Cheng Lian

unread,
Dec 26, 2013, 1:16:10 AM12/26/13
to spark...@googlegroups.com
Hi Kaden.  Are you running your application on a Spark cluster?  If so, `f' is executed by the executor process on the worker node, thus you should look for your debugging output file on the worker node.
Reply all
Reply to author
Forward
0 new messages