Where are stderr messages logged if not running on EMR?

[Roving Richard]

I am having a really hard time debugging some MRJob scripts. I can troubleshoot errors if the job runs to completion, but my life would be much easier if I could just print() arbitrary outputs.

I found an old thread where someone said you can log using sys.stderr.write() (link: https://groups.google.com/d/msg/mrjob/0VL02lfWTv0/qLoEQTvKwTkJ). The discussion says where log files are saved on EMR instances, but where are MRJob logs saved  on a local Linux system?

I've looked at the directories in the documentation (https://pythonhosted.org/mrjob/guides/configs-hadoopy-runners.html#option-hadoop_log_dirs ) under /var/ and /mnt/ without luck.

[Damjan Krstajic]

I am having the same problem. I cannot find MRJob logs on a local Linux system. My colleague has informed me and I can confirm that logs are created during the execution and then after successful completion the folder containg them is automatically removed unless an error is detected.

Any advice?

[Roving Richard]

I gave up and switched to PySpark and I am so glad I did.