Where is the training log ?

[alexandre benoit]

Hi all,

must be a stupid question but until recently, one could access training log from /tmp/caffe.INFO simlink (linux user) but it seems that it does not work anymore.

And i do not see any log file in other folders.

Then, how to access training/test loss history now ?

Thanks.

[Diego Rueda]

there are many ways, but one of them could be using script, just type script log.txt before training and it saves all your output in the file

[alexandre benoit]

Hi,

thank you for this first reply.

Well, yes, i tested again using command line calls to caffe and log works fine (thank you for your additional hint).

however, how to get log when caffe is called from python

when calling :

solver = caffe.SGDSolver(solver_train_pathname)
solver.solve()

In this configuration, 

_How to collect the  global log at the end of the solving process (as with the command line call when getting file pointed by /tmp/caffe.INFO)

_And also how to collect log in real time during the solving process (i guess one should replace the automatic "solve" method by several "run_forward" and "run_backward" if so how and is there another way using the solve method ?)

Thanks for help.

[Suleman]

Any updates on this?

[Jan C Peters]

In python you can redirect stdout and/or stderr to a file or wherever you want to capture the output, e.g. do

import sys

file = open('output.log', 'wt')
sys.stdout = file
sys.stderr = file

every output after that will go into the file output.log. Even regular print commands. So that may not be entirely what you want, but it can be a workaround.

Jan

[Suleman]

This works for print statements that I have in my python code, but does not write solver output to file when I call solver.solve()

[Amir Abdi]

If you run the solver in command line, you can do this:
/build/tools/caffe train --solver=solver.prototxt  2>&1 | tee outputfile.txt

And of course you can do the same system call in python.

[Jan C Peters]

Mhm, actually redirecting in python should work, it sure does for me. But I too find it annoying that the log is so difficult to handle, and that accuracy values and other output blobs and statistics cannot be gathered other than by parsing the log.

Jan