Print commands

[Martijn Starmans]

Hey all,

I use ``print'' commands in my tools (e.g. in Python) in order to keep track of the progress and output of my tools. However, FASTR does not seem to recognize or suppresses these as the outputs are not shown on the command line. Is there a way to show these on the command line upon execution?

Thanks in advance,

Martijn

[Hakim Achterberg]

Hi Martijn,

It depends where you use the print statements. Is it inside the executable called by the Tool? Fastr executes tools in the background (might even be on a different host in case of a cluster/cloud execution) and the stdout/stderr is captured. There is no easy way to access it during execution, but afterwards it is in the __fastr_result__.pickle.gz (which is a pickled job object). We are looking at different way to make it easier to access this type of data, as it could help with debugging and tracking logs. The main problem is that the execution is not always performed on the same machine as you are looking at the console.

A possibility would be to both log the tool stdout/stderr to memory and a file at the same time. This file could be accessible during execution and could be inspected for progress/debug purposes. If this is desirable I will look into it. Would that solve your problem?

Cheers,

Hakim

[Martijn Starmans]

Hi Hakim,

The print statements are indeed inside the executable called by the tool. The issue is indeed accessing the data during execution to see the progress inside the executables called by the tool. Your suggestion would indeed solve that issue, so I hope to hear from you if it works out. Thanks for looking into it!

Kind regards,

Martijn

[Hakim Achterberg]

Dear Martijn,

We are planning to release a new fastr version coming Friday. The release after that will focus on logging and error reporting improvements. We will look into the mentioned issues and would welcome any suggestions and feedback regarding those topics.

Cheers,

Hakim

[Baldur van Lew]

Hi Hakim

For the EYR project I made the logging (also from the worker nodes) based on this example in the Python Cookbook https://docs.python.org/2/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network plus a file handler to write the output. What I didn't create, and is really needed, was a log viewer to filter the log messages by worker/job/time slot etc. but given the right log format that should be too difficult.

Cheers

Baldur

[Hakim Achterberg]

Hi Baldur,

That sound interesting, but if you collect from multiple workers I suppose we need to track that so that users can filter per worker/job to avoid an overload of messages. For the viewer, what did you have in mind? console based/curses or more a GUI/webbased solution? We once made fastr log to javascript using websockets connection, that was also nice but we never really continued that project.

Cheers,

Hakim