Suggestions for extra log files

[Whitcomb, Mr. Tim]

I'm very interested in using the "extra log files" config option to allow for logfile access via the Cylc interface rather than hunting through directory paths. For most tasks, this is relatively straightforward as I can just redirect the executable log output to something like ${CYLC_TASK_LOG_ROOT}.log

There's one case, though, where this isn't as clean and I'm looking for a solution:

The forecast model writes out a log file that is currently placed under
$CYLC_SUITE_SHARE_DIR/dynamic/$CYLC_TASK_CYCLE_POINT/logs
This is done so that other tasks (e.g. a log watcher emitting messages back to the suite controller about model progression) can have a consistent place to look for the log file and (most importantly) not need to know the name of the task that was responsible for the forecast model execution[*]

Unfortunately, the extra log files require that the file reside in the $CYLC_TASK_LOG_DIR directory, which is linked to the task name. I thought about having a post-script/err-script copy or link the log file from the share directory, but some users have let me know that they like to be able to see the log file as the task is running, which means that the linkage would need to happen at the beginning, rather than the end, of a task.

Are there any suggestions/best practices for how to handle this? I see there are some tricks with "tee" sending output to multiple files, but that would result in the log being duplicated, instead of linked.

Tim


[*] The reason for several tasks is to allow for cold start forecasts as well as short (06Z/18Z cycles) and long (00Z/12Z cycles) forecasts with different task names (and therefore different options resource requirements set) even though the forecast model underlying the tasks is the same, and the watcher doesn't care about short/long/etc (for now).

[Hilary Oliver]

Hi Tim,

Interesting. I guess your model executable is writing the log file directly to a single configurable location, so you can't easily make it duplicate the file.

In which case, one idea off the top of my head: have the model write to the task-specific log root, and use another "watcher" task that watches that file and live-duplicates it to the shared location. Something like "tail -f LOG >> LOG2" with a bit of additional machinery to detect when the file is complete.  (Not very elegant if you need separate new watcher tasks for all three model variants, I suppose, but you might be able to make one do it all?)

Hilary

--

---
You received this message because you are subscribed to the Google Groups "cylc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cylc+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Whitcomb, Mr. Tim]

That’s correct – “log file” in this case meaning “capturing the standard out written by the forecast model executable”.

 

For your second suggestion, setting up a link (hard or soft) is probably an easier way to go than the tail option (to reduce duplication and make sure that there is a single authoritative log) – it’s an interesting idea.

[Hilary Oliver]

> That’s correct – “log file” in this case meaning “capturing the standard out written by the forecast model executable”.

Ah, in that case, whatever is doing the "capturing" could presumably duplicate the stream then, with "tee", couldn't it?

H

[Whitcomb, Mr. Tim]

That’s correct, at the cost of duplicating the information (rather than linking it)