How to do logging for batch jobs?

[Johan Andersson]

Hi,

I would like some input on how to do logging when running batch jobs. We need frequent log updates from the jobs as well as persisting the log. We are currently using Cloudwatch Logs and would like to keep doing so if possible.

I see that the "Log Shipper Pattern" is recommended in the docs, but I don't understand how that would work for batch jobs as there is no way for the log task to know when the main task has completed. I understand that this would work just fine for long running services, but not for batch jobs.

We have discussed a few solutions:

1. Reading the logs from the clients. We don't want to talk to the clients (I know a proxy feature is being worked on). It will also be a bit tricky to persist the logs.

2. Piping the output from the command to a script that uploads the logs "command | upload-logs". I haven't found how to do piping in Nomad, so maybe it's not even an option?

3. Running the script that uploads the logs directly and send the actual command as argument "run-command-and-upload-logs command".

I don't really like any of the solutions. Wouldn't it be possible for Nomad to handle the logging? Various logging services could be supported, which would be configured in the jobspec. This might be a bad idea for some reason and you are welcome to shoot it down.

Would sure appreciate your thoughts on this!

Thanks!

[Alex Dadgar]

Hey Johan,

You are totally right that this is an issue. We will fixes this soon by introducing the concept of a leader task such that when that exits, other tasks are told to exit within the task group.

Until then, option 2 and 3 that you listed are probably the best work arounds. You can do piping if using exec by just running your command as “/bin/bash” and your args become [“-c”, “command | next-command”].

In the future we would like to have logging plugins that support all the common log endpoints so users don’t have to roll their own.

Thanks,
Alex Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/41e430c7-46ea-4900-bc4e-4bb6b23c54c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johan Andersson]

Make sense, thanks for working on this!