Tracking process states through pending, running, completed.

[Matthew Marshall]

Hi,

I have been having issues with using `nextflow log` and `nextflow run -with-trace` in that for both instances I can only track completed processes (and with `nextflow log` only if I halt the workflow, use the logging utility, and then resume - I'm surprised this is a limitation).

No alternatives seem to be present in the documentation, except Tower but this doesn't suit my needs from the looks, I want a CLI solution.

I was wondering if perhaps I am mistaken in my above understanding or, if not, if there is another way to track process states from pending through to complete on the command line?

Thanks,

Matthew

[Sean Davis]

While not exactly what you are looking for, take a look at:

https://www.nextflow.io/docs/latest/tracing.html#weblog-via-http

A single endpoint that accepts a POST request will get you real-time monitoring of tasks, runs, etc. from an arbitrary number of simultaneous Nextflow pipelines. Write those records to a small database (or even a file) and you have a streaming log that you can query as you see fit. We've found this approach to be remarkably flexible and it forms the basis of a distributed workflow system monitor that leverages both cloud and HPC resources. 

Sean

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/cb3cae86-687b-4b03-92d0-a48cc236b523n%40googlegroups.com.

--

Sean Davis, MD, PhD

Center for Cancer Research

National Cancer Institute

National Institutes of Health

Bethesda, MD 20892

https://seandavi.github.io/

https://twitter.com/seandavis12