Is there a way to access the output directory of the job inside a tool definition?

[porter...@gmail.com]

My use case for this is that I'm trying to write a CWL tool definition for a tool that asks you to supply as an input via the command line a directory in which it should place logs. This is a reasonably common pattern and there are other examples like this (e.g. passing a scratch directory in which to put temporary files, etc.). At first I thought I would just require the logging directory to be passed in as an input as part of the job, something similar to:

inputs:
  # . . . other inputs
  - id: #log_dir
   type: string
   inputBinding:
     position: 1
     prefix:'--log-dir'

However, this doesn't work, since the path specified (which is on the host system) won't exist inside the docker container. Using a dummy File input suffices to get the implementation to mount another directory into the docker container, but this fails with the tool complaining about trying to open a log file for writing on a read-only filesystem, since the reference implementation mounts input directories into the container as read-only. What I really want to do is specify that the tool should be passed --log-dir $OUTDIR where $OUTDIR is the output directory passed to cwltool via the --outdir argument (I'm assuming this is the "designated output directory" referred to in the specification). I was hoping a JsonPointer expression with job/outdir might suffice, but the outdir doesn't appear to be available on the job object.

I think my question boils down to the fact that the specification says:

> Output files produced by tool execution must be written to the **designated output directory**.

But I can't find a way to make this happen in the case of a tool that takes command line inputs specifying locations it should write to, since I can't figure out how to access the designated output directory in order to tell the tool to use it. Moreover, this seems difficult to do in the general case of poorly behaved tools that, e.g. just write temporary files / output files in the same directory as some input file. Apologies for the noise if I'm missing something obvious!

Thanks,

James Porter

[Peter Amstutz]

Hi James,

The output directory is the default starting working directory, so you
might be able to use a relative path like '.'

However I agree that is an oversight, there should be a way to access
the paths to the output directory and temporary directory when
building the tool invocation. Please make a github issue.

For the case of badly behaved tools that want to put their output in
the same directory as their input:

https://common-workflow-language.github.io/draft-2/#createfilerequirement

"If the value is an expression that evaluates to a File object, this
indicates the referenced file should be added to the designated output
directory prior to executing the tool.

Files added in this way may be read-only, and may be implemented
through bind mounts or file system links in such a way as to avoid
unnecessary copying of the input file."

IIRC that feature may not yet be implemented in the reference
implementation, so that also needs a github issue.

- Peter

> --
> You received this message because you are subscribed to the Google Groups
> "common-workflow-language" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to common-workflow-la...@googlegroups.com.
> To post to this group, send email to
> common-workf...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-workflow-language/a3b93d77-47a8-4322-bd52-0707d2793c7c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[porter...@gmail.com]

Thanks, that makes sense. Issue filed for accessing the output directory: https://github.com/common-workflow-language/common-workflow-language/issues/62

CreateFileRequirement appears to be already implemented: https://github.com/common-workflow-language/common-workflow-language/blob/c07bae7b79797b78ef639746411a8876fd0b0846/reference/cwltool/draft2tool.py#L309-L313