Anduril 1.x: Handline output filename prefixes

[Alexander Kanitz]

Dear Anduril developers,

I have the following Anduril 1.x-related problem:
A program I'm interested to include in my Anduril workflow invariably adds a suffix to the required output filename. So basically, it's not actually the output filename, but just a prefix. As a consequence, the instance's output port will be mapped to an empty file in the instance directory, while the real output file will be an orphan with no connection to any accessible port and whose filename is composed of the prefix and the automatically added suffix.

I'm not sure I have explained this well, so here's an example:

// Component 'problematic_component' has a single outport 'outfile'
instance = problematic_component(infile = previous_instance.outfile)

The instance directory will then contain the following files:
outfile (empty but accessible via 'instance.outfile')
outfile.suffix (contains actual results but inaccessible)

I'm quite sure I could find a dirty solution to the problem relatively easily, but since this is surely a relatively common case: Is there perhaps an elegant/preferred but most of all *clean* (i.e. not messing with states and suchlike) solution to the problem?

Thanks a lot and best,
Alex Kanitz

[Alexander Kanitz]

Wanted to add that I see the following two solutions currently:
1. Renaming the output file with @out and overwriting it worth the real result file. That's easy but very dirty because the upstream component is changed.
2. Using Folder2Array on the instance folder and then keeping only the file that ends with the suffix. While clumsy that's probably more sustainable. However, I don't know how to access the instance folder?

Anything more elegant (has to be Anduril 1.x!) would be much appreciated, as well as a clue on how to get the name of an instance or at least the execution folder from within an Anduril network file.

Thanks for your help and best,
Alex Kanitz

[Alejandra Cervera]

Hi,
If you are running the program in BashEvaluate, what I would do is cd to folder1 in the BashEvaluate script, run the program there giving whatever suffix you want, and then either FolderExtractor or Folder2Array on folder1 to get the file you want or inside BashEvaluate link (or move) the file you are interested in to optOut1

myProgram = BashEvaluate(var1 = inputFile,
                                                  script ="""cd @folder1@
                                                                   run program @var1@ suffix
                                                                   ln -s myFile.suffix @optOut1@""")


                                                                    

[Ville Rantanen]

Note, use:

ln -sf myFile.suffix @optOut1@

this is because optOut1 file already exists, and ln will fail unless -f is forcing the output to write over.

[Ville Rantanen]

Typically, when dealing with programs that produce an output whose filename i can not control:

1. in the component, create temporary folder (API has a command for this)
2. run the command in the temporary folder
3. move the output to the actual output port

It's not okay for a component to leave orphan files in the execution folder. Your problem must be fixed in the upstream component.
Also, downstream component must not modify the upstream component execution folder (as you mentioned)

to a question of yours "what is the instance name of an upstream component?", the folder name is basically the answer.
if you KNOW the input is coming from an actual component (and not INPUT), the

instance_name=$( basename $( dirname $inputFile ) )

You can of course test the existence of _command file to have some certainty of it being a component:

test -f $( dirname $inputFile )/_command

[Alexander Kanitz]

Thanks guys!

So basically you are suggesting to either wrap the command in BashEvaluate() or twiddle with the component itself, via the API. Correct? For now I would probably rather stick with the former, because it allows me to solve the problem in the network file itself. In fact, in this particular case I may even simply change the program behavior itself because I have access to the source code. But in the longer run I may come up with a more sustainable solution along the lines of the second suggestion. I am anyway using a rather unusual setup, where I am using the Python API, exclusively, as a generic, standardized wrapper for every single component (Python or not), like so:

#!/usr/bin/env python

# Import modules
import sys
import anduril
import anduril_custom_functions
from anduril.args import *

# Command template
command = <COMMAND_TEMPLATE> # The command template is a formalized representation of the actual command with all potential inputs, outputs, parameters and their arguments that is rendered during execution

# Execute command
exit_status = anduril_custom_functions.main(component, command, tempdir) # Here the formal, rendered command template is sanitized and then executed, among other things

# Return exit status
sys.exit(exit_status)

One the one hand, this enables me to generate the components automatically, based on a simple tabular template, while at the same time allowing me to customize execution modes, logging, error handling, input file validation etc more freely ("anduril_custom_functions"). This is also a good place to handle unusual cases (such as suffix appending) in a more generic manner, as I am already doing e.g. with optional outports. I know it's a quirky setup and basically reduces Anduril's functionalities to its absolute core. But in this way it gives me the freedom that I want while maintaining clean components and network files, so that there is essentially only one place to make changes in.

@Ville:
I think I was perhaps not very clear about the instance name problem. What I actually wanted to know is if there's a way of getting the absolute path of an instance's execution directory from within the network file? Alternatively, the main execution directory (--execution-dir option in the "anduril run" call) would also do of course.

Thanks again and best,
Alex

[Ville Rantanen]

For the instance file question, if using Anduril Script, you can find out some things with iterators

inst=BashEvaluate(...)

for file: std.iterdir(inst.folder1) {

     // absolute path to a file in folder1 ==  file.path

    //  use std.strReplace & other string functions to get the dir

}

also, you can take a file in an Array, and iterate that. std.iterArray  also returns a record with the  full path to a file

Withtout running Anduril, it may be difficult to deduce the actual folder name of an instance (by just parsing the network file). Esp. if it involves functions and inline component calls. 

You could make use of the  _state file in the main execution directory.

[Alexander Kanitz]

Thanks, good to know.

Best,
Alex