Passing input or variables to Python script block

[Pietro Marchesi]

Hi, 

I was wondering how inputs are passed to (and retrievable from) script blocks that are not BASH (in this case Python). The use case is the following: I have many pickled Python objects that I would like to load in Python and combine. Of course I can write an external Python script to be called in BASH that receives all files as input, but since it would only be a few lines of code, it feels unnecessary to introduce another script, and it would be nicer to just do it in the script block. 

The process would look something like:

process pythonprocess{

  input:

  file 'file*' from files.collect()
  
  output:
  file 'combined_object' into combined

  """

  #!/usr/bin/env python3

  # load files and combine them using Python...

  """

But I cannot figure out how I would access the inputs from the script.

Thanks a lot, 

Pietro

[Paolo Di Tommaso]

Hi Pietro, 

There's no magic solution here. Likely the best option is to save your python script to a separate executable file (as described in the previous email) and pass the parameter files as command line arguments. 

Hope it helps. 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Marc Logghe]

Hi Pietro,

Not sure if I understand your issue correctly but I guess you can access the files in your script block with regular python, taking into account that the staged files will have names like file1, file2, etc. (docs).

In the script you do an os.listdir and loop over the files and filter out the wanted ones with a regex '^file\d+$' 

HTH,

Marc

Op di 13 feb. 2018 om 17:42 schreef Paolo Di Tommaso <paolo.d...@gmail.com>:

Hi Pietro, 

There's no magic solution here. Likely the best option is to save your python script to a separate executable file (as described in the previous email) and pass the parameter files as command line arguments. 

Hope it helps. 

p

On Tue, Feb 13, 2018 at 5:23 PM, Pietro Marchesi <pietroma...@gmail.com> wrote:

Hi, 

I was wondering how inputs are passed to (and retrievable from) script blocks that are not BASH (in this case Python). The use case is the following: I have many pickled Python objects that I would like to load in Python and combine. Of course I can write an external Python script to be called in BASH that receives all files as input, but since it would only be a few lines of code, it feels unnecessary to introduce another script, and it would be nicer to just do it in the script block. 

The process would look something like:

process pythonprocess{

  input:

  file 'file*' from files.collect()
  
  output:
  file 'combined_object' into combined

  """

  #!/usr/bin/env python3

  # load files and combine them using Python...

  """

But I cannot figure out how I would access the inputs from the script.

Thanks a lot, 

Pietro

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Steve]

You might also consider a bash heredoc

params.samples_list = ['Sample1', 'Sample2', 'Sample3', 'Sample4']
Channel.from( params.samples_list ).into{ samples_list; samples_list2 }

samples_list2.println()

process make_file {
    echo true
    executor "local"
    input:
    val(sample_ID) from samples_list

    output:
    file  "${sample_ID}.txt" into samples_files

    script:
    """
    touch "${sample_ID}.txt"
    """
}

process python_test {
    echo true
    executor "local"

    input:
    file 'file*' from samples_files.collect()

    script:
    """
    python - file* <<E0F
import sys
print(sys.argv)
E0F
    """

}

output:

N E X T F L O W  ~  version 0.27.2
Launching `pytest.nf` [elated_fermat] - revision: bb7bdd8c19
Sample1
Sample2
Sample3
Sample4
[warm up] executor > local
[5a/ef208c] Submitted process > make_file (4)
[e7/0a711c] Submitted process > make_file (3)
[c2/aa3c49] Submitted process > make_file (2)
[ff/c61434] Submitted process > make_file (1)
[af/8a6889] Submitted process > python_test
file1 file2 file3 file4
['-', 'file1', 'file2', 'file3', 'file4']

[Pietro Marchesi]

Thanks everyone, 

What I found easiest was to loop and match through os.listdir() to find the staged files, as mentioned by Marc. However, in the regex expression, both the \d and the $ need to be escaped for them to work in nextflow (not exactly sure why), so what works is actually '^file\\d+\$'. 

process collectfiles{

  echo true

  input:

  file 'file*' from files.collect()

  """

  #!/usr/bin/env python3

  import os

  import re

  print(list(filter((lambda x: re.search(r'^file\\d+\$', x)), os.listdir())))

  """

}

Thanks, 

P