Sending pipeline process output files as email attachments

[Steve]

I was reviewing the documentation on how to send emails with Nextflow, but could not figure out how to send files produced by pipeline processes as email attachments. Given a demo pipeline that looks like this:

params.samples_list = ['Sample1', 'Sample2', 'Sample3', 'Sample4']
Channel.from( params.samples_list ).into{ samples_list; samples_list2 }

process make_txt {
    tag { sample_ID }
    executor "local"
    echo true

    input:
    val(sample_ID) from samples_list

    output:
    file "${sample_ID}.txt" into samples_files, samples_files2

    script:
    """
    echo "[make_txt] ${sample_ID}"
    echo "[make_txt] ${sample_ID}" > "${sample_ID}.txt"
    """
}

I tried the following methods:

sendMail {
    to 'm...@email.com'
    attach samples_files.toList()
    subject 'Catch up'

    body
    '''
    Hi there,
    Look! Multi-lines
    mail content!
    '''
}

output:

N E X T F L O W  ~  version 0.27.3
Launching `sendMail.nf` [fabulous_plateau] - revision: a152128b7d
[warm up] executor > local
ERROR ~ Invalid attachment argument: DataflowVariable(value=null) [class groovyx.gpars.dataflow.DataflowVariable]

 -- Check script 'sendMail.nf' at line: 25 or see '.nextflow.log' file for more details

another method:

m...@email.com

same error:

N E X T F L O W  ~  version 0.27.3
Launching `sendMail.nf` [boring_jang] - revision: 5c56ba8b5e
[warm up] executor > local
ERROR ~ Invalid attachment argument: DataflowVariable(value=null) [class groovyx.gpars.dataflow.DataflowVariable]

 -- Check script 'sendMail.nf' at line: 38 or see '.nextflow.log' file for more details

tried making the email step itself a process:

process send_mail {
    input:
    file(attachments: "*") from samples_files.toList()

    exec:
    sendMail( to: 'm...@email.com',
              subject: 'Catch up',
              body: 'Hi, how are you!',
              attach: attachments )

}

output:

N E X T F L O W  ~  version 0.27.3
Launching `sendMail.nf` [big_varahamihira] - revision: cea783231f
[warm up] executor > local
[ff/c0f36c] Submitted process > make_txt (Sample2)
[6e/5b2299] Submitted process > make_txt (Sample4)
[2b/3f1088] Submitted process > make_txt (Sample3)
[8c/ccc337] Submitted process > make_txt (Sample1)
[make_txt] Sample2
[make_txt] Sample4
[make_txt] Sample3
[make_txt] Sample1
[f2/343ffe] Submitted process > send_mail
ERROR ~ Error executing process > 'send_mail'

Caused by:
  Invalid attachment argument: Sample2.txt Sample4.txt Sample3.txt Sample1.txt [class nextflow.util.BlankSeparatedList]

Source block:
  sendMail( to: 'm...@email.com',
            subject: 'Catch up',
            body: 'Hi, how are you!',
            attach: attachments )

Work dir:
  /Users/kellys04/projects/nextflow-samplesheet-demo/work/f2/343ffee5ff5498a229e6a56d01149a

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details

Any suggestions on how to implement this? Thanks!

[Paolo Di Tommaso]

You need to resolve the channel data type to the target value(s). Try 

    attach samples_files.toList().getVal()

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Paolo Di Tommaso]

Also the version you are using is affected by a bug. Upgrade to the latest version. 

p

[Steve]

Thanks, this works.

I saw that you can send workflow metadata as part of a completion event email  as per the docs here: https://www.nextflow.io/docs/latest/mail.html#mail-notification

I was wondering if it is possible to also send files like the trace, timeline.html, and report.html with this too, and combine them with this workflow process output?

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Lavi Bharath]

I am also trying to attach report.html as attachment. 

report.html is generated as the last step and hence sending report.html as an attachment fails in workflow.onComplete

Any workaround for this? Thanks.

[Paolo Di Tommaso]

Just use a simple bash wrapper to launch your pipeline execution and send the email when it completes. 

p

[Lavi Bharath]

Thanks Paolo.

[Steve]

Is the `sendMail` function available from outside of Nextflow? Its a lot more convenient than trying to write a `sendmail` command in bash, from what I have seen, especially in terms of dealing with attachments. `mutt` is a good bash alternative but it is not always installed on systems.

[Paolo Di Tommaso]

What about creating a script just sending a email ? 

 

#!/bin/env nextflow 

sendMail {

  from <address>

  to <adress>

  """
  Ciao

  """

}

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Steve]

Not sure how you are supposed to get the shebang to work there. I do not keep Nextflow in my PATH, none of these work:

$ (env nextflow=nextflow ./email2.nf)
/bin/env: nextflow: No such file or directory

$ (export nextflow=nextflow; ./email2.nf)
/usr/bin/env: nextflow: No such file or directory

Also, I cannot tell if this method allows access to parameters in the 'nextflow.config' file.

Just use a simple bash wrapper to launch your pipeline execution and send the email when it completes.

I also tried this, and it keeps breaking.

Channel.fromPath("${params.output_dir}/email/attachments/*").set { email_attachments_channel }

String subject_line = new File("${params.output_dir}/email/subject.txt").text
def body = new File("${params.output_dir}/email/body.txt").text
def attachments = email_attachments_channel.toList().getVal()
println "${attachments}"

// pause a moment before sending the email; 3s
sleep(3000)

sendMail {
  from "${params.email_to}"
  to "${params.email_from}"
  attach attachments
  subject subject_line
  """
  ${body}
  """.stripIndent()
}

$ ./nextflow run email.nf
N E X T F L O W  ~  version 0.29.0
Launching `email.nf` [cheesy_archimedes] - revision: 25126a44ff
[/ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/RunParameters.tsv, /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/multiqc_report.html, /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/SampleSheet.csv, /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/demultiplexing_report.html, /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/RTAComplete.txt, /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/output/email/attachments/Demultiplex_Stats.htm]
ERROR ~ Broken pipe

 -- Check script 'email.nf' at line: 11 or see '.nextflow.log' file for more details

$ cat .nextflow.log
May-29 16:58:10.914 [main] DEBUG nextflow.cli.Launcher - $> ./nextflow run email.nf
May-29 16:58:13.479 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 0.29.0
May-29 16:58:13.497 [main] INFO  nextflow.cli.CmdRun - Launching `email.nf` [cheesy_archimedes] - revision: 25126a44ff
May-29 16:58:13.513 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: nextflow.config
May-29 16:58:13.516 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/nextflow.config
May-29 16:58:14.189 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
May-29 16:58:16.802 [main] DEBUG nextflow.Session - Session uuid: bd08c218-d707-498f-b5fe-5cbceb64f468
May-29 16:58:16.803 [main] DEBUG nextflow.Session - Run name: cheesy_archimedes
May-29 16:58:16.804 [main] DEBUG nextflow.Session - Executor pool size: 32
May-29 16:58:16.820 [main] DEBUG nextflow.cli.CmdRun -
  Version: 0.29.0 build 4803
  Modified: 24-04-2018 08:09 UTC (04:09 EDT)
  System: Linux 2.6.32-642.11.1.el6.x86_64
  Runtime: Groovy 2.4.15 on OpenJDK 64-Bit Server VM 1.8.0_171-b10
  Encoding: UTF-8 (ANSI_X3.4-1968)
  Process: 19968@phoenix2 [192.168.10.3]
  CPUs: 32 - Mem: 126 GB (12.9 GB) - Swap: 244.1 GB (229.6 GB)
May-29 16:58:16.977 [main] DEBUG nextflow.Session - Work-dir: /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/work [nfs]
May-29 16:58:18.301 [main] DEBUG nextflow.Session - Session start invoked
May-29 16:58:18.305 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
May-29 16:58:18.306 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /ifs/data/molecpathlab/production/Demultiplexing/180522_NB501073_0046_AH2LNJAFXY/trace.txt
May-29 16:58:18.433 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
May-29 16:58:18.911 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
May-29 16:58:18.927 [main] DEBUG nextflow.Channel - files for syntax: glob; folder: output/email/attachments/; pattern: *; options: null
May-29 16:58:25.601 [main] DEBUG nextflow.Session - Session aborted -- Cause: Broken pipe
May-29 16:58:25.730 [main] ERROR nextflow.cli.Launcher - Broken pipe
java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:95)
        at com.sun.mail.util.QPEncoderStream.output(QPEncoderStream.java:197)
        at com.sun.mail.util.QPEncoderStream.write(QPEncoderStream.java:140)
        at com.sun.mail.util.QPEncoderStream.write(QPEncoderStream.java:95)
        at javax.activation.DataHandler.writeTo(DataHandler.java:309)
        at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
        at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:865)
        at javax.mail.internet.MimeMultipart.writeTo(MimeMultipart.java:462)
        at com.sun.mail.handlers.multipart_mixed.writeTo(multipart_mixed.java:103)
        at javax.activation.ObjectDataContentHandler.writeTo(DataHandler.java:889)
        at javax.activation.DataHandler.writeTo(DataHandler.java:317)
        at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
        at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1773)
        at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1749)
        at nextflow.mail.Mailer.sendViaSysMail(Mailer.groovy:255)
        at nextflow.mail.Mailer.send(Mailer.groovy:431)
        at nextflow.mail.Mailer.send(Mailer.groovy:496)
        at nextflow.mail.Mailer$send.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
        at nextflow.Nextflow.sendMail(Nextflow.groovy:385)
        at nextflow.Nextflow$sendMail.callStatic(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:197)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:209)
        at _nf_script_68b8c80f.run(_nf_script_68b8c80f:11)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:341)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:165)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:223)
        at nextflow.cli.Launcher.run(Launcher.groovy:428)
        at nextflow.cli.Launcher.main(Launcher.groovy:582)

This method was working fine until it randomly started breaking today.

On Wednesday, May 2, 2018 at 1:03:53 PM UTC-4, Paolo Di Tommaso wrote:

What about creating a script just sending a email ? 

 

#!/bin/env nextflow 

sendMail {

  from <address>

  to <adress>

  """
  Ciao

  """

}

p

On Wed, May 2, 2018 at 6:59 PM, Steve <skbi...@gmail.com> wrote:

Is the `sendMail` function available from outside of Nextflow? Its a lot more convenient than trying to write a `sendmail` command in bash, from what I have seen, especially in terms of dealing with attachments. `mutt` is a good bash alternative but it is not always installed on systems.

On Monday, April 30, 2018 at 3:38:29 AM UTC-4, Paolo Di Tommaso wrote:

Just use a simple bash wrapper to launch your pipeline execution and send the email when it completes. 

p

On Sat, Apr 28, 2018 at 8:29 PM, Lavi Bharath <lavib...@gmail.com> wrote:

I am also trying to attach report.html as attachment. 

report.html is generated as the last step and hence sending report.html as an attachment fails in workflow.onComplete

Any workaround for this? Thanks.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

When using /usr/bin/env <something> shebang declaration, `something` need to be in directory in your PATH 

https://stackoverflow.com/a/10383546/395921

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.