Permission errors with docker and nextflow?

[Lauren Kleine]

Hi Paolo,

Here is the repository for the this nextflow project: https://github.com/laurenkleine/taxonomy-nf 

I have a process stuck right now and I keep getting this error:

laurenrk@stengleinlab101:~/projects/taxonomy-nf$ nextflow run main_practice_docker.nf -profile docker -resume

N E X T F L O W  ~  version 0.26.0

Launching `main_practice_docker.nf` [suspicious_meninsky] - revision: 6832b5f676

[warm up] executor > local

[41/135b63] Cached process > RunCutAdapt (Croc7776)

[b4/6d6514] Cached process > RunPreFastQC (Croc7776)

[f1/2477ad] Cached process > cd_hit_est (Croc7776)

[83/4342f5] Cached process > reconcile_reads (Croc7776)

[6b/c60c5f] Cached process > RunPostFastQC (Croc7776)

[28/973514] Submitted process > runBowtie2 (Croc7776)

ERROR ~ Error executing process > 'runBowtie2 (Croc7776)'

Caused by:

  Process `runBowtie2 (Croc7776)` terminated with an error exit status (1)

Command executed:

  bowtie2 -x /home/databases/croc/croc_genome --local -q -U Croc7776_R1_fu.fastq --sensitive --score-min C,60,0 --time -p 12 -S Croc7776_R1_fu_bt.sam 2> Croc7776_R1_bt.log

  

  /home/laurenrk/projects/taxonomy-nf/bin/fasta_from_sam -f Croc7776_R1_fu.fastq -r Croc7776_R1_fu_bt.sam > Croc7776_R1_fuh1.fastq

  

  /home/laurenrk/projects/taxonomy-nf/bin/reconcile_read2_file Croc7776_R1_fuh1.fastq Croc7776_R2_fu.fastq f2 > Croc7776_R2_fuh1.fastq

  

  bowtie2 -x /home/databases/croc/croc_genome --local -q -U Croc7776_R2_fu.fastq --sensitive --score-min C,60,0 --time -p 12 -S Croc7776_R2_fu_bt.sam 2> Croc7776_R2_bt.log

  

  /home/laurenrk/projects/taxonomy-nf/bin/fasta_from_sam -f Croc7776_R2_fuh1.fastq -r Croc7776_R2_fu_bt.sam > Croc7776_R2_fuh.fastq

  

  /home/laurenrk/projects/taxonomy-nf/bin/reconcile_read2_file Croc7776_R2_fuh.fastq Croc7776_R1_fuh1.fastq > Croc7776_R1_fuh.fastq

Command exit status:

  1

Command output:

  (empty)

Command error:

  .command.run.1: line 101:    12 Terminated              nxf_trace "$pid" .command.trace

Work dir:

  /home/laurenrk/projects/taxonomy-nf/work/28/97351471e32d7ba7564658a69a42e2

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details

Here is the nextflow.log info from running this:

Apr-23 14:44:35.869 [Task submitter] INFO  nextflow.Session - [08/318e16] Submitted process > runBowtie2 (Croc7776)

Apr-23 14:44:36.987 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: runBowtie2 (Croc7776); status: COMPLETED; exit: 1; error: -; w

orkDir: /home/laurenrk/projects/taxonomy-nf/work/08/318e16b6e9dbdfbe920b4ee43ef0db]

Apr-23 14:44:37.037 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'runBowtie2 (Croc7776)'

I've tried many different bowtie2 images, I originally built my own, but now I keep trying different ones because I thought that was my issue. This might be an issue with the image? I've gotten similar errors with this process, mostly with permission errors, before I got this error I had one that said something along the lines of "Permission denied: Croc7776_R1_bt.log" ?

[Paolo Di Tommaso]

Does that happen only for the bowtie process? Also, what's the running system? 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Lauren Kleine]

Yes, this only happens with the bowtie process. The server is Ubuntu 14.04.5.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

What makes you think it's a permission problem? 

It seems more a command error.  I would suggest to change in the task work dir and debug it running `bash .command.run`.

p

[Lauren Kleine]

Oh I was wrong, it wasn't a permissions problem! 

I found the issue with Bowtie2 in the work directory for the process. Thank you!

[Lauren Kleine]

The last issue I was having didn't have to do with permissions, but this issue does.

I have a process that runs BLASTn remotely:

process BLASTcontigs_vs_nt {
	publishDir "${params.output}/BLAST_Results", mode: "copy"
	tag { dataset_id }
	input:
	set dataset_id, file("${dataset_id}_spade_contigs.fa") from (SPAdes_results2)
	output:
	set dataset_id, file("${dataset_id}.fa.bn_nt") into (BLASTresults1, BLASTresults2)
	set dataset_id, file("${dataset_id}_spade_contigs_n.fa") into (BLASTresults)
	"""
	blastn -query ${dataset_id}_spade_contigs.fa -db nt -remote -evalue 1e-8 -task megablast -outfmt 6 | $baseDir/bin/consolidate_blast_output > ${dataset_id}.fa.bn_nt
	$baseDir/bin/fasta_from_blast -f ${dataset_id}_spade_contigs.fa -r ${dataset_id}.fa.bn_nt > ${dataset_id}_spade_contigs_n.fa
	"""
	}

I was having trouble writing a Docker image for BLAST (I'm still very new to Docker), so I used biocontainers/blast:latest and configured it to this process (https://github.com/laurenkleine/taxonomy-nf/blob/551eb6cabe3e3d52a10ce055db454513d8b40940/config/docker.config#L19).

When this process tries to run, I get the error: 

  touch: cannot touch '.command.trace': Permission denied

I also have the  fixOwnership = true turned on for my docker.config file. Do you suppose this is why I am having this permission issue, because I did not create this biocontainers docker image?

[Paolo Di Tommaso]

I guess this happens because that container defines a custom user. 

Try adding the setting in your NF config: 

docker.runOptions = '-u $(id -u):$(id -g)'

(note the single quotes) 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Lauren Kleine]

That fixed it, thank you!

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Xin Gong]

Hello Paolo,

I tried docker.runOptions = '-u $(id -u):$(id -g)' but in .command.run I only see -u $(id-u) I didn't see :$(id -g)

Thank you!

Xin

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Xin Gong]

our sysadmin said id -g gives me primary group id. hard coding, docker.runOptions = ' -u <user id>: <group id> ' worked.