run-asm-pipeline.sh caused tail: error writing 'standard output': Broken pipe

[Michał T. Lorenc]

Hi all,
I ran 3d-dna with this command `singularity shell 3d-dna.simg /3d-dna/run-asm-pipeline.sh --editor-repeat-coverage 4 3d/pilon.fasta 3d/merged_nodups.txt` but I got
Broken pipes:

###############
Finilizing output:
... -s flag was triggered, treating all contigs/scaffolds shorter than 15000 as unattempted.
... -l flag was triggered. Output will appear with headers of the form pilon_hic_scaffold_#.
... -g flag was triggered, making gap size between scaffolded draft sequences to be equal to 500.
Analyzing the merged assembly
...trimming N overhangs
...adding gaps
tail: error writing 'standard output': Broken pipe
...
tail: error writing 'standard output': Broken pipe
########################### Job Execution History #############################
JobId:66477.flm1
UserName:michallorenc
GroupName:qris-qut
JobName:3d-dna
SessionId:15569
ResourcesRequested:mem=200gb,ncpus=24,place=free,walltime=150:00:00
ResourcesUsed:cpupercent=2344,cput=386:04:13,mem=52520684kb,ncpus=24,vmem=55736996kb,walltime=42:27:31
QueueUsed:Single
AccountString:qris-qut
ExitStatus:0

Please find below my singulirity container:

BootStrap: docker
From: debian:stretch-backports

%help
  A container with 3d-dna

%post
  apt-get update && apt-get -y upgrade
  apt-get -y install \
    build-essential \
    wget \
    bzip2 \
    ca-certificates \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    procps \
    locales \
    git

  rm -rf /var/lib/apt/lists/*
  apt-get clean

  sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && \
    update-locale LANG=en_US.UTF-8

  # download and install Miniconda
  wget -c https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
    /bin/bash Miniconda2-latest-Linux-x86_64.sh -bfp /usr/local

  ln -s /usr/local/etc/profile.d/conda.sh /etc/profile.d/conda.sh
  rm -rf Miniconda2-latest-Linux-x86_64.sh

  # add bioconda channels
  conda config --add channels defaults
  conda config --add channels conda-forge
  conda config --add channels bioconda
  conda update conda

  # install 3d-dna dependecies
  conda install -y lastz=1.0.4
  conda install -y openjdk=8.0.152
  conda install -y gawk=4.2.1
  conda install -y coreutils=8.30
  conda install -y parallel=20180522
  conda install -y python=2.7
  conda install -y numpy=1.15.1
  conda install -y scipy=1.1.0
  conda install -y matplotlib=2.2.3
  conda clean --all --yes

  # install 3d-dna
  git clone https://github.com/theaidenlab/3d-dna.git

%environment
  export PATH=/usr/local/bin/:$PATH
  export LANG=en_US.UTF-8

%runscript
  source /etc/profile.d/conda.sh
  conda activate 3d-dna

How can I fix the Broken Pipes?

Thank you in advance.

Michal

[Olga Dudchenko]

Hi Michal,

I have received a similar inquiry just a few days ago and have been trying to investigate. This may be system specific. See this link below on input exhaustion and suggestions that they list re dd:

https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error

Let me know if this solved the problem. you'd want to insert in lines 62 and 64 of 3d-dna/finalize/construct-fasta-from-asm.sh

Thank!

Olga

[Michał T. Lorenc]

Hi Olga,

Maybe your tail syntax is OS specific because I tried it on Suse and Debian which caused the same error. Which OS do you use?

Thank you in advance,

Michal

[Michał T. Lorenc]

P.s. Do you SIGPIPE signal handler set to 'ignore' in your script ( https://unix.stackexchange.com/questions/481174/tail-error-writing-standard-output-broken-pipe )?

[Michał T. Lorenc]

P.S. What should insert in lines 62 and 64 of 3d-dna/finalize/construct-

fasta-from-asm.sh ? Could you please provide an example?

Thank you in advance.

Best wishes,

Michal

On Monday, November 12, 2018 at 9:01:57 AM UTC+10, Olga Dudchenko wrote:

[Michał T. Lorenc]

Hi Olga,

Do you think this would help https://unix.stackexchange.com/questions/481174/tail-error-writing-standard-output-broken-pipe#comment880649_481174 ?

Or have you already find a solution?

Thank you in advance,

Michal

[Olga Dudchenko]

Hey Michal,

I have a solution but I am traveling and cannot properly commit. As a temporary workaround, dump the output into a temporary file and then read it for reverse-fasta...

Hope this helps,

Olga

[Michał T. Lorenc]

Hi Olga,

Do you mean something like this:

cp pilon.FINAL.fasta pilon.FINAL.fasta.bak
bioawk -c fastx '{print ">"$name;print revcomp($seq)}' pilon.FINAL.fasta.bak

or
cat pilon.FINAL.fasta.bak | while read L; do  echo $L; read L; echo "$L" | rev | tr "ATGC" "TACG" ; done

Thank you in advance.

Michal

[Olga Dudchenko]

Oh, I actually meant something silly like

tail -c +${index[${contig}]} ${fasta} | awk '$0~/>/{exit}1' | awk -f ${pipeline}/utils/reverse-fasta.awk -

tail -c +${index[${contig}]} ${fasta} > tmp.fasta && awk '$0~/>/{exit}1' tmp.fasta | awk -f ${pipeline}/utils/reverse-fasta.awk

Btw, this has worked for another person with the same problem: https://github.com/theaidenlab/3d-dna/issues/29

Best,

Olga

[Michał T. Lorenc]

Hi Olga,

where should I insert or replece those two of your lines of code ?

Thank you in advance,

Best wishes,

Michal