Using pick_open_reference_otus.py with a large .fna file (32GB) on an lsf cluster (orchestra@HMS)

[David Gootenberg]

Hello there,

I am attempting to use pick_open_reference_otus.py on a large .fna file (32GB, can figure out the exact number of sequences if desired) on an lsf cluster (orchestra at Harvard Medical School). This .fna file is a concatenation of several qiime-demultiplexed MiSeq runs that were performed on fecal samples from 100s of individuals. I have attempted picking 3 times and it has "stalled" (displayed no file changes and very low CPU usages ~1 CPUsecond/hr for 24 hours) once on step3 and twice on step1 after a few days of running. I've made some tweaks to the /tmp/ file location as well as the assignment of cores and memory (in more detail below). I've followed the protocol for submitting Qiime Errors as below:

Step 1 (small test data set)

We recently completed picking on a 9GB .fna file (~22*10^6 sequences) that was a subset of this cohort. This took approximately 4 days to finish and the final usage summary was:

CPU time   : 285841.00 sec.

  Max Memory :     24785 MB

  Max Swap   :     37556 MB

  Max Processes  :        37

  Max Threads    :        46

The original call was:

bsub -q priority -W 720:00 -B -n 8 -R “usage[mem=16000]” pick_open_reference_otus.py -i /groups/.../bal_and_boston_seqs.fna -o /groups/.../2017_04_18_bal_and_boston_pickopenotus_97v2 -r /groups/shared_databases/GreenGeneOTU/gg_13_8_otus/rep_set/97_otus.fasta -n 2017_04_18_bal_and_boston_pickopenotus_97 -a -O 8 -p /groups/.../uclust_parameters_no_rev_strand_match.txt

so I am surprised that 46 threads were used when I called for 8 in the original call. Also it seems I used less that the 8*16GB=128GB of RAM that I asked for.

Step 2 (exact command and error)

The calls differed but the most recent call I made was:

bsub -q priority -W 720:00 -B -n 12 -R "rusage[mem=14000]" pick_open_reference_otus.py -i /groups/.../seqs_all_boston_uganda_botswana_2017_04_13.fna -o /groups/.../2017_04_13_pickopenotus_97v4 -r /groups/shared_databases/GreenGeneOTU/gg_13_8_otus/rep_set/97_otus.fasta -n 2017_04_13_pickopenotus_97 -a -O 12 -p /groups/.../uclust_parameters_no_rev_strand_match.txt

Which is asking for 12 cores each with 14GB of RAM.

The previous attempts were:

1.  12 cores, 8GB RAM/core, not using full file paths and /tmp/ was /tmp/ (reached step3 but after 2 days I saw 24 hours of no file changes and ~1 CPU-second/hr)

2.  4 cores, 48GB RAM/core, using full file paths and /tmp/ was /tmp/ (reached step1 but after 5 days I saw 24 hours of no file changes and ~1 CPU-second/hr)

3.  8 cores, 24GB RAM/core, using full file paths and /n/scratch2/ as /tmp/ (reached step1 but after 4 days I saw 24 hours of no file changes and ~1 CPU-second/hr)

As indicated above, I never saw any errors per se but I killed these jobs after ~24 hours of no changes to the target files and usage of ~1 CPU-second/hr as measured by bjobs.

The resource usage on the 3rd attempt before killing was 

CPU time   :  93947.00 sec.

   Max Memory :     53403 MB

   Max Swap   :     63873 MB

   Max Processes  :        37

   Max Threads    :        37

Step 3 (qiime config)

(My assumption is that the failures are "extraneous" bits of qiime. I had no errors only print_qiime_config.py -t [below])

print_qiime_config.py -tf

System information

==================

         Platform: linux2

   Python version: 2.7.6 (default, Aug  3 2015, 17:43:52)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)]

Python executable: /opt/python-2.7.6/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.3

Dependency versions

===================

                QIIME library version: 1.9.1

                 QIIME script version: 1.9.1

      qiime-default-reference version: 0.1.3

                        NumPy version: 1.10.1

                        SciPy version: 0.15.1

                       pandas version: 0.16.2

                   matplotlib version: 1.4.3

                  biom-format version: 2.1.4

                         h5py version: 2.5.0 (HDF5 version: 1.8.16)

                         qcli version: 0.1.1

                         pyqi version: 0.3.2

                   scikit-bio version: 0.2.3

                       PyNAST version: 1.2.2

                      Emperor version: 0.9.51

                      burrito version: 0.9.1

             burrito-fillings version: 0.1.1

                    sortmerna version: SortMeRNA version 2.0, 29/11/2014

                    sumaclust version: SUMACLUST Version 1.0.00

                        swarm version: Swarm 1.2.19 [Mar 30 2016 11:33:38]

                                gdata: Installed.

RDP Classifier version (if installed): Not installed.

          Java version (if installed): 1.6.0_27

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: None

      pick_otus_reference_seqs_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /n/scratch2/dg102/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: blastall

                 seconds_to_sleep: 1

QIIME full install test results

===============================

.FFFF..FFFF.F....FF..FFF..F

======================================================================

FAIL: test_INFERNAL_supported_version (__main__.QIIMEDependencyFull)

INFERNAL is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 498, in test_INFERNAL_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Infernal not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_R_supported_version (__main__.QIIMEDependencyFull)

R is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 693, in test_R_supported_version

    % ('.'.join(map(str, minimum_version)), version_string))

AssertionError: Unsupported R version. 2.12.0 or greater is required, but running /programs/x86_64-linux/system/opt/linux-centos6-x86_64/gcc-4.9.2/R-3.3.2-7vupd4rahkuofhlx2sbphkjjnrpgnkrp/rlib/R/bin/exec/R: /opt/gcc/4.8.5/lib64/libgomp.so.1: version `GOMP_4.0' not found (required by /programs/x86_64-linux/system/opt/linux-centos6-x86_64/gcc-4.9.2/R-3.3.2-7vupd4rahkuofhlx2sbphkjjnrpgnkrp/rlib/R/lib/libR.so)

.

======================================================================

FAIL: test_ampliconnoise_install (__main__.QIIMEDependencyFull)

AmpliconNoise install looks sane.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 382, in test_ampliconnoise_install

    "$PYRO_LOOKUP_FILE variable is not set. See %s for help." % url)

AssertionError: $PYRO_LOOKUP_FILE variable is not set. See http://qiime.org/install/install.html#ampliconnoise-install-notes for help.

======================================================================

FAIL: test_blast_supported_version (__main__.QIIMEDependencyFull)

blast is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 470, in test_blast_supported_version

    % ('.'.join(map(str, acceptable_version)), version_string))

AssertionError: Unsupported blast version. 2.2.22 is required, but running 2.2.26.

======================================================================

FAIL: test_cdbtools_supported_version (__main__.QIIMEDependencyFull)

cdbtools is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 477, in test_cdbtools_supported_version

    "which components of QIIME you plan to use.")

AssertionError: cdbtools not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_cdhit_supported_version (__main__.QIIMEDependencyFull)

cd-hit is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 621, in test_cdhit_supported_version

    "which components of QIIME you plan to use.")

AssertionError: cd-hit not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_chimeraSlayer_install (__main__.QIIMEDependencyFull)

no obvious problems with ChimeraSlayer install

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 429, in test_chimeraSlayer_install

    self.assertTrue(chim_slay, "ChimeraSlayer was not found in your $PATH")

AssertionError: ChimeraSlayer was not found in your $PATH

======================================================================

FAIL: test_clearcut_supported_version (__main__.QIIMEDependencyFull)

clearcut is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 601, in test_clearcut_supported_version

    "which components of QIIME you plan to use.")

AssertionError: clearcut not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_denoiser_supported_version (__main__.QIIMEDependencyFull)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 570, in test_denoiser_supported_version

    "Denoiser flowgram aligner not found or not "

AssertionError: Denoiser flowgram aligner not found or not executable. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_mothur_supported_version (__main__.QIIMEDependencyFull)

mothur is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 558, in test_mothur_supported_version

    % ('.'.join(map(str, acceptable_version)), version_string))

AssertionError: Unsupported mothur version. 1.25.0 is required, but running 1.38.1.

======================================================================

FAIL: test_muscle_supported_version (__main__.QIIMEDependencyFull)

muscle is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 519, in test_muscle_supported_version

    "which components of QIIME you plan to use.")

AssertionError: muscle not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_raxmlHPC_supported_version (__main__.QIIMEDependencyFull)

raxmlHPC is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 594, in test_raxmlHPC_supported_version

    % ('.'.join(map(str, acceptable_version)), version_string))

AssertionError: Unsupported raxmlHPC version. (7, 3, 0).(7, 3, 0) is required, but running 7.4.8.

======================================================================

FAIL: test_rtax_supported_version (__main__.QIIMEDependencyFull)

rtax is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 629, in test_rtax_supported_version

    "which components of QIIME you plan to use.")

AssertionError: rtax not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================

FAIL: test_sourcetracker_installed (__main__.QIIMEDependencyFull)

sourcetracker is installed

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 412, in test_sourcetracker_installed

    ("SOURCETRACKER_PATH is not set. This is "

AssertionError: SOURCETRACKER_PATH is not set. This is only important if you plan to use SourceTracker.

======================================================================

FAIL: test_usearch_supported_version (__main__.QIIMEDependencyFull)

usearch is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/opt/qiime-1.9.1/bin/print_qiime_config.py", line 650, in test_usearch_supported_version

    "which components of QIIME you plan to use.")

AssertionError: usearch not found. This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

Ran 27 tests in 20.665s

FAILED (failures=15)

print_qiime_config.py -t

System information

==================

         Platform: linux2

   Python version: 2.7.6 (default, Aug  3 2015, 17:43:52)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)]

Python executable: /opt/python-2.7.6/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.3

Dependency versions

===================

          QIIME library version: 1.9.1

           QIIME script version: 1.9.1

qiime-default-reference version: 0.1.3

                  NumPy version: 1.10.1

                  SciPy version: 0.15.1

                 pandas version: 0.16.2

             matplotlib version: 1.4.3

            biom-format version: 2.1.4

                   h5py version: 2.5.0 (HDF5 version: 1.8.16)

                   qcli version: 0.1.1

                   pyqi version: 0.3.2

             scikit-bio version: 0.2.3

                 PyNAST version: 1.2.2

                Emperor version: 0.9.51

                burrito version: 0.9.1

       burrito-fillings version: 0.1.1

              sortmerna version: SortMeRNA version 2.0, 29/11/2014

              sumaclust version: SUMACLUST Version 1.0.00

                  swarm version: Swarm 1.2.19 [Mar 30 2016 11:33:38]

                          gdata: Installed.

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: None

      pick_otus_reference_seqs_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /opt/qiime-1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /n/scratch2/dg102/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: blastall

                 seconds_to_sleep: 1

QIIME base install test results

===============================

.........

----------------------------------------------------------------------

Ran 9 tests in 0.347s

OK

Summary and additional data

I had suspected some of my issues were a result of not using the absolute file path but instead using the relative file path from the calling location, but I corrected that after the 1st attempt. I troubleshooted what I thought was /tmp/ filling up (I think Qiime will use the local node /tmp/ which is ~100G) so I changed the qiime_config to use /n/scratch2 which is a large temporary file system. I'm not sure if the method is being core-limited, RAM-limited, I/O-limited or something else which is why I tweaked some of those factors. Also, I haven't been able to find a start_parallel_jobs.py command specific to lsf servers (other than one from 2010 https://gist.github.com/laserson/769114 ) so I wonder if that is playing into this, since I seem to be using far more processes/threads than I originally ask for.

I have saved all of the intermediate files from the failed runs, which includes the timing of when qiime finished each step.

My params file is uclust_parameters_no_rev_strand_match.txt

pick_otus:enable_rev_strand_match False

pick_otus:max_accepts 1

pick_otus:max_rejects 8

pick_otus:stepwords 8

pick_otus:word_length 8

Any help would be appreciated. Thanks!

-David

[Jai Ram Rideout]

Hi David,

Thanks for the detailed post! I'm not sure what's going wrong but I suspect that either 1) your concatenated .fna file may not meet the requirements required by QIIME/uclust; or 2) the temp directory used by QIIME is running out of disk space (there's a bug related to that, detailed below).

1) Can you run validate_demultiplexed_fasta.py on your concatenated .fna file to make sure it's valid for OTU picking?

2) If step 1 looks good, let's try to get QIIME to use the correct temporary directory. There are still some lingering bugs in QIIME where /tmp is used instead of your QIIME config's temp_dir. It sounds like you want to use /n/scratch2/dg102/ as your temporary directory. Make sure you have temp_dir pointing to that location in your QIIME config (it looks like you do already, but please confirm that). Next, set the TMPDIR environment variable to also point to that temp directory. You'll want the environment variable to be set on the node that pick_open_reference_otus.py is running on, and how you accomplish this will depend on your cluster's configuration / queueing system. If you're unsure about how to set an environment variable before running a command on your cluster I recommend talking with your system administrator.

If I were running this directly (i.e. not through a queueing system), there's a couple of ways to set the variable:

export TMPDIR=/n/scratch2/dg102/

pick_open_reference_otus.py -i ...-r ...

OR

TMPDIR=/n/scratch2/dg102/ pick_open_reference_otus.py -i ... -r ...

Let us know how it goes!

Jai

[David Gootenberg]

Just an update, I'm working on 1) but validate_demultiplexed_fasta.py has been running on the 32GB .fna for about 5 hours on 4 cores and still hasn't produced any output beyond the initial empty output directory. I guess big files take a long time.  I'll keep you updated.

2) I'll contact my system administrator about setting the environment variable.

-David

[Jai Ram Rideout]

Thanks David! validate_demultiplexed_fasta.py may take quite some time to run-- it doesn't run in parallel so only a single core will be used.

Jai

[David Gootenberg]

Oh, duh. Well guess I'm holding up 4 cores for nothing...

[David Gootenberg]

Hi Jai, thanks for your patience.

I had a chance to run 1) validate_demultiplexed_fasta.py using a partial mapping file (which is why some of these numbers are wonky) and I got the following, so I realize I have to eliminate duplicates that arose from concatenating multiple demultiplexed .fna files with unassigned samples. My plan is just to use sed to replace every unassigned with unassigned[date] in the original separate .fna files which should eliminate duplicates and then I'll re-concatenate them. But there were some other issues that I think are more core to the problem and I don't seem to see previously referenced in the forums. (see below: HOWEVER)

# fasta file seqs_all_boston_uganda_botswana_2017_04_13.fna validation report

Percent duplicate labels: 0.000

Percent QIIME-incompatible fasta labels: 0.000

Percent of labels that fail to map to SampleIDs: 0.896 (this is just because I don't have a full 1000+ sample mapping file finished)

Percent of sequences with invalid characters: 0.000

Percent of sequences with barcodes detected: 0.005

Percent of sequences with barcodes detected at the beginning of the sequence: 0.000

Percent of sequences with primers detected: 0.000

Duplicate labels found:

Unassigned_4049647

Unassigned_2931256

Unassigned_2116795

Unassigned_4476327

Unassigned_5543589

Unassigned_4533228

Unassigned_4868118

Unassigned_6446321

Unassigned_1039100

Unassigned_6323880

Unassigned_746031

... [continues]

HOWEVER,

It turns out one of my problems was cluster-intrinsic for orchestra, since their queue was having a problem with long pending times which is why some of my picking calls were waiting in the queue for a long time.  That seems to have resolved and my most recent pick got to step3 before "failing." Upon looking at the output files, it seems that there might some files missing from step3 that might be the prime reason causing the pipeline to hang (because I think I may have seen things like this before)

As a reminder, my original call was 

bsub -q priority -W 720:00 -B -n 12 -R "rusage[mem=14000]" pick_open_reference_otus.py -i /groups/.../seqs_all_boston_uganda_botswana_2017_04_13.fna -o /groups/.../2017_04_13_pickopenotus_97v4 -r /groups/shared_databases/GreenGeneOTU/gg_13_8_otus/rep_set/97_otus.fasta -n 2017_04_13_pickopenotus_97 -a -O 12 -p /groups/.../uclust_parameters_no_rev_strand_match.txt

But in the step3 folder I think we are missing files that should be associated with thread 0 and 9. Based on my understanding of qiime this could cause the re-aggregator script to loop infinitely waiting for those files to be made.

Thu May 04 12:44:45 dg102@mezzanine:/groups/.../2017_04_13_pickopenotus_97v4/step3_otus$ ls -a

total 6.9G

drwxrwsr-x 3 dg102 kwon 1.6K May  3 15:55 ./

drwxrwsr-x 5 dg102 kwon  124 May  3 15:10 ../

drwxrwsr-x 2 dg102 kwon  576 May  3 15:55 POTU_OB9G_/

-rw-rw-r-- 1 dg102 kwon 356M May  3 15:50 POTU_OB9G_.0_clusters.uc

-rw-rw-r-- 1 dg102 kwon 368M May  3 15:50 POTU_OB9G_.10_clusters.uc

-rw-rw-r-- 1 dg102 kwon  50M May  3 15:50 POTU_OB9G_.10_failures.txt

-rw-rw-r-- 1 dg102 kwon  807 May  3 15:50 POTU_OB9G_.10_otus.log

-rw-rw-r-- 1 dg102 kwon 118M May  3 15:50 POTU_OB9G_.10_otus.txt

-rw-rw-r-- 1 dg102 kwon 295M May  3 15:44 POTU_OB9G_.11_clusters.uc

-rw-rw-r-- 1 dg102 kwon  43M May  3 15:44 POTU_OB9G_.11_failures.txt

-rw-rw-r-- 1 dg102 kwon  807 May  3 15:44 POTU_OB9G_.11_otus.log

-rw-rw-r-- 1 dg102 kwon 139M May  3 15:44 POTU_OB9G_.11_otus.txt

-rw-rw-r-- 1 dg102 kwon 262M May  3 15:42 POTU_OB9G_.1_clusters.uc

-rw-rw-r-- 1 dg102 kwon  30M May  3 15:42 POTU_OB9G_.1_failures.txt

-rw-rw-r-- 1 dg102 kwon  805 May  3 15:42 POTU_OB9G_.1_otus.log

-rw-rw-r-- 1 dg102 kwon 130M May  3 15:42 POTU_OB9G_.1_otus.txt

-rw-rw-r-- 1 dg102 kwon 376M May  3 15:52 POTU_OB9G_.2_clusters.uc

-rw-rw-r-- 1 dg102 kwon  54M May  3 15:52 POTU_OB9G_.2_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:52 POTU_OB9G_.2_otus.log

-rw-rw-r-- 1 dg102 kwon 107M May  3 15:52 POTU_OB9G_.2_otus.txt

-rw-rw-r-- 1 dg102 kwon 405M May  3 15:53 POTU_OB9G_.3_clusters.uc

-rw-rw-r-- 1 dg102 kwon 341M May  3 15:49 POTU_OB9G_.4_clusters.uc

-rw-rw-r-- 1 dg102 kwon  52M May  3 15:49 POTU_OB9G_.4_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:49 POTU_OB9G_.4_otus.log

-rw-rw-r-- 1 dg102 kwon 123M May  3 15:49 POTU_OB9G_.4_otus.txt

-rw-rw-r-- 1 dg102 kwon 394M May  3 15:52 POTU_OB9G_.5_clusters.uc

-rw-rw-r-- 1 dg102 kwon  50M May  3 15:53 POTU_OB9G_.5_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:53 POTU_OB9G_.5_otus.log

-rw-rw-r-- 1 dg102 kwon 115M May  3 15:53 POTU_OB9G_.5_otus.txt

-rw-rw-r-- 1 dg102 kwon 325M May  3 15:46 POTU_OB9G_.6_clusters.uc

-rw-rw-r-- 1 dg102 kwon  35M May  3 15:47 POTU_OB9G_.6_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:47 POTU_OB9G_.6_otus.log

-rw-rw-r-- 1 dg102 kwon 114M May  3 15:47 POTU_OB9G_.6_otus.txt

-rw-rw-r-- 1 dg102 kwon 362M May  3 15:50 POTU_OB9G_.7_clusters.uc

-rw-rw-r-- 1 dg102 kwon  47M May  3 15:51 POTU_OB9G_.7_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:51 POTU_OB9G_.7_otus.log

-rw-rw-r-- 1 dg102 kwon 108M May  3 15:51 POTU_OB9G_.7_otus.txt

-rw-rw-r-- 1 dg102 kwon 404M May  3 15:55 POTU_OB9G_.8_clusters.uc

-rw-rw-r-- 1 dg102 kwon  63M May  3 15:55 POTU_OB9G_.8_failures.txt

-rw-rw-r-- 1 dg102 kwon  806 May  3 15:55 POTU_OB9G_.8_otus.log

-rw-rw-r-- 1 dg102 kwon 102M May  3 15:55 POTU_OB9G_.8_otus.txt

-rw-rw-r-- 1 dg102 kwon 372M May  3 15:52 POTU_OB9G_.9_clusters.uc

For reference, ls of POTU_OB9G and content of some the contained files (deletion list, expected out files, merge ma, jobs)

Thu May 04 12:46:41 dg102@mezzanine:/groups/.../2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_$ ls

total 28G

-rw-rw-r-- 1 dg102 kwon 5.8K May  3 15:28 deletion_list.txt

-rw-rw-r-- 1 dg102 kwon 5.3K May  3 15:28 expected_out_files.txt

-rw-rw-r-- 1 dg102 kwon 4.3K May  3 15:28 merge_map.txt

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:19 POTU_OB9G_.0.fasta

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:27 POTU_OB9G_.10.fasta

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:28 POTU_OB9G_.11.fasta

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:20 POTU_OB9G_.1.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:21 POTU_OB9G_.2.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:22 POTU_OB9G_.3.fasta

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:22 POTU_OB9G_.4.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:23 POTU_OB9G_.5.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:24 POTU_OB9G_.6.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:25 POTU_OB9G_.7.fasta

-rw-rw-r-- 1 dg102 kwon 1.9G May  3 15:26 POTU_OB9G_.8.fasta

-rw-rw-r-- 1 dg102 kwon 2.0G May  3 15:26 POTU_OB9G_.9.fasta

-rw-rw-r-- 1 dg102 kwon  17K May  3 15:28 POTU_OB9G_jobs.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.10.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.11.fasta

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/merge_map.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/deletion_list.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/expected_out_files.txt

deletion_list.txt (END) 

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_clusters.uc

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_failures.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_clusters.uc

expected_out_files.txt (END) 

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.txt   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.txt  /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.txt  /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//failures_otus.txt

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.log   /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_otus.log  /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_otus.log  /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//failures_otus.log

/groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_failures.txt       /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.10_failures.txt      /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.11_failures.txt      /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//failures_failures.txt

merge_map.txt (END) 

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.0_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.0_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.1_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.1_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.2_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.2_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.3_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.3_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.4_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.4_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.5_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.5_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.6_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.6_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.7_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.7_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.8_clusters.uc /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.8_clusters.uc ; exit

/bin/bash;  pick_otus.py -i /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9.fasta -r /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step2_otus//step2_rep_set.fna -m uclust_ref --suppress_new_clusters -o /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_ -s 0.97    --max_accepts 1 --max_rejects 8 --stepwords 8 --w 8   ; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9_otus.log /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.log; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9_otus.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_otus.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9_failures.txt /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus//POTU_OB9G_.9_failures.txt; mv /groups/kwon/david/2017_04_03_all_16S_analysis/2017_04_13_pickopenotus_97v4/step3_otus/POTU_OB9G_/POTU_OB9G_.9_clusters.uc 

POTU_OB9G_jobs.txt 

Thanks for your help!

[Greg Caporaso]

Hi David,

Thanks for the info. You're right - it seems like the poller is waiting on the 0 and 9 jobs to complete. If those are no longer running, the poller will wait for them indefinitely. If they're not still running (which sounds like the case here) we need some more information to determine why that is. Does your queueing system create logs of standard out and standard error? Those might have error messages in them for the 0 and the 9 jobs. Alternatively, is it possible that the queueing system killed the job because it ran for too long, or used more memory than was allocated for the job? 

Also, a couple of tips that are helpful as you're trying to debug these types of issues. If you call identify_missing_files.py passing the expected_out_files.txt as -e, you'll get a list of the files that the poller is still waiting on. This is useful for figuring out which jobs may have failed. As long as the poller is running, if you re-run the job that failed (you can find the command in the <job-name>_jobs.txt file, in your case POTU_OB9G_jobs.txt), the poller should notice when the missing files are generated and allow the job to complete (i.e., aggregate all of the results into the final output files). 

Best,

Greg

[David Gootenberg]

Great, thanks Greg! I'll follow up on the lsf logs to see what happened with those jobs. And thanks for the tips on debugging and interpreting the job files. I'll check them out and report back.

Have a good weekend!

[David Gootenberg]

Just as a follow-up/bookend it appears that my issue was the duplicate sampleIDs arising from the unassigned samples. I renamed those by date of sequencing to make them unique and my command seemed to finish. I'm running further analysis now so I'll see if there was any bugs but I got all the expected output files. The run stats FYI:

Started at Fri May 12 11:13:45 2017
Results reported at Sat May 20 23:53:49 2017

Successfully completed.

Resource usage summary:

    CPU time   : 901491.00 sec.
    Max Memory :     58858 MB
    Max Swap   :     74007 MB

    Max Processes  :        53
    Max Threads    :        66

Thanks all for your help!

[Colin Brislawn]

Hello David,

Thanks for posting your log file. This line jumped out at me:

    Max Swap   :     74007 MB 

That's way too much swap! Reading and writing files to swap disk is literally thousands of times slower than reading files from RAM, so this heavy use of swap space would slow your run to a crawl. 

In this case, using fewer threads may reduce your run times; while -O 6 may use 1/2 the CPU time, it will avoid the -1000x performance penalty of running out of RAM and using the Swap disk instead.

Every data set is a little different, but I think experimenting with setting to minimize your swap usage could be very worthwhile.

Colin

[David Gootenberg]

Yeah thanks for the tip. I think this may just be a quirk in our lsf reporting system. When I spoke to our core staff they said that is just a mis-naming of total memory in the log (because that's how our lsf build outputs it). It does not actually use that much swap, but I will keep that in mind in the future if I find I am actually using that much swap.

-David