too many open files

[Xiaopeng Bian]

Hi,

 I got a "Too many open files" error while running a tumor/normal pair. I've run many pairs and never  have such problems. I tried the trick at:

https://github.com/chapmanb/bcbio-nextgen/issues/457

but that did not seem to fix the problem, do I need to be an admin to use "ulimit -u 20000"?

Other ways to fix it?

Thanks.

Sat Oct 28 05:19:13 EDT 2017] picard.sam.ReorderSam done. Elapsed time: 1,070.09 minutes.

Runtime.totalMemory()=3547725824

To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp

Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/MB/bcbiotx/tmpuhXmPZ/sortingcollection.7790529915946323988.tmp (Too many open files)

at htsjdk.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:463)

at htsjdk.samtools.util.SortingCollection$MergingIterator.<init>(SortingCollection.java:402)

at htsjdk.samtools.util.SortingCollection.iterator(SortingCollection.java:272)

at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:213)

at picard.sam.ReorderSam.doWork(ReorderSam.java:133)

at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:205)

at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:94)

at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:104)

Caused by: java.io.FileNotFoundException: /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/MB/bcbiotx/tmpuhXmPZ/sortingcollection.7790529915946323988.tmp (Too many open files)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at htsjdk.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:457)

... 7 more

' returned non-zero exit status 1

[Brad Chapman]

Xiaopeng;
Thanks for the report. It looks like the picard preparation process from
pre-aligned BAMs is using too many file handles for your current system. Based
on the issue you reference there are some detailed docs about file handles:

https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html#open-file-handles

Practically you'll just need more file handles on your system, either your
ulimit -u or ulimit -n. The docs offer tips on checking the hard and soft
limits to see what you can change without administrator privileges.

If you have a low hard limit and can't change them, then the only other option
is to try to avoid the problematic process. You can either realign the inputs
or use a more lightweight `bam_clean` option like `fixrgs` if you have a
compatible input BAM
(https://bcbio-nextgen.readthedocs.io/en/latest/contents/configuration.html#alignment).
Hope this helps,
Brad


> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.
> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/5b14a279-d78d-496b-ae11-c35088340a9b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Xiaopeng Bian]

Hi, Brad:

Thanks for your replay.

I checked with "ulimit -u" and "ulimit -n" both return 4096, I changed "unlimit -u 20000" but won't be able to change "ulimit -n" because "Operation not permitted"

/etc/security/limits.conf showed:

*                -       memlock         unlimited

*                soft    nofile          4096

*                hard    nofile          4096

I don't think I can change that for no admin privilege.

So I I use the fixrg option and hope that will work!

Thanks again.

Xiaopeng

[Xiaopeng Bian]

Hi, Brad:

I used the 'fixrgs' option, I still got the same error, what else I can do?

FYI, I attached the yaml file and posted the error message below.

Thanks.

Xiaopeng

[Thu Nov 02 03:51:40 EDT 2017] picard.sam.ReorderSam done. Elapsed time: 999.82 minutes.

Runtime.totalMemory()=3547725824

To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp

Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/MB/bcbiotx/tmpYaipVI/sortingcollection.1398367637786018823.tmp (Too many open files)

at htsjdk.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:463)

at htsjdk.samtools.util.SortingCollection$MergingIterator.<init>(SortingCollection.java:402)

at htsjdk.samtools.util.SortingCollection.iterator(SortingCollection.java:272)

at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:213)

at picard.sam.ReorderSam.doWork(ReorderSam.java:133)

at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:205)

at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:94)

at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:104)

Caused by: java.io.FileNotFoundException: /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/MB/bcbiotx/tmpYaipVI/sortingcollection.1398367637786018823.tmp (Too many open files)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at htsjdk.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:457)

... 7 more

' returned non-zero exit status 1

__________________________________

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Thanks for following up and including your configuration file. It looks like
the configuration file has `bamclean: picard` set for the second sample; it's
only changed to `bamclean: fixrg` for the first. My guess is the second sample
triggers the issue as well and you'd want to switch them both over. Hope this
helps,
Brad

> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/529c1473-68ae-4e38-97e5-be0b0ece65bf%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

> [ mb.yaml: application/octet-stream ]

[Xiaopeng Bian]

Hi, Brad:

Thanks for the quick reply. I fixed the yaml file and relauched but got halt for a long time(for more than a day) with warning message:

[2017-11-05T18:30Z] Warning: bcbio no longer support explicit setting of mark_duplicate algorithm. Using b

est-practice choice based on input data.

Does this mean I need to check duplication following Broad best practice before setting up bcbio and set "mark_duplicates: false"? Otherwise the process will get halt?

Thanks.

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Thanks for the report and sorry for the continued issues. The warning
shouldn't have anything to do with the longer hang you're seeing, that's just
the last thing logged to the screen.

Can you identify what bcbio is doing while it hangs? If not, could you try
rerunning with a single core (`-n 1`) and then doing a ctrl-c when it seems to
hang for a long period of time. That could help identify what it's doing
that's causing such a long wait time.

Hope this helps with debugging,
Brad


> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/13b75929-21b9-43fd-a0c1-7dccdbf11e97%40googlegroups.com.

[Xiaopeng Bian]

Hi, Brad:

Thanks for your reply. Here are some final logged messages from several long hang process:

1.

[2017-11-08T19:52Z] Timing: variant calling

[2017-11-08T19:52Z] multiprocessing: variantcall_sample

[2017-11-08T19:52Z] Genotyping paired variants with FreeBayes

2.

[2017-11-10T01:12Z] Timing: variant calling

[2017-11-10T01:12Z] multiprocessing: variantcall_sample

[2017-11-10T01:12Z] Genotyping paired variants with FreeBayes

3.

[2017-11-09T00:01Z] tabix index aml_full-Y_0_16096953.vcf.gz

[2017-11-09T00:02Z] GATK: VariantAnnotator

[2017-11-09T00:02Z] INFO  19:02:03,377 HelpFormatter - ----------------------------------------------------------------------------------

[2017-11-09T00:02Z] INFO  19:02:03,379 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18

[2017-11-09T00:02Z] INFO  19:02:03,379 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute

[2017-11-09T00:02Z] INFO  19:02:03,379 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk

[2017-11-09T00:02Z] INFO  19:02:03,379 HelpFormatter - [Wed Nov 08 19:02:03 EST 2017] Executing on Linux 2.6.32-504.16.2.el6.x86_64 amd64

[2017-11-09T00:02Z] INFO  19:02:03,379 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_102-b14

[2017-11-09T00:02Z] INFO  19:02:03,383 HelpFormatter - Program Args: -T VariantAnnotator -R /fdb/bcbio-nextgen/current/genomes/Hsapiens/g1k_v37_no_decoy2/seq/g1k_v37_no_decoy2.fa --variant /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/freebayes/Y/aml_full-Y_0_16096953.vcf.gz --out /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bcbiotx/tmp2O7paa/aml_full-Y_0_16096953-gatkann.vcf.gz -L /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/freebayes/Y/aml_full-Y_0_16096953.vcf.gz -I /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bamclean/aml-tumor_full/AML31_primary-tumor_wgs_na_hiseq_bwa-0.5.9-reorder-fixrgs-gatkfilter-dedup.bam -I /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bamclean/aml-normal_full/AML31_normal_wgs_na_hiseq_bwa-0.5.9-reorder-fixrgs-gatkfilter-dedup.bam -A BaseQualityRankSumTest -A FisherStrand -A GCContent -A HaplotypeScore -A HomopolymerRun -A MappingQualityRankSumTest -A MappingQualityZero -A QualByDepth -A ReadPosRankSumTest -A RMSMappingQuality --allow_potentially_misencoded_quality_scores -U ALL --read_filter BadCigar --read_filter NotPrimaryAlignment

[2017-11-09T00:02Z] INFO  19:02:03,391 HelpFormatter - Executing as bianxi@cn0318 on Linux 2.6.32-504.16.2.el6.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_102-b14.

[2017-11-09T00:02Z] INFO  19:02:03,391 HelpFormatter - Date/Time: 2017/11/08 19:02:03

[2017-11-09T00:02Z] INFO  19:02:03,391 HelpFormatter - ----------------------------------------------------------------------------------

[2017-11-09T00:02Z] INFO  19:02:03,392 HelpFormatter - ----------------------------------------------------------------------------------

[2017-11-09T00:02Z] INFO  19:02:03,401 GenomeAnalysisEngine - Strictness is SILENT

[2017-11-09T00:02Z] INFO  19:02:03,505 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 250

[2017-11-09T00:02Z] INFO  19:02:03,512 SAMDataSource$SAMReaders - Initializing SAMRecords in serial

[2017-11-09T00:02Z] INFO  19:02:03,602 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.09

[2017-11-09T00:02Z] INFO  19:02:04,014 IntervalUtils - Processing 5670 bp from intervals

[2017-11-09T00:02Z] WARN  19:02:04,019 IndexDictionaryUtils - Track variant doesn't have a sequence dictionary built in, skipping dictionary validation

[2017-11-09T00:02Z] INFO  19:02:04,082 GenomeAnalysisEngine - Preparing for traversal over 2 BAM files

[2017-11-09T00:02Z] INFO  19:02:05,033 GenomeAnalysisEngine - Done preparing for traversal

[2017-11-09T00:02Z] INFO  19:02:05,033 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]

[2017-11-09T00:02Z] INFO  19:02:05,034 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining

[2017-11-09T00:02Z] INFO  19:02:05,034 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime

[2017-11-09T00:02Z] INFO  19:02:05,087 StrandBiasTest - SAM/BAM data was found. Attempting to use read data to calculate strand bias annotations values.

[2017-11-09T00:02Z] WARN  19:02:05,599 HaplotypeScore - Annotation will not be calculated, must be called from UnifiedGenotyper, not org.broadinstitute.gatk.tools.walkers.annotator.VariantAnnotator

[2017-11-09T00:02Z] INFO  19:02:35,038 ProgressMeter -       Y:9316443      2455.0    30.0 s       3.4 h       19.2%     2.6 m       2.1 m

[2017-11-09T00:03Z] INFO  19:03:05,040 ProgressMeter -      Y:13474008    210553.0    60.0 s       4.7 m       70.5%    85.0 s      25.0 s

[2017-11-09T00:03Z] INFO  19:03:33,642 VariantAnnotator - Processed 4999 loci.

[2017-11-09T00:03Z] 

[2017-11-09T00:03Z] INFO  19:03:33,653 ProgressMeter -            done    320588.0    88.0 s       4.6 m      100.0%    88.0 s       0.0 s

[2017-11-09T00:03Z] INFO  19:03:33,653 ProgressMeter - Total runtime 88.62 secs, 1.48 min, 0.02 hours

[2017-11-09T00:03Z] INFO  19:03:33,653 MicroScheduler - 1077117 reads were filtered out during the traversal out of approximately 3947714 total reads (27.28%)

[2017-11-09T00:03Z] INFO  19:03:33,653 MicroScheduler -   -> 0 reads (0.00% of total) failing BadCigarFilter

[2017-11-09T00:03Z] INFO  19:03:33,654 MicroScheduler -   -> 1077117 reads (27.28% of total) failing DuplicateReadFilter

[2017-11-09T00:03Z] INFO  19:03:33,654 MicroScheduler -   -> 0 reads (0.00% of total) failing FailsVendorQualityCheckFilter

[2017-11-09T00:03Z] INFO  19:03:33,654 MicroScheduler -   -> 0 reads (0.00% of total) failing MalformedReadFilter

[2017-11-09T00:03Z] INFO  19:03:33,654 MicroScheduler -   -> 0 reads (0.00% of total) failing NotPrimaryAlignmentFilter

[2017-11-09T00:03Z] INFO  19:03:33,654 MicroScheduler -   -> 0 reads (0.00% of total) failing UnmappedReadFilter

[2017-11-09T00:03Z] ------------------------------------------------------------------------------------------

[2017-11-09T00:03Z] Done. There were 2 WARN messages, the first 2 are repeated below.

[2017-11-09T00:03Z] WARN  19:02:04,019 IndexDictionaryUtils - Track variant doesn't have a sequence dictionary built in, skipping dictionary validation

[2017-11-09T00:03Z] WARN  19:02:05,599 HaplotypeScore - Annotation will not be calculated, must be called from UnifiedGenotyper, not org.broadinstitute.gatk.tools.walkers.annotator.VariantAnnotator

[2017-11-09T00:03Z] ------------------------------------------------------------------------------------------

Thanks again for your help!

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Thanks for following up with all these details. Unfortunately I don't see
anything in those logs that help me provide a definitely clue about where it's
hanging. It appears to be running FreeBayes. If you run `top` on the machine
where it's running, do you see one or more freebayes processes running? If so,
things are working correctly and freebayes is just taking a long time on your
samples. This can happen with very repetitive regions or high depth samples.
Does that help explain what is happening?

Brad


> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/c4fd8195-9575-4d25-a3a1-ddf4c94a8b09%40googlegroups.com.

[Xiaopeng Bian]

Hi, Brad:

Thanks for your reply.

I ran top but did not see any freebayes process,

[bianxi@biowulf ~]$ top | grep bianxi

28006 bianxi    20   0 18412 4496  832 R 22.5  0.0   0:00.42 top                                   

28006 bianxi    20   0 18416 4616  948 R  5.3  0.0   0:00.59 top                                   

28006 bianxi    20   0 18416 4616  948 R  6.0  0.0   0:00.78 top  

and here's log for yet another long hang process:

[2017-11-08T19:52Z] Timing: hla typing

[2017-11-08T19:52Z] Resource requests: freebayes, gatk, mutect, picard, vardict, varscan; memory: 2.00, 3.50, 3.50, 3.00, 2.00; cores: 16, 1, 1, 1, 1, 16

[2017-11-08T19:52Z] Configuring 32 jobs to run, using 1 cores each with 3.50g of memory reserved for each job

[2017-11-08T19:52Z] Timing: alignment post-processing

[2017-11-08T19:52Z] multiprocessing: piped_bamprep

[2017-11-08T19:52Z] Timing: variant calling

[2017-11-08T19:52Z] multiprocessing: variantcall_sample

[2017-11-08T19:52Z] Genotyping paired variants with FreeBayes

Thanks.

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Xiaopeng Bian]

And another long hang:

[2017-11-11T19:03Z] Genotyping with VarDict: Inference

[2017-11-11T19:03Z] Genotyping with VarDict: Inference

[2017-11-11T19:03Z] Genotyping with VarDict: Inference

[2017-11-11T19:03Z] Genotyping with VarDict: Inference

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Xiaopeng Bian]

A question,

when a process breaks and I'd like to re-run, I usually leave everything already generated as is and just re-run the process in the same folder. Could that be a problem and end up in failure?

Thanks.

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Unfortunately these log messages don't provide much additional information to
go on. It's definitely running callers but I'm not sure what is happening to
the processes. Is it possible they're being killed by the operating system due
to memory constraints on the running processes? Is this running multicore on a
single machine or distributed on a cluster?

bcbio does restart where you left off so running an analysis in place is
exactly the way to do it.

My best suggestion would be to look at `log/bcbio-nextgen-commands.log` to see
the calls that get started at the end of a run. If these are consistent
between hangs then you've identified problem regions and could try running the
command line standalone and monitoring with top to see the resource usage.

Hope this helps,
Brad

> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/15d9b528-80d3-4501-8bcc-cd3fa8f0bc32%40googlegroups.com.

[Xiaopeng Bian]

Hi, Brad:

I am running on NIH Bioowulf system, I think it is a distributed system.

I contacted our admin staff, they checked it for me and here is their reply:

In this case I've gone ahead and looked at this job. It still has a freebayes process (single thread) running as well as the bcbio_nextgen processes.  However, none of the files these processes have opened have been touched since Nov 11 and none of the processes are reading or writing any data. I suspect your job is stuck in a bad state and will not produce output. My recommendation would be to cancel it.

This is the last line from the log file you mentioned:

[2017-11-11T19:03Z] unset R_HOME && unset JAVA_HOME && export PATH=/fdb/bcbio-nextgen/current/anaconda/bin:$PATH && export VAR_DICT_OPTS='-Xms750m -Xmx3000m -XX:+UseSerialGC -Djava.io.tmpdir=/gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bcbiotx/tmpPJo2Tt' && vardict-java -G /fdb/bcbio-nextgen/current/genomes/Hsapiens/g1k_v37_no_decoy2/seq/g1k_v37_no_decoy2.fa -f 0.1 -N aml-tumor_40 -b "/gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bamclean/aml-tumor_40/AML31_primary-tumor_wgs_na_hiseq_bwa-0.5.9.41x-reorder-fixrgs-gatkfilter-dedup.bam|/gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bamclean/aml-normal_40/AML31_normal_wgs_na_hiseq_bwa-0.5.9.40x-reorder-fixrgs-gatkfilter-dedup.bam" -c 1 -S 2 -E 3 -g 4 -Q 10 /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/vardict/Y/aml_40_1109-Y_58821543_59373566-regions-glimit-regionlimit.bed | testsomatic.R | var2vcf_paired.pl -P 0.9 -m 4.25 -f 0.1  -M  -N "aml-tumor_40|aml-normal_40" | /fdb/bcbio-nextgen/current/anaconda/bin/py -x 'bcbio.variation.vcfutils.add_contig_to_header(x, "/fdb/bcbio-nextgen/current/genomes/Hsapiens/g1k_v37_no_decoy2/seq/g1k_v37_no_decoy2.fa")' | bcftools filter -m '+' -s 'REJECT' -e 'STATUS !~ ".*Somatic"' 2> /dev/null | /fdb/bcbio-nextgen/current/anaconda/bin/py -x 'bcbio.variation.vardict.depth_freq_filter(x, 0, "False")' | bcftools filter -i 'QUAL >= 0' | sed 's/\\.*Somatic\\/Somatic/' | sed 's/REJECT,Description=".*">/REJECT,Description="Not Somatic via VarDict">/' | /fdb/bcbio-nextgen/current/anaconda/bin/python -c 'from bcbio.variation import freebayes; freebayes.call_somatic("aml-tumor_40", "aml-normal_40")'  | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDX]/, "N", $4) } {print}' | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDX]/, "N", $5) } {print}' |  awk -F$'\t' -v OFS='\t' '$1!~/^#/ && $4 == $5 {next} {print}' | /fdb/bcbio-nextgen/current/anaconda/bin/vcfstreamsort | bgzip -c > /gpfs/gsfs3/users/nextgen2/Xiaopeng/DCEG/bcbio/aml/bcbiotx/tmpPJo2Tt/aml_40_1109-Y_58821543_59373566-raw.vcf.gz

I am going to try more as you suggested.

Thanks.

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Thanks for the additional details and digging into the problems. NIH Biowulf
is an HPC cluster and it sounds like you might be submitting jobs but I don't
know enough about it to give useful advice. If your jobs are running on
external nodes after submission that would explain why you don't see them in
top on the submission machine.

From the reply below, I agree with re-submitting the jobs and then working
with the HPC folks there to be able to monitor them. This might give more
insight into if variant calling is taking a long time on your samples or if
something problematic is happening with the jobs.

Sorry to not have a specific answer but hope this general advice helps,
Brad


> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/c4937182-e807-4b0f-a41d-48153723114a%40googlegroups.com.

[Xiaopeng Bian]

Hi, Brad:

Here is long hang from a re-submission, seems stuck in Freebayes, should I take it out from yaml?

[2017-11-17T23:01Z] Timing: hla typing

[2017-11-17T23:01Z] Resource requests: freebayes, gatk, mutect, picard, vardict, varscan; memory: 2.00, 3.50, 3.50, 3.00, 2.00; cores: 16, 1, 1, 1, 1, 16

[2017-11-17T23:01Z] Configuring 32 jobs to run, using 1 cores each with 3.50g of memory reserved for each job

[2017-11-17T23:01Z] Timing: alignment post-processing

[2017-11-17T23:01Z] multiprocessing: piped_bamprep

[2017-11-17T23:01Z] Timing: variant calling

[2017-11-17T23:01Z] multiprocessing: variantcall_sample

[2017-11-17T23:01Z] Genotyping paired variants with FreeBayes

Thanks

Xiaopeng

On Monday, October 30, 2017 at 10:03:52 AM UTC-4, Xiaopeng Bian wrote:

[Brad Chapman]

Xiaopeng;
Thanks, I agree that removing FreeBayes from this sample makes sense as it
seems to be taking an unreasonably long time or have resource issues with your
samples. Hope this gets your analysis finished,
Brad

> [ text/plain ]

> --
> You received this message because you are subscribed to the Google Groups "biovalidation" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to biovalidatio...@googlegroups.com.
> To post to this group, send email to bioval...@googlegroups.com.
> Visit this group at https://groups.google.com/group/biovalidation.

> To view this discussion on the web visit https://groups.google.com/d/msgid/biovalidation/f206ece0-7ef6-435a-8be9-61a18b034406%40googlegroups.com.