idba_ud job dies with "killed" message

[Bill Nelson]

I have an assembly job that's not that big, but it dies after the kmer 80 step as shown below. Any idea why it's doing this? There's no way it should run out of memory, because this is running on a machine with 256G of RAM.

thanks,
Bill

$ idba_ud -o Oscar_02122013 --min_contig 250 -r ITZY.trimmed.pe.fasta.keep --num_threads 20number of threads 20
reads 19001636
long reads 0
extra reads 0
read_length 133
kmer 20
kmers 97431570 97970577
merge bubble 123784
contigs: 363585 n50: 498 max: 16776 mean: 177 total length: 64688017 n80: 163
aligned 10135819 reads
confirmed bases: 37160465 correct reads: 7462356 bases: 2895251
distance mean 215.592 sd 10.1272
seed contigs 70344 local contigs 727170
kmer 40
kmers 76288774 76262481
merge bubble 24922
contigs: 54543 n50: 7812 max: 315353 mean: 1226 total length: 66870677 n80: 1223
aligned 16051171 reads
confirmed bases: 57624929 correct reads: 13662265 bases: 2508074
distance mean 215.448 sd 9.99197
seed contigs 28032 local contigs 109086
kmer 60
kmers 67773727 67746461
merge bubble 2227
contigs: 27402 n50: 21567 max: 553743 mean: 2434 total length: 66708025 n80: 2693
aligned 16448562 reads
confirmed bases: 59468739 correct reads: 14055182 bases: 163881
distance mean 215.448 sd 9.98938
seed contigs 19713 local contigs 54804
kmer 80
kmers 65865460 65857545
merge bubble 580
Killed

[Yu PENG]

Can you try deleting "contig_graph.MergeSimilarPath();" in src/release/idba_ud.cpp and try again? Please remember to rebuild the program by "make clean && make". 

It seems there is a bug in MergeSimilarPath causing failure. 

Thanks,

Yu Peng

--
You received this message because you are subscribed to the Google Groups "hku-idba" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hku-idba+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bill Nelson]

Yes, that's it.

Thanks Yu.

Bill

[5heikki]

Hello,

I'm running into this same problem with one of my samples. I know for a fact that a subset of these reads can be assembled both into a circular phage genome and into a prophage with host dna in both ends. Could this be the underlying cause? How does removing "contig_graph.MergeSimilarPath();" affect idba_ud's behavior?

[Yu PENG]

There was a bug inside MergeSimilarPath(). I have fixed in my github repo. You can check it out here. 

Thanks,

Yu Peng

For more options, visit https://groups.google.com/d/optout.

[Theo Allnutt]

Hi,

I had same problem and tried the fix but idba_ud then used all memory (80GB) and 'killed' again. Is this fixed in current 1.1.1? I was using 1.1.1 already.

Thanks,

Theo

[Yu PENG]

What is the size of input data? It may run out of memory on big input. 

Thanks,

Yu Peng

[Theo Allnutt]

Input is about 1GB, so not super big. I will try just reducing reads slightly.

[Theo Allnutt]

N.B. I posted this as an issue on the GitHub page.

[Theo Allnutt]

Using 1.1.2 now and same problem.

[Theo Allnutt]

Actually, I had not deleted 1.1.1 so ignore that last one.. going to try 1.1.2 now, but I notice that the maximum read length has gone down to 124 again, where it was 700 in idba 1.1.1. So now have to use -l option. Does this make any difference? Should the maximum read length be increased because reads >300 bp are very common now?

Thanks,

Theo

[Theo Allnutt]

Unfortunately now getting a different error with idba 1.1.2:

~/d/031$ ~/bin/idba-1.1.2/bin/idba_ud -l ~/d/031/reads/int_fasta/2882_D1.fasta -o ~/d/031/idba/2882_D1 --num_threads 24  --mink 114 --maxk 124 --step 10
number of threads 24
reads 0
long reads 1675036
extra reads 0
read_length 0
kmer 114
kmers 5039992 5030496
merge bubble 844
contigs: 200 n50: 64214 max: 184664 mean: 14755 total length: 2951126 n80: 26521
aligned 0 reads
confirmed bases: 0 correct reads: 0 bases: 0
distance mean -nan sd -nan
invalid insert distance
kmer 124
kmers 2968570 2968775
merge bubble 41
contigs: 175 n50: 64234 max: 184674 mean: 16851 total length: 2949006 n80: 26531
terminate called after throwing an instance of 'std::logic_error'
  what():  SequenceReader::SequenceReader() istream is invalid
Aborted

[Theo Allnutt]

Ok, its working now.

I changed /idba-1.1.2/src/sequence/short_sequence.h to longer kMaxShortSequence to:

static const uint32_t kMaxShortSequence = 500;

and changed the max kmer size in idba-1.1.2/src/basic/kmer.h byt changing the number of bits to:

static const uint32_t kNumUint64 = 16;

Then recompiled:

./configure
make

Now IDBA is working with my 300bp paired end data with kmers of 100, 200 and 300. Assembly looks much better so far than it was with kmers limited to 124 bp. I don't really understand why assemblers limit kmer size, but I'm not a mathematician. The highest kmer size always seems to give the 'best' assembly.

Theo

[Jianshu Zhao]

Dear Yu Peng, I am running a very big data set, 30gb for merged fasta file on a linux platform. I set 400 cores and 1000 gb memory. It get killed very time when the generated Kmer file reaches 138 gb (hard drive is 7 tb). I tried a lot to times. no contig.fa file and the program stopped with a 138 gb kmer file there... Can you please help. no error output.

To unsubscribe from this group and stop receiving emails from it, send an email to hku-...@googlegroups.com.