abyss-sealer scales not very well on many CPUs?

[Sven]

Hi,

I want to close gaps in a 60,000 gaps genome assembly using ~400mio PE reads for testing. I omitted LMP reads.

I chose a bloom filter size of 60GB; we have plenty of high memory machines.

I observe a very long run time with B=6000,P=100,j=48, maybe due to chosen paramater or probably due to the fact that only one CPU is used.
Which parts of sealer run multi-threaded? What is the botleneck?

I started another job with B=3000,P=10,j=78 to reduce runtime. But here CPU usage is constantly at about 150%.

What do I miss? Any idea?

I run sealer with ~14 kmers ranging from 250 down to 60 for testing purposes.

As with abyss I need to play around with parameters and try to optimise the results by trial&error ..

best,
Sven

[Sven]

I changed bloom filter size to only 1G ... and it still runs with 125% CPU load :-(

Using e.g. --max-paths=nolimit and --max-branches=nolimit CPU load increased temporarily to ~450% (just seconds, "Finding flanks") and then falls back to ~125%.

Here I used j=24.

There is definitely something I missed related to parametrization of sealer ..

It takes >1 week for sealer to work on my dataset using just one CPU.

Ideas are welcome :-)

best,
Sven

[Ben Vandervalk]

I observe a very long run time with B=6000,P=100,j=48, maybe due to chosen paramater or probably due to the fact that only one CPU is used.

Do I understand correctly that you are running Sealer with 48 threads on a single CPU?  If so, that is probably the issue. The general idea with multithreading is to allocate one CPU core per thread  -- otherwise you will be "oversubscribing" the CPU core(s) and actually making the performance worse than if you used a single thread.

Which parts of sealer run multi-threaded? What is the botleneck?

All significant computations in the Sealer code are multithreaded: Bloom filter loading, graph searching, gap filling.   We have done performance tests which showed that the performance scales well with the number of threads.

I run sealer with ~14 kmers ranging from 250 down to 60 for testing purposes.

As with abyss I need to play around with parameters and try to optimise the results by trial&error ..

If you are still having issues, please see this thread about the effect of various Sealer parameters on performance: https://github.com/bcgsc/abyss/issues/79

- Ben

[Sven]

Hi Ben,

I am afraid that I didn't get the threads issue:

I have multicore highmem servers, e.g. 4x Intel E7-4870 (10 cores per cpu, 20 threads) with loads of RAM. I want to run sealer using e.g. 40 threads. I thought sealer would use 40 cpu cores to distribute 40 threads?
How do I run sealer to use 40 cores / multi-threaded then? Or is sealer always using one cpu and according to the type of cpu I have to choose the correct number of threads (in my example 10 / 20)?

Thanks for pointing me to the parameter thread.

best,
Sven

[Ben Vandervalk]

Sorry, Sven.  I must have misunderstood your description.

Yes, Sealer should automatically distribute its threads across the available 40 CPU cores.  That's the way all multithreaded programs behave, and Sealer is no different.

Are you using a job scheduler to submit jobs to your cluster? (e.g. SGE, LSF, PBS, SLURM)

The only reason I can imagine for Sealer only using only one CPU is that your are using a job scheduler and only one CPU was allocated in the job submission script.

Have you previously run other multithreaded programs (e.g. bwa) on the same machine successfully?

- Ben

--
You received this message because you are subscribed to the Google Groups "ABySS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to abyss-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sven]

I noticed it while submitting some sealer jobs to our cluster. Jobs on our cluster are not distributed across machines; one job, one server/node.

Then I ran all these tests locally on a 64 core machine with 256GB RAM. Yes, other programs run fine on multiple CPUs. On different machines it behaves the same (e.g. Intel 80 core machine).

I have used the latest git for compilation,

commit 940e20b5217f97c55c2673f55292803880044ebb
Author: Ben Vandervalk <xxx@xxx>
Date:   Tue Oct 13 15:15:13 2015 -0700

maybe I should simply recompile 1.9.0 release ..

Ideas welcome :-)

best,

Sven

[Sven]

ok, I recompiled abyss 1.9.0 release and ran sealer again on my dataset with the same result, my -j=48/bloom-size=48G job run with 100/160% CPU load .... :-(
Memory consumption corresponds ~ bloom-size.

Server is a 80core/1TB Intel machine ... other sw runs fine multithreaded.

This way it takes ages for sealer to finish the job ..

best,
Sven

[Sven]

It seems with large values of k (>100) where there are no gaps closed, cpu load is around 100/150%.
With decreasing k values, when sealer starts to close gaps, cpu load raises, currently k=100 (36 gaps closed) @ 550-750%.
Not yet >4000% though ... but raising ...

Sven

[Sven]

Just to illustrate this issue ... sealer runs since 5 days and will get killed tomorrow :-)
I am attaching a "load" screenshot of the machine, indicating the start of sealer 5 days ago and the subsequent load since then with --threads=48.

I have no idea what causes this "unusual" behavior.

best,
Sven

[Ben Vandervalk]

Sorry Sven, unfortunately I don't understand it either.

I will let you know if I make any significant improvements to Sealer, but in the meantime I think you will have to try some other gap filling software.

- Ben

[Lijing Bu]

Could it because the compilation didn't include the sparsehash program?