Comet memory usage

[Frank Koopmans]

I would like to use Comet to search label-free DDA data from an AB SCIEX TrippleTOF 5600. After converting the .wiff files using msconvert and setting up the Comet parameter file, I started the search but the process ran out of RAM and then crashed while processing input spectra.

Computer specs: 4core CPU, 12GB RAM, windows8, comet.2013020.win64.exe

input wiff size: ~600mb

resulting mzML: ~5gb

Is there anything I can do to optimize/configure, or should I wait until more efficient data preprocessing is implemented before working with Comet?

Cheers,

Frank

[Jimmy Eng]

Oops, looks like I corresponded with Frank directly instead of via this group.  Anyways, for anyone else who may come across this problem, make use of either the "use_sparse_matrix" or the "spectrum_batch_size" parameters.  In Frank's case, setting "use_sparse_matrix" seemed to run 3 to 4 fold faster than making use of "spectrum_batch_size".  This was for a fully tryptic search of ~230,000 ms/ms spectra.  On my linux box with phospho semi-tryptic searches, "spectrum_batch_size" searches are faster than "use_sparse_matrix".  The take home message is that there are options to address memory constraints but you should try both methods out on your system and search environment to see which one is optimal for you.  I'll try and do a study on this when I have time.

- Jimmy

[Frank Koopmans]

Thanks for the great help so far !

So I have run an extensive set of tests on my computer, on both Windows 8.1 64bit and Ubuntu 13.10, using the latest Comet release (Release 2013.02 rev. 1). Some observations:

Linux: running a batch search takes only 17 minutes, which is great. However, the RAM usage increases with each subsequential batch (after 10 batches of 7500 spectra, RAM has doubled from 5gb to 10gb, also occurs with previous release).

Windows: when I do a batch search, Comet never uses more than 38% CPU while searching and is _very_ slow (extrapolating the search time of a single block, about 10-fold slower than linux). A search with sparse-matix settings still consumes a whopping 11.5GB and took 6 hours.

So something funny is going on here, maybe you can investigate the threading on windows and profile the code when doing a batch search?

It would be great if we can get the windows build to perform like the linux build, since this allows me setup Comet on our computer in the lab. Please let me know if there is anything I can help with.

Cheers,

Frank

[Jimmy Eng]

Thanks for the feedback Frank.  We'll definitely investigate the issues you're observing.