MSstats::dataProcess RAM overflow for SpectroNaut input

329 views
Skip to first unread message

Christian Schori

unread,
Mar 21, 2022, 10:11:23 AM3/21/22
to MSstats
Dear MSstats Team

We've already processed quite some datasets exported from SpectroNaut with MSstats (4.2.0 with R 4.1.2), but this recent file (see here) always crashed during subplot summarization due to insufficient memory (despite having 180GB RAM (!) available). Can you explain what's going on here?

Thank you very much already for looking into this issue!

Best,
Christian


> SN_import <- SpectronauttoMSstatsFormat(input = input,
+                            annotation = annot,
+                            intensity = 'PeakArea',
+                            filter_with_Qvalue = TRUE,
+                            qvalue_cutoff = 0.01,
+                            useUniquePeptide = TRUE,
+                            removeFewMeasurements = TRUE,
+                            removeProtein_with1Feature = FALSE,
+                            summaryforMultipleRows = max)

INFO  [2022-03-21 15:02:34] ** Raw data from Spectronaut imported successfully.
INFO  [2022-03-21 15:02:35] ** Raw data from Spectronaut cleaned successfully.
INFO  [2022-03-21 15:02:35] ** Using provided annotation.
INFO  [2022-03-21 15:02:35] ** Run labels were standardized to remove symbols such as '.' or '%'.
INFO  [2022-03-21 15:02:35] ** The following options are used:
  - Features will be defined by the columns: PeptideSequence, PrecursorCharge, FragmentIon, ProductCharge
  - Shared peptides will be removed.
  - Proteins with single feature will not be removed.
  - Features with less than 3 measurements across runs will be removed.
INFO  [2022-03-21 15:02:36] ** Intensities with values not smaller than 0.01 in PGQvalue are replaced with NA
INFO  [2022-03-21 15:02:36] ** Intensities with values not smaller than 0.01 in EGQvalue are replaced with 0
INFO  [2022-03-21 15:02:37] ** Features with all missing measurements across runs are removed.
INFO  [2022-03-21 15:02:38] ** Shared peptides are removed.
INFO  [2022-03-21 15:02:43] ** Multiple measurements in a feature and a run are summarized by summaryforMultipleRows: max
INFO  [2022-03-21 15:02:43] ** Features with one or two measurements across runs are removed.
INFO  [2022-03-21 15:02:46] ** Run annotation merged with quantification data.
INFO  [2022-03-21 15:02:49] ** Features with one or two measurements across runs are removed.
INFO  [2022-03-21 15:02:49] ** Fractionation handled.
INFO  [2022-03-21 15:02:54] ** Updated quantification data to make balanced design. Missing values are marked by NA
INFO  [2022-03-21 15:02:54] ** Finished preprocessing. The dataset is ready to be processed by the dataProcess function.

> SN_dataProcess <- dataProcess(SN_import, censoredInt = "0")
INFO  [2022-03-21 14:28:03] ** There are 215484 intensities which are zero or less than 1. These intensities are replaced with 1
INFO  [2022-03-21 14:28:06] ** Log2 intensities under cutoff = 4.9  were considered as censored missing values.
INFO  [2022-03-21 14:28:06] ** Log2 intensities = 0 were considered as censored missing values.
INFO  [2022-03-21 14:28:06] ** Use all features that the dataset originally has.
INFO  [2022-03-21 14:28:07]
 # proteins: 2040
 # peptides per protein: 1-5089
 # features per peptide: 1-6
INFO  [2022-03-21 14:28:07] Some proteins have only one feature:
 P19001,
 P35441,
 Q9CQI6,
 Q9D3I6,
 Q9WVC3 ...
INFO  [2022-03-21 14:28:07]
                    KO-ex KO-sed WT-ex WT-sed
             # runs     6      6     6      6
    # bioreplicates     6      6     6      6
 # tech. replicates     1      1     1      1
INFO  [2022-03-21 14:28:09]  == Start the summarization per subplot...
  |=                                        |   0%<simpleError: vector memory exhausted (limit reached?)>
INFO  [2022-03-21 14:35:00]  == Summarization is done.
Error in merge.data.table(input[, colnames(input) != "newABUNDANCE", with = FALSE],  :
  Elements listed in `by` must be valid column names in x and y
In addition: Warning message:
In merge.data.table(input[, colnames(input) != "newABUNDANCE", with = FALSE],  :
  You are trying to join data.tables where 'y' argument is 0 columns data.table.

Mateusz Staniak

unread,
Mar 22, 2022, 7:00:08 AM3/22/22
to MSstats
Hi,

alternatively, if intensities are normalized, you can try using the extended workflow of MSstats v4+ (MSstats* functions such as MSstatsSummarize) to process the data in chunks.
Moreover, we are working on an MSstats workflow for large Spectronaut files (but it will take some additional time before it's ready)


Kind regards,
Mateusz

Christian Schori

unread,
Mar 22, 2022, 12:10:11 PM3/22/22
to MSstats
Hi Mateusz,

Thank you very much for your quick reply! I've just tried to reprocess this file with the suggested settings for R (R_MAX_VSIZE=300Gb) on our Ubunutu-Desktop with 188Gb RAM and monitored the RAM (with htop). As before, dataProcess runs nicely for about 5 to 10minutes, with an average 6-8Gb of RAM usage, and then skyrockets using the complete 188Gb of RAM resulting in R to crash. Interestingly the progress bar for this step is stuck at 0% for the whole time. Would there be a way to monitor the vector memory usage during this process? - Although I'm not so sure if this is really the root cause of this issue... In general, I don't really understand, why we don't have this issue for other (even bigger) Spectronaut datasets...

Could this behavior even be sample-type dependent? (I know this sounds far-fetched and a bit desperate... :-) ) In this particular sample we're analyzing muscle samples and ~700'000 lines of the total 4'000'000 lines belong to Titin (A2ASS6). Could this possibly interfere with the subplot summarization? - But again, manual removal of all Titin-related entries did not fix the memory overflow issue...

Regarding the suggested MSstatsSummarize function: Is this equivalent to the dataProcess function? I will definitely look into this. Do you have an example workflow on how to implement this new function available?

Thank you once more for all your help!

Best regards,
Christian

Mateusz Staniak

unread,
Mar 22, 2022, 12:13:31 PM3/22/22
to MSstats
Hi,

please try MBimpute = FALSE option.
For the alternative interface, please see the help for MSstatsSummarizationOutput function.


Kind regards,
Mateusz

Christian Schori

unread,
Mar 23, 2022, 5:32:10 AM3/23/22
to MSstats
Hi Mateusz,

Thank you for this tip. MBimpute = FALSE did the trick. - But I still don't really understand, why this is happening? How can the imputation process chew up all the RAM? Based on the number of censoredInt, only about 8% of all intensities need imputation. Is there something wrong with our input file, or is there a bug in the imputation process?

In addition, I've also checked with the suggested MSstatsSummarize method:

raw = SN_to_MSstatsformat_import
method = "TMP"
cens = "NA"
impute = TRUE
MSstatsConvert::MSstatsLogsSettings(FALSE)
input = MSstatsPrepareForDataProcess(raw, 2, NULL)
input = MSstatsNormalize(input, "EQUALIZEMEDIANS")
input = MSstatsMergeFractions(input)
input = MSstatsHandleMissing(input, "TMP", TRUE, "NA", 0.999)
input = MSstatsSelectFeatures(input, "all")
processed = getProcessed(input)
input = MSstatsPrepareForSummarization(input, method, impute, cens, FALSE)
input_split = split(input, input$PROTEIN)
summarized = MSstatsSummarize(input_split, method, impute, cens, FALSE, TRUE)
output = MSstatsSummarizationOutput(input, summarized, processed,
method, impute, cens)

Again, the process produced a memory overflow at the "MSstatsSummarize" step, resulting R to crash. Using the same approach with "impute = FALSE" did run through...

Best regards,
Christian

Christian Schori

unread,
Jun 2, 2022, 10:44:33 AM6/2/22
to MSstats
Dear all,

Since we've recently run into a similar problem again, I'd like to share here our current workaround to overcome the issue with non-converging imputation, until the announced MSstats workflow for large SpectroNaut files will be released:

Setting the dataProcess parameter "featureSubset" to "topN" with "n_top_feature" = 1000 did the trick.
(In our current sample there was only a single protein (titin) in our (muscle) sample with > 1000 peptides per protein (4522), which likely also was the reason for the stalled imputation.)

Best,
Christian

vitek...@gmail.com

unread,
Jun 2, 2022, 8:41:26 PM6/2/22
to MSstats
Dear Christian,

Thanks so much for sharing. We are now also discussing other export options with Biognosis, but meanwhile this should work.

Many thanks again
Olga

Reply all
Reply to author
Forward
0 new messages