Errore in get_matter_list_sublist(x, i, j) : failed to read data elements

[Marco Fidaleo]

Hello,

I have multiple files for analysis. These files are mostly similar and originate from the same apparatus. While I can read and convert many of them successfully, there are some that produce the following error: "Error in get_matter_list_sublist(x, i, j) : failed to read data elements." Do you believe this issue is solvable, or is there an error in the original data, making the dataset unusable in Cardinal?

> mse_DRG_pos_20um_2ss <- readMSIData(DRG_pos_20um_2ss)

parsing imzML file: ‘C:\Users\ACER\Desktop\DRG_pos_20um_2ss.imzml’

detected 'processed' imzML

detected 'profile' spectra

returning MSImagingArrays

done.

 

> mse_DRG_pos_20um_2ss

MSImagingArrays with 130460 spectra

spectraData(2): mz, intensity

pixelData(3): x, y, run

coord(2): x = 1...593, y = 1...220

runNames(1): DRG_pos_20um_2ss

experimentData(10): spectrumType, spectrumRepresentation, instrumentModel, ..., scanType, lineScanDirection, pixelSize

centroided: FALSE

continuous: FALSE

 

 

> conv_DRG_pos_20um_2ss <- convertMSImagingArrays2Experiment(mse_DRG_pos_20um_2ss)

Errore in get_matter_list_sublist(x, i, j) : failed to read data elements

[kbemis]

Will need to see some more details.

If you paste the output of running traceback() immediately after the error occurs, that will help me know what was happening when the error occurred.

You can also try accessing data arrays directly

mz(my_data)[[i]]

intensity(my_data)[[i]]

for different values of i and see if you are able to access some data arrays and not others.

It's possible it's a bug, but it's also possible the ".ibd" file was incompletely written.

-Kylie

[Marco Fidaleo]

I executed the commands you recommended (please see the results below). How can I determine if the .ibd file has been incompletely written?

> traceback() 4: get_matter_list_sublist(x, i, j) 3: mzlist[i] 2: mzlist[i] 1: convertMSImagingArrays2Experiment(mse_DRG_pos_20um_2ss)

> mz(mse_DRG_pos_20um_2ss)[[200]] [1] 99.00683 99.01295 99.02522 99.02726 99.02930 99.03340 99.03544 99.03748 99.04362 99.04566 [11] 99.04771 99.04975 99.05793 99.05997 99.06611 99.06815 99.07019 99.07224 99.07429 99.07633 [21] 99.07837 99.08041 99.08247 99.08451 99.08656 99.08859 99.09064 99.09882 99.10087 99.10291 [31] 99.10496 99.10905 99.11109 99.11518 99.11723 99.12540 99.12746 99.14381 99.14586 99.14791 [41] 99.15199 99.15405 99.15609 99.15813 99.16018 99.16222 99.16427 99.16632 99.16836 99.17040 [51] 99.17245 99.17450 99.17859 99.18063 99.18269 99.18473 99.19086 99.19700 99.19905 99.20110 [61] 99.20314 99.20928 99.21133 99.21337 99.21746 99.21951 99.22155 99.22360 99.22565 99.22770 [71] 99.22974 99.23178 99.23588 99.23793 99.24612 99.24815 99.25430 99.25634 99.28091 99.28295 [81] 99.28500 99.28909 99.29114 99.29318 99.29728 99.30137 99.33413 99.33617 99.34641 99.34846 [91] 99.35050 99.35255 99.35460 99.36074 99.36279 99.36484 99.36893 99.37099 99.37917 99.38123 [101] 99.38326 99.38737 99.39351 99.39555 99.40580 99.40784 99.40989 99.41194 99.45290 99.45495 [111] 99.45700 99.45905 99.46111 99.47134 99.47339 99.47544 99.48569 99.48774 99.48978 99.49183 [121] 99.50413 99.50618 99.51437 99.51642 99.52257 99.52461 99.52872 99.53077 99.53691 99.53896 [131] 99.54716 99.54921 99.55126 99.55536 99.55741 99.57175 99.57381 99.57586 99.58815 99.59020 [141] 99.59840 99.60046 99.60251 99.61070 99.61275 99.61890 99.62096 99.62301 99.62505 99.63941 [151] 99.64761 99.64967 99.65376 99.65582 99.65787 99.65991 99.66196 99.66402 99.67017 99.67837 [161] 99.68042 99.68247 99.68657 99.68862 99.71119 99.71324 99.71529 99.71734 99.71939 99.72145 [171] 99.73170 99.73375 99.74401 99.75221 99.75426 99.75632 99.75838 99.77068 99.77274 99.78094 [181] 99.78300 99.78915 99.79120 99.79326 99.79530 99.79736 99.79941 99.80352 99.80556 99.80762 [191] 99.81377 99.81583 99.81788 99.82609 99.82814 99.83635 99.83840 99.84456 99.84661 99.84867 [201] 99.86098 99.86304 99.87330 99.87535 99.87946 99.88151 99.89178 99.89383 99.91026 99.91230 [211] 99.91436 99.91641 99.91846 99.92052 99.92257 99.92873 99.93078 99.93284 99.93695 99.93900 [221] 99.95132 99.95338 99.95543 99.95748 99.96159 99.96365 99.96980 99.97186 99.97392 99.97802 [231] 99.98007 99.98624 99.98829 99.99034 100.00472 100.00677 100.00883 100.01089 100.01499 100.01705 [241] 100.01910 100.02116 100.02527 100.02732 100.02938 100.03143 100.04581 100.04787 100.06225 100.08485 [251] 100.08691 100.11979 100.12184 100.13418 100.13624 100.13829 100.14034 100.14240 100.14445 100.14856 [261] 100.15062 100.16296 100.16502 100.16707 100.16912 100.17324 100.17529 100.17735 100.17941 100.19380 [271] 100.19585 100.20407 100.20613 100.21230 100.22053 100.22258 100.23698 100.23904 100.24727 100.24931 [281] 100.25137 100.26372 100.26578 100.26783 100.26988 100.27194 100.27400 100.28429 100.28634 100.29046 [291] 100.29252 100.29868 100.30898 100.31104 100.33984 100.34189 100.34601 100.34807 100.36040 100.36246 [301] 100.36452 100.37482 100.37687 100.37893 100.38099 100.38511 100.38716 100.39128 100.39745 100.39951 [311] 100.40157 100.40363 100.40775 100.40981 100.41804 100.42010 100.42216 100.42421 100.43244 100.43450 [321] 100.43656 100.43862 100.44067 100.44686 100.44892 100.45097 100.45303 100.46127 100.46333 100.46745 [331] 100.47774 100.47980 100.48186 100.49010 100.49216 100.49422 100.49627 100.49833 100.50039 100.50245 [341] 100.50451 100.51069 100.51275 100.51893 100.52099 100.52305 100.52716 100.52922 100.53952 100.54158 [351] 100.54364 100.54570 100.55394 100.56011 100.56217 100.56424 100.56630 100.56836 100.57454 100.57660 [361] 100.58278 100.58484 100.59514 100.60545 100.60750 100.60956 100.61162 100.62193 100.62399 100.62604 [371] 100.62811 100.63223 100.63428 100.64458 100.64665 100.66313 100.66520 100.67757 100.67963 100.68169 [381] 100.68581 100.70023 100.70229 100.70436 100.72292 100.72498 100.74353 100.74559 100.75383 100.75590 [391] 100.76620 100.76827 100.77033 100.77858 100.78065 100.78889 100.79095 100.79920 100.80126 100.80332 [401] 100.80539 100.80951 100.81158 100.81364 100.81570 100.82188 100.82395 100.82602 100.82807 100.83014 [411] 100.83220 100.83427 100.85077 100.85283 100.85490 100.85695 100.85902 100.86108 100.86314 100.88171 [421] 100.88377 100.88996 100.89203 100.89409 100.89615 100.92917 100.93123 100.93330 100.94361 100.94568 [431] 100.94775 100.94980 100.95187 100.95393 100.95600 100.96838 100.97044 100.97251 100.97457 100.97663 [441] 100.97870 100.98077 100.99109 100.99315 100.99521 100.99728 100.99934 101.00141 101.00347 101.00553 [451] 101.00760 101.00967 101.01173 101.01379 101.01585 101.01792 101.01999 101.02824 101.03031 101.03237 [461] 101.03857 101.04063 101.04476 101.04683 101.04890 101.05715 101.05922 101.06128 101.06335 101.07367 [471] 101.07574 101.07780 101.07986 101.08607 101.08813 101.09019 101.09225 101.09432 101.09639 101.09845 [481] 101.10052 101.10465 101.10671 101.11085 101.11498 101.11704 101.11911 101.12118 101.12325 101.12531 [491] 101.12737 101.12943 101.13150 101.13357 101.13563 101.13770 101.13976 101.14183 101.14389 101.14596 [501] 101.15216 101.15422 101.15629 101.16042 101.16248 101.16455 101.16662 101.16869 101.17075 101.19348 [511] 101.19555 101.19762 101.20795 101.21001 101.21622 101.21828 101.22241 101.22448 101.22861 101.23067 [521] 101.23274 101.23482 101.24721 101.24928 101.25134 101.25342 101.25961 101.26168 101.26375 101.26581 [531] 101.26788 101.26995 101.27408 101.27615 101.27821 101.29889 101.30716 101.30923 101.31129 101.31543 [541] 101.31750 101.31956 101.32163 101.32370 101.32577 101.32784 101.33404 101.33611 101.33817 101.34024 [551] 101.34231 101.34438 101.35058 101.35265 101.35472 101.35885 101.36092 101.37333 101.37540 101.37746 [561] 101.37953 101.38160 101.39194 101.39815 101.40228 101.41056 101.41263 101.41469 101.41883 101.42091 [571] 101.42297 101.42711 101.42918 101.43124 101.43331 101.43952 101.44159 101.44779 101.44987 101.45193 [581] 101.46021 101.46227 101.48504 101.48711 101.48918 101.49746 101.51608 101.51814 101.52022 101.52229 [591] 101.52436 101.53264 101.53471 101.53677 101.53884 101.54092 101.54506 101.55126 101.55334 101.55747 [601] 101.55955 101.57404 101.57610 101.58438 101.59267 101.59473 101.59681 101.59888 101.60095 101.60302 [611] 101.60509 101.61337 101.61544 101.61752 101.62373 101.62579 101.62994 101.63200 101.63615 101.63822 [621] 101.64236 101.64443 101.64651 101.64857 101.65064 101.65893 101.66100 101.67550 101.67757 101.68378 [631] 101.69415 101.69621 101.69828 101.70036 101.70864 101.71071 101.71899 101.72107 101.72314 101.73351 [641] 101.73557 101.73972 101.74179 101.74386 101.74593 101.76251 101.76458 101.76665 101.76872 101.77080 [651] 101.77701 101.77908 101.78116 101.78323 101.78530 101.81224 101.81432 101.81639 101.82260 101.82468 [661] 101.82675 101.82883 101.83089 101.83297 101.83504 101.83712 101.84541 101.84748 101.84955 101.85162 [671] 101.85370 101.85577 101.86407 101.86614 101.88686 101.88894 101.89101 101.89309 101.89723 101.89931 [681] 101.90138 101.90346 101.90553 101.91175 101.91383 101.91589 101.92626 101.92834 101.93456 101.93663 [691] 101.94285 101.94493 101.94908 101.95116 101.95322 101.95530 101.95737 101.96359 101.96567 101.96982 [701] 101.97189 101.97397 101.97604 101.97812 101.98019 101.98641 101.98849 101.99263 101.99471 102.00301 [711] 102.00508 102.00716 102.00923 102.01131 102.01338 102.01546 102.01753 102.02791 102.02998 102.03205 [721] 102.03413 102.03620 102.03828 102.05488 102.05695 102.05902 102.06110 102.06318 102.06525 102.06940 [731] 102.07148 102.07355 102.08186 102.08393 102.08600 102.09223 102.09431 102.10468 102.10676 102.10883 [741] 102.11091 102.12337 102.12543 102.12752 102.13374 102.13582 102.15242 102.15658 102.15865 102.16074 [751] 102.17734 102.17942 102.18149 102.18357 102.18565 102.19603 102.19810 102.20018 102.20226 102.20433 [761] 102.20849 102.21056 102.21264 102.21471 102.21680 102.21887 102.22095 102.22302 102.22510 102.22925 [771] 102.23133 102.23341 102.23549 102.23756 102.24379 102.24587 102.25002 102.25210 102.25418 102.26248 [781] 102.26456 102.27702 102.27910 102.28118 102.28326 102.28534 102.28741 102.28949 102.29365 102.29572 [791] 102.30403 102.31650 102.31857 102.32066 102.33520 102.33727 102.33936 102.34143 102.34975 102.35182 [801] 102.37260 102.37468 102.37884 102.38092 102.39338 102.39546 102.39962 102.42249 102.42456 102.42664 [811] 102.42873 102.43079 102.43704 102.44328 102.44535 102.44743 102.44951 102.47446 102.47654 102.48070 [821] 102.49525 102.49733 102.50565 102.50981 102.51189 102.51397 102.51605 102.52021 102.52229 102.52437 [831] 102.52644 102.52853 102.53061 102.53269 102.53477 102.54101 102.54309 102.54725 102.54932 102.55141 [841] 102.55349 102.55765 102.55973 102.56181 102.56389 102.57013 102.57221 102.57429 102.60549 102.60757 [851] 102.61590 102.61798 102.62630 102.62839 102.63255 102.63462 102.63670 102.67208 102.67416 102.68249 [861] 102.69707 102.70331 102.70539 102.71163 102.71371 102.71996 102.72204 102.73037 102.73245 102.74078 [871] 102.74286 102.75119 102.75327 102.75951 102.76160 102.76785 102.76993 102.77201 102.77409 102.77618 [881] 102.79283 102.79492 102.79700 102.79908 102.80325 102.80534 102.81158 102.83032 102.83241 102.83449 [891] 102.83657 102.83865 102.84074 102.84282 102.84490 102.85115 102.85324 102.86157 102.86366 102.86990 [901] 102.87199 102.88448 102.88657 102.89073 102.89282 102.90323 102.90532 102.91991 102.92199 102.93240 [911] 102.93449 102.93658 102.93866 102.94074 102.94283 102.94908 102.95116 102.95325 102.95533 102.95741 [921] 102.96367 102.96575 102.97200 102.97409 102.97617 102.97826 102.98035 102.98242 102.99911 103.00118 [931] 103.00327 103.00536 103.00744 103.00953 103.01369 103.01578 103.01787 103.01994 103.02203 103.02412 [941] 103.02621 103.02829 103.03037 103.03246 103.03455 103.03663 103.03872 103.04079 103.04288 103.04497 [951] 103.04706 103.04913 103.05748 103.05956 103.06998 103.07625 103.07832 103.08041 103.08459 103.08668 [961] 103.08875 103.09710 103.09918 103.10127 103.10336 103.10545 103.10752 103.10961 103.11169 103.12004 [971] 103.12212 103.12630 103.13464 103.13673 103.13881 103.14090 103.15550 103.15759 103.16176 103.16385 [981] 103.16593 103.16801 103.18262 103.19097 103.19306 103.19514 103.20557 103.20766 103.21184 103.21393 [991] 103.22018 103.22227 103.22436 103.22644 103.22853 103.23062 103.23688 103.23897 103.24105 103.24314 [ reached getOption("max.print") -- omitted 91202 entries ]

> intensity(mse_DRG_pos_20um_2ss)[[1]] [1] 12 6 11 71 3 22 12 12 2 56 24 68 78 106 14 21 22 60 39 14 60 94 96 61 54 [26] 12 26 19 25 50 12 27 27 12 6 10 10 23 63 2 18 9 94 43 6 12 24 43 30 10 [51] 18 24 12 6 19 7 12 6 7 14 6 24 6 4 29 4 16 15 32 6 12 12 6 4 16 [76] 15 10 47 9 20 17 52 4 24 21 20 21 29 2 28 4 218 73 6 12 6 28 30 16 6 [101] 12 12 69 13 16 28 32 28 23 17 12 25 74 12 6 20 42 26 36 10 24 39 19 4 28 [126] 6 41 38 17 48 52 59 58 10 12 6 13 28 17 26 14 54 13 44 5 8 96 44 12 12 [151] 12 43 17 14 7 18 35 6 12 16 6 45 13 30 282 9 16 4 18 9 30 16 59 14 21 [176] 56 56 47 32 24 5 25 12 77 4 44 3 6 12 22 24 22 57 12 50 3 99 25 20 31 [201] 20 7 35 22 12 24 19 6 24 6 4 28 35 38 49 8 12 19 37 57 7 16 12 6 9 [226] 38 24 20 25 39 16 12 20 6 22 31 6 52 45 16 6 26 23 26 7 19 3 3 41 6 [251] 12 26 7 14 7 39 11 16 4 22 5 19 16 6 10 10 16 6 14 27 53 31 27 14 16 [276] 8 43 38 16 30 21 26 13 29 29 4 12 6 2 24 37 28 8 29 23 16 16 4 81 4 [301] 47 206 6 31 23 34 5 20 21 500 22 24 6 24 4 28 6 121 29 21 3 17 12 12 12 [326] 13 41 14 23 23 8 39 15 20 6 12 40 30 11 12 6 110 13 16 4 5 20 12 31 4 [351] 20 10 16 14 34 8 14 20 29 5 10 10 91 39 118 4 35 16 6 12 75 7 29 4 4 [376] 16 7 14 203 21 19 5 6 24 6 18 9 38 16 20 32 7 35 27 12 6 16 4 36 25 [401] 10 15 40 112 12 19 44 6 19 23 48 42 56 22 45 12 12 12 22 16 40 120 20 35 93 [426] 42 12 12 12 24 16 6 25 55 6 12 6 30 34 13 31 18 18 12 6 102 29 79 33 6 [451] 88 22 39 6 24 91 43 12 33 3 38 12 6 7 28 10 16 205 39 4 33 34 16 29 23 [476] 33 7 23 55 25 20 26 10 17 14 23 14 7 5 20 123 8 16 4 46 46 10 40 112 1 [501] 19 27 17 11 21 19 14 26 35 21 37 5 16 4 7 50 10 6 12 23 16 4 16 4 24 [526] 4 18 23 52 60 7 14 7 14 20 84 72 2 49 35 38 11 112 41 7 14 81 42 7 131 [551] 27 16 33 16 24 50 17 27 32 19 14 7 4 35 26 30 9 18 10 15 19 21 103 12 21 [576] 12 6 12 6 9 18 12 14 20 16 4 7 14 6 57 12 30 12 12 12 21 33 32 29 17 [601] 32 3 22 46 10 19 8 22 34 55 9 19 25 6 12 50 19 5 20 90 23 16 45 107 40 [626] 39 21 3 19 3 56 44 19 29 41 55 35 13 31 20 86 16 12 15 20 20 34 5 81 33 [651] 49 6 46 14 23 42 55 44 110 27 16 19 3 40 3 65 65 20 15 18 73 15 39 15 21 [676] 3 7 30 12 6 21 29 4 47 23 20 9 18 6 12 40 45 12 24 9 19 28 35 31 4 [701] 16 14 11 9 31 65 12 9 20 19 5 20 6 12 31 6 19 6 24 27 17 12 21 16 38 [726] 10 25 12 19 21 26 16 122 36 23 8 45 8 32 44 71 61 19 33 12 6 12 32 31 19 [751] 11 29 12 42 53 19 72 230 4 12 16 16 28 28 6 34 29 25 232 48 6 12 26 11 13 [776] 21 24 43 23 15 2 48 15 13 12 6 20 54 16 28 14 25 12 24 6 12 15 10 19 3 [801] 15 20 28 12 6 19 22 170 32 19 12 26 52 23 35 4 23 4 33 8 12 23 10 28 58 [826] 7 132 44 56 29 5 15 34 12 21 65 187 162 199 54 38 30 44 16 4 16 134 33 19 17 [851] 31 22 115 29 27 39 28 7 33 16 4 26 15 33 4 16 21 15 10 41 155 6 46 12 6 [876] 19 12 6 16 24 12 6 28 68 19 12 25 16 4 12 20 7 24 12 12 20 27 11 12 41 [901] 47 14 20 11 29 32 14 29 6 21 10 15 80 5 9 25 14 19 5 20 12 12 64 44 12 [926] 6 21 16 8 10 10 92 35 17 9 27 21 8 27 11 12 6 16 30 56 33 12 27 16 4 [951] 7 26 12 23 14 32 85 4 7 21 4 16 106 100 44 16 40 23 22 19 16 6 20 10 16 [976] 21 49 4 16 15 32 19 21 6 12 16 4 6 12 12 33 55 40 112 20 7 14 20 24 25 [ reached getOption("max.print") -- omitted 92593 entries ]

[kbemis]

Can you try intensity(mse_DRG_pos_20um_2ss)[[length(mse_DRG_pos_20um_2ss)]]?

(You don't need to give me the results, just whether or not you get the same error.)

-Kylie

[kbemis]

Actually:

Can you try intensity(mse_DRG_pos_20um_2ss)[c(1L, length(mse_DRG_pos_20um_2ss))]?

[Marco Fidaleo]

Same error:

intensity(mse_DRG_pos_20um_2ss)[c(1L, length(mse_DRG_pos_20um_2ss))]

Errore in get_matter_list_sublist(x, i, j) : failed to read data elements

[yonghu...@gmail.com]

Dear Kylie,

I got the same error when I tested Cardinal V3.6 using the dataset described here: https://github.com/kuwisdelu/Cardinal/issues/31 

Best regards,

Yonghui Dong

[kbemis]

Okay. This is helpful.

Marco: This does suggest there may be either a problem with the imzML file offsets or the ".ibd" file may be incomplete.

You can try tail(matter::atomdata(intensity(mse))@offset) and making sure those are (large) positive numbers.

You can also try any(lengths(intensity(mse)) == 0) to see if you actually have the same issue as Yonghui, which is plausible...

Yonghui: For your data, I get "subscript out of bounds" so a slightly different error, although possibly the same issue.

In your case the issue appears to be due to empty (zero-length) spectra, which is causing a bug that I will fix and push to release as soon as I can.

-Kylie

[kbemis]

Yonghui, the error should be fixed by matter 2.6.1 which I've pushed to release but may take a day or two to show up on Bioconductor.

Marco, let me know if your error persists.

-Kylie

[Marco Fidaleo]

If I receive this output after running the commands you suggested, what steps can I take to continue my work?

> tail(matter::atomdata(intensity(mse_DRG_pos_20um_2ss))@offset)

[1] 72641808840 72642312576 72642815540 72643319912 72643819664 72644316172

> any(lengths(intensity(mse_DRG_pos_20um_2ss)) == 0)

[1] FALSE

[kbemis]

Hi Marco,

Is your ".ibd" file at least 72.64 GB?

-Kylie

[Marco Fidaleo]

I think you got it! The .ibd file is approximately 32 GB in size. Does this strongly indicate that the .ibd file is incomplete? It's possible that during the file conversion process, it wasn't properly finalized?

M

[kbemis]

Yes, the imzML offsets are telling Cardinal to load spectra that don't exist in the ibd file, so the writing of the ibd file was probably interrupted somehow.

I've seen this a few times, so I should probably add a check in Cardinal that warns if this is the ibd file is too small for the offsets indicated by the imzML file.

-Kylie