ts_import failing

[G Perez Goncalves]

Hi all,

I am trying to get started with processing some data collected using SerialEM in warp, and I was able to get what I think were sucessful mc and ctf, but then my pipeline keeps failing on import. It is able to parse the mdoc files, but it seems to remove every single tilt "based on selection status, max tilt angle and masked fraction criteria" even with the defaults. It does not outright fail, the job finishes saying that all of the tilts were removed but does not write into tomostar. I have run it with a single tilt series worth of tilts right now at a time to try to troubleshoot, but nothing has worked.

 Has anyone run into a problem that sounds like this/could help to troubleshoot? Here is my input command:

WarpTools ts_import   

[tab]--mdocs ./test_symLinked/test_mdoc_ts1

[tab]--frameseries test_frameseries_ts1

[tab]--tilt_exposure 2.93

[tab]--output test-tomostarls 

Here is the output of that command:

Running command ts_import with:
mdocs = ./test_symLinked/test_mdoc_ts1
pattern = *.mdoc
frameseries = test_frameseries_ts1
tilt_exposure = 2.93
dont_invert = False
override_axis = null
auto_zero = False
tilt_offset = null
max_tilt = 90
min_intensity = 0
max_mask = 1
min_ntilts = 1
output = test-tomostar

Found 35 MDOC files, searching for *.mdoc in /vf/users/goncalvesgm/CIHP-1_32C/20260306_20260302_GPG7_1/test_symLinked/test_mdoc_ts1
Looking for frame series...
35/35                                                                                                                                                  
Parsing MDOCs and creating .tomostar files...
0/35No images left for 20260227_GPG7_1_ts1_00064_-33.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00096_-45.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00032_-21.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00000_-9.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
4/35No images left for 20260227_GPG7_1_ts1_00068_18.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00100_30.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00004_-6.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00036_6.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
8/35No images left for 20260227_GPG7_1_ts1_00104_33.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00040_9.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00008_-3.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00072_21.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
12/35No images left for 20260227_GPG7_1_ts1_00108_-48.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00044_-24.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00012_-12.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00076_-36.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
16/35No images left for 20260227_GPG7_1_ts1_00080_-39.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00112_-51.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00048_-27.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00016_-15.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
20/35No images left for 20260227_GPG7_1_ts1_00052_12.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00116_36.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00020_-0.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00084_24.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
24/35No images left for 20260227_GPG7_1_ts1_00056_15.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00024_3.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00120_39.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00088_27.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
28/35No images left for 20260227_GPG7_1_ts1_00124_-54.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00060_-30.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00028_-18.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
No images left for 20260227_GPG7_1_ts1_00092_-42.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
32/35No images left for 20260227_GPG7_1_ts1_00128_-57.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
33/35No images left for 20260227_GPG7_1_ts1_00132_0.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
34/35No images left for 20260227_GPG7_1_ts1_00136_0.0.tif.mdoc after removing tilts based on selection status, max tilt angle and masked fraction criteria
35/35                                                                                                                                                  
Successfully parsed 35 MDOC files, 0 failed

[Alice Eruera]

Hi Gerardo,

I had the exact same issue you're describing while reconstructing a dataset just the other day. Long story short, after spending several hours troubleshooting, I tried changing the selection status for all the frames. After I set all the selection statuses to 'selected', my tilt-series imported normally and the mdoc files parsed just fine. I don't know why the frames were automatically deselected for some reason. 

Perhaps the same thing has happened to you. I suggest you try it and see if it helps. In your project directory, run:

WarpTools change_selection --settings warp_frameseries.settings --select path/to/your/frames/*tif

and then,

WarpTools ts_import  --mdocs ./test_symLinked/test_mdoc_ts1 --frameseries test_frameseries_ts1 --tilt_exposure 2.93 --output test-tomostarls 

 

Hope this helps!

Alice   

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/d0ee6559-4a2b-4d92-889f-125349cd931fn%40googlegroups.com.

--

Alice Eruera, PhD

Research Fellow, SB Grid Consortium

Harvard Medical School 

Department of Biological Chemistry and Molecular Pharmacology | 45 Shattuck Street, 2nd Floor, Boston, MA, 02115

sbgrid.org

[Alice Eruera]

(For reference, my ts_import command is as follows but I see that yours is different:)

WarpTools ts_import \
--mdocs 13014/data/ \
--pattern "*.mdoc" \
--frameseries warp_frameseries \
--tilt_exposure 2.03 \
--dont_invert \
--output tomostar \
--override_axis 85