Problem running fs_etomo_fiducials: wrpDose column not detected

[Moritz Kirchner]

Hello,

I am currently processing a tomography dataset and have been having problems with running the fs_etomo_fiducials protocol.

After running fs_motion_and_ctf I have created tomostar files for the project. Then I wanted to align running fs_etomo_fiducials using the following command:

WarpTools ts_etomo_fiducials --settings warp_nanobdella_sha05g6.settings --angpix 10 --fiducial_size 10 --initial_axis 84.8 --device_list 1

However, the protocol fails to process the files and the error logs (attached) show that either the wrpDose and/or wrpTilt column seems to be missing from the tomostar files. When I have a look the the corresponding tomostar file (attached) I can clearly see that the wrpDose column is present and the file has the same columns in the same order as the example given in the Quick start Guide. I am not sure why WarpTools is not detecting the column it should (which I guess is wrpDose and not wrpTilt at this point because wrpTilt is not in the example). Has anyone else had problems with this and would know of a solutions? Many thanks in advance!

Best wishes,

Moritz

[Warp Bot]

Hi Moritz,

A couple of clarifications would help pin this down. The exact required columns are wrpAngleTilt and wrpDose (note: not wrpTilt), and the parser will reject a file where either is missing or where the column header lines aren't formatted exactly as expected.

1. How were the tomostar files created? The supported route is WarpTools ts_import (which generates them from the MDOCs). If you wrote them manually, that would be the first thing to revisit, because the parser is strict about the header section.

2. Can you paste the first 10 lines of one tomostar file verbatim into a reply? That includes the data_ line, the loop_ line, every _wrp* column line, and the first data row. Also useful: the output of

head -c 200 your_file.tomostar | od -c | head

so we can see whether there's a stray BOM, leading whitespace before any _wrp line, or unusual line endings. The column-header parser stops as soon as it hits a line that does not start with an underscore, so a single line with leading spaces or a stray character can cause it to think a column is missing even when you can see it in the file.

3. Could you paste the exact error message from the log (not paraphrased)? It should mention either wrpAngleTilt or wrpDose specifically.

4. Which version of WarpTools are you running (WarpTools --version)?

One more note: ts_etomo_fiducials needs the wrpAngleTilt column too, so if your file only has wrpAngleTilt missing or named differently (e.g. wrpTilt) that would explain the error you're seeing.

— Warp Bot
THIS IS AN AUTOMATED MESSAGE GENERATED BY AN LLM. IT MAY OR MAY NOT SOLVE YOUR PROBLEM. IF YOU'D LIKE TO SPEAK TO A HUMAN, SAY SO IN YOUR MESSAGE.

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/e475e635-3e65-48bb-ad6f-210a9aa81a5dn%40googlegroups.com.

[Moritz Kirchner]

Hello,

here are some clarifications to hopefully find the problem. 

1. The tomostar files were created by using ts_import like so:

WarpTools ts_import --mdocs mdocs --frameseries warp_frameseries --tilt_exposure 3.38 --min_intensity 0.3 --dont_invert --output tomostar 

2. The first lines of one tomostar file up to the first data row looks like this:

data_

loop_
_wrpMovieName #1
_wrpAngleTilt #2
_wrpAxisAngle #3
_wrpDose #4
_wrpAverageIntensity #5
_wrpMaskedFraction #6
  ../warp_frameseries/sha05g6_squares_1_A1_ts_002_-60.0_Mar19_22.48.52.tif  -60.01  84.800   135.20001  51.126  0.000

The output of  head -c 200 your_file.tomostar | od -c | head is the following:

0000000  \n   d   a   t   a   _  \n  \n   l   o   o   p   _  \n   _   w
0000020   r   p   M   o   v   i   e   N   a   m   e       #   1  \n   _
0000040   w   r   p   A   n   g   l   e   T   i   l   t       #   2  \n
0000060   _   w   r   p   A   x   i   s   A   n   g   l   e       #   3
0000100  \n   _   w   r   p   D   o   s   e       #   4  \n   _   w   r
0000120   p   A   v   e   r   a   g   e   I   n   t   e   n   s   i   t
0000140   y       #   5  \n   _   w   r   p   M   a   s   k   e   d   F
0000160   r   a   c   t   i   o   n       #   6  \n           .   .   /
0000200   w   a   r   p   _   f   r   a   m   e   s   e   r   i   e   s
0000220   /   s   h   a   0   5   g   6   _   s   q   u   a   r   e   s

3. The contents of worker0.out is the following:

Running on GPU #1 (23690 MB free), port 43327

Received "LoadGainRef", with 5 arguments, for GPU #0, 23690 MB free:
Loaded gain reference and defect map: 5760, 4092, 1, normal, real, 1 A/px, ID = -1, False, False, False,
Execution took 0.983 seconds

Received "TomoStack", with 2 arguments, for GPU #0, 23690 MB free:
System.Exception: STAR file has no wrpDose or wrpTilt column.
   at Warp.TiltSeries.InitializeFromTomoStar(Star table) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 293
   at Warp.TiltSeries..ctor(String path, String dataDirectoryName) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 286
   at WarpWorker.WarpWorker.EvaluateCommand(NamedSerializableObject Command) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpWorker/WarpWorker.cs:line 339
Received "TomoStack", with 2 arguments, for GPU #0, 23690 MB free:
System.Exception: STAR file has no wrpDose or wrpTilt column.
   at Warp.TiltSeries.InitializeFromTomoStar(Star table) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 293
   at Warp.TiltSeries..ctor(String path, String dataDirectoryName) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 286
   at WarpWorker.WarpWorker.EvaluateCommand(NamedSerializableObject Command) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpWorker/WarpWorker.cs:line 339
10 seconds without heartbeat, exiting
, for GPU #1, 23690 MB free:
System.Exception: STAR file has no wrpDose or wrpTilt column.
   at Warp.TiltSeries.InitializeFromTomoStar(Star table) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 293
   at Warp.TiltSeries..ctor(String path, String dataDirectoryName) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpLib/TiltSeries.cs:line 286
   at WarpWorker.WarpWorker.EvaluateCommand(NamedSerializableObject Command) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1737588682413/work/WarpWorker/WarpWorker.cs:line 339
10 seconds without heartbeat, exiting

As you can see it mentions that either wrpDose or wrpTilt is missing instead of referencing one specific column. 

4. The version of WarpTools is 2.0.0.

To your note: As you can see above the wrpAngleTilt column is actually present in the tomostar file. It seems to be looking for something called wrpTilt instead, judging from the error. 

Thank you very much for your help. 

Best wishes

Moritz 

[Warp Bot]

Hi Moritz,

Thanks, that's very useful.

The error message in your build is misleading. The text says "wrpDose or wrpTilt" but the parser actually checks for wrpDose AND wrpAngleTilt. The "wrpTilt" string was a typo in the error message and was corrected in a later release. So the parser is looking for the right thing; the message just printed the wrong column name.

One more thing worth doing: your build (warp_1737588682413, late January 2025) is over a year old. Many bug fixes have landed since then, including the corrected error message. If you installed via conda:

conda env remove -n warp
conda create -n warp -c conda-forge -c nvidia/label/cuda-12.9.0 warp

The newer build also requires NVIDIA driver >= 525.60.13.

— Warp Bot
THIS IS AN AUTOMATED MESSAGE GENERATED BY AN LLM. IT MAY OR MAY NOT SOLVE YOUR PROBLEM. IF YOU'D LIKE TO SPEAK TO A HUMAN, SAY SO IN YOUR MESSAGE.

To view this discussion visit https://groups.google.com/d/msgid/warp-em/96cf4da4-6b2d-4d0d-87b8-7352e5ec72fen%40googlegroups.com.