2d particle extraction issue

Reika Watanabe

unread,

Jan 15, 2025, 8:41:42 PM1/15/25

to Warp

Dear all,

I encountered issues with 2d particle extraction step and I could not figure out.

I have total of 67 tomograms and I could extract 2d particle series data of 60k particles from all these 67 tomogrames without any issue.

Using the same tilt series setting, now I tried to extract 224K particles, I got the error. Among 67, I have four tomograms which I can not extract particles from.

error message is as following

File search will be relative to /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/tomostar
67 files found
Parsing previous results for each item, if available...
67/67, previous metadata found for 67
Found 224780 particles in 67 tilt series
Connecting to workers...
Connected to 2 workers
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_129.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_122.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_125.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_123.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
67/67, 4 failed, 00:00 remaining
Finished processing in 00:30:44

I used 1) Warptool change selection with --selection

or

2) Warptool change selection with --null

The outcome show

1)

Before change: 4 deselected, 0 selected, 63 null
After change: 0 deselected, 67 selected, 0 null

2)

Before change: 4 deselected, 0 selected, 63 null
After change: 0 deselected, 0 selected, 67 null

after both change selection, I got the same errors and 4 folders corresponding 4 tomograms were missing in the particle series folder and particle information are not found in matching data star.

It looks like original warp_run_data.star used for extraction causes this, but I could not find any difference between problematic tomograms (TS_122, TS_123, TS_125, TS_129) and others.

I also made run_data.star which contains just four tomograms data and tried to exract separately, again same unselected error.

I also checked the log file, and others shows proper extraction but 4 problematic ones shows last extraction from the last 60K extraction log.

Have you encountered this strange issues? How can I solve this issue?

Thank you very much for your kind attention in advance.

Reika

Alister Burt

unread,

Jan 15, 2025, 10:23:26 PM1/15/25

to Reika Watanabe, Warp

Hi Reika,

Could you update your installation and try again? We recently made a change which might fix the cause of this issue (https://github.com/warpem/warp/pull/316)

If that doesn’t work, please also share the contents of the worker logs which will show the error

Cheers,

Alister

Sent from mobile - apologies for brevity

On Jan 15, 2025, at 17:41, Reika Watanabe <reika.wa...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/4de167d8-2e1f-4ad2-bc6c-9b3d46a51fa6n%40googlegroups.com.

Reika Watanabe

unread,

Jan 16, 2025, 1:51:31 PM1/16/25

to Warp

Dear Alister,

Thank you very much for coming back to me.

I updated warp version like this.

(warp) [rwatanabe@c110144 ~]$ conda update warp -c warpem -c nvidia/label/cuda-11.8.0 -c pytorch -c conda-forge
Channels:
- warpem
- nvidia/label/cuda-11.8.0
- pytorch
- conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

# All requested packages already installed.

(warp) [rwatanabe@c110144 ~]$ conda list warp
# packages in environment at /home/rwatanabe/mambaforge/envs/warp:
#
# Name Version Build Channel
warp 2.0.0dev31 py311_0 warpem

After this, again I use warptool change_selection

(warp) [rwatanabe@c110144 ~]$ WarpTools change_selection \
> --settings /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries.setting \
> --null \
>
Running command change_selection with:
select = False
deselect = False
null = True
invert = False
settings = /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries.setting
input_data = { }
input_data_recursive = False
input_processing = null
output_processing = null
input_norawdata = False

No alternative input specified, will use input parameters from warp_tiltseries.setting

File search will be relative to /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/tomostar
67 files found
Parsing previous results for each item, if available...
67/67, previous metadata found for 67

67/67 processed

Before change: 4 deselected, 0 selected, 63 null
After change: 0 deselected, 0 selected, 67 null

I tried to extract 2 D particles but without success (same errors).

(warp) [rwatanabe@c110144 ~]$ WarpTools ts_export_particles \
> --settings /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries.setting \
> --input_star /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/Relion5_for_ALL/warp_run_data.star \
> --coords_angpix 12.8928 \
> --output_star /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/Relion5_for_ALL/matching12.89A_run_data.star \
> --output_angpix 12.8928 \
> --box 32 \
> --diameter 240 \
> --relative_output_paths \
> --2d \
>
Running command ts_export_particles with:
input_star = /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/Relion5_for_ALL/warp_run_data.star
input_directory = null
input_pattern = *.star
coords_angpix = 12.8928
normalized_coords = False
output_star = /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/Relion5_for_ALL/matching12.89A_run_data.star
output_angpix = 12.8928
box = 32
diameter = 240
relative_output_paths = True
2d = True
3d = False
dont_normalize_input = False
dont_normalize_3d = False
n_tilts = null
max_missing_tilts = 5
device_list = { }
perdevice = 1
workers = { }
settings = /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries.setting
input_data = { }
input_data_recursive = False
input_processing = null
output_processing = null
input_norawdata = False

No alternative input specified, will use input parameters from warp_tiltseries.setting

File search will be relative to /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/tomostar
67 files found
Parsing previous results for each item, if available...
67/67, previous metadata found for 67
Found 224780 particles in 67 tilt series
Connecting to workers...
Connected to 2 workers
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_129.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_122.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_125.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
Failed to process /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_123.tomostar, marked as unselected
Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.
Use the change_selection WarpTool to reactivate this item if required.
67/67, 4 failed, 00:00 remaining

Finished processing in 00:31:57
Saying goodbye to all workers... Done

In order to obtain workers log, I tried

(warp) [rwatanabe@c110144 ~]$ export WARP_DEBUG=1
(warp) [rwatanabe@c110144 ~]$

I did not get anything, could you please let me know how I can get workers log to see the necessary information.

Thank you very much for your kind attention in advance!

Reika

Alister Burt

unread,

Jan 16, 2025, 3:20:40 PM1/16/25

to Reika Watanabe, Warp

Check logs in /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/logs for more info.

To view this discussion visit https://groups.google.com/d/msgid/warp-em/6d259a53-43fe-4400-b898-10214dec6c97n%40googlegroups.com.

Reika Watanabe

unread,

Jan 16, 2025, 4:01:31 PM1/16/25

to Warp

Hi Alister,

I did checked that log already. I did not find any obvious errors in this log. Please let me know if any other critical test to be done.

For example, this is TS_122 log, which cannot extracted with data.star containing 224k particles in total. The information here is corresponding to the process that I use data.star containing just 64K particle.

I did this extraction to check if some tomostar or tilt series setting are messed up. I could extract without any issue when I used smaller data.star.

2025-01-15 19:24:00.010 Received "TomoExportParticleSeries", with 6 arguments, for GPU #1, 23333 MB free:
2025-01-15 19:26:34.253 Exported 5666 particles for /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_122.tomostar
2025-01-15 19:26:34.254 Execution took 154.242 seconds

2025-01-15 19:26:34.255

Another data is TS_121. The particles from this tomogram is extracted perfectly.

2025-01-16 18:15:24.510 Received "TomoExportParticleSeries", with 6 arguments, for GPU #1, 23819 MB free:
2025-01-16 18:16:11.802 Exported 7705 particles for /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_121.tomostar
2025-01-16 18:16:11.803 Execution took 47.291 seconds
2025-01-16 18:16:11.804

Thank you very much!

Reika

Alister Burt

unread,

Jan 16, 2025, 4:06:19 PM1/16/25

to Reika Watanabe, Warp

Hi Reika,

Please rerun the job that fails then look at the logs for that job - logs are overwritten with each job

Cheers,

Alister

On Jan 16, 2025, at 13:01, Reika Watanabe <reika.wa...@gmail.com> wrote:

To view this discussion visit https://groups.google.com/d/msgid/warp-em/2b21724d-45cc-417e-8a13-163ea4d1b4f8n%40googlegroups.com.

Reika Watanabe

unread,

Jan 16, 2025, 5:19:25 PM1/16/25

to Warp

Hi Alister,

I just rerun the job. Although the new particle series (except problematic four tomograms), the log was not updated today's extraction.

The log of the particle extracted-tomogram (TS_121).

2025-01-16 21:53:34.239 Received "TomoExportParticleSeries", with 6 arguments, for GPU #0, 23514 MB free:
2025-01-16 21:54:22.503 Exported 7705 particles for /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_121.tomostar
2025-01-16 21:54:22.504 Execution took 48.263 seconds
2025-01-16 21:54:22.505

The log of particle not-exctracted-tomogram (TS_122)

2025-01-15 19:24:00.010 Received "TomoExportParticleSeries", with 6 arguments, for GPU #1, 23333 MB free:

2025-01-15 19:26:34.253 Exported 5666 particles for /mnt/staging/rwatanabe/DataProcess/Warp_tool/NP_delta601_all/warp_tiltseries/TS_122.tomostar
2025-01-15 19:26:34.254 Execution took 154.242 seconds

2025-01-15 19:26:34.255

There is not obvious errors.

If we use the same data.star, it seems that they do not renew the log.

Thank you for your guidance!

Reika

Reika Watanabe

unread,

Jan 16, 2025, 5:21:00 PM1/16/25

to Warp

Sorry for my mistake,

Although the new particle series (except problematic four tomograms) were successfully created , the log was not updated today's extraction.

Reika Watanabe

unread,

Jan 22, 2025, 5:40:21 PM1/22/25

to Warp

Hello Alister,

I investigate more detail about problematic tomograms. All four tomograms contain more particles (between 9.3K to 10K) compared to other successful tomograms for extraction.

Using equivalent data-star, with Warp1.09 version, I could extract subtomo version without any issue.

Are there any limitation of numbers of particle series extraction per tomogram in the current Warp_tool version (see attached)?

# Name Version Build Channel
warp 2.0.0dev31 py311_0 warpem

Or do you think this is limitation of my workstation? The spec of my workstation below.

Thank you very much for your kind attention in advance!

With best wishes,

Reika

Alister Burt

unread,

Jan 22, 2025, 6:34:50 PM1/22/25

to Reika Watanabe, Warp

Hi Reika,

The only similar issue I've seen is https://github.com/warpem/warp/issues/315, I pushed a temporary fix for that in https://github.com/warpem/warp/pull/316

I checked and the fix in #316 is not included in 2.0.0dev31 so I pushed 2.0.0dev32 - this will be online once https://github.com/warpem/warp/actions/runs/12919012914/job/36028674870 completes.

side note: 10k particles per tomogram is a lot... whilst I agree that Warp should be able to handle arbitrary numbers of particles, it might be worth looking at your particle picking approach?

Cheers,

Alister

To view this discussion visit https://groups.google.com/d/msgid/warp-em/0612f969-aacf-4061-9e3d-f9b751058c96n%40googlegroups.com.

Reika Watanabe

unread,

Jan 22, 2025, 9:35:27 PM1/22/25

to Alister Burt, Warp

Thank you very much, Allister, for your prompt reply.

I'm looking forward to testing newer version when it's available.

I'm working on ebolavirus filamentous nucleocapsid in cells.

The filamentous nucleocapsids are filament traced and Dynamo- aligned. I am extracting 4 repeating unit in each box without over-wrapping. Each particles are unique and all contribute to averaging. I think I cannot miss these particles. What I can think is to duplicate tomostar with different names, split data_star, and extract separately. I might have to go back processing step again if I want to do this.

Thank you very much again for your kind support 😉

With best wishes,

Reika

Alister Burt

unread,

Jan 23, 2025, 1:15:42 PM1/23/25

to Reika Watanabe, Warp

Hi Reika,

I tried to reproduce your issue this morning and was able to export 10k particles from a single tilt series.

Here is what the worker log looks like:

2025-01-23 18:06:53.055 Received "TomoExportParticleSeries", with 6 arguments, for GPU #0, 39850 MB free:
2025-01-23 18:08:00.998 Exported 10300 particles for .../EMPIAR-10491-5TS/particle_export_test_10k/TS_1.tomostar
2025-01-23 18:08:00.999 Execution took 67.944 seconds
2025-01-23 18:08:00.999 

And the resulting star file contains all particles.

The fact that you said the worker log was not created/updated makes me think this tilt series got deselected at some point - please use the change_selection tool and try again. Set --output_processing to a new directory to be sure that what you are looking at is indeed the result of the latest run in each case.

Cheers,

Alister

Reika Watanabe

unread,

Jan 23, 2025, 6:04:46 PM1/23/25

to Alister Burt, Warp

Thank you very much, Alister,

Just let me confirm: Are you using the original 2.0.0dev31, right?

I have used select tool to deselect but never tried output-processing folder.

I will let you know if this works.

With best wishes,

Reika

Alister Burt

unread,

Jan 23, 2025, 6:16:32 PM1/23/25

to Reika Watanabe, Warp

this was tested on 2.0.0dev32

Reika Watanabe

unread,

Jan 23, 2025, 10:17:34 PM1/23/25

to Warp

Hi, Alister

I updated to dev32 version.

Now I could extract ALL particles including ones from tomograms containing more than 10K particles.

I really appreciate this great improvement.

Thank you very very much and have a great evening!

With best wishes,

Reika

Alister Burt

unread,

Jan 23, 2025, 11:22:42 PM1/23/25

to Reika Watanabe, Warp

Cool! Glad to hear this solved your issue :-)

Sent from mobile - apologies for brevity

On Jan 23, 2025, at 19:17, Reika Watanabe <reika.wa...@gmail.com> wrote:

To view this discussion visit https://groups.google.com/d/msgid/warp-em/75bb8edd-7d1b-4b1c-b180-b6f5bc6d86d2n%40googlegroups.com.

Reply all

Reply to author

Forward