ts_particles_export crash with Relion .star file
Karine Mazmanian
hi all,
I am running particle extraction with WarpTools ts_export_particles after Relion5 3D refinement, and it keeps crashing on me. I used run_data.star from relion 5 refine3d to re-extract my particles at a different pixel size with ts_export_particles. it worked before but stopped working after our admin updated warptools (my current version is 2.0.0dev39).
In the terminal i get a c# duplicate-key error, but if i check the worker log for the same tilt series, no error is reported and the particles look like they were successfully exported. (see the error messages below). I did some troubleshooting with Claude and Deepwiki, and workaround that worked for me was removing 10 columns from data_particles (rlnTomoParticleId, rlnTomoParticleName, rlnImageName, rlnTomoVisibleFrames, rlnGroupNumber, rlnClassNumber, rlnNormCorrection, rlnLogLikeliContribution, rlnMaxValueProbDistribution, rlnNrOfSignificantSamples).
I want to check if anybody had met this error before and if I am doing something wrong.
command: WarpTools ts_export_particles \ --settings warp_tiltseries.settings \ --input_star relion_5.72Apx/Refine3D/job023/run_data.star \ --output_star relion_2.9Apx/extracted_proteasome_2.9Apx_224.star \ --coords_angpix 5.72 \ --output_angpix 2.86 \ --box 224 \ --diameter 450 \ --relative_output_paths \ --2d terminal
error: Failed to process /data/.../warp_tiltseries/Position_8.tomostar, marked as unselected
System.ArgumentException: An item with the same key has already been added. Key: rlnTomoParticleId at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) at Warp.Star.AddColumn(String name, String[] values) in .../WarpLib/Star.cs:line 591 at WarpTools.Commands.ExportParticlesTiltseries.<>c__DisplayClass0_0.<Run>b__1(WorkerWrapper worker, TiltSeries tiltSeries) in .../WarpTools/Commands/Tiltseries/ExportParticlesTiltseries.cs:line 341
worker log for the same tilt series, no error: 2026-05-14 03:21:39.837 Received "TomoExportParticleSeries", with 6 arguments, for GPU #2, 11022 MB free: 2026-05-14 03:22:08.243 Exported 486 particles for /data/.../warp_tiltseries/Position_8.tomostar 2026-05-14 03:22:08.243 Execution took 28.405 seconds
I am happy to share the input star if needed.
Thanks!
Karine
Warp Bot
This is a real bug — your workaround is correct. Here is what is happening:
For 2D output, the worker first writes a temporary per-tilt-series particle table that already contains rlnTomoParticleId, rlnTomoParticleName, rlnImageName and rlnTomoVisibleFrames (see WarpLib/TiltSeries/TiltSeries.ReconstructParticleSeries.cs:74-91). After the worker returns, ts_export_particles tries to add every column from your input star that is not in a small "known columns" set (WarpTools/Commands/Tiltseries/ExportParticlesTiltseries.cs:172-188), and that set does not list those four columns. So when your run_data.star already has them, AddColumn throws "An item with the same key has already been added" at Star.cs:591, and the host marks the tilt-series unselected.
This also explains why the worker log shows a clean success: the particle images and the temp star are written, then the crash happens in the host while assembling the final output table.
The four columns that actually collide are rlnTomoParticleId, rlnTomoParticleName, rlnImageName, rlnTomoVisibleFrames. The other columns you removed (rlnGroupNumber, rlnClassNumber, rlnNormCorrection, rlnLogLikeliContribution, rlnMaxValueProbDistribution, rlnNrOfSignificantSamples) are not in the temp star and do not strictly need to be dropped, but it is harmless to do so.
I have filed a GitHub issue to track the fix:
{{ISSUE_URL}}
Until that lands, your workaround of stripping the colliding columns from data_particles before passing the star to ts_export_particles is the right thing to do.
— Warp Bot
THIS IS AN AUTOMATED MESSAGE GENERATED BY AN LLM. IT MAY OR MAY NOT SOLVE YOUR PROBLEM. IF YOU'D LIKE TO SPEAK TO A HUMAN, SAY SO IN YOUR MESSAGE.
--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/861a0bea-834b-4f8e-ad45-5025c29ce763n%40googlegroups.com.
Florent Waltz
We are trying WarpM + Missalignment from our tomoguide ribosome test dataset, and we ran into a problem when trying to re-extract particles at bin1.
Just to take you through what we did, for template matching, we used pytom_match_pick at bin4. We then exported the particles at bin2 (3.82 Å/px) as either 2D stacks (for RELION 5.0.1 --tomo) or 3D subtomos (for RELION 5.0.1 SPA). The idea was also to test if one would perform better than the other.
Both exports worked fine. We worked with exactly the same set of particles.
As expected, the 2D workflow was faster and took much less disk space. After RELION 3D refine at bin2, both yielded a bin2 Nyquist average, with the average from the 2D stacks looking slightly better than the 3D subtomos.
After that, we ran both through MCore without problem, with the 2D still better. We reached very close to Nyquist bin1.
Since we are not experienced WarpM users, we are unsure if the following part is 1) this is the correct way to go, and 2) an expected behavior.
The idea was to reextract the particles at bin1 using ts_export_particles, refine in Relion, and run another round of M refinement.
But we ran into a similar problem to Karin.
Using WarpTools ts_export_particles (with --2d or --3d pointing to the respective Refine3D folders and run_data.star), the particles are successfully created, but still errors are reported.
The errors:
For 2D (--2d): System.ArgumentException: An item with the same key has already been added. Key: rlnTomoParticleId
For 3D (--3d): System.ArgumentException: An item with the same key has already been added. Key: rlnCtfMaxResolution
Again, we are not experienced WarpM users, and maybe some conversion of the run_data.star is needed?
Removing columns from the .star file appears to "fix" the crash, but we are concerned that if _rlnTomoParticleId and _rlnTomoParticleName are removed, the link between the original bin2 extracted particles and the bin1 re-extracted particles will be lost (e.g. bin2 particle 34 is now bin1 particle 67 or whatever).
Using this trick i managed to re-extract the --2d particles at bin1, but refinement in Relion then gives me little improvement compared to my original bin2 Refine BEFORE M, which suggests that no information has been carried over?
Thanks in advance
Best,
Florent
Lukas Bauer
Following up on Karine's and Florent's reports about the rlnTomoParticleId duplicate-key crash when using ts_export_particles with a RELION5 run_data.star — we ran into the same issue and wrote a small Python script to fix it. But we are not sure so far if this is causing any downstream issues, or if it can be seen as a final fix for the issue.
The script makes two minimal changes:
1. Renames rlnTomoName → rlnMicrographName (required for WarpTools to match particles to tilt series)
2. Drops the 4 columns that WarpTools writes internally and therefore collide: rlnTomoParticleId, rlnTomoParticleName, rlnImageName, rlnTomoVisibleFrames
All other columns are left completely untouched — Euler angles, shifts (rlnOriginXAngst/Y/Z), class numbers, GroupNumber, NormCorrection, and the full optics block are preserved as-is.
We have not tested this as extensively as Florent, so we would be very happy about any feedback on whether this reliably fixes the bug — and whether the refined orientations are correctly carried over into the re-extracted particles.
Lukas