ts_export_particles without raw data

[Mart Last]

Hi,

I'm trying something that I think should be possible (?) but can't figure out how to do. After doing the whole raw data to tomograms workflow in WarpTools I've offloaded my raw frames to a hard disk to free up memory. But I do still have the frame averages in warp_frameseries/average which as raw a version of the data as I suppose I need.

When I run WarpTools ts_export_particles I get errors due to missing raw data files:

System.IO.DirectoryNotFoundException: Could not find a part of the path '/cephfs/mlast/compu_projects/easymode/datasets/034_DICTYO/raw/TS_008_052_-60.0.tif'

Which I'm guessing is because the tomostar files point at files like:

  ../warp_frameseries/TS_004_056_-64.0.tif

Which don't exist, but corresponding ../warp_frameseries/*.xml's do and they in turn point at the missing ../raw/ directory.

So my question is: is there a simple way to tell Warp to sample from the previously generated frame averages instead of from the raw data? (A way that is simpler than just copying the data back onto the cluster 🙃) Any input much appreciated!

Best,

Mart

[Hamidreza Rahmani]

Hi Mart,

Have you tried making dummy soft links or text files with the correct name in the rawdata directory?

I am not sure if this will work but maybe WarpTools only checks if that path exists and moves on to aligned micrographs.

Best,

Hamid

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/1f139e04-bd84-45a4-8be2-29293ec59f4an%40googlegroups.com.

[Mart Last]

Hi Hamid,

That's a good suggestion, I'll try... at some point, I actually just deleted the frame averages when I got a bit overzealous with cleaning up  storage... will update this thread once I run into the same problem again and get to try your suggestion.

Op dinsdag 25 november 2025 om 16:23:41 UTC schreef hami...@gmail.com:

[Mart Last]

Sadly faking the raw data files doesn't work. I get this error instead:

{"type":"Processing error","title":"Command execution failed","status":500,"detail":"System.NullReferenceException: Object reference not set to an instance of an object.\n   at Warp.Headers.HeaderEER..ctor(String path, Stream stream) in /cephfs/mlast/_compute/warp/WarpLib/Headers/EER.cs:line 60\n   at Warp.Headers.MapHeader.ReadFromFile(BinaryReader reader, String path, int2 headerlessSliceDims, Int64 headerlessOffset, Type headerlessType, Stream stream) in /cephfs/mlast/_compute/warp/WarpLib/Headers/Headers.cs:line 35\n   at Warp.Headers.MapHeader.ReadFromFile(String path, int2 headerlessSliceDims, Int64 headerlessOffset, Type headerlessType, Boolean isBigEndian, Stream stream) in /cephfs/mlast/_compute/warp/WarpLib/Headers/Headers.cs:line 97\n   at Warp.Headers.MapHeader.ReadFromFile(String path, int2 headerlessSliceDims, Int64 headerlessOffset, Type headerlessType, Stream stream) in /cephfs/mlast/_compute/warp/WarpLib/Headers/Headers.cs:line 86\n   at Warp.Headers.MapHeader.ReadFromFile(String path) in /cephfs/mlast/_compute/warp/WarpLib/Headers/Headers.cs:line 107\n   at Warp.TiltSeries.LoadMovieData(ProcessingOptionsBase options, Movie[]& movies, Image[]& movieData, Boolean doOddEven, Image[]& movieDataOdd, Image[]& movieDataEven, Boolean useDenoised) in /cephfs/mlast/_compute/warp/WarpLib/TiltSeries/TiltSeries.cs:line 1476\n   at Warp.TiltSeries.ReconstructParticleSeries(ProcessingOptionsTomoSubReconstruction options, float3[] positions, float3[] angles, String tablePath, Star& tableOut) in /cephfs/mlast/_compute/warp/WarpLib/TiltSeries/TiltSeries.ReconstructParticleSeries.cs:line 33\n   at WarpWorker.WarpWorkerProcess.EvaluateCommand(NamedSerializableObject Command) in /cephfs/mlast/_compute/warp/WarpWorker/WarpWorker.cs:line 861\n   at WarpWorker.API.ServiceController.EvaluateCommand() in /cephfs/mlast/_compute/warp/WarpWorker/API/Service.cs:line 47","traceId":"00-0d1316522c97dce7bae2d88d3a72eb61-2b47007da152ca38-00"}

Maybe I need to fake the file a bit better and write a fake .eer with normal header but without the data ... 

Op dinsdag 25 november 2025 om 19:47:31 UTC schreef Mart Last:

[Mart Last]

Okay for future reference, there is a workaround does do the trick. For every missing raw data file, create a symbolic link with the expected filename that points to a single real raw image file.

WarpTools ts_export_particles seems to only really get image data from the frame averages, but initially does look at the raw files' headers to figure out their shape, size, bitdepth, etc. 

In my current case I made 40.000 symlinks that all point at a single real .eer file with a header that's valid for all the raw data:

(anaconda3)mlast@hal[raw]: header test.eer RO image file on unit 1 : temp Size= 75629 K This is a TIFF file. Number of columns, rows, sections ..... 16384 16384 225 Map mode .............................. 0 (byte) Start cols, rows, sects, grid x,y,z ... 0 0 0 16384 16384 225 Pixel spacing (Angstroms).............. 1.000 1.000 1.000 Cell angles ........................... 90.000 90.000 90.000 Fast, medium, slow axes ............... X Y Z Origin on x,y,z ....................... 0.000 0.000 0.000 Minimum density ....................... 0.0000 Maximum density ....................... 1.0000 Mean density .......................... 0.92034E-03 tilt angles (original,current) ........ 0.0 0.0 0.0 0.0 0.0 0.0 Space group,# extra bytes,idtype,lens . 0 0 0 0 0 Titles :

WarpToolst ts_export_particles then runs normally.

Op maandag 15 december 2025 om 14:44:47 UTC schreef Mart Last:

[Alister Burt]

Thank you for reporting back Mart, this is ugly and I’m sorry it’s necessary but glad you found a workaround!!

Cheers,

Alister

Sent from mobile - apologies for brevity

On Dec 15, 2025, at 11:11, Mart Last <mgf...@gmail.com> wrote:



To view this discussion visit https://groups.google.com/d/msgid/warp-em/ecacdf5f-cd90-409a-9ef1-4c4bbd826788n%40googlegroups.com.