Warp Worker is not working in WarpTools

[Huabin Zhou]

Dear all, 

I was trying to run the tutorial for WarpTools. After creating the setting files for frames and TS, I got a problem running the fs_motion_and_ctf. It looks the warp works was initialized but killed after a few seconds.  Following is what I used:

Software:  # Name                    Version                   Build  Channel
warp                      2.0.0dev33              py311_0    warpem

I also tried the build on SBGrid, same error.

Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server

GPU: A100 or V100, Cuda version 12.4

The command and error are as following.  Any suggestions or insights would be helpful!

-------------------------------------------------------

WarpTools fs_motion_and_ctf \
> --settings warp_frameseries.settings \
> --m_grid 1x1x3 \
> --c_grid 2x2x1 \
> --c_range_max 7 \
> --c_defocus_max 8 \
> --c_use_sum \
> --out_averages \
> --out_average_halves
Running command fs_motion_and_ctf with:
m_range_min = 500
m_range_max = 10
m_bfac = -500
m_grid = 1x1x3
c_window = 512
c_range_min = 30
c_range_max = 7
c_defocus_min = 0.5
c_defocus_max = 8
c_voltage = 300
c_cs = 2.7
c_amplitude = 0.07
c_fit_phase = False
c_use_sum = True
c_grid = 2x2x1
out_averages = True
out_average_halves = True
out_thumbnails = null
out_skip_first = 0
out_skip_last = 0
device_list = {  }
perdevice = 1
workers = {  }
settings = warp_frameseries.settings
input_data = {  }
input_data_recursive = False
input_processing = null
output_processing = null
input_norawdata = False
strict = False

No alternative input specified, will use input parameters from warp_frameseries.settings
File search will be relative to 

~/warp-test/frames
328 files found
Parsing previous results for each item, if available...
328/328, previous metadata found for 0                                        
Connecting to workers...
Unhandled exception. System.Exception: Couldn't connect to newly created worker at localhost:40388 for 20 seconds, something must be wrong
   at Warp.WorkerWrapper..ctor(Int32 deviceID, Boolean silent, Boolean attachDebugger) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpLib/WorkerWrapper.cs:line 114
   at WarpTools.Commands.DistributedOptions.GetWorkers(Boolean attachDebugger) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpTools/Commands/DistributedOptions.cs:line 57
   at WarpTools.Commands.MotionCTFFrameseries.Run(Object options) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpTools/Commands/Frameseries/MotionCTFFrameseries.cs:line 198
   at WarpTools.WarpTools.Run(Object options) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpTools/Program.cs:line 30
   at Warp.Tools.CommandLineParserHelper.ParseAndRun(String[] args, Func`2 run, Type[] verbs, String appName) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpLib/Tools/CommandLineParserHelper.cs:line 26
   at WarpTools.WarpTools.Main(String[] args) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1741217580690/work/WarpTools/Program.cs:line 17
   at WarpTools.WarpTools.<Main>(String[] args)
Aborted (core dumped)

[Alister Burt]

Hi Huabin,

Sorry you're running into issues - I assume there are no worker logs produced?

It's possible your HPC environment is configured to reroute all http traffic which is disrupting communication between the WarpTools process and the WarpWorker processes, please see https://github.com/warpem/warp/issues/28#issuecomment-2436353364 for troubleshooting hints

Cheers,

Alister

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/aaad3042-25dc-4569-9a84-af8ab3cc8512n%40googlegroups.com.

[Huabin Zhou]

Hi Alster,

Thank you so much for your quick reply. It's fixed by changing the proxy setting for local traffic.

Best,

Huabin