lyse heartbeat failure

[Yoan Spahn]

Hello everybody,

when running my multishot analysis on a large amount of single shots (with rather high amounts of data per shot) I get the lyse "heartbeat failure" error. Does anybody know where this error originates from, as lyse does not provide further details?

Thanks

Yoan

[Edvinas Gvozdiovas]

Hi Yoan,

What does your memory usage in task manager look like when your script is running? Are you topping out at 100%?

--
You received this message because you are subscribed to the Google Groups "the labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/f0adb387-1f7a-497b-811a-daaa9194329dn%40googlegroups.com.

[Yoan Spahn]

Hi,

thanks for the response. My memory usage is at about 40% and doesn't top out. I´ve had issues with RAM in the past but the error message was then a different one (stating the lack of memory). 

My analysis script failed due to "heartbeat failure" while calling data() after importing al my python packages. But somehow not only the Worker seems to have a problem, the entire lyse programm feezes everytime this error occurs. I´ve increased the timeout of data, but this didn´t help.

Best regards

Yoan

[Philip Starkey]

Hi Yoan,

This sounds like the zprocess heartbeat timeout. Technically it's configurable, but it looks like you'd have to edit the zprocess source to do so. I wouldn't expect this to happen unless it's taking more than 10 seconds to transmit the data frame (excluding the time it takes to make a copy of the data frame). That's quite a long time, so it's probably worth seeing if we can cut that down.

The data function takes keyword arguments that allow you to request/receive only a subset of rows and columns. Do you need all of the rows and columns in your multishot analysis script? If not, I highly recommend using n_seqiences and filter_kwargs to restrict the transmitted data. See here for documentation: https://docs.labscriptsuite.org/projects/lyse/en/latest/api/_autosummary/lyse.data/#lyse.data

Cheers, 

Phil

[Yoan Spahn]

Hi Philip,

I already feared i might have to look into the zprocess source code. 

Using the filter function and loading my data into different data frames helped a lot (thanks !!!)  but sometimes the time it takes to transmit the data frames still reaches the 10s mark. Unfortunaly I have to use a lot of data due to the nature of my work. I will try to look into the zprocess source code and try to increase the 10s to 30s or something like that.

Bests from Germany

Yoan

[Jan Deppe]

Hey Yoan,

Just out of interest, how many single shots did you load into lyse and how did it perform ?

We also want to analyze a large amount of single shots but ours are quite small (20mb) and we are wondering if lyse can handle this many shots.

Greetings from Germany :)

Jan

[Yoan Spahn]

Hi Jan,

so my individual hdf5 files for each single shot have a size of a few mb and I´m using 1000-1500 shots per evaluation at the moment. But I´ve allready used larger data sets, the limiting factor seems to be the size of the data frame you want to load within the multi shot routine. This can be solved by using the filter_kwargs as mentioned by Philip and spliting your big data frame into smaller data frames.

Bests

Yoan

[Ian B. Spielman]

Dear Yoan,

I know that what follows is not a “canonical” lyse use pattern, but unless you are grabbing 10’s of records that contain only numbers or strings (and for sure no large arrays) I would avoid pulling the dataframe like the plague. If you like lyse, my suggested solution in this case is to store larger chunks of information in your own external file (h5 or whatever you like) and pull that path from the data frame.

— Ian

Ian B. Spielman

Fellow, Joint Quantum Institute
National Institute of Standards and Technology and the University of Maryland

----- WEB -----
http://ultracold.jqi.umd.edu

----- EMAIL -----
spie...@jqi.umd.edu

----- ZOOM -----
https://umd.zoom.us/j/7984811536

----- PHONE -----
(301) 246-2482

----- MAIL -----
UMD:
2207 Computer & Space Sciences Bldg.
College Park, MD 20742

NIST:
100 Bureau Drive, Stop 8424
Gaithersburg, MD 20899-8424 USA

----- OFFICE -----
UMD: Physical Sciences Complex, Room 2153
NIST: Building 216, Room B131

> To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/110c56da-a8fc-4a0d-af41-178370766e40n%40googlegroups.com.