Working with 2D arrays in the dataframe of lyse

[Fabian Lenz]

Hello everyone,

in Lyse we are saving 2D-data from single-shot-routines using the 'save_results' method. However this stores the array as whole object making it harder to access in multishot routines. At the moment there is no elegant way of saving multidimensional data?

After looking at the code in 'dataframe_utilities' I tried to cast the data to a dictionary before saving it. This resulted in the dictionary getting loaded as object:

After removing the shot and re-adding it, the dataframe got generated in the form I was hoping to:

Which of the two generated dataframes corresponds to the intended behaviour?

If it is the latter, then would this be a legitimate way to save multidimensional data?

Best regards

Fabian Lenz

[Zak V]

Hi Fabian,

The suggested way to save/load arrays is to use the `save_result_array()` and `get_result_array()` methods of lyse's `Run` class. Alternatively you could also use the convenience methods `save_result_arrays()` and `get_result_arrays()` if you'd like to save/load multiple arrays at once. Some of those methods have had docstrings added for them, but there hasn't been a new release of the labscript suite since then. You can read them on github at https://github.com/labscript-suite/lyse/blob/master/lyse/__init__.py. Alternatively, if you have a developer install you can pull the latest changes and have those docstrings in your local copy of lyse.

Note that those methods will save/load the array to/from the hdf5 file but won't add it to the lyse dataframe. I believe that behavior was chosen to avoid making the lyse dataframe too large, which would make it slow to send it over ZeroMQ. To access those arrays from other lyse routines, you'll need to create a `Run` instance then use its `get_result_array()` method.

Cheers,

Zak

[Fabian Lenz]

Hi Zak,

Thank you for your answer. While playing around a little and testing different approaches, I found that there is a routine_storage Object, an instance of the Class _RoutineStorage(Object) declared in the __init__.py of lyse, that is empty.

My approach is now to perform all time-costly computations like flattening the dataframe, filtering and grouping only once, and then saving all intermediate results to the routine_storage. When running the script again, it loads everything from the storage and skips all deselected computations.

The advantage is that small changes take only milliseconds when rerunning the routine.

The bug when saving dictionaries still consists though. For my approach in using the routine_storage this does not cause any trouble, so I am fine with that. But I guess in a future version of lyse it would be better to fix this. The cause is probably, that results from a routine do not use the methods in dataframe_utilities.py but some others in the __main__.py of lyse that must work differently.

Cheers,

Fabian

[Zak V]

Hi Fabian,

Glad you got something working! I should mention that, as you likely already know, things stored in `routine_storage` aren't saved to disk so they'll go away when lyse is closed. Also, if I remember correctly, there is a separate `routine_storage` instance for each lyse script. So if you store a result array in `routine_storage` in one lyse script, other lyse scripts won't be able to access it.

When using `save_result_array()` the results will be written to the shot's hdf5 file on disk. That means that they'll still be there if lyse is closed and it means that other lyse scripts can access those results by using `get_result_array()` to load the array back from the disk. If saving/loading to/from disk is too slow but you still want to share data between lyse scripts, then you may find the discussion in lyse issue #26 (https://github.com/labscript-suite/lyse/issues/26) to be helpful.

If you'd like you can report the bug you are seeing on the lyse issue tracker (https://github.com/labscript-suite/lyse/issues). It'll be more likely to get fixed if you include instructions and example code that others can use to reproduce your issue. Also, since the code is open source you are welcome to submit a pull request to fix the issue yourself!

Cheers,

Zak

[Fabian Lenz]

Hi Zak,

the ' routine_storage' being accessible from one script only is totaly fine to me and saving to disk can just be done inside the routine itself. So no worry in that.

I reported the bug in github. But I am not really sure how to fix it.

Cheers

Fabian