Hello everyone,
We've been using labscript for sevral years now and started to experience some serious issues with runmanager, which worsened over time.
Basically, the software would start freezing for a few seconds every time a global was updated, and it took several seconds for runmanager to create the shots files and send them over to BLACS.
I've traced back the issue to the hdf5 file used by runmanager to store the globals (we went for a single file approach, which holds ever single global variable, splitted into groups), which was actually growing in size over time (incremented each time a globals value is changed in runmanager), to the point that when I checked yesterday its size was about 500Mb. I've temporarily replaced it with one of our recent experimental shot file and the freezing issues are gone!
However, the file size (of the new file) is still increased when a global is changed. This is very easy to diagnose, we just have to click multiple times on some boolean variable checkbox and watch the file size grow by increments of ~10kb, live. This was tested on our experimental setup and on a second one, which runs a more recent version of labscript.
We first thought that the issue was actually a feature which we were not aware of, i.e. this globals file was initially thought to hold "memory" of the previous globals values, and that modifying one global would simply append a new "line" to the file with the new set of globals. However, when opening the hdf5 file (by hand or with e.g. python), we can only access the most recent state of runmanager. It seems like the globals effectively get replaced in the hdf5 structure of the file but that somehow the old values do not get deleted from the binary itself.
Moreover, the issue do not seem to come from the updating of the globals values but rather from the updating of the expansions in the file. Indeed, when looking at the expansion subgroups in the hdf5 file, they seem to hold memory of variables previously deleted from runmanager. This was again tested on two setups, by creating "dummy" variables, and deleting them right away. They would still appear in the expansions of the hdf5, albeit not in the globals.
I've not pushed the analysis / deciphering of the source code much further because I first wanted to know if we were doing something obviously wrong or if we were missing some feature of the software (or if this has been patched in recent versions).
For reference, we are using an older version of the software dating fromearly 2017 which we never really took the time to update, and the other version that was tested dates from early 2019 I believe.
Best regards
Victor Helson