lyse dataframe order

[Edvinas Gvozdiovas]

Hello,

Is there a way to get the indices of shots as marked below by the red arrow?

The pandas dataframe has some sort of sorting going on which means I cannot trust the order of shots to be the same as in the user interface.

[dihm....@gmail.com]

Well I can't say I knew this before you asked, but it does not appear to be possible. That number is merely counting the rows in the Qt display box that is showing the dataframe (ie it only lives in the Qt display widget). The underlying dataframe has the shots inserted into their correct order (by default using the filepath, which should ultimately sort based on date and time of shot execution) upon first load, completely ignoring the order in which they were added to lyse. Note that an alternate method of sorting is possible based on integer counting of runs/shots within the run, configurable in the labconfig setup using the `integer_indexing` key. If you are running into issues where you have renamed things from their defaults but want sensible ordering based on when shots executed, this latter option is probably what you are looking for.

While I'm open to hearing a counter argument, I doubt that this is something we would want to support. Typical usage will always have shots arriving in time sorted order anyway, and lyse does not support any facility for re-ordering the dataframe display rows anyway beyond removing shots and re-adding them by hand, one (few?) at a time. My general suggestion (not knowing what you are trying to do, mind you) is that if you need some sort of custom ordering of rows in the dataframe, that ordering should be scripted in the analysis file (via standard pandas operations) rather than manually fiddled with in the GUI. 

-David

[Edvinas Gvozdiovas]

Hi David,

Thanks for the response.

This would be useful for filtering out undesired data quickly and painlessly. Every once in a while we have shots loaded in a messy order, and we have a script that lets us hover over data points in a plot to see which .h5 file they came from. I was pulling the indices as they came in the dataframe assuming they matched with that index in the picture until I realized it's not consistent. Supporting this would allow us to very quickly find "problem data" and delete it from lyse as it is incredibly easy to find a number in a table of increasing numbers, whereas matching a filepath can be painful.

In the end, not a huge deal. We'll be looking at filepaths from now on.

[dihm....@gmail.com]

Ah, that makes sense. 

You probably should consider giving the integer indexing a try. Each row gets labeled by three integers (forget exactly what each one means) but they are much easier to parse visually than the full paths and recreating them for the tooltip should be trivial as the indexes draw from columns in the dataframe.

[Ian B. Spielman]

Maybe this is crazy, but what about just creating a new field in the data frame that is equal to the GUI index number? In my mind there is nothing special about the lyse dataframe (since it is basically a convince tool to get a data that otherwise is stored in shot files), so adding a new field seems reasonable.

In addition Edvinas: it would be a trivial addition to the lyse API to add a function that instructs lyse to remove one or many rows from the dataframe (and the GUI) from within a scrip.

— Ian


Ian B. Spielman

Fellow, Joint Quantum Institute
National Institute of Standards and Technology and the University of Maryland

----- WEB -----
http://ultracold.jqi.umd.edu

----- EMAIL -----
spie...@jqi.umd.edu

----- ZOOM -----
https://umd.zoom.us/j/7984811536

----- PHONE -----
(301) 246-2482

----- MAIL -----
UMD:
2207 Computer & Space Sciences Bldg.
College Park, MD 20742

NIST:
100 Bureau Drive, Stop 8424
Gaithersburg, MD 20899-8424 USA

----- OFFICE -----
UMD: Physical Sciences Complex, Room 2153
NIST: Building 216, Room B131

> The pandas dataframe has some sort of sorting going on which means I cannot trust the order of shots to be the same as in the user interface.
>

> --
> You received this message because you are subscribed to the Google Groups "the labscript suite" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/61ec4306-3edd-4181-93ac-3dc0086057ebn%40googlegroups.com.