Importing lyse

[Roger Ding]

Hi,

There's probably something obvious that I'm missing, but I'm having trouble importing lyse (the module, e.g., using "import lyse" or "from lyse import *") when trying to run python scripts outside of lyse (the GUI interface).

Example program (saved as test.py in "New folder (3)" as scratch space):

    from lyse import *
    print('hello')

Thrown error (e.g., from IDLE):

    Traceback (most recent call last):
      File "C:\Users\reyalp\Downloads\New folder (3)\test.py", line 1, in <module>
        from lyse import *
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\lyse\__init__.py", line 16, in <module>
        from lyse.dataframe_utilities import get_series_from_shot as _get_singleshot
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\lyse\dataframe_utilities.py", line 13, in <module>
        import labscript_utils.h5_lock, h5py
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\labscript_utils\__init__.py", line 101, in <module>
        zprocess.disable_quick_edit()
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\utils.py", line 170, in disable_quick_edit
        [
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\utils.py", line 171, in <listcomp>
        a is not None and a.isatty() and a.fileno() >= 0
    io.UnsupportedOperation: fileno

I'm currently using a "Regular installation" inside a conda environment and the issue with importing lyse comes up using the same conda environment that labscript is installed in ("py38" in this case).

I prefer using something like Jupyter or Spyder to write/test/debug code so this issue has been a bit of a head scratcher for me. Also, I like to learn if there's a "workflow" others prefer (or at least better ways of writing programs) that I could implement.

Thanks,
Roger

[Chris Billington]

Hi Roger,

It's very much intended to be able to use lyse from outside the GUI program, so we'd like it to work!

The problem you're seeing is similar to this one:

https://github.com/labscript-suite/labscript-utils/pull/78

in which some other function inspecting stdout.fileno() failed with the same exception in the context of a Jupyter notebook. You're using IDLE, but it sounds like it must be pretty similar.

The fix is easy enough, since the disable_quick_edit() function is not relevant to IDLE or Jupyter notebooks anyway. I've just made a commit to zprocess to simply catch the io.UnsupportedOperation and turn it into an invalid file number, which should fix the problem.

Could you test that the changes from the below commit (you can just copy-paste them into C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\utils.py to test) fix the issue?

https://github.com/chrisjbillington/zprocess/commit/2d00311fa93515caf1cbc935c3b0fe89f4ba4002

Regards,

Chris

--
You received this message because you are subscribed to the Google Groups "the labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/0d8b3642-75c3-4f17-adbb-b594c67f7b26n%40googlegroups.com.

[Roger]

Hi Chris,

Thanks for looking into this so quickly. I definitely prefer something like Jupyter or Spyder over IDLE, but figured IDLE was a more barebones way to see if I'm screwing something up.

Copying the lines from your commit (https://github.com/chrisjbillington/zprocess/commit/2d00311fa93515caf1cbc935c3b0fe89f4ba4002), I get the following error (using IDLE):

    Traceback (most recent call last):

      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\security.py", line 343, in instance
        return cls._instances[cls, shared_secret]
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\weakref.py", line 131, in __getitem__
        o = self.data[key]()
    KeyError: (<class 'zprocess.security.SecureContext'>, 'pdd5Y5t02EJ8PG/mR1xWxqsI2j/coIVzj/ifotXXAkeiESin4EqvvkHOWQL0V3n5BU5aMIV6KZzgMSLd8UEjBw==')

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\reyalp\Downloads\New folder (3)\test.py", line 1, in <module>
        from lyse import *
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\lyse\__init__.py", line 16, in <module>
        from lyse.dataframe_utilities import get_series_from_shot as _get_singleshot
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\lyse\dataframe_utilities.py", line 13, in <module>
        import labscript_utils.h5_lock, h5py

      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\labscript_utils\h5_lock.py", line 96, in <module>
        connect_to_zlock_server()
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\labscript_utils\ls_zprocess.py", line 322, in connect_to_zlock_server
        client.ping(timeout=0.05)
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\zlock\__init__.py", line 105, in ping
        self._new_socket(timeout)
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\zlock\__init__.py", line 78, in _new_socket
        context = SecureContext.instance(shared_secret=self.shared_secret)
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\security.py", line 345, in instance
        instance = cls(io_threads, shared_secret=shared_secret)
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\security.py", line 322, in __init__
        _check_versions()
      File "C:\Users\reyalp\anaconda3\envs\py38\lib\site-packages\zprocess\security.py", line 40, in _check_versions
        if sys.stderr is not None and sys.stderr.fileno() >= 0:
    io.UnsupportedOperation: fileno

I did notice a slight discrepancy between the utils.py verion I have from anaconda (located at C:\Users\reyalp\anaconda3\envs\py38\Lib\site-packages\zprocess\utils.py) and your most recent utils.py (from https://github.com/chrisjbillington/zprocess/blob/master/zprocess/utils.py) so I tried replacing utils.py with your most recent version and got a similar error. (The slight discrepancy I noticed was the inclusion of "from pathlib import Path" in your version compared to the utils.py I got from anaconda which didn't have that line.)

Interestingly, running "python test.py" (from the anaconda powershell) doesn't throw an error with either version of utils.py.

Roger Ding

[Chris Billington]

Hi Roger,

Looks like just the same issue in a few more places throughout zprocess. I've pushed fixes for them. Could you test?

Since there are a few files, rather than patching in the changes yourself, you might instead want to install the latest commit from our anaconda repository's "test" label with the following command:

conda install -c labscript-suite/label/test zprocess==2.20.5.dev8

Regards,

Chris

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/005177b0-708b-47aa-b3f1-a99b06f0f9ben%40googlegroups.com.

[Chris Billington]

Just adding to this thread the following message that I initially sent to Roger only, until he confirmed he'd changed his encryption key:

---

Hi Roger,

Unfortunately the traceback in your previous email contains an encryption key (it is not great that Python prints this in any circumstance, I'll think about how to ensure this doesn't happen!). Since your encryption key is on the internet now, you should generate a new one and not use the old one.

To generate a new one, first delete the old one from the C:\Users\reyalp\labscript-suite\labconfig directory, it will be called zpsecret-xxxxxxxx.key where the x's are random hex digits. Then with a miniconda terminal, cd to that directory and generate a new key by running "python -m zprocess.makesecret". That will generate a similarly named file. Then edit the [security] section of your labconfig file to update it to use the new file just generated.

There are two long-running programs, zlock and zlog, that will then not work until restarted with the new encryption key. You can either reboot the computer, or find these two processes in Windows' system monitor or whatever it's called and end them (I prefer process explorer). They will be Python processes with a command line that looks like "python -m labscript_utils.zlock" or "python -m labscript_utils.zlog".

If you're using multiple computers, you'll want to copy the key to all of them in the same location and update their labconfig files too.

Whilst it's not super high risk since someone does need to be on your local network in order to access the computers in question, I would definitely want to change that key, as access to a labscript suite program over the network does allow execution of arbitrary code.

[Roger]

Hi Chris,

Your latest commit seems to have worked!

Using test.py:
    from lyse import *
    ser = data(path)
    print(ser)

With lyse (the GUI) running, I'm able to run test.py in Spyder and it pulls up the expected information from the shot .h5 file loaded in lyse.

In Jupyter (with the lyse GUI still running), I'm able to import lyse (the module) without issue but it errors out at "ser = data(path)" with (I'll put the traceback below):
    FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '-f', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

It seems like it can't find the file path so providing an explicit path to the shot .h5 file seems to get around this issue.

Anyways, thanks again for all your time and help Chris!

Roger Ding


---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [22], in <cell line: 2>()
      1 from lyse import *
----> 2 ser = data(path)
      3 #ser = data(p)
      4 print(ser)

File ~\anaconda3\envs\py38\lib\site-packages\lyse\__init__.py:147, in data(filepath, host, port, timeout, n_sequences, filter_kwargs)
     82 """Get data from the lyse dataframe or a file.
     83
     84 This function allows for either extracting information from a run's hdf5
   (...)
    144     the lyse dataframe, or a subset of it, is returned.
    145 """    
    146 if filepath is not None:
--> 147     return _get_singleshot(filepath)
    148 else:
    149     if n_sequences is not None:

File ~\anaconda3\envs\py38\lib\site-packages\lyse\dataframe_utilities.py:124, in get_series_from_shot(filepath)
    123 def get_series_from_shot(filepath):
--> 124     nested_dict = get_nested_dict_from_shot(filepath)
    125     flat_dict =  flatten_dict(nested_dict)
    126     s = flat_dict_to_flat_series(flat_dict)

File ~\anaconda3\envs\py38\lib\site-packages\lyse\dataframe_utilities.py:29, in get_nested_dict_from_shot(filepath)
     28 def get_nested_dict_from_shot(filepath):
---> 29     row = runmanager.get_shot_globals(filepath)
     30     with h5py.File(filepath,'r') as h5_file:
     31         if 'results' in h5_file:

File ~\anaconda3\envs\py38\lib\site-packages\runmanager\__init__.py:920, in get_shot_globals(filepath)
    916 """Returns the evaluated globals for a shot, for use by labscript or lyse.
    917 Simple dictionary access as in dict(h5py.File(filepath).attrs) would be fine
    918 except we want to apply some hacks, so it's best to do that in one place."""
    919 params = {}
--> 920 with h5py.File(filepath, 'r') as f:
    921     for name, value in f['globals'].attrs.items():
    922         # Convert numpy bools to normal bools:
    923         if isinstance(value, np.bool_):

File ~\anaconda3\envs\py38\lib\site-packages\labscript_utils\h5_lock.py:53, in File.__init__(self, name, mode, driver, libver, **kwds)
     51     self.zlock.acquire()
     52 try:
---> 53     _File.__init__(self, name, mode, driver, libver, **kwds)
     54 except:
     55     if hasattr(self, 'zlock'):

File ~\anaconda3\envs\py38\lib\site-packages\h5py\_hl\files.py:507, in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, **kwds)
    502     fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
    503                      locking, page_buf_size, min_meta_keep, min_raw_keep, **kwds)
    504     fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
    505                      fs_persist=fs_persist, fs_threshold=fs_threshold,
    506                      fs_page_size=fs_page_size)
--> 507     fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    509 if isinstance(libver, tuple):
    510     self._libver = libver

File ~\anaconda3\envs\py38\lib\site-packages\h5py\_hl\files.py:220, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    218     if swmr and swmr_support:
    219         flags |= h5f.ACC_SWMR_READ
--> 220     fid = h5f.open(name, flags, fapl=fapl)
    221 elif mode == 'r+':
    222     fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

File h5py\_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py\_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py\h5f.pyx:106, in h5py.h5f.open()

FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '-f', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

[Chris Billington]

Hi Roger,

Good to hear that issue is resolved! IDLE and Jupyter are pretty odd environments from the perspective of the Python interpreter, but many use Jupyter at least so good to have this fixed.

As for the error you're getting now, the lyse module sets the path variable to None if the script is not being run from within lyse, or to sys.argv[1] if the script is being run with command line arguments. This way, you can write a script and run it from the command line on specific HDF5 files by passing them in as command line arguments. Looks like Jupyter has "-f" as the first command line argument. I have no idea why, but it's obviously not a file.

But unless you're intentionally running scripts with command line arguments, you should avoid using the path variable in scripts run outside of lyse. Instead, if you call lyse.data() with no arguments (or with None as the only argument - that's what's effectively happening in Spyder for you), you will get back a pandas dataframe with data from all shots currently loaded into lyse. You can then have your code pick out one filepath from the dataframe to continue analysis on, or whatnot.

Regards,

Chris

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/884b49f0-5ffa-49c2-b7aa-1cc4d1b06958n%40googlegroups.com.