Lyse on OSX

[Fred Jendrzejewski]

Hi,

I have been looking into the possibility of python for data treatment on ultracold atoms and wanted to figure out a bit the possibilities of lyse. Did you ever try to make it run on OSX ? And is it anyway a good idea to employ lyse for this purpose or is rather for dataviewing as they come in?

Cheers,

Fred

[Chris Billington]

Hi Fred,

Lyse should work fine on OSX. It was very hard to get working in the past when we used the GTK GUI toolkit, but now that it uses the Qt toolkit it should simply be a matter of installing the Anaconda Python distribution, which makes it easy to satisfy lyse's dependencies. If you were install the Anaconda Python distribution (with Python 2.7), and follow the installation instructions for the labscript suite, I think you will have lyse working with no problems. To actually run lyse, you will need to run "python -m lyse" from the command line, since we have not made proper app launchers for OSX or linux.

As for whether lyse is likely to be useful for "offline" analysis of data that wasn't acquired using the rest of the labscript suite, I'm not so sure. Lyse is really just a program that hands off control to the user's code and lets them manage which analysis routines will run in which order, and lyse runs all the analysis routines on each new experiment shot that comes in. Lyse is also used for offline analysis, but this is mostly useful when one is analysing data already packed into HDF5 files in the way lyse expects. If you already have all the data, it is probably not worth packing it into HDF5 files and then using lyse to get it out again - this is probably more work than it is worth and you ought to just run analysis code on the data as you already have it.

On the other hand, if you want "online" data analysis (i.e whilst your experiment is running), then lyse might be a good choice. it's not just for displaying the data, you can do whatever calculations you like and save or display the results. If this would be useful to you, despite you not using the rest of the labscript suite, then it might be worth adding some code to your current setup that will a) turn whatever format your data is in into a HDF5 file in the format lyse expects, and b) send the resulting file to lyse.

To do this, you would have to write something approximately like the following (untested):

import os
import zprocess
import runmanager
import h5py
from labscript_utils.labconfig import LabConfig

h5_filename = "test_shot_file_001.h5"

params = {'laser_detuning': 7.0, 'magnetic_field_on': True}

# Make a HDF5 file and put the parameters of this shot into it
runmanager.make_single_run_file(filename=h5_filename,
                                sequenceglobals=None,
                                shot_globals=params,
                                sequence_id='test',
                                sequence_index=0,
                                notes='',
                                run_no=0,
                                n_runs=1)
                                
with h5py.File(h5_filename) as f:
    # Put all your other data in the HDF5 file here
    pass
    
# Send the hdf5 file to lyse, assuming it is running on localhost:
config = LabConfig()
port = int(config.get('ports','lyse'))
data = {'filepath': os.path.abspath(h5_filename)}
response = zprocess.zmq_get(port, 'localhost', data)
if response != 'added successfully':
    raise Exception(response)

Hope this might be useful!

Chris

--
You received this message because you are subscribed to the Google Groups "The labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuite+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Billington
School of Physics and Astronomy
Monash University
Victoria 3800, Australia
Ph: +61 423952456

[Fred Jendrzejewski]

Hi Chris,

thanks so much for those hints. It actually worked out quite nicely. I see lyse as a good first step to transition from our homemade system into labscriptsuite without disrupting the production cycle to heavily. So I will definitely try to make our datas compatible with lyse.

I am now trying to reconstruct my images into the appropiate hdf5 file. This is actually a lot of puzzling, especially to find out how the pictures are expected etc. Would you have by any chance the super standard example of:

— absorption pics (probe, reference, background)

— thermal fit on this ?

This would save an incredible amount of time and I would be happy to bring it into shape such that it can become part of the lyse examples.

Cheers,

Fred

You received this message because you are subscribed to a topic in the Google Groups "The labscript suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/labscriptsuite/ZKAhU3fQFs0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to labscriptsuit...@googlegroups.com.

[Fred Jendrzejewski]

Hi,

as mentionned your suggestioned worked nicely and I also found a way for the analysis to run on images etc. Now I have one final question. How can I set lyse to listen to the appropiate folder, which is something like:

/Usr/Data/2016-11-14/

here, the last part changes obviously for each day. I have been trying to make this part work as I really like the idea of having the shots automatically added. However, I am still struggling.

Again thanks for the help and awesome job with the program.

-- Fred

[Chris Billington]

Hi Fred,

Glad you made some progress with it!

Lyse doesn't listen to folders, it listens on a TCP socket. If you want to automatically add new shots, some code running somewhere will need to tell lyse about them via this TCP socket (more specifically it's a TCP socket being used via the ZeroMQ library, which is what all the different parts of the labscript suite use to communicate).

So I assume you are creating a folder full of HDF files? If so, then the code that is creating the HDF5 files should also tell lyse about them via the TCP socket, like so:

# Send the hdf5 file to lyse, assuming it is running on localhost:
config = LabConfig()
port = int(config.get('ports','lyse'))
data = {'filepath': os.path.abspath(h5_filename)}
response = zprocess.zmq_get(port, 'localhost', data)
if response != 'added successfully':
    raise Exception(response)

here, the config stuff is just about getting the TCP port number that lyse is listening on. Unless you have changed it, there will be a default configuration file (in labscript_suite/labconfig) with the default port number. Lyse reads the config file to see what port it should listen on, and the above code similarly reads the config file to see what port to send data to. It sends lyse a dictionary containing the absolute filepath to the HDF5 file in question. We then use the zprocess library to connect to lyse and send data, and get a response. Lyse should respond "added successfully" if it worked, so that's what the error checking is doing there.

So, it would make most sense to add the above code to whatever program is creating the HDF5 files - that way you can tell lyse about them as soon as they have been created.

If however for some reason you can't do this, then you could write a standalone program that watched the folder and send any new filepaths it sees to lyse.

Something like this (also mostly untested):

import os
import time
import zprocess
from labscript_utils.labconfig import LabConfig

    
def submit_to_lyse(h5_filename):

    # Send the hdf5 file to lyse, assuming it is running on localhost:
    config = LabConfig()
    port = int(config.get('ports','lyse'))
    data = {'filepath': os.path.abspath(h5_filename)}
    response = zprocess.zmq_get(port, 'localhost', data)
    if response != 'added successfully':
        raise Exception(response)

        

def watch_for_new_h5_files():
    previous_h5_files = set()
    while True:
        # Check shot folder every 1 second for new files and submit them to lyse:
        shot_folder = time.strftime('/Usr/Data/%Y-%m-%d') # Use new date each loop so code can run overnight
        if os.path.isdir(shot_folder):
            h5_files = set(os.path.join(shot_folder, f) for f in os.listdir(shot_folder) if f.endswith('.h5'))
            new_h5_files = h5_files.difference(previous_h5_files)
            for h5_file in new_h5_files:
                submit_to_lyse(h5_file)
            previous_h5_files = h5_files
        else:
            print("Shot folder doesn't exist (yet?)")
        time.sleep(1)
        
if __name__ == "__main__":
    watch_for_new_h5_files()

Hope that helps!

-Chris

--

You received this message because you are subscribed to the Google Groups "The labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuite+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fred Jendrzejewski]

Hi Chris,

this worked beautifully. Thank you very much.

-- Fred

import os
import time
import zprocess
from labscript_utils.labconfig import LabConfig

# Send the hdf5 file to lyse, assuming it is running on localhost:
config = LabConfig()
port = int(config.get('ports','lyse'))
data = {'filepath': os.path.abspath(h5_filename)}
response = zprocess.zmq_get(port, 'localhost', data)
if response != 'added successfully':
    raise Exception(response)

To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fred Jendrzejewski]

Hi,

I recently felt like adventures and decided to update lyse, even trying to run it under python 3, which did not work out. Is this supposed to work ?

Additionially, I tried to reinstall the python2 version with the current labscript installer. However, this did not work either as I always get the following error:

Traceback (most recent call last):
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/fnj/labscript_suite2/lyse/__main__.py", line 2143, in <module>
    logger = setup_logging('lyse')
  File "labscript_utils/setup_logging.py", line 88, in setup_logging
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 156, in __init__
    check_access(self.filepath)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 200, in check_access
    return _zmq_log_client.check_access(filepath, timeout)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 120, in check_access
    raise exc_class(message)
OSError: [Errno 2] No such file or directory: u'/Users/fnj/labscript_suite2/lyse/lyse/lyse.log'

Would anyone ever have seen such an error message ? I checked with a collegue that made his system work in windows and he did not have any issues (but the versions all seemed to match).

Cheers,

Fred

[Chris Billington]

HI Fred,

I've been running lyse with Python 3, and had considered its port to Python 3 complete. If you report any issues you come up against, I'll look into them!

The error you're seeing now is some path issue, I've seen it before. It's something about exactly which folder the current working directory is set to when you run lyse. How are you starting lyse when you get this error, and if you're staring it from the terminal, which folder are you in when you run the command? I suspect this is the same class of errors as we fixed in BLACS by removing all reference to __file__ when computing paths. It can be wrong in Python 2 in some contexts due to an inconsistency between absolute and relative paths, which has been resolved in Python 3.

In the error message, there is an extra '/lyse/' in the filepath to the log, and the zlog server is complaining that the folder doesn't exist, in which it is supposed to create the log file. I could make zlog create intermediate folders in log paths, or better, I should fix whatever has led to the incorrect calculation of the log file path. I do not think the Python logging library creates intermediate folders for log files, so I'm not sure why this issue would be new, I don't think it is unique to us having moved to a logging server (zlog).

In the meantime, creating the folder '/Users/fnj/labscript_suite2/lyse/lyse/' ought to stop the error message from appearing - the only consequence is that the log file will be in an extra subfolder.

-Chris

[Fred Jendrzejewski]

Hi,

Am 24.10.2018 um 17:01 schrieb Chris Billington <chrisjbi...@gmail.com>:

HI Fred,

I've been running lyse with Python 3, and had considered its port to Python 3 complete. If you report any issues you come up against, I'll look into them!

Ok, let me have a shot at it. I might open an extra thread or keep it in here ?

The error you're seeing now is some path issue, I've seen it before. It's something about exactly which folder the current working directory is set to when you run lyse. How are you starting lyse when you get this error, and if you're staring it from the terminal, which folder are you in when you run the command? I suspect this is the same class of errors as we fixed in BLACS by removing all reference to __file__ when computing paths. It can be wrong in Python 2 in some contexts due to an inconsistency between absolute and relative paths, which has been resolved in Python 3.

In the error message, there is an extra '/lyse/' in the filepath to the log, and the zlog server is complaining that the folder doesn't exist, in which it is supposed to create the log file. I could make zlog create intermediate folders in log paths, or better, I should fix whatever has led to the incorrect calculation of the log file path. I do not think the Python logging library creates intermediate folders for log files, so I'm not sure why this issue would be new, I don't think it is unique to us having moved to a logging server (zlog).

In the meantime, creating the folder '/Users/fnj/labscript_suite2/lyse/lyse/' ought to stop the error message from appearing - the only consequence is that the log file will be in an extra subfolder.

alright. This allowed me to get it started. Perfect.

Again thanks a lot for your work.

Cheers,

Fred

-Chris

On Wed, Oct 24, 2018 at 10:44 AM Fred Jendrzejewski <fred.jend...@gmail.com> wrote:

Hi,

I recently felt like adventures and decided to update lyse, even trying to run it under python 3, which did not work out. Is this supposed to work ?

Additionially, I tried to reinstall the python2 version with the current labscript installer. However, this did not work either as I always get the following error:

Traceback (most recent call last):
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/fnj/labscript_suite2/lyse/__main__.py", line 2143, in <module>
    logger = setup_logging('lyse')
  File "labscript_utils/setup_logging.py", line 88, in

 setup_logging
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 156,in __init__
    check_access(self.filepath)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 200,in check_access
    return _zmq_log_client.check_access(filepath, timeout)
  File "/Users/fnj/miniconda2/envs/lyse2/lib/python2.7/site-packages/zprocess/zlog/__init__.py", line 120,in check_access
    raise exc_class(message)

-- 
You received this message because you are subscribed to a topic in the Google Groups "The labscript suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/labscriptsuite/ZKAhU3fQFs0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to labscriptsuit...@googlegroups.com.

[Chris Billington]

Excellent.

Maybe a new thread would be good for lyse Python 3 issues.

-Chris