how to import connection_table

[Andreas Trenkwalder]

Hi all, I would like to import "connection_table.py" into each experiment script file. The reason is, that we have many different experiment scripts (>100) due to historic reasons andevery time we are changing the name of a digital or analog channel, or when we change the calibration (unit conversion), we would need to update each experiment script, which is impractical.

I tried following line in the experiment script:

"from labscriptlib.FPGA_test.connection_table import *"

But when I compile this in "runmanager" I get the exception at the bottom which tells me that a file has been imported twice. I get "first time imported, as labscriptlib.FPGA_test" and "second time imported, as user_devices" which are both folders and not files.

The connection_table.py is located in the experiment folder "labscriptlib/FPGA_test/" where also my experiment script "FPGA_test.py" is located.

The device class is defined in "user_devices/FPGA_device.py" which is a PseudoClock device with a single hard-coded Clockline and analog and digital channels connected to two classes of IntermediateDevice. Probably, FPGA_test.py is the file which is imported twice? but it seems that above line is executed twice which I guess causes the error.

I tried other variations of above code to import connection_table but with not much luck.

So my question is: how can I import "connection_table" into an experiment script file?

Many thanks,

Andreas Trenkwalder

here the exception:

----------------------------------------------------------

Traceback (most recent call last):

File "/home/andi/labscript-suite/userlib/labscriptlib/FPGA_test/FPGA_test.py", line 22, in <module>

from labscriptlib.FPGA_test.connection_table import *

RuntimeError: Double import! The same file has been imported under two different names, resulting in two copies of the module. This is almost certainly a mistake. If you are running a script from within a package and want to import another submodule of that package, import it by its full path: 'import module.submodule' instead of just 'import submodule.'

Path imported: /home/andi/labscript-suite/userlib/labscriptlib/FPGA_test/namespace

Traceback (first time imported, as labscriptlib.FPGA_test):

------------

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 94, in <module>

batch_processor = BatchProcessor(to_parent,from_parent,kill_lock)

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 43, in __init__

self.mainloop()

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 50, in mainloop

success = self.compile(*data)

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 73, in compile

exec(code, self.script_module.__dict__)

File "/home/andi/labscript-suite/userlib/labscriptlib/FPGA_test/FPGA_test.py", line 22, in <module>

from labscriptlib.FPGA_test.connection_table import *

------------

Traceback (second time imported, as user_devices):

------------

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 94, in <module>

batch_processor = BatchProcessor(to_parent,from_parent,kill_lock)

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 43, in __init__

self.mainloop()

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 50, in mainloop

success = self.compile(*data)

File "/home/andi/Andi/programming/labscript-suite/.venv/lib/python3.6/site-packages/runmanager/batch_compiler.py", line 73, in compile

exec(code, self.script_module.__dict__)

File "/home/andi/labscript-suite/userlib/labscriptlib/FPGA_test/FPGA_test.py", line 22, in <module>

from labscriptlib.FPGA_test.connection_table import *

File "/home/andi/labscript-suite/userlib/labscriptlib/FPGA_test/connection_table.py", line 12, in <module>

from user_devices.FPGA_device import FPGA_board, DigitalChannels, AnalogChannels, PRIMARY_IP, SECONDARY_IP, DEFAULT_PORT

------------

Compilation aborted.

[Zak V]

Hi Andreas,

Does using `import_or_reload()` instead of `import` resolve your issue? To use it, add something like the following at the top of your labscripts:

```

from labscript_utils import import_or_reload

import_or_reload('labscriptlib.FPGA_test.connection_table')

```

Cheers,

Zak

[Andreas Trenkwalder]

Hi Zak, thanks for the fast response!

No, unfortunately using import_or_reload() gives the same exception as import, except that now importlib.import_module is used.

I am not very much familiar with python modules but from the error it seems that user_devices is a module and labscriptlib as well and both need a reference to FPGA_device which is part of user_devices. 

Is it correct that there is an empty __init__.py file in the labscriptlib folder while there is none in the user_devices folder?

Maybe I should move FPGA_device (my device class) into another folder outside of user_devices and labscriptlib and import it from both modules? Python should allow to use the same file from several modules?

Maybe a good place is pythonlib? 

The labscribtlib has also a "common" subfolder with empty __init__.py file.

I can test this later.

For completeness, I use Python 3.6 on Ubuntu LTS 18.04 and Pycharm Community 2021.1

Many thanks!

Andreas

--
You received this message because you are subscribed to a topic in the Google Groups "the labscript suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/labscriptsuite/4y-ndD3qZ-0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to labscriptsuit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/ea3e4f1e-769b-4205-a4aa-0ccf9e0d0c51n%40googlegroups.com.

[Chris Billington]

Hi Andreas,

Your files look to be in the right place. One this issue is fixed, you will need to use import_or_reload(), otherwise the import will only work the first time you compile a shot, but that's not the issue you're facing yet!

This looks like it might be a bug in the "double import denier", which is some code designed to prevent a mistake that I don't think you're making. Normally it detects when the same file has been imported under multiple names, but here it seems to be firing thinking that a file in labscriptlib is being imported twice, when that doesn't look to be what your code is doing - the second import of some stuff from user_devices is not the same file.

And in fact that 'file' is not a file at all, it's "labscriptlib/FPGA_test/namespace". I assume you do not have a file called "namespace" there.

I suspect this is a misbehaviour of the "double import denier" code in the presence of "namespace packages".

Could you let me know where out of the following you do and don't have __init__.py files, empty or otherwise?

/userlib/labscriptlib/__init__.py

/userlib/labscriptlib/FPGA_test/__init__.py

/userlib/user_devices/__init__.py

Also, are you by any chance doing any manipulation of the Python import path, as in, modifying sys.path in-code, or setting the PYTHONPATH environment variable?

Creating more empty __init__.py files wherever you don't have them already might fix the issue. But I'd be interested in which ones' absence might be causing the issue, since then I might be able to fix the underlying problem!

Regards,

-Chris.

You received this message because you are subscribed to the Google Groups "the labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAA-%2BackiCumNfi_nntDNmrXi2H%2BomjdPJFt%3DYXkRWYwipVgV4g%40mail.gmail.com.

[Andreas Trenkwalder]

Hi Chris, thanks for taking care of this issue!

I confirm I do not use any "namespace" or manipulate the python path. I also checked before that there are no circular references (like importing connection_table from FPGA_device).

Below I mark where empty __init__.py files exist with * and ! where there is none:

/userlib/labscriptlib/__init__.py *

/userlib/labscriptlib/FPGA_test/__init__.py !

/userlib/user_devices/__init__.py !

i.e. only one init file exists in these folders.

When I add an empty __init__.py file to /userlib/user_devices/ or to /userlib/labscriptlib/FPGA_test or both I always get "NameError: name 'primary' is not defined" which is the error I get when connection table is not loaded ('primary' is the name of the first PseudoClock device). I use import_or_reload as suggested.

I can make a minimal example to be sure it's not a problem in my code.

Many thanks!

Andreas

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAKy3cVvB9w8xczVjB5GZGacAOzhqLPkV1T%2Bmtx2-J19v%3D8FazQ%40mail.gmail.com.

[Chris Billington]

Hi Andreas,

That seems to have resolved the double import issue, now it looks like we're back in the realm of having to ensure the connection table module is reloaded, what import_or_reload is intended to achieve.

Have you restarted runmanager, or at least it's compilation subprocess (there's a "restart subprocess" button), to clear out whatever messed up interpreter state the previous issue might have left behind?

If so and it still doesn't work, a minimal breaking example would be excellent.

Thanks,

Chris

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAA-%2BacmwpJ9mF6vW7L_EWR3HhsbpFKqUYHORPmGYijp-04YnJA%40mail.gmail.com.

[Andreas Trenkwalder]

Hi Chris,

yes I restarted runmanager between each test. 

Using DummyPseudoclock and DummyIntermediate device works with import_or_reload but this is a different scenario since it does not involve user_ device.

I will provide a minimal example using the code of one of the standard labscribt devices (maybe PulseBlaster) as custom device.

Thanks,

Andreas

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAKy3cVuYjjTHWL_eS_ME4A3H8U_YXVpkSuH0DRFWUEL8_kzriA%40mail.gmail.com.

[Andreas Trenkwalder]

Hi all,

the problem is solved: 

1. add an empty __init__.py file into the folder /labscribtlib/<experiment> which contains the experiment script and connection_table.py 

2. in experiment script use following instead of import:

from labscript_utils import import_or_reload

import_or_reload('labscriptlib.<experiment>.connection_table.

3. restart blacs and runmanager. Maybe open/save/close __init__.py to have it with the actual time and date. Maybe repeat this step twice?

4. the last step is not clear what to do since after steps 1-3 it still did not worked. I heavily modified connection_table.py, experiment script file, device class file and register_classes.py and then it worked.

I sometimes see that after a code change the old code is still executed. Maybe the Python Cache is not always updated although the source has been modified? This might explain why steps 1-3 did not immediately work.

What I actually did after steps 1-3 was: I copied the code of PulseBlaster (with modified class name to force python to use my code and not the labscript device) and modified all step 4 files for this. Then it worked since I still had the __init__ files in the /labscribtlib/<experiment> and /userlib/user_devices folders. removing both __init__ files generated the "Double Import" exception again. Adding __init__.py to /labscribtlib/<experiment> solved the problem - now immediately!!

Many thanks to Zak and Chris!

Andreas

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAKy3cVuYjjTHWL_eS_ME4A3H8U_YXVpkSuH0DRFWUEL8_kzriA%40mail.gmail.com.

[Lars Pause]

Hi Andreas,

hi Chris and Zak,

when reading your conversation I realized, that conceptually we do the same as Andreas since we started with labscript but without importing connection_table.py directly. However I am not sure if a) our way is helpful for Andreas or any other labscript-user as an alternative and b) it is a good way to go. So here is what we do to avoid changing both connection_table.py and experiment-script-files:

Our connection_table.py-file looks like this:
from labscript import *
from labscriptlib.ct_build import *

build_full_connectiontable()

# Experiment logic
start()
config.supress_mild_warnings = False
stop(1)

The head of each experimentfile looks the same, just having the experiment code in between start() and stop(t)

On top we have this ct_build.py-file saved under userlib/labscriptlib/ where we import all devices and define all connections for them.

def build_full_connectiontable():
    build_pulseblaster_connections()
    build_pci_6713_1_connections()
    ...

and e.g build_pulseblaster_connections() look like:

def build_pulseblaster_connections():
    PulseBlaster_SP2_24_100_4k(name='pulse_blaster')
    ClockLine(name='A1_clockline', pseudoclock=pulse_blaster.pseudoclock, connection='flag 0')
    DigitalOut(name='pb_flag8', parent_device=pulse_blaster.direct_outputs, connection='flag 8', default_value=0)

It basically outsources the device definitions into an extra file which can be imported by both experiment scripts and the connection_table-file. In my opinion this only counteracts the labscript-idea of having all connections of your experiment defined in connection_table.py and only the specifically used connections described in the header of your experiment script.

Lars

[Andreas Trenkwalder]

Hi Lars,

sorry for my very late response.

Thank you very much for your alternative solution!

I have tested it and it works perfectly.

I have tested a similar approach when trying to solve my problem but it failed since I did not define a function to regenerate the connection table on each call. If you only import you get after the second and further compilations with runmanager the error: 'No toplevel deviced and no master pseudoclock found'.

You could also use the alternative solution of Chris in the experiment script file:

from labscript_utils import import_or_reload

import_or_reload('labscriptlib.ct_build') 

where now you execute the code not anymore within a function.

The only disadvantage of your approach compared to "my" solution is that BLACS does not tell you that connection_table has been changed.

Greetings,

Andreas

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/6ec54133-eb6a-4f69-9877-48f2bd193e49n%40googlegroups.com.