Shot to shot delay from updating parameters

[Allen Chiu]

Hello Everyone, 

While we are surveying different control platforms, we heard from other labs running labscript that when they sweep through a parameter, shot-to-shot, there's a ~200-500 ms delay. The labs uses NI cards with Pulseblaster or Pineblaster.  Can anyone share if they have similar experiences with this?

I did a small research and I’m speculating a few things that might lead to this 500 ms delay

1) USB interface of the pulsblaster. From people in spincore, they told me they did a small test on it and for a 4k instruction USB interface has a 1 sec load time vs. PCI interface has a 200 ms load time and 2 ms for PCIe interface. Unfortunately, we don't have many pulseblaster boards to try this. The 500 ms delay lab uses usb interfaces.

2) For changing parameters shot-to-shot, does Labscript rewrite the whole instruction into NI cards and pulseblaster? or can it only change/update the parameters changed?

3) Just due to the time it takes to compile and process for a relatively old computer. 

Any comments or suggestions are welcome!

Thanks!

Best, 
Allen

[Lars Pause]

Hi Allen,

we are using labscript with mostly PCI-NI-Cards and a PCIe-Pulseblaster. Our programming time (or "delay" if you would like to call it like this) is roughly 300ms between two shots.

1) I have no exact values for this. I can just tell you that for our Pulseblaster just the programming is max a few ms which fits well with the data you have. What is taking time is the transition_to_manual and transition_to_buffered procedure labscript goes through after each shot. A rough timing guess: Programming takes 50ms of the 300ms.

2) Yes, it is the case that instructions are written newly onto the devices if parameters change. This is because you compile your experiment script for each new parameter.

3) Compilation is done by runmanager (which handles all you variables and compiles the shots). Runmanager is decoupled from blacs (managing your hardware and running the shots) and can in principle run on any other computer from which it loads the compiled shots into blacs. Blacs is then reading the instructions from the shot-files and programming the devices. During this step there will be a timing difference between old and new computers.

What can be a solution to not depend on changing delays between shots for cold atom experiments: Start your MOT at the end of the last shot and use a "filling level" of your MOT/BEC monitored by a camera at the beginning of a new shot to start 'real physics'. This is of course only possible if you do not change MOT parameters.

Lars

[Allen Chiu]

Hello Lars, 

Thanks for sharing! 

I now wonder if it is possible to surpass the loops (Fig 6.4 and 6.6 in Phillip's thesis) it has to go through. I agree these adds stability and sanity checks to the system, but for experiments where cycle times are crucial, we're hoping to minimize the shot-to-shot delays for our tweezer experiment. I'm wondering if we can hack this by directly not going into manual mode in the loop. I wonder if anyone every tried this before. 

Thanks!

Best, 

Allen

[Tomáš Lamich]

Hi Lars and Allen,

I was trying to do a similar thing in our single atom experiment, basically what I did is add a wait at the end of the experiment to see if the atom survived and use lyse to change the globals in order to reflect the state of the atom.

I see that the compilation time of a shot is around 40 ms, analysis time around 25 ms, but the shot to shot delay is 1.17 s. I have no idea where the extra 1.1 s of delay comes from, my suspicion is with the transition procedures and communication between the components of Labscript.

We are using PrawnBlaster through USB as the master pseudoclock and NIDAQ cards through PCI for the testing of the script (we didn't get around to using it in the experiment yet).

As for surpassing the transition_to_manual procedure, I didn't look into that yet.

Allen, you said you are running a tweezer experiment, could you please specify a bit more why is the shot to shot delay so important to you? 

In our experiment we need to reload the atom quite often, so the shot to shot isn't as crucial as long as we don't want to recycle the atom.

Have a great day everyone!

Tom

--
You received this message because you are subscribed to the Google Groups "the labscript suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to labscriptsuit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/bc0b4f62-958c-42d8-aed0-e8ea7f9cf2b7n%40googlegroups.com.

[Ian B. Spielman]

One thing I would like to add to this discussion the FunctionRunner device that is designed to help with this problem. This allows you to run a script at the end of a shot, but before transition to manual and before invoking lyse. The idea here is that if there are time critical computations (such as those that inform a upcoming shot) they should be performed here without the overhead associated with lyse.

— Ian


Ian B. Spielman

Fellow, Joint Quantum Institute
National Institute of Standards and Technology and the University of Maryland

----- WEB -----
http://ultracold.jqi.umd.edu

----- EMAIL -----
spie...@jqi.umd.edu

----- ZOOM -----
https://umd.zoom.us/j/7984811536

----- PHONE -----
(301) 246-2482

----- MAIL -----
UMD:
2207 Computer & Space Sciences Bldg.
College Park, MD 20742

NIST:
100 Bureau Drive, Stop 8424
Gaithersburg, MD 20899-8424 USA

----- OFFICE -----
UMD: Physical Sciences Complex, Room 2153
NIST: Building 216, Room B131

> To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/CAE_aAjLS2mVReRih9F_KmCa_%3DV%2BG9L6PqWSUjWU%3DRuP_Hwwv%3Dw%40mail.gmail.com.

[Zak V]

Hi all,

For those who are curious about what limits their shot repetition rate, I'd suggest taking a look at the blacs log file. The log messages have timestamps which can help you figure out how long different steps take, which device is the slowest, and so on.

I might also add that some (but not all) devices support smart programming so that their instructions aren't reprogrammed if they are the same as in the previous shot, even if the instructions for other devices have changed. When I first set up Labscript in my old lab we had an ancient Pulseblaster with very outdated firmware that took a long time to program. I found that if I scanned over durations of some stage in our sequence, the transition to buffered duration in blacs was ~5 seconds or more because the Pulseblaster program had to be changed each time. However, if I just scanned the voltage of some NI card output, the pulseblaster didn't need to be reprogrammed so the transition to buffered duration was much shorter and the experiment ran much faster. So if you have one slow device which supports smart programming, you can sometimes be a bit clever about the ordering of your shots to reduce programming time. For example, when doing a 2D scan of duration and output voltage we always scanned over duration in the outer loop and output voltage in the inner loop to minimize the number of Pulseblaster reprogrammings required.

We eventually had our Pulseblaster's firmware updated and then it was no longer the rate-limiting device even if its instructions were changed. After doing that and optimizing our custom PCO camera labscript device code, we were able to get our transition to manual time down to 150 ms to 200 ms if I recall correctly. I've forgotten how long transitioning to manual took at the end of a shot, but I think it was less than that, so the total time between the end of one shot and the start of another was typically a few hundred milliseconds.

Cheers,

Zak

[Tomáš Lamich]

Hi all,

I tried to look into what Zak suggested, and I found that the limiting factor isn't firmware of the devices we use (prawnblaster and NIDAQ cards) but handling the h5 file before the shot, which takes around 1 second (shortened timestamps below). Does anyone have any suggestions on how to speed it up? The experimental sequence I am using is attached to this email. 

In addition, is it possible to use the function runner to extract the timeout status of a specified wait? I could figure out how to access the information since it is not written in the h5 file and as of now, I do not have analogue inputs available on my card.

Thank you very much for your help and have a great day,

Tomas

Simplified timestamps:

2022-07-06 11:31:29,436 INFO BLACS.queue_manager.thread: All devices are back in static mode.

... processes form last to next shot

2022-07-06 11:31:29,437 INFO BLACS.AnalysisSubmission.mainloop: Processed signal: file

2022-07-06 11:31:29,463 INFO BLACS: received filepath: Z:\Experiments\test_apparatus\feedback_tst\2022\07\06\0131\2022-07-06_0131_feedback_tst_0.h5

2022-07-06 11:31:29,463 INFO BLACS: local filepath: C:\Experiments\test_apparatus\feedback_tst\2022\07\06\0131\2022-07-06_0131_feedback_tst_0.h5

2022-07-06 11:31:29,463 DEBUG BLACS.ConnectionTable: Parsing connection table from C:\Experiments\test_apparatus\feedback_tst\2022\07\06\0131\2022-07-06_0131_feedback_tst_0.h5

2022-07-06 11:31:29,469 INFO BLACS: Request handler: Experiment added successfully

2022-07-06 11:31:30,443 INFO BLACS.queue_manager.thread: Got a file: C:\Experiments\test_apparatus\feedback_tst\2022\07\06\0131\2022-07-06_0131_feedback_tst_0.h5

... preparing the h5 file

2022-07-06 11:31:30,474 DEBUG BLACS.DIO_card.mainloop: Processing event _transition_to_buffered

... programming the devices

2022-07-06 11:31:30,523 DEBUG BLACS.queue_manager.thread: About to start the master pseudoclock

2022-07-06 11:31:30,524 DEBUG BLACS.prawn.mainloop: Processing event _start_run

2022-07-06 11:31:30,525 DEBUG BLACS.prawn.mainloop: Instructing worker main_worker to do job start_run

... running the shot

2022-07-06 11:31:30,627 INFO BLACS.queue_manager.thread: All devices are back in static mode.

... ending the shot

2022-07-06 11:31:30,657 INFO BLACS: Request handler: Experiment added successfully

2022-07-06 11:31:31,634 INFO BLACS.queue_manager.thread: Got a file: C:\Experiments\test_apparatus\feedback_tst\2022\07\06\0132\2022-07-06_0132_feedback_tst_0.h5

To view this discussion on the web, visit https://groups.google.com/d/msgid/labscriptsuite/84102b1b-303b-4a90-b293-8561250d4e7bn%40googlegroups.com.

[Philip Starkey]

Hi Tomas,

It looks like in that log file you are not adding a new shot to BLACS until the previous has finished. In that case, it's likely the delay is caused by this sleep statement: https://github.com/labscript-suite/blacs/blob/75ef8612aa170c16a8b9c3e7c59157cd16e2ef29/blacs/experiment_queue.py#L542

Feel free to change that to a fraction of a second (e.g. 0.01) and you should hopefully see the delay decrease. This won't change the time between shots if you have more than one in the queue at a time though as the sleep statement should not be executed.

Cheers,

Phil