workflow for simultaneous Intan RHD2000 and image processing

[Polina Litvak]

Dear Goncalo Lopes,

I am using Bonsai for motion detection of rodent in ROI (in a maze) while collecting physiological data using a IDS uEye camera and Intan hardware (RHD2000).
I'd like to know how I can add a mark in my .rhd file when I detect motion in the ROI  (needed for off line data analysis) or if there is another good way to synchronize the two.

Also, what's the best way to learn about the various nodes in Bonsai.Ephys package ? I am trying to go through the Ephys sample workflow you've provided here but have difficulties deciphering it.

Many thanks,

Polina

[goncaloclopes]

Hi Polina,

If you are using an IDS uEye camera, you may be able to configure it to pulse one of the GPIO ports using a strobe configuration every time a frame is acquired. You can then just connect the GPIO pin to one of the ADC ports in the OpenEphys and the pulses will show up in the board ADC data matrix. This is actually my preferred way for synchronizing video with Ephys with millisecond precision.

If you do it this way, then you can just save the results of the ROI tracking to its own file, and you can then rebuild exactly when each frame happened by looking at the ADC values. I never did it with the IDS personally, but it looks very similar to the strategy you would use with the PointGrey cameras. You should look into the IDS manual and search for "flash strobe" settings or something like that, they should have example circuit wiring diagrams.

If you do it later in software, you will be adding latencies and extra jitter between the ephys times and camera times...but I guess it's possible to add extra information to the RHD data file by using CombineLatest. Conceptually, it would look something like:

The idea is that every time you get a packet of ephys, you also get whatever is the latest result of the image processing and bring it along. You can then save it to the same or to different files.

As I said, the problem is that this can have the jitter on the order of 10 ms in the alignment, give or take because of USB and other intervening buffers... again, I really recommend for precisely synchronizing video and ephys to go the GPIO route and use a hardware sync pulse.

How are you recording the IDS in Bonsai? Are you using the VideoCaptureDevice node or a dedicated IDS node?

[Polina Litvak]

Thanks for your reply, I am recording with CameraCapture node.

You are saying that by synchronizing using camera's GPIO I am guaranteed that the time stamp saved in ROI tracking results file will match the time stamps showing up in the ADC matrix, correct ?

So what would the basic workflow look like then ? Should Rhd2000 and CameraCapture nodes be in two parallel branches or should they be connected sequentially ?

Best,

Polina

[goncaloclopes]

Actually, you don't even need any timestamps in this case, because the GPIO pulses are recorded directly on the same clock as the Intan acquisition board. More importantly, it is important to have access to the hardware frame counter in order to keep precise track of dropped frames. In case you don't have hardware frame counters, then I guess you can still use Timestamp to make sure you're not drifting.

The strategy I am mentioning is actually all implemented in hardware, so from the point of view of Bonsai you just record your data as normal. There just so happens to be an ADC channel that will signal whenever a video frame is acquired. For example if you take this channel and count the number of pulses, there should be as many as there are frames. In this case, then you know that the 3rd pulse happened when frame #3 happened, etc.

If you're using an IDS camera, I recommend you use a dedicated node for that. I think at some point João shared an experimental package with support for these cameras that included the frame counters... (João, do you still have it?)

I can take a look tomorrow. Anyway, you can still test the synchronization without this, even by just using the IDS user-interface. They should allow you to configure the flash strobe and turn it on and off. If you record from the BoardAdcData of the Rhd2000EvalBoard node during this time, you should see one of the channels reflect the pulse periodically. They probably have instructions on how to setup and wire everything in their user manual, but I've never done it with an IDS camera myself.

[Polina Litvak]

I am having a hard time setting up my IDS uEye camera to send a pulse (using strobe configuration) every time a frame gets acquisition. I've connected the GPIO pin on the camera to one of the ADC ports in the RHD2000 Intan board but don't see any pulses showing up in the board ADC data matrix. If anyone had better luck with this or can point me to the right source on the web, please share your expertise.

Thank you!

On Tuesday, April 26, 2016 at 2:10:15 PM UTC+2, Polina Litvak wrote:

[goncaloclopes]

Hi Polina,

Which GPIO pin are you using? If you are using GPIO pin 1, which is the default strobe output, please keep in mind this output pin is opto-isolated. This means that you actually need to connect the pin across a pull-up resistor to an external voltage in order to get it to work. Opto-isolated pins are used to keep external circuits electrically isolated from camera circuits, to prevent damage to camera electronics from voltage or current spikes. However, because the two circuits are effectively isolated electrically, you need to always have some kind of external voltage source.

If you don't want to deal with this, you can also just use pins 3 or 4, which are not opto-isolated (careful that you need to change your ground pin also to match these outputs). You should be able to treat these as normal "wires", but bear in mind that their TTL is 3.3V rather than 5V.

Let me know if you have any difficulties.

[Polina Litvak]

Hello Gonçalo,

Thanks for your reply. I am now using pin 3 (like you've recommended) and some tick marks now show up in Intan's data matrix. I have a few questions though:

1. What would you say the flash strobe duration should be set to in order to line up well with the video (fps) ? I have it currently set to 1000 milliseconds. 

2. What's the default frame rate for Bonsai's uEye camera node ? Can it be changed ?

3. You've mentioned I should activate and record the hardware frame counter from the camera (to keep track of skipped frames) - I am having a hard time finding this setting on the camera (uEye usb 3.0). Any input is greatly appreciated!

Another important aspect is how to best set up Intan recording - is it best to control it from within Bonsai workflow using the specialized Intan node and add it as a branch parallel to video acquisition or should Intan recording be started externally first, and then the Bonsai workflow (which would do video, ROI detection and etc.) Which way would be easiest for later data analysis ? 

Many thanks.

Polina

On Tuesday, April 26, 2016 at 2:10:15 PM UTC+2, Polina Litvak wrote:

[goncaloclopes]

Hi Polina,

Great to hear you got it to work. Actually, my original advice was for PointGrey cameras, I forgot you were using an IDS uEye camera! I don't know what their pin configuration is, but if you see the strobe data, then I guess it should be working...

1. Regarding the strobe duration, again I don't know about IDS, but for PointGrey, if you set the duration to zero it actually makes the strobe be the same as shutter exposure time, which is usually slow enough that you'll see the pulses clearly on ephys samples and gives you maximum information about image acquisition.

2. If I remember correctly, the Bonsai IDS node takes a uEye configuration file as input. Whatever settings you have saved in the configuration file will be used for acquisition, including frame rate and any other settings. You can generate these files using the provided IDS uEye camera viewer.

3. You can find the frame counter in the ImageInfo.FrameNumber property (right-click on the IDS node). You may need to activate it in the configuration file using the uEye camera viewer, but I don't remember the menus by heart.

Finally, regarding initialization order, yes, this will be critical for easy analysis. In general, you want to start ephys acquisition first and only after start video acquisition. Now this depends actually on how the uEye camera operates with regard to strobing.

Can you check if the camera keeps sending pulses to the ephys even if there is no active acquisition going on? This happens with PointGrey and makes everything much more complicated as you have no way of properly controlling the start of acquisition with software... but let's worry about this when we get there, there are a number of possible workarounds.

Hope this helps.

[Polina Litvak]

Hello Gonçalo,

Thanks for your reply. What I am confused about is how do we make the camera aware of the presence or absence of ephys given that they should be started sequentially and not in parallel, from two distinct workflows? While troubleshooting the pin configurations, I've connected the camera's GPIO to an oscilloscope and hence can confirm that given my current configuration, the camera sends pulses when I start video acquisition regardless of ephys. So what is the best way to proceed to make data analysis as straight forward as possible ?

Many thanks,

Polina

[goncaloclopes]

Hi Polina,

That's good to know, you're in luck! This means you can control the acquisition strobe through software :-)

In that case, a workflow like this should be enough:

Basically you can use SubscribeWhen to control when a branch of the workflow starts running. If you use the ephys data itself as trigger, this will start the camera when the first packet of ephys data is acquired, which should be enough to ensure the sequential order.

Hope this helps.

[Polina Litvak]

Hello Gonçalo,

Thanks for the attached workflow. Is there a way to introduce a delay after Intan starts but before video kicks in ? For some reason I am unable to link my ImageSource node to a plain Delay node. The reason I'd like to try this is because the total number of frames I am seeing is larger than the total number of pulses I am tracking being sent from the camera, which doesn't make a lot of sense. Could it be that the time to start Intan and actually track GPIO is so long that there are already some frames recorded ? Would a delay of a sec or so between starting Intan and camera solve this ?

Many thanks, as always!

Polina

On Tuesday, April 26, 2016 at 2:10:15 PM UTC+2, Polina Litvak wrote:

[goncaloclopes]

Hi Polina,

Interesting, it's worth figuring out why this happens. It can be a delay in initiation, but it can also be a matter of dropped frames somehow. Did you check the hardware frame counter coming from the camera frames? Are the numbers always sequential? If you never dropped any frames, the difference between frame counts should be always one. If it is bigger, at least you will know when and how many you dropped.

Depending on your hardware setup, frame rate, resolution and processing load, it is not so uncommon to drop some frames unfortunately. In my case I was using 120 fps image acquisition with PG and I would drop on average around 1-2 frames every 30 minutes, which is kind of already pretty good in my opinion. However, it is crucial to know exactly when these drops happen or else interpreting the pulse data is impossible!

About delaying the initiation of the node, I think you probably want to use DelaySubscription instead of Delay.

Let me know what you find out, I'm curious to know more about how to get this to work with uEye IDS cameras.