PointGray Blackfly USB 3.0 Camera keeps sending GPIO strobe out pulses even after the aquisition was stopped.

[Evgeny Resnik]

Hey guys,

I am  going to use PointGray Blackfly USB 3.0 Camera for video tracking animals in synch with the OpenEphys.
Video tracking will be done with Bonsai, which receives all the metadata (Timestamps,FrameCounter) from the camera.
In order to avoid blinking LED-synchronization, my plan was to send strobe_out digital pulses from the camera GPIO pins directly to the OenEphys.
Planning that i assumed that the camera generates these digital output pulses only when frame acquisition is initiated by Bonsai and running.
Unfortunately the camera turned out to keep sending the pulses even after i stop the bonsai pipeline (or stop video acquisition in the FlyCap2 software).

I wonder whether it is normal behavior of these PointGray  cameras?
Can one change settings in a way that it stops sending pulses when the acquisition is stopped?


Because of this problem the camera pulses will be recorded by the OpenEphys even before the actual Bonsai tracking starts.
A way to overcome this i see at the moment would be to connect an Arduino to Bonsai, which will send a start/end TTL pulses to the OpenEphys.
But ideally i would like to avoid additional elements....

Thanks!

[Evgeny Resnik]

I'v found info on that myself :)

The issue has been already documented in a knowledge base of PointGray:
"GPIO strobe signal continues after isochronous image transfer stops"
https://www.ptgrey.com/KB/10078

[goncaloclopes]

Yes, this is a known problem, unfortunately. One way to work around it is to activate the camera's trigger mode. When the camera is set to trigger mode, it stops acquisition until a trigger is read. In this mode, no strobe pulses are sent out. You can then turn off trigger mode and the strobe pulses and acquisition will resume again.

You can do this manually, or more recently I've also been prototyping a way to set this dynmically from the PointGrey node. I'll look up where that version is and I can try sharing that with you.

[jorge.cla...@gmail.com]

Hi Gonçalo,

We have a similar challenge to solve, not related with end of acquisition but the start of acquisition process.

Is there a way of toggling trigger mode, TTL output control or camera power GPIO registers inside Bonsai?




Cheers,
Jorge

[goncaloclopes]

Hi Jorge and Evgueny,

I recovered the experimental version that I developed for a colleague. It adds a new property "ToggleTrigger" to the PointGrey source. If set to "True", the source will toggle the trigger mode when starting and toggle it again when stopping.

Keep in mind that this is highly experimental and I can give no guarantees at the moment that the API will remain this way in the future, but it may provide you with a quick workaround for the moment.

I have to admit that when performing my own recordings I actually toggled the Trigger mode manually in the FlyCapture dialog every session. The reason was that doing it automatically raised more complicated timing constraints for initializing the setup and I preferred to make sure that everything was stable before starting the acquisition. Anyway, I understand the convenience and advantage of automating it of course.

More generally regarding PointGrey and other camera driver settings, I opted early on to try and avoid as much as possible replicating configuration settings in these nodes. The reason is that most packages allow you to configure the camera driver settings from their own UI and then either save a configuration file (IDS) or keep them saved in camera memory (EEPROM). There are usually a lot of these settings, at least dozens and dozens of possible configurations that can change very fast across driver updates and camera manufacturers. Being a single developer, it would be extremely hard to keep all these settings up-to-date and working across all camera models and versions and it would make adoption of new features and cameras much slower.

Also keep in mind that it is possible to modify the source code of the Bonsai plugin itself for your purposes. You can find inside the attached symbols package the source code for the node which you could in principle modify in order to configure dynamically any setting that you might specifically need (I did this myself in the past for implementing cool features such as controlling exposure on a frame-by-frame basis for high-dynamic range photography, etc).

In the future, I am considering the option of providing a more direct access to the camera object so that it would be possible to script these changes in, but this will probably not happen in the immediate short-term.

Hope this package can help for now and thanks for the feedback!

[Evgeny Resnik]

Hi Gonsalo,

Great, thanks a lot for the plugins!!
How should i properly add them to bonsai library?
Just copying to the folder C:\Users\evgeny\AppData\Local\Bonsai\Packages\Bonsai.PointGrey.2.1.1 didn't work....

Thanks!
Evgeny

[Gonçalo Lopes]

https://bitbucket.org/horizongir/bonsai/wiki/InstallCustomPackageTutorial
Don't forget to select "Include prerelease" in order to have it show up in the list.

---

From: Evgeny Resnik
Sent: ‎09/‎05/‎2016 07:54
To: Bonsai Users
Subject: [bonsai-users] Re: PointGray Blackfly USB 3.0 Camera keeps sendingGPIO strobe out pulses even after the aquisition was stopped.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/2291beff-42ae-431a-a85d-f79fa6e42102%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Warren]

Hi,

I have attempted to synchronize my ephys and pointgrey camera using the scheme described in this post, but I've run into an issue. I often lose frames at the very beginning or the very end of acquisition. This is problematic because I can't use the frame counter metadata to resolve which frames were lost (e.g. if the first 10 frames are dropped you will not see the frame counter skipping frames, which is otherwise necessary to identify where missed frames have occurred). In other words, I think it may not be possible to tell whether frames were lost at the beginning or the end of the recording, which makes synchronization impossible when frames are lost at the beginning or end. My solution has been to trigger frame acquisition with externally generated TTLs. Throughout the session I put brief pauses in the TTL, which effectively divides the session into epochs. I can then perform the synchronization within each epoch, which avoids this issue for the most part. This is working, but seems pretty inelegant. Is anyone else experiencing frame loss at the beginning or end of acquisition? Is there some way to tell whether frames were lost at the beginning or end in this circumstance?

Thanks,

Rick

[Gonçalo Lopes]

Hi Richard,

This may be a clunky workaround, but you can try the following sequence:

1. Tick the checkbox that sets the PointGrey camera to trigger mode (FlyCap2 -> Settings -> Advanced)

2. Start the FlyCapture bonsai source. This should initialize the camera but block waiting for frames.

3. Uncheck the camera from trigger mode. Frames should start being acquired.

4. Before stopping the workflow, set the camera to trigger mode again.

5. Stop the workflow.

6. Uncheck the camera from trigger mode one last time.

Let me know if by following this method you can account for all frames.

The problem with the PointGrey cameras running in free-mode is that they keep grabbing frames even when no application is looking for them. TTLs only stop if the camera is in trigger mode.

I may consider adding a way to soft trigger the camera start to work around this issue, but let me know first how it works for you.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/452b978f-2a3e-47af-90cb-45b98245ea00%40googlegroups.com.

[Richard Warren]

Thank you for the quick reply. I tried this method initially, but was losing frames at the beginning and/or end of recording and was unable to use the camera metadata to resolve whether the frames were lost at the beginning or the end.

To be more concrete, in Bonsai I acquire a certain number of frames, and in my acquisition software I record a certain number of exposure TTLs that is greater than the number of frames in Bonsai due to frames being lost during acquisition. I then locate missing frames using the camera metadata by identifying successive frames where the difference between frame counts is greater than 1. Ideally, the number of acquired frames + the number of detected missed frames would equal the number of exposure TTLs recorded. However, if you drop frames at the beginning or end of a recording, I can't find a way to identify / locate these missed frames using the camera metadata, because there will be no identifiable frame count differences that are greater than 1. I hope I've explained this well. Perhaps there is an easy way to resolve this problem that I've missed?

Thanks,

Rick

[goncal...@gmail.com]

Did you confirm that you are not getting TTLs in the ephys system while the trigger mode is checked?

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/aadb6704-b406-47ba-ad78-20e60d72b107%40googlegroups.com.

[Richard Warren]

Yes, there are only TTLs when it is not in Trigger mode. I observe the same behavior of losing frames in the beginning and/or end when triggering frame acquisition with TTLs from an Arduino. It commonly drops tens of frames at the very beginning (at 250 fps). I suspect all of the dropped frames are at the beginning, with none at the end, but I haven't been able to verify this. If they are all at the beginning than I could reconstruct the frame times easily, but I'm reluctant to make that assumption.

[goncal...@gmail.com]

I think I remember a colleague reporting a similar issue with specific versions of the FlyCapture drivers. Which version of FlyCap are you using?

 

I will try to search for that email and report back if I learn any clues.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/a4e69d6d-dca6-4693-ba94-42d4e035d607%40googlegroups.com.

[Richard Warren]

Excellent, thanks. I'm using FlyCapture version 2.5.3.4.

[Gonçalo Lopes]

Hey Richard,

I've found the e-mail about the delay initializing the camera. Apparently this happened while updating from FlyCap 2.9.3.11 to 2.10.3.169, which are both newer versions than the one you have.

To be honest, now that I think about it, that version seems quite old. Some of the older FlyCap versions also had issues with triggers. The oldest version that I know works stably is 2.6.3.2.

Also, for the record, can you let me know what is the firmware version installed in your camera? This can also influence trigger and initialization behaviour. It should show up in the camera selection screen when you start up FlyCap.

Unfortunately PointGrey (now FLIR) makes it very hard to keep track of software versions, as they only make available in their website a couple of the latest ones.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/a60a1125-315b-4e1a-89ce-451bac1609fd%40googlegroups.com.

[Richard Warren]

I've updated FlyCapture to the newest version, 2.11.3.164, and the firmware on the Grasshopper3 camera to the latest, 2.22.3.0. I've been testing again by manually toggling the trigger mode off and on at the beginning and end of recordings. Unfortunately I am experiencing the same issue. Namely, after toggling the trigger mode off the camera begins acquiring frames, and I begin seeing exposure TTLs in my acquisition software. However, Bonsai only starts seeing these frames with some delay, usually 1-4 seconds. In the most recent test Bonsai misses the first 560 frames. Importantly, this is the same behavior I observe when using the camera in trigger mode, with Bonsai often missing the fist chunk of frames. So I think this is not a problem of camera initialization. The issue seems to be that Bonsai takes a moment to 'wake up' and realize there are frames coming in sometimes... What do you think? Thanks again for the help.

Rick

[Gonçalo Lopes]

That is strange indeed, I have to say I have never observed this behavior. The Bonsai source is actually quite simple, all the code comes from PointGrey's own C# API, so they control how frames are acquired, etc.

If I was having this issue myself I would try to run the C# examples that come with their API and see if the issue is still there. If it was, I would contact PointGrey's support directly and have them deal with it.

Unfortunately I'm not sure how to help much more than this... what happens if you turn on the TTLs, let the camera "warmup", then turn them off for a while, and then turn them on again? Do you see a second "warmup"?

By the way, can you share the workflow you are using for the acquisition? I just want to make sure that there are no other potential side effects there.

Do you see the dropped frames even if there is only a single FlyCapture source in the workflow?

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/b8f03893-ae12-43e4-aa65-52484e36358a%40googlegroups.com.

[Richard Warren]

Thanks for the reply. I tried the 'warmup' idea, and it worked in one test, but the occurrence of this bug is very sporadic so its difficult to draw conclusions without repeated tests. Most recently, in tests where I didn't loose initial frames, I actually stored one more frame in Bonsai than there were TTLs in my acquisition software.

I've attached my workflow, as well as a simplified test workflow I also used that ended up exhibiting the same behavior.

At any rate, it seems this bug is peculiar to my setup/CPU, so I don't want to bother you too much about it. I'll run some more careful tests in a couple weeks and if all else fails I have a functioning work-around. If I externally trigger frame acquisition, and introduces brief pauses in the TTLs at key moments throughout the experiment, I can reconstruct >99% of the 'epochs' created by the TTL breaks.

Thanks again for the help, for which I'm grateful.

Rick

[Gonçalo Lopes]

Thanks for the update. Keep up posted if you ever figure out what is going on.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/970dda61-d78c-48c6-aa59-b1197642b809%40googlegroups.com.