Closed-loop Arduino control

[c.joseph...@gmail.com]

Hi all,

This is maybe more a Bonsai-to-Ardunio question, as I'm still somewhat new to both Bonsai and Arduino. I'm trying to use DeepLabCut tracking to conditionally turn on or off Arduino-controlled optogenetic stimulation. I've attached a pilot workflow in Bonsai (still trying to figure out the Arduino script - see below), and I've described what I'd like to do below. 

I have a stable DeepLabCut model capable of tracking two different animals, allowing for output to video and csv for post-processing. From the model, at each frame I'm extracting a body part from each animal (giving x- and y-coordinates), then zipping the positions together. To determine how close the points are I'm taking the difference between the two coordinates, giving them an absolute value, and applying a less-than Boolean giving a True (1) statement only when the distance along a dimension (x or y) is less than my threshold (my pilot for this has been 30 pixels). Up to this point, all of this works. I get confused as to how to use this to send an output to the Arduino: I need to determine when conditions of both Booleans to be true (i.e. the x- and y-coordinates are below threshold) before turning on stimulation. The easiest way to do this hardware-wise seems to be by USB connection, so I'm wondering whether I should directly send the results of the Boolean as a serial output and let the Arduino handle the determination of the conditional, or if Bonsai should conduct more processing here and send a simple Go/No-Go signal to the Arduino. Or should I bypass serial output entirely and try to generate direct TTL input to the Arduino, and if so how that would work.

Overall, my questions:

1. Is there a simpler way to accomplish the determination of body-part position and thresholding how close the body parts are in space?
2. What would be the easiest method of communicating with the Arduino using the comparison of body part positions? And how would I program the Arduino to handle these inputs?

If any other details would be helpful here, let me know.

Thanks all!

-Joey

[brunocruz]

Hi Joey,

Without knowing more about your experimental needs, what you are doing seems like a reasonable approach. When controlling optogenetic light sources I would stay away from generating the stimuli using Bonsai (and communicating with Arduino with firmata) since we need millisecond precision. With that in mind there are a few possible solutions. The two easiest ones I can think of:

1. Do what you are probably already doing. Code your time sensitive logic in Arduino and use serial communication to turn it on and off by sending a character from Bonsai.

2. Use an arduino running firmata (look in the forums there are several examples) and use it as a slave to send digital triggers to a second pulse generator hardware (e.g. pulse pal, another arduino, etc...).

Now, on to your actual question. If I am understand your experiment correctly, the easiest thing is to run all the logic in Bonsai and simply send a turn on/off character to Arduino. If you just want to check if both conditions are met, replace the "CombineLatest" by a "Zip". Your two streams are tied to the same frame, as a result for each frame you are guaranteed to have paired  "synchronised" values, so you can simply use "Zip". To check if both conditions are met you can simply add a "BitwiseAnd" node in front of Zip which compares pairs of values. If you want to use this output to filter some event, add a condition node (either after BitwiseAnd or even encapsulating the BitwiseAnd inside the condition node). This event can then trigger a character to be sent to arduino (solution 1) or a digital output (solution 2).

I hope this helps,

cheers,

B

[Giuseppe Giannotti]

Regarding your second question: did you try a python node in the bonsai workflow? you could use a python node with an IF statement (if value is lower than threshold) to generate a string to send to arduino. On arduino you could readstreang in loop and generate the TTL (pin HIGH) when that specific string is received.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/50424f34-ac3c-4f8b-98ad-cded7622fc25n%40googlegroups.com.

[c.joseph...@gmail.com]

I got it working! As you both recommended, within Bonsai I used a Python transform to handle the BitwiseAnd statement and exported a character using SerialStringWrite; the Arduino then only has to turn on/off stimulation based on the incoming character.

Thanks for your help, I really appreciate it!