combining multiple ROI-thresholded streams into one serial output

[franz kuchling]

Hi everyone,

I recently started using Bonsai for my project, which involves putting small light sensing organisms in different channels and measuring how fast they go towards the light. Each channel (I have 18 of them observed by 6 cameras, so 3 channels per camera) has 2 individually controllable LEDs, one on each side. 

The goal is to have my bonsai script to detect, based on a ROI threshold, trigger the Arduino to switch the LEDs of one channel once sufficient organisms reach one side.

My first attempt is attached with separate channels, which works somewhat for one channel, but I don't know how to scale it up:

Camera capture is cropped into 2 different region-channels, a threshold is set and pixels counted and exported into an output stream.

Based on a condition of pixels greater than 80% I pass each output stream along to a skip while based on lesser than 80% (this is duplicate but was necessary to restart the stream after skip while was triggered, better suggestions are welcome! I don't really understand how to restart streams very well). Each channel is than transformed into an output of 1 or 0, written to the Arduino serial port to determine if the the lights need to be turned on on the left or right side. This works fine as long as not both ROIs are own (in  which case the lights flicker), which is fine for now, but better suggestions are welcome here too!

The problem is now how to scale this up to 18 channels! I attached an attempt to combine channels as well, where the idea was to zip up the channels as integer inputs and just add them up, which can then be decoded by the Arduino. 

The problem with this is that it doesn't add it up if only one stream is active...

Ideally what I was thinking was that with 18 channels you essentially would just encode this as an 18 bit problem, so each channel would be, based on which side has its ROI activated, be one bit of 1 or 0, so if for example all 18 channels where activated by the left side, you just pass along one serial stream of the number 262143, which is the integer representation of an 18bit of 1s. Naturally this would have to be dynamic, so if one channel flips, then somewhere in the 18bit you flip a 1 to a 0 and pass that number along to Arduino for it to decode.

I hope this all makes sense, any suggestions are welcome, whether along the lines I was thinking or an entirely different way to solve the problem.

Many thanks in advance and have a great day,

Franz

[brunocruz]

I am not sure i fully understood the protocol so I am just going to try to help you with bits and pieces of specific problems you mention :P

"Based on a condition of pixels greater than 80% I pass each output stream along to a skip while based on lesser than 80% (this is duplicate but was necessary to restart the stream after skip while was triggered, better suggestions are welcome! I don't really understand how to restart streams very well). Each channel is than transformed into an output of 1 or 0, written to the Arduino serial port to determine if the the lights need to be turned on on the left or right side. This works fine as long as not both ROIs are own (in  which case the lights flicker), which is fine for now, but better suggestions are welcome here too!"

I am not sure i followed the logic. If you just want to turn on the light if a condition is met you just connect the output of your comparator (i.e. `GreaterThan(x)`) to your event (i.e. sending the string via serial). If you just want the toggles of this event you do add a "DistinctUntilChanged". If you want to reduce the flicker, you could use some sort of node that implements a low-pass filter on the output. The `Throttle` operator might be a good place to start.

" Ideally what I was thinking was that with 18 channels you essentially would just encode this as an 18 bit problem, so each channel would be, based on which side has its ROI activated, be one bit of 1 or 0, so if for example all 18 channels where activated by the left side, you just pass along one serial stream of the number 262143, which is the integer representation of an 18bit of 1s. Naturally this would have to be dynamic, so if one channel flips, then somewhere in the 18bit you flip a 1 to a 0 and pass that number along to Arduino for it to decode."

This sounds like a sensible thing to do. For simplicity I would just use a 32bit representation which allows you to use a native type (int32) [unless there is some sort of 24bit data type i am not aware of?]. 

A simple way to achieve what you are describing is to output a default value of 0 when the channel should be off or a int value corresponding to the channel number (2^n, where n = channel number). Simply zipping a boolean (on/off) and the corresponding (2^n value) and passing them into an ExpressionTransform(it.Item2 * (it.Item1 ? 1 : 0) ).

 

Then, given the probably assynchronous nature of the update process, I would just use a combineLatest->Add to pool across all branches and sum all the integers inside (I would group them in groups of 3 and then groups of 6, since the tuple size limit is 7 at the moment).

I am not sure if this will be of much help, but perhaps you could expand a bit more the logic you are trying to get.

Cheers,

 Bruno

[franz kuchling]

Hi,

Thank you for your detailed response, and sorry for my late response back, I was out ill for a while and couldn't work on this. I have a couple of follow up questions where I didn't quite understand or wasn't sure how to implement your suggestions:

On Friday, May 27, 2022 at 3:48:20 AM UTC-4 brunocruz wrote:

I am not sure i fully understood the protocol so I am just going to try to help you with bits and pieces of specific problems you mention :P

"Based on a condition of pixels greater than 80% I pass each output stream along to a skip while based on lesser than 80% (this is duplicate but was necessary to restart the stream after skip while was triggered, better suggestions are welcome! I don't really understand how to restart streams very well). Each channel is than transformed into an output of 1 or 0, written to the Arduino serial port to determine if the the lights need to be turned on on the left or right side. This works fine as long as not both ROIs are own (in  which case the lights flicker), which is fine for now, but better suggestions are welcome here too!"

I am not sure i followed the logic. If you just want to turn on the light if a condition is met you just connect the output of your comparator (i.e. `GreaterThan(x)`) to your event (i.e. sending the string via serial). If you just want the toggles of this event you do add a "DistinctUntilChanged". If you want to reduce the flicker, you could use some sort of node that implements a low-pass filter on the output. The `Throttle` operator might be a good place to start.

 

The problem I had with this simple passing of the output to the serial is that it either just passed the integer I specified after my GreaterThan if that was met, or using, the DistinctUntilChanged, passed nothing at all. What I really wanted is for it to pass 0 if GreaterThan is not met, and 1 if GreaterThan is met. But I think your answer below gets at that, though I haven't quite managed to implement that.

" Ideally what I was thinking was that with 18 channels you essentially would just encode this as an 18 bit problem, so each channel would be, based on which side has its ROI activated, be one bit of 1 or 0, so if for example all 18 channels where activated by the left side, you just pass along one serial stream of the number 262143, which is the integer representation of an 18bit of 1s. Naturally this would have to be dynamic, so if one channel flips, then somewhere in the 18bit you flip a 1 to a 0 and pass that number along to Arduino for it to decode."

This sounds like a sensible thing to do. For simplicity I would just use a 32bit representation which allows you to use a native type (int32) [unless there is some sort of 24bit data type i am not aware of?]. 

A simple way to achieve what you are describing is to output a default value of 0 when the channel should be off or a int value corresponding to the channel number (2^n, where n = channel number). Simply zipping a boolean (on/off) and the corresponding (2^n value) and passing them into an ExpressionTransform(it.Item2 * (it.Item1 ? 1 : 0) ). 

 

1) So is ProcessingOutput here just my CameraCapture apture up to GreaterThan branch?

2) If that is true, then, as above, this doesn't change the boolean and int afterwards, no matter what my processing output is, does it? 

3) Can you please elaborate on the ExpressionTransform? I didn't catch this at all, do you mean scripting a C# Transform node? If so, I haven't gotten into the  scripting yet, could you point me to some resources of I could accomplish this? 

 

Then, given the probably assynchronous nature of the update process, I would just use a combineLatest->Add to pool across all branches and sum all the integers inside (I would group them in groups of 3 and then groups of 6, since the tuple size limit is 7 at the moment).

I think this makes sense, I should be able to do this after I understand the other parts above. 

I am not sure if this will be of much help, but perhaps you could expand a bit more the logic you are trying to get.

Regarding the logic I am trying to get, for thirst first stage, for each channel the logic would be the following:

1) Each channel with organisms inside and LEDs on each side has two ROIs being captured by a Camera, one ROI on each side.

2) Each channel would pass (through the 32bit operator we discussed above) one bit of information to the arduino, where 1 corresponds to the organism(s) being detected in one ROI, and 0 if on the other.

3) The arduino interprets each bit by switching the LEDs on opposite of the side where organisms where detected in the ROI. 

4) Each time an ROIs are triggered so that the bit per channel flips, the time of this event is recorded into a csv file (I haven't mentioned this above yet because I assumed it should be fairly simple once everything else stands) 

 

Many thanks again for taking the time in helping me with this and apologies if I'm being thick, I am still very new to this.

Best,

Franz

[brunocruz]

1) So is ProcessingOutput here just my CameraCapture apture up to GreaterThan branch?

That would work. That said, I would probably add a distinctUntilChanged to make sure you dont send duplicated events...

2) If that is true, then, as above, this doesn't change the boolean and int afterwards, no matter what my processing output is, does it? 

Sorry, i should have been a bit more clear here. The "Boolean" is meant to represent the output of step 1, i.e. the output of the "GreaterThan" node. I just left it there to make it explicit.

3) Can you please elaborate on the ExpressionTransform? I didn't catch this at all, do you mean scripting a C# Transform node? If so, I haven't gotten into the  scripting yet, could you point me to some resources of I could accomplish this? 

You can simpy use an ExpressionTransform instead of a c# node. This node allows for small expressions to be used without much knowledge of c# language. 

The one I used above: "it.Item2 * (it.Item1 ? 1 : 0)", for instance, will check what is the value of it.Item1 (the first element of the tuple, ie. the boolean). If this value evaluates to True, it will output a "1", else it will output a 0. It will then multiply this value by "it.Item2" (i.e. the channel number). In other words, if Boolean is True: return ChannelID; else: return 0;

Then, given the probably assynchronous nature of the update process, I would just use a combineLatest->Add to pool across all branches and sum all the integers inside (I would group them in groups of 3 and then groups of 6, since the tuple size limit is 7 at the moment).I think this makes sense, I should be able to do this after I understand the other parts above.

There are probably other ways to make this more scalable. For instance, if you use a DistinctUntilChanged, you could simply keep suming the output of every single node that gets updated as either +ChannelID or -ChannelID (simply changing the "0" in the ExpressionTransform to a "-1"), depending on the state of that channel. I attached a small example of how this might work. 

I hope this gets you started :)

Cheers,

Bruno

[franz kuchling]

Brilliant, thanks!!!

I've attached my now working example (again many thanks to Bruno!) in case anyone is interested with explanations below for 1 camera and 2 channels (so 4 ROIs). 

I ended up not actually using the ExpressionTransform, not because it doesn't work (it does), but because I wanted the stimulus to switch to the other side once one ROI was activated. 

For that, I found an example on this forum that just sets a repeat loop of sequentially doing one task for one ROI and then switches to the other:

https://groups.google.com/g/bonsai-users/c/KowhBkOEAeQ/m/kxQtQnqkAAAJ

Quick bonus question: Could anyone please explain why using and Amb node that select the first sequence to react out of two observable sequences, followed by a repeat, stays on the first sequence instead of again checking both which one goes first? Just in case I didn't want to initialize which side the organism has to go first in order to activate a trigger.

Main workflow (for now just for the first camera, which here captures 2 channels simultaneously by different ROIs):

Arduino Update receives integers once from each channel once an ROI has been activated, accumulates all together and passes it to the Serial Port to the Arduino, which decodes it bitwise to set the LEDs in each channel. (Arduino code attached as well)

The Add here just adds 1 to the total 32 bit in, the reason being that Serial.parseInt in arduino returns back 0 immediately after a Serial String has been sent,  negating the setting of the number, so I make sure it's always at least 1 and then just substract 1 again when decoding bitRead. 

Camera 1 Workflow:

4 ROIs are cropped, pixels after thresholding are counted and published for ROI Activation below. 

For each channel, a repeat loop checks first one side, and once Condition is met, passes it on to the other side.

Channel 1 workflow for one side Roi1A:

Condition contains the GreaterThan Threshold that will trigger sending of the channel ID as bits 2^n (so 2, 4, 8, etc....) to Arduino Update

Channel 1 workflow for other side Roi1B:

multiplies by -1 to remove the count in accumulate like Bruno suggested above using ExpressionTransform

Any further suggestions for improvement are welcome, especially regarding the above mentioned issues with setting the repeat loop such that you don't have to fix the order in which it first has to get activated.

Best and again many thanks,

Franz