Some basic questions ( closed-loop demo experiment)

[Bruno Cruz]

Hey everyone, 

I need some help with a couple of problems i am having with bonsai ^^

1. (more out of curiosity) can i use the small NI USB board with bonsai? (this one ). This is because when i try to connect with it using the standard NI module it gives me an error about the clock signal value (which i am assuming its because this one doesnt have an internal clock).

2. I am trying to prepare a little experiment. The overall goal is to deliver a stimulus in close loop with the movement of the subject. So far i have successfully tracked the animal and calculated the difference in pixels from the image as a proxy of the total movement of the animal ( i might use a python node to compute the distance if this does not work). Afterwards i want to sum this index for a given interval of time (e.g. 1 sec) and if it is greater than X, give the stimulus with a 50% probability(catch trials and what not). The stimulus will be generated from FunctionGenerator node, so i guess i need some kind of 'Gating signal', however i cant seem to combine a combinelatest with function generator and feed it into the DigitalOutput of Arduino or National instruments board.

After delivering this stimulus, i add a T delay which will prevent another stimulus from being delivered during this period, after which the workflow will keep running.

Let me know if it is not clear :)

Thanks,

Bruno

[goncaloclopes]

Hi Bruno,

1. I would assume it should work, although I never tried directly. Do you get meaningful values out? I would let that be the ultimate judge of whether it works or not.

2. For this it would be helpful to go over a concrete workflow, but I think there are a number of questions embedded here that I will try to clarify:

a) Integrating measures over time: this is a great example where sliding windows may be useful. You can create automatically sliding windows in Bonsai by using ElementCountWindow followed by SelectMany. Inside the SelectMany you can Sum over all the elements in the window and you'll get your sliding integration over time. This is very similar to Figure 3D in the Bonsai paper, but using Sum instead of Average (you can also use whatever you want). I'm reproducing the idea below:

And inside the SelectMany you have the Sum for each Window:

You can also just hook up a PythonTransform and do a recursive running average (e.g. result = 0.99 * input + 0.01 * result) or any other transformation filter, but I decided to use this as another example for sliding windows using SelectMany :-)

b) Probabilities: there is no built-in random number generator in Bonsai yet, but it's easy enough to cook up one using Python scripting. Here's an example for a PythonTransform that spits out a uniformly distributed random number (between 0 and 1) for every input tick:

from System import Random

generator = Random()

@returns(float)

def process(value):

  return generator.NextDouble()

c) "Gating": now this is where I was a bit confused. You mention using FunctionGenerator for generating a digital output for the Arduino, but you shouldn't need this at all since DigitalOutput takes a Boolean value directly (i.e. True or False). You can simply run your condition test on the movement integration signal (e.g. with a GreaterThan node) and then feed the result of the comparison directly to DigitalOutput.

Now this is in case you just want to turn something on directly in the Arduino depending on a condition, but maybe what you meant is to generate a stimulus train when some event happens?

In this case then maybe Arduino will not be good enough, depending on timing resolution needed for the train. If you want to do it with the Arduino I would recommend running a dedicated sequence in the microcontroller to ensure precise timing, using solution 3 mentioned in this post. Otherwise you should maybe use some programmable stimulator like PulsePal or a proper NI-DAQ PCIe card (I don't know what is the timing resolution in the small USB boxes, but I was under the impression it wasn't as good as the regular NI-DAQ blocks).

Anyway, in case you do need to use a buffered output, then I would recommend designing two waveforms using FunctionGenerator and/or ScalarBuffer, one with the stimulus train and other other with a constant zero value, then CombineLatest with your criteria. You can then use a simple ExpressionTransform to flip between them, as in the following example using the X-coordinates of the mouse:

Hope this helps.

G

[Bruno Cruz]

hi!

Ok so i got it 'kinda' working. But i am starting to better a better idea how bonsai is suppose to work.

First of all the window into select many is super useful and worked like a charm! Second the number generator is also working great. The place where i had more trouble so far was actually getting the pulses working.

Right now i am using arduino so send a digital output to the NI board that then triggers the Laser ( i am not so worried about getting perfect timing right now). So to trigger it works great, the problem i am having is actually to give him a 30sec idle period, that is: after giving a stimulus, how do i prevent other stimulus from being triggered in Bonsai. Right now i am doing this by having the signal coming to the laser last for 30sec which prevents it from being re-activated even if it receives a trigger from Bonsai. I know its not the best way but seems to work so far =)

Anyway thanks for the help! and if someone knows how to implement these timers let me know ;)

B

[goncaloclopes]

Hi,

I think the workflow below can be used as a basis to implement this kind of logic:

In this example there is some kind of condition tested on MouseMove (e.g. if the X position is greater than some value), but we want to impose a refractory period after each positive result.

One way to do this is to use the sequence of combinators DistinctUntilChanged, a simple boolean Condition and GateInterval. In order to understand why, it's easier to go in over what each node is doing to the data stream (it also helps to visualize what each node is sending out):

1) After GreaterThan, we have a constant stream of Boolean values for each mouse movement (this could easily be a camera or any other digital signal).

2) DistinctUntilChanged will now filter the stream such that new values will only be emitted when they are different from the previous value. For example, applying DistinctUntilChanged to the sequence [False, False, True, True, True, False, False, False] will generate the values [False, True, False].

Basically you get the edges of the boolean stream.

3) The Condition simply filters the stream by saying we only want rising edges (i.e. positive, or True values). This works because we know that when DistinctUntilChanged sends out a True, it must mean the signal just changed from False to True; a positive rising edge). Usually it is necessary to specify what is the condition inside the green nested workflow (like an IF statement in programming languages), but in this case because the data stream is already made of Boolean values, we can just place the Condition node in there and it will be happy (Booleans are their own conditions).

4) GateInterval is an interesting beast: as the name implies, it gates the signal according to a time interval. What this means is that after the first value goes through, the gate is closed by the specified amount of time (i.e. any values arriving during that time will be dropped). After the time elapses, the gate is opened again and any new value will be allowed through, and so on.

Basically in the end you will have built an event source with a refractory period, which you can then feed into a SelectMany to trigger a stimulation sequence, etc.

Hope this helps.

[Jay McNally]

Hi,

I am attempting to use Bonsai in a similar manner to that described by Bruno at the beginning of this thread. Essentially what I am trying to do is to trigger a stimulus pulse via threshold detection, and then wait a certain period of time following the stimulation before applying further stimulus pulses. Currently I am using an approach similar to that outlined in Goncalo's post from Aug 1st. 

     


For my protocol, "Source.X" is a voltage reading from a Intan Rhd2000, and the output format is listed as "double". This output is fed into the Greater Than -> DistinctUntilChange -> Condition -> GateInterval series of nodes as in the figure above. The Boolean output of this series is then fed into a digital output node, which allows an arduino to trigger a pulse simulator. The output of this node is then fed into a python transform node which switches the boolean output True -> False, which is then used to turn the arduino's digital output back off.

This approach works reasonably well, but I am have some issues with the precision of the Gate Interval timing. I have the interval set to 30 seconds (00:00:30), which should in theory stop my protocol from triggering a stimulation for 30 seconds after a stimulation has been given. The gate seems to be functioning, because it does block data that passes the Greater Than node for a certain period of time. The problem is that this period of time seems to change from trial to trial. Have any of you seen such an issue before with the Gate Interval node, or is there perhaps another issue with my approach that I am not seeing that might be causing this issue?

Thanks for any help you can provide,

Jay            

[goncaloclopes]

Hi Jay,

Indeed, you're right that GateInterval may not be the most reliable way to implement a refractory period. In fact, I can see 3 possible ways to implement it now, and this even turned out to uncover an important limitation of the current version that I will fix in the upcoming Bonsai 2.2, so thanks for going deeper with this question!

Here are the three ways to interpret a "refractory period":

1) Refractory period as an absolute periodic "switch"

This was the original answer to the question. The way to think about what Gate is doing here is to visualize it literally as a "gate" or "switch" that opens for a single element to pass through, and then immediately closes until some "master" signal tells it to open again. Once the gate is open, it will remain open until a new element arrives. In the case of GateInterval, the master is a periodic temporal signal. In your case, you can imagine that there is a clock firing every 30s to signal the Gate should open.

The problem with this approach, which you detected, is that the clock is not necessarily aligned with the production of elements. It literally fires every 30s, regardless of when the event actually happens. Now, if you imagine that Gate is in the open state but the event only arrives 27s later, the Gate will only close for 3s, because after that a new clock signal will arrive and make it open again. Clearly not what you might want from a "refractory period".

2) Refractory period as an enforced "silent period" between two trials

This one uses directly the simple Gate combinator, which allows us to specify what the "master" signal is. Here what we are saying is that Gate does not open periodically, but rather it opens when a specified "silent period" elapses. This can be detected using the Throttle operator. Throttle is another combinator used to control the rate of events in a data stream, and the way it works is that it drops all events except if they are followed by a specified silent period. Imagine detecting pauses when typing at the keyboard. In this case, for example, the Gate would open when you stop typing for 30s.

The possible issue with this interpretation is that the "silent period" starts counting from the last event, not necessarily the one that made it through the gate. This means that if your subject is impatient and can never stop completely for 30s, he will never get another stimulation trial.

I can imagine this being useful actually, and maybe it's what you want. If not, there's the final interpretation.

3) Refractory period as an enforced "dead period" starting from the trigger event

This one seems easy at first glance, but there's a catch, which led me to the limitation I was mentioning at the start of the post. The idea here is that you stem the flow of data completely for a given interval, and then resume. Take(1) specifies that only a single element from the data stream will be taken, everything else will be ignored. After this you add a Delay equal to your desired "dead period". After the delay elapses, the sequence will complete, because there is nothing else to propagate. At that point, we want to Repeat the whole sequence, i.e. again Take a single element, Delay for 30s, and so on. In this case, you want to add your stimulation sinks between Take and Delay.

This is exactly what we wanted. The catch is that if your workflow has only one branch, Repeat will restart the whole workflow, including shutting down the data source and reinitializing it again. If this is a camera or an acquisition board, you may not want to do that as it can disrupt continuous data acquisition. Now this is where it gets tricky. I may be missing something obvious, but it doesn't look like there is an easy way to avoid this on the current version of Bonsai apart from doing the following "hack":

What I've done here is add a single branch after the point that I want to avoid reinitializing. The branch does nothing (MemberSelector by itself is a "no-op"). However, branching in Bonsai has a side effect we can exploit in this case: specifically, it guarantees that until all branches are dead, the source will not be reinitialized. Repeat only applies to the top branch, so because the bottom branch remains alive, the source is prevented from reinitializing.

This should implement is perhaps the most "intuitive" implementation of a refractory period. I definitely don't like the need for this hack, so I will consider adding variations of Publish to allow implementing this scenario more easily for Bonsai 2.2.

Thanks again for the feedback and I hope this helps.

[Jay McNally]

Thanks, Goncalo!

I have tried replacing the Gate Interval portion of my workflow with the Take -> Delay -> Repeat (w/hack) method you described above. Simply following the output of the Take node, this looked like it might work. However, after I added back the nodes for turning on and off my Arduino's digital output  (DigitialOutput -> Python Transform Node ->DigitalOutput) used to control the stimulus generator, my refractory period still seems to be quite variable. Similar to what I described with the GateInterval the stimulus generator is often triggered before the full 30s of the delay period has elapsed, or a stimulus pulse is generated without a discernible event that would have made it through the GreaterThan node. As it seemed to be working before inclusion of the ardiuno controls, perhaps my issue is with the way I incorporated the digital output controlling nodes? Is it an issue to use two digital output nodes to turn on, and then turn off the output as I do in the attached workflow?   

Thanks in advance for any further advice.

- Jay  

[goncaloclopes]

Hi Jay,

Looking at your workflow, I know what's going on and yes, your intuition that it is the outputs is correct.

You can actually use as many output nodes as you want, it turns out that this is not the problem. Rather, the Arduino nodes have a particular way of working that is probably not clear at the surface, so I'll take the time to explain it and hopefully why the fix works will also become clear.

As you already noticed, whenever you use Arduino digital or analog inputs and outputs, you can set them to share the same COM port. However, there can only be a single connection open to a specific COM port at any given time, so for this to work, these nodes have to share information. Bonsai handles this for you behind the scenes using a shared manager object. The way this works is that whenever there is a subscription to an input or output node, Bonsai checks whether the connection is already open. If it is, it reuses the connection, if not, it creates and opens the connection. This means that the first node to ask for a subscription actually triggers the opening of the connection.

But how about closing connections? Somehow Bonsai needs to decide whenever a connection is not used anymore. The way this is done currently is that we also track when subscriptions are terminated. Whenever all of the subscriptions to a given connection are terminated, Bonsai will close the connection. This allows you to open and close connections to different Arduinos dynamically by controlling when node subscriptions start and stop. Usually for sequential pipelines you never notice anything, because the connection is simply open all the time.

However, in this case we introduced the Repeat node. The behavior of the Repeat node is interesting, because what it does is listen to when a sequence is complete (in this case, when the Delay node fires), and at that time it terminates the old subscription and creates a new one. This is crucial because it means the two output nodes will be closed, which in turn will close the connection to the Arduino. When the new subscription is created, the Arduino connection will open again.

The problem is that connecting to the Arduino actually takes time (around two seconds, because of limitations of the Firmata protocol). This is the jitter you are seeing. Typically you want to open the connection at the beginning of the workflow and keep it open throughout.

The way the connection lifetime is specified is planned for improvement, precisely because of issues like these, but this is part of a deeper reorganization of the way Bonsai works that will only be implemented in Bonsai 3.0 (which will probably not be out until next year). For now, the workaround is to add an AnalogInput or DigitalInput source node elsewhere in the workflow. This will ensure that the connection to the Arduino remains open no matter what happens with the Repeat node.

I hope the explanation made sense. I'm sorry that all this feels a bit clunky, but thanks for pushing Bonsai in this way, these are exactly the use cases I will reflect upon to determine how the future architecture should be specified, so it's very useful. Adding the extra inputs should not add any significant pressure to the system, since Firmata is already streaming them anyway (they are simply not going anywhere).

Hope this helps. Let me know how it goes.

Best,

[Jay McNally]

Thanks for the above suggestion.

Adding the extra digital input node did the trick!

The timing of the stimulus generation in workflow works well now.

[黄康]

I've been looking for such answer almost the whole day!!!!, thank you guys, Bonsai is great!

在 2015年8月2日星期日 UTC+8上午11:02:19，goncaloclopes写道：