high frequency events

[Mostafa Safaie]

Hi all,

Is there an upper bound for the frequency of incoming hardware interrupts/events? I realise that time resolution in PyControl is 1ms, does that mean that hardware interrupts with higher rate would cause problems?

Bests,

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

There  will be an upper limit on the rate at which pyControl can process external events, but exactly what this rate is will be quite context specific, depending on factors including whether the event rate is continouous or a short burst and what the task does in response to each event.  I have not characterised this in detail but am likely to do so for the next revision of the pyControl manscript.  My current understanding is that two different input events happening within 1ms of each other would be no problem, but a continuous stream of input events at 1KHz would probably overload the frameworks ability to process them.

What is the practical use case you are thinking about?

best,

Thomas

[Mostafa Safaie]

Hi Thomas,

I just wrote an SPI interface for a mouse sensor (PMW3360DM-T2QU) to track 3D treadmill movements. It has a hardware interrupt for motion detection that I couldn't manage to get to work. In that case, the events would be bursts of high rate events (when motion occurs) and I imagined that's the reason why it doesn't work (and that's when I asked the question), which makes sense based on your comment.

It does however work perfectly if I poll the motion registers every 1ms (and not every 0.1ms—I didn't try any interval in between).

Thanks for clearing that up!

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa

To integrate this sensor you might be able to use a similar approach to the Rotary_encoder, which inherits most of its functionallity from the Analog_input class (defined in hardware.py).  The analog input reads an analog pin at a specified frequency, with the sample read happening on a callback triggered by a micropython hardware timer (these are different from the pyControl timer functionallity, and allow a function to be called regularly at a specific frequency with much less overhead).  Samples are stored in a buffer and transmitted to the GUI once the buffer is full (a pair of buffers is used so the reads can continue without waiting for the full buffer to be transmitted).  A threshold can be defined with framework events generated when the signal crosses it in the rising and/or falling direction, and you can also access the current value of the signal from task code.   The Rotary_encoder class inherits from the Analog_input class, changing the initialisation code and replacing the read_sample method, to read data from a rotary encoder rather than analog pin.

Integrating your sensor in this way would let you take advantage of existing code for transfering analog signals to the GUI at high rates (not tested extensively but I think the upper limit is between 5 and 10KHz), triggering events on threshold crossings, displaying signals in the GUI, and saving data.   The current analog input functionality is only for 1D signals, but if you can get it working for one component of the sensor signal, and the approach looks like it would work for your application, it should be possible to extend the functionallity to multidimensional signals, and I would be happy to help with this.  There are restrictions on memory allocation in timer callbacks, but I think it should be possible to read from the SPI as long as you read the data into a preallocated buffer (this might be why you were having trouble using hardware interrupts, as they also have the same memory allocation restrictions, see here for more info). 

best,

Thomas

PS: I am on holiday this week so probably won't get back to further emails before next Monday.

[Mostafa Safaie]

Hi Thomas,

Thanks for all the tips. I spent some time understanding the Analog_input and Rotary_encoder logic, and midway through adapting the code for my application, I gave up:

- As I understand, the trick in Rotary_encoder is using the alternative function of timers. However, in my case, that's impossible, since I need to detect the threshold crossing of an interrupt signal in order to read the motion registers off of the sensor.

- Still, since the data that needs to be stored is 4 bytes  (2 bytes for delta_x and 2 for delta_y) I could just use the analog input to detect the threshold crossing in the interrupt signal and then read the registers and return the values back to back (delta_y+delta_x) so they could be stored in the buffers, but then the problem is that _timer_ISR() uses the last value of the buffer to detect thresholds.

- I tried overloading _timer_ISR() but I couldn't get the event to be detected in the GUI. As soon as I tried to interface with the sensor, even the threshold crossing detection stopped working. The sensor interface only uses utime.sleep_us() and a memory buffer (a list of bytearrays called self.motionBuffer) defined in the class and accessed in the ISR (I attached the code snippet in case you want to have a look).

At this point, I stopped going down the rabbit hole :D

I think it is more straightforward to directly set up a hardware timer at 1kHz and access the sensor registers through a callback (basically polling), instead of using the interrupt signal.

Now my question is, how can I raise a GUI event without using the Pycontrol timers, to avoid the overhead that you mentioned? Or is there another alternative that I'm missing?

Looking forward :)

Thanks,

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

I think I did not explain myself very clearly in my previous email .

- I was not proposing that you use the hardware timer alternate functions to read the sensor - these are used by the rotary encoder class (but not the analog input class) because there is special functionallity implented in the micropython hardware timers for decoding quadrature signals, but this is not relevant to your sensor.

- I am proposing that you poll the sensor value at a specified frequency (e.g. 1KHz) using a hardware timer.  However rather than implementing this yourself from scratch, I am suggesting that you use the machinery in the analog_input class (which also polls its input regularly at a specified frequency using a hardware timer), in order to stream the data back to the computer, visualise it in the GUI, and generate framework events on threshold crossings. 

If you send your PMW3360DM class definition, I can send an example of what I am thinking of.

best,

Thomas

[Mostafa Safaie]

Hi Thomas,

I realised using the alternate functions is not possible, but I still tried to use the threshold crossing mechanism for the sensor interrupt line. Like your suggestion, I now just want to use a hardware timer for polling.

In this repo: https://github.com/BeNeuroLab/PyTreadmillTask under /devices/_PMW3360DM.py, PMW3360DM() is the sensor class, and the read_pos() method reads the motion registers (FYI, the test_motion_sensor_2 branch contains my attempt at subclassing Analog_input).

Looking forward to seeing what you have in mind.

Bests,

Mostafa

[Mostafa Safaie]

My new attempt is here :)
https://github.com/BeNeuroLab/PyTreadmillTask/blob/test_motion_sensor_3/devices/_PMW3360DM.py#L241 (not tested yet)

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

The attached code shows how I would try to integrate your sensor class with the pyControl Analog_input class.  It defines a Motion_detector class which inherits from Analog_input, adds additional init code to instantiate the sensor, and overwrites the analog input's read_sample method to return the x component of the sensor data (Analog_input currently only works with 1D data but if we can get this working we can think about how to extend to 2D).  The read_sample method defined in the Motion_detector class will be called by the Analog_input._timer_ISR  method, which is triggered by a hardware timer at the specified sample rate.

Due to the functionallity it inherits from Analog_input, you should be able to use it just like you would use a normal pyControl analog input, i.e. it will stream data back to the computer, display it in the GUI and save it as a binary file, and generate threshold events on threshold crossings.

Let me know how you get on.

T

[Mostafa Safaie]

Hi Thomas,

After a long delay, I got back to this project.

Thanks for the suggestion. I did try it out, but I think it can't work because the functionality of the Analog_input._timer_ISR() depends on the threshold value which in this context doesn't really exist. So the ISR needs to be overloaded.

Instead, as you can see here, I used the code in Analog_input class and fit it to my application. After a lot of trial and error to solve the memory allocation limitations of writing an interrupt routine, it is now kinda working: the timer sends the events to the GUI, the ISR reads the X and Y values off the sensor (each value is 2 bytes) then I concatenate them together and fill the data buffer, with data_type set to 'L'  (4 bytes = 2 bytes for X + 2 bytes for Y).

Obviously, the values transmitted to the GUI are meaningless per se, but in processing, I can just cast them to 4 bytes, separate the 2 values, and convert them back to integers. The real-time values will also be available in the task using the self.delta array, similar to the self.position in the Rotary_encoder class.

Everything kind of works, without much change to the framework except this, but the GUI crashes with different types of errors (see attached). Does the data_type = 'L'  not mean a 4-byte integer? Why the overflow error happens then? Also, without any other logic in the task file (even the print statement commented out), I get memory allocation errors if I go higher than ~100Hz sampling rate, which is very confusing because using the pyControl set_timer() function and much less efficient interaction with the sensor (without any preallocated buffers), I could go as high as 1kHz. Also even when things are working, the events plot can't keep up (see attached), could be nice to turn off plotting for some events/analog inputs, I couldn't find if that's implemented.

So I guess my main question is whether there is some other data rate limit in the serial communication to the GUI? And how can I get around that?

Any other insight is also highly appreciated :)

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

Before getting into the details of your code, it's worth briefly going over the Analog_input classes functionallity, to make sure were on the same page about how an adaptation for your sensor would work.

The analog input can do two things: 

1) Streams continous analog data to the GUI, which is plotted and saved in a file.  This process does not generate any framework events.

2) Generates framework events on rising and/or falling threshold crossings.  This only happens if rising_event or falling_event arguments are specified when it is initialised and these events are used by the current task.

As event generation triggered by threshold crossings is completely optional, I don't see why it would prevent you from subclassing the analog input.  Subclassing the analog input would also make it much easier to understand your code as it would make it clear what functionallity has been modified.  I would suggest not having seperate PMW3360DM and MotionDetector classes (because you are never going to use the  PMW3360DM class indepenent of the MotionDetector) but rather having a single class that subclasses Analog_input.

Regarding the memory allocation errors when you try and acquire at high sample rates, I think the issue is caused by the fact your input is generating framework events every time it aquires a sample (unlike the analog input class which only does so on threshold crossings).  Specifically,  your timer_ISR function (called regularly using a hardware timer to read each new sample) puts the input's ID in the interrupt_queue, causing the framework to call it's _process_interrupt method, which generates a framework event (by putting an event in fw.event_queue).  This is almost certainly not what you want because these events would be completely uniformative to the task (as they are just periodic at the sampling frequency), and will put a lot of load on the framework as it processes the high frequency stream of events.  You don't need to be generating framework events in order to stream the analog input to the computer - analog data is sent to the computer by the _send_buffer method, which is called by the framework when the input puts its ID in the stream_data_queue.

Regarding the overflow_error, I think this is happening because although it analog input class says that data type 'L' (unsigned 4 byte integer) is allowed, it looks like when the data is saved to disk by the data_logger class, it is converted to a 4 byte signed integer.  I have created an issue here to record this bug.  A workaround for you should be to just specify a datatype of 'l' (signed 4 byte integer) as you are anyway using the 4 bytes to store two 2 byte integers, rather than an unsigned 4 byte integer.

Regarding the IndexError, i'm not exactly sure what is causing this but it will be easier to debug once the other issues are fixed.

best,

Thomas

[Mostafa Safaie]

Hi Thomas,

Thanks again for your tips and for clearing many things up. I hadn't noticed that streaming data to the computer is relatively independent of framework events.

I agree that I do not need every timer interrupt as a framework event, but I do need some events in order to advance the task logic (like in the encoder example). That's why I still cannot see how I can make it work without, at least, overloading the _timer_ISR method. Note that in lines 331-333, events are only generated based on a comparison between a threshold and the last value in the buffer. I have to use the buffer to save both x and y coordinates, so finding a threshold to satisfy any condition (5 cm movement for instance), is not trivial.

If I understood everything, I have one option basically: rewrite the _timer_ISR (in a subclass of Analog_input, or my current MotionDetector class), and modify a threshold value in the task file to generate framework events depending on the task state and/or the movement (i.e., the aggregate of the sensor data, not its last value).

Do you agree with that? Or is there anything else I missed?

Bests,

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

You are right that if you want to return both the x and y coordinates (so need to fit 2x 2 byte integers into each element of the buffer), then you would need to overload the _timer_ISR method to make the input generate framework events based on the signal, as simply detecting threshold crossings on the latest sample in the buffer is not going to be meaningful.

Taking a step back from implementation, I think it would be useful to think about how you want the sensor to interact with behavioural tasks, and hence when it makes sense for the sensor to generate events.  How would you ideally use the motion signal in your tasks?  Would it be sufficient to specify a threshold on a single component of the motion velocity, such that two different framework events was generated on rising and falling threshold crossings?

T

[Mostafa Safaie]

Hi Thomas,

I think it cannot be too general because it depends on the task requirements (eg: x-movements only, or movement angle, velocity, etc) and that varies from case to case. In my task, at the moment at least, I need the distance travelled which with a little trick to avoid calculating the square root, I implemented it here.

There is still one (probably small) problem: as soon as the sensor's readout is non-zero (movement in front of the lens), the data stops being recorded to disk and plotted in the GUI, with no error messages in the terminal (see the image attached). I tried saving the data as unsigned here (and saving it as an 8-byte int, instead of 4) and also setting the data_type to both 'l' and 'L', in all cases, with no luck.

I know the issue is related to concatenating two 2-byte ints (movement along x and y) and streaming it as a single 4-byte int, because if I fill the buffer with only one of the values (x, or y only, ie 2 bytes, in this line) the problem disappears. So, it is probably due to the sign of the integer somewhere in the framework that I haven't managed to pinpoint yet.

Do you have any suggestions? Is there any way to get a verbose log of what's going on under the hood?

Thanks again :)

Mostafa

[thoma...@neuro.fchampalimaud.org]

Hi Mostafa,

Apologies for the delay getting back to this.

This issue with the data stopping being recorded and plotted is a bit mysterious because of the way it depends on the signal, and the fact it does not generate an error.

If I underderstand correctly, it does not crash the GUI, just stops the saving and plotting of analog data.  To debug it I would probably work forward from where the data is read from the serial to try and work out where the problem is happening.  The serial data from the pyboard is processed by the process_data method of the Pycboard class.  For debugging you could add a line at 366 which prints to the terminal or a log file each time a chunk of analog data with a correct checksum is recieved (perhaps reporting the analog data chunk's timestamp  variable to make it easy to see when any dropouts occured)  - this would let you see whether when there is a movement signal the data is being recieved and translated into the data_array variable OK.

If the data is being recieved and decoded OK by the Pycboard.process_data, I would then try and work out whether it is the plotting or the saving of the data that is causing problems.  The Pycboard.process_data method passes the recieved data to an instance of the Data_logger class, which then saves it to disk, and passes on to any additional data_consumers - specifically the Task_plot class that handles the plotting.  You could comment out the code in Data_logger.process_data that either saves the analog data to disk or that passes it on to other data_consumers, to see if these leaves the other function (saving or plotting) intact.

Let me know how you get on.

T