Use analog input TTL to trigger digital input sampling

[Martin Spacek]

Originally sent via http://www.open-ephys.org/contact/ on 2016-12-05:

Hello, I'm in Laura Busse's lab at LMU Munich, and we're about to buy a bunch of Intan RHD2000 systems. We want to record a 16 bit digital word from our visual stimulus system on every screen refresh. Since the OE board only has 8 digital in lines, we have to go with the RHD2000. However, we'd like to use the OE GUI for acquisition.

Ideally, we'd like to sample the 16 bit digital word only once per screen refresh, triggered by the vsync signal from the video card. Since the RHD2000 has 16 digital in, I'd like to use one of the analog inputs to trigger sampling of the digital word. I know you can use an analog input to trigger start (and stop?) of recording, but can the same be done to trigger sampling of the digital input lines? I know I asked Reid Harrison at Intan about this a year or two ago, and he said that's not possible at the moment, at least for the RHD2000 software. I was hoping it might be different for the OE GUI. If not, would it be difficult to implement? I could likely lend a hand. Ideally, we'd like to not have to save both the digital input and vsync signal at 30 kHz each, just to go and discard 99.9% of that data later. All we care about are the digital words and their timestamps, which will arrive at 120 Hz.

Thanks,

Martin

[Martin Spacek]

Jakob Voigts replied on 2016-12-06 via email:

Hi Martin,

I don't think that's easily possible in the firmware. The sampling of the digital inputs is linked to the neural data sampling and triggering independent updates would require an extensive rewrite of the firmware.

I do think there's a simple solution though that just uses some software: If your screen runs at ~120Hz you could just record everything at 30khz, pull the 16 lines into the 16 digital inputs, and use a photodiode on the screen on an analog channel to measure the screen refresh timing and then you can extract the digital data later. Or you could even just write a pretty simple plugin that interprets these data and generates annotated events, or a state encoded in a continuous data stream etc. I wouldn't worry about this being 'inefficient' at all, the cpu overhead would be negligible.

Also, if you already have some open ephys boards, you could use them for 16 inputs, as long as you can add a level shifter to the higher 8 lines. We added a .1" header on the board that gives access to these, so you have the lower 8 accessible via the front panel hdmi, with a level shifter, and the higher 8 on the internal header, without level shifter, at 3.3V logic level. If you're ok with soldering a level shifter onto the prototype area and having some bodge wires on there it shouldn't take too long to make this work.

best,

jakob

[Martin Spacek]

Hi Jakob,

Thanks for the detailed reply. So the OE board can actually do 16 bits digital in, but only 8 lines are wired to the HDMI connector? That's excellent news! The 8 bit input limitation was the sole reason we were considering going with the RHD2000 from Intan. Does the OE plugin-GUI software support all 16 bits? From the screenshots I saw, it looked like only 8 bits. Perhaps this should be clarified somewhere on the main website.

We do actually have an OE board, v2.1 rev 3 (May 2013) with a USB 2.0 connector. Looking at the bottom of the board, I see that 8 holes are labelled DIGITAL_IN_9 through 16, so that must be the header you're talking about.

As for the level shifter, I emailed Reid Harrison about plugging 5V directly into the 3.3V digital inputs on the RHD2000. He said that doing so causes no problems in their testing. I tried it myself with the RHD2000 board in my previous lab (Nick Swindale at UBC in Vancouver), and it worked fine. But maybe it's worth doing the level bitshifting anyway.

The grid of holes in the top left quadrant of the OE board, is that disconnected from all the rest of the circuitry? If so, I guess it's meant to be used as a prototyping board or breadboard. Are the rows or columns in the grid connected to each other within the PCB, or all they all disconnected?

Also, is there easy access somewhere to the first 8 bits of digital input? Ideally, we'd have a single bundle of 16 digital lines coming out the back of the OE case, wired to a DB25 connector, so that we can easily connect to and disconnect from our DataPixx stimulus system, which itself has a DB25 digital out connector.

Thanks again,

Martin 

[Jan Zimmermann]

Hey all,

I have been down the same path and wrote some hacky tweaks for myself to get 8bit/16bit words out of the TTL lines.

@Martin, lets just write a little plugin that saves the words correctly and gives us GUI access to switch between 8/16bit

Plexon does this by having x lines plus a strobe line which is not neccessary. So lets just make a plugin that checks all lines for a change and given a switch of any line recomputes the word and saves that with its timestamp.

@Aaron Jakob and Josh where in the OE files / kwick files should we save this? make it a new event identifier? I just want to be consistent with your efforts.

All the best

Jan

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at https://groups.google.com/group/open-ephys.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/ed6d17c1-73e1-4866-9f19-005c93ca55d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jakob Voigts]

Hi guys,

sounds good - I think such a plugin would be very useful. I think a separate plugin that generates annotated events would be the cleanest approach.

As far as i see it, it should be possible to select 8 bits (only hdmi plug) or 16, as Martin is working on as inputs, and for triggering we could either do one of the bits, or use an analog channel threshold for instance for use with a photodiode taped to a screen or something like that.

Aaron is currently re-writing and cleaning up all of the event type code, so exactly this kind of thing will be much nicer soon, but its not all done yet so im not sure what to do about writing code now - Aaron likely has a specific recommendation of what to do.

About the hardware: We have the 8 upper inputs on this row on .1 inch headers, think they should be labelled - i attached a screenshot of the relevant section. The other 8 pins in that row are the 8 upper outputs that we didnt fit on the hdmi port.

The prototype area is just unconnected vias for soldering, like an old school project board, but nothing is connected. Only exception are a GND and a 5V line at the bottom, they are all marked. On the right side, we added vias to free FPGA pins that you could use as well (see below).

Even though we added the 8 'missing'  header pins (9-16)  We screwed up thinking ahead and have not added headers to all of the pins for 1-8, so the easiest would be to either solder to the pins on LS_IN, just with some small diameter copper wire, or to re-route these TTL in pins to some of the FPGA pins that we have already pulled out to the prototype area. You'd just have to edit the .ucf file and re-compile the firmware, but you'd end up having to use your hacked firmware, so i'd likely go for the hardware approach. Depends on how much you like soldering over installing xilinx ise :)

Even easier, you could just use an hdmi cable to get access to 1-8, pull these to some external prototype board, and then solder a .1" header onto the board to get 9-16, also get these to the external board, add a level shifter for these, and then you can have a 16 pin 5V compatible db25 connector. Would look a bit strange, but it should be very easy to make.

I havent tested it but i'd rather not try putting 5V straight into the FPGA - level shifters are cheap and opal kelly boards, and lost experiment time are not.

[Martin Spacek]

Hi Jan and Jakob,

Wow, lots to think about. Thanks for all the suggestions. OK, so at the moment there are 3 issues (for me at least):

1. Level shifting and wiring the upper 8 bits.

2. Writing a plugin that allows switching acquisition between 8 or 16 digital input lines.

3. Writing a plugin and/or modifying the firmware to acquire or annotate the digital input triggered on some digital or analog input event.

Issue 1:

Jakob, I noticed you've designed an extension board that wires an extra digital in HDMI port (stacked on top of the existing one), through a level shifter and into the upper 8 bit header on the acquisition board:

https://github.com/open-ephys/acquisition-board/tree/master/extension-board

Here's a .png of the design I extracted from Eagle:

The extension also adds another 4 headstage ports, wired to the free FPGA pins, for a grand total of 1024 channels (!). The README suggests there's some uncertainty about ensuring that signal timing/trace length is not a problem for the extra headstage ports, and also the need for firmware changes to make the extra headstage ports work. But, I would be very interested in having this board just for the purposes of nicely expanding the digital inputs by 8 bits. That part presumably doesn't require any design iterations. It looks like there might be space to add an extra HDMI port for the upper 8 digital out lines as well, which could be stacked above the existing digital out port. That would bring the OE board right up to the I/O capabilities of the RHD2000.

Is there any chance this extension board could be built in the relatively near future? My supervisor might even be willing to fund a bit of development. Also, getting an initial extension board out quickly for the sake of immediate digital I/O extension would be a good way to test out the more risky and complicated headstage port expansion. We don't need the extra headstage ports (for now), but I'd be willing to help test them.

The extension board says it's compatible with v2.2+ of the acquisition board. We currently only have a v2.1 rev 3 acquisition board, but we're about to order a new one.

Otherwise, as you said, I need to do something hacky, probably in the prototype area. Any particular level shifter you would recommend? Intan has an application note that recommends the SN74LVC4245A as an 8 bit level shifter, but that's only in surface mount:

http://www.intantech.com/files/Intan_RHD2000_IO_level_shifters.pdf

An older version of that document included a voltage divider (Figure 2, 3.3 and 6.8 kohm resistors) as a simpler option, but presumably that caused some issues and that figure has been removed.

Also, I wonder if there's an HDMI connector out there that can be soldered to the 0.1" prototyping holes. And those 8 pull-down (?) resistors on the level shifters on the acquisition and extension boards, are they really necessary? I'm looking for shortcuts :)

Issues 2 and 3:

Jan, if you could share whatever you've written for selecting 8/16 bits, that would be great. I have zero experience with the OE code. I do have a little bit of C++ experience. I have no verilog experience. I haven't even tried building and running the plugin-GUI yet, so maybe I should get onto that...

Ideally, we would like to use one of the analog input lines to trigger sampling of all 16 digital input lines (and another analog in to trigger recording start/stop), either using the vsync pulse from the graphics card, or a photodiode on our stimulus screen. But I can see that others would want to use one of the digital lines as a trigger, or even all of them as a trigger (sample on bit change) as you suggested, Jan.

Jakob, is there some way to include Aaron in this conversation, to clarify the status of the event code rewrite? It would be good to know what to branch off of so that merging later won't be a headache.

Thanks again everyone.

[Martin Spacek]

Whoops, that extension board .png was missing the traces on the back side of the board. Here it is again in all its glory:

[Aarón Cuevas]

Hi all,

Regarding the TTL word treatment on the software, the Rhythm node does sample all 16 bits and generate TTL events whenever each of those bits change. Each of those events also contain the whole TTL word (as a 64bit unsigned integer) accessible by something like

uint64 ttl = *(event->getRawData() + 6); //Where event is the MidiMessage object

In the new system I'm designing, the whole TTL words will be automatically saved, which does not happen currently. In fact, although we do have a BINARY_MSG type of event, which could be useful to send this kind of data too, none of the record engines support it so unless you wanted to quick-hack them to do so, I believe the best option would be to make a very simple plugin to extract the TTL world, convert it to a string and send it as a text message event, which will be written to disk. Again, once the new system is online, all this won't be necessary.

Best,

Aarón.

[Jan Zimmermann]

Fully agree with Aaron's approach as I am currently using that exactly.

This however only covers my on bit change messaging.

I think a plugin to specify how exactly one wants to save these events is clearly necessary.

@Aaron once you know where those words should go in terms of the data files I will write the plugin.

@Martin for the time being you could always do what Aaron described plus save it via

void GenericProcessor::addEvent(MidiBuffer& eventBuffer,

                                uint8 type,

                                int sampleNum,

                                uint8 eventId,

                                uint8 eventChannel,

                                uint8 numBytes,

                                uint8* eventData,

                                bool isTimestamp)

All the best

Jan

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at https://groups.google.com/group/open-ephys.

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/91d4eead-bb0f-4175-8532-7a4933508312%40googlegroups.com.

[Aarón Cuevas]

Excuse me if I'm a bit confused here, but I think I'm not understanding what you mean. Only recordengine plugins can write directly to the data files, which might be a bit an overshot for this at this moment, Is that what you're wanting to make or what exactly do you plan for your plugin to do?

[Martin Spacek]

Hello everyone, sorry for the long silence, and thanks for the input. After some consideration (and input from Nikolas Karalis), I've decided to simplify things so that we don't need so many digital input lines. Now all that I want to do is detect when two input lines are both high: the vsync signal from the stimulus monitor, and a "success" bit from the stimulus system indicating that the draw loop for that frame has completed. The vsync will arrive on every monitor refresh, but the success bit will only be high until a successful buffer flip, and then will be set immediately low again for the next iteration of the draw loop.

It sounds like the ideal solution right now is to write a plugin of some kind that detects when both of these input lines (probably digital, maybe analog) are high, and generates an event message. Presumably, this would be the same kind of message as that generated by the text message input box at the bottom of the open-ephys window.

Aaron wrote:

"I believe the best option would be to make a very simple plugin to extract the TTL world, convert it to a string and send it as a text message event, which will be written to disk."

My question now is, what kind of plugin should I write? A processor plugin that would be dragged to the signal chain, or some other kind? Is there a particular existing one I should look at as an example? I've only just started looking at the code, and I'm not quite sure where to start.

Thanks!

Thanks!

[Martin Spacek]

On 2017-02-03 07:52 PM, Jan Zimmermann wrote:

> Do you need this msg to be processed in real time in terms of its output or is

> it enough for you to have it logged in the file ?

Hi Jan, no, we don't really need the output in real time. We just need it to be logged with a precise timestamp, the same way text messages seem to be in one of the files generated in the binary recording format.

[Nikolas Karalis]

I guess you don't need to write a plugin for that, unless you need it in real time.

If the two inputs are connected to the digital IO ports, the timestamps will be saved as TTLs. If you connect them to the analog IO ports, then you can save the two extra channels and extract them during the post-processing.

[Martin Spacek]

Hmm, yes, it could be done offline after recording is done, but since it's such a simple step, I'd rather get it out of the way online.

[Martin Spacek]

After quite a struggle, I finally got a rough version to work. Here it is. The filter plugin is called a "BlixxProcessor", named after our stimulus system:

https://github.com/mspacek/plugin-GUI/tree/blixx/Source/Plugins/BlixxProcessor

The meat of it is in BlixxProcesssor.cpp.

It detects rising edge TTL events on channel 0 (1st digital input line), and outputs a MESSAGE text event with the text "BLIXXFRAME", which gets saved along with a timestamp to an .eventsmessages file (I'm using the binary recording engine).

This is the typical output to stdout:

*** 2 events received by Blixx node 101
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 1, 100, 0, 0, 0, 0
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 0, 100, 100, 0, 1, 0
*** 11 events received by Blixx node 101
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 1, 100, 0, 4, 0, 0
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 0, 100, 100, 0, 1, 0
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 789
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 789
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 5, 101, 0, 0, 1, 789
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 5, 101, 0, 0, 1, 789
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 1, 0, 0, 790
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 792
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 5, 101, 0, 0, 1, 792
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 1, 0, 0, 794
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 799
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 5, 101, 0, 0, 1, 799
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 1, 0, 0, 800
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 809
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 5, 101, 0, 0, 1, 809

eventType 3 is TTL, and 5 is MESSAGE. The others don't matter much for now, I think.

One issue that remains is that I sometimes get JUCE Assertion (jassert) errors, and what looks like corruption, like this:

*** 4 events received by Blixx node 101
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 1, 100, 0, 4, 0, 0
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 0, 100, 100, 0, 1, 0
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 0, 0, 0, 500
*** added event
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 3, 100, 1, 0, 0, 502
JUCE Assertion failure in juce_MidiMessage.cpp:105
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 0, 0, 0, 0, 0, 20250624
eventType, sourceNodeId, eventId, eventChannel, save, samplePosition: 53, 7, 0, 0, 0, 0

I wonder, is this because I'm reading and writing events to the same buffer? Should I instead first read all the TTL events in the buffer, save the data from them in some temporary buffer, and then add all the new MESSAGE events to the same buffer all at once at the very end? Maybe I'm hitting some kind of a race condition...

[Aarón Cuevas]

Hi,

Although I don't think it is exactly a race condition (all the processing gets done in the same thread) Juce documentation says that is isn't safe to modify a MidiBuffer while an iterator is being used, which is the case when you call checkForEvents and try to add some inside the handleEvent method. If that's the case, I think your solution is the best way to go: add all the necesary events into a different MidiBuffer and then, in the process method, at any points after the checkForEvents call, you can merge them by just doing eventBuffer.addEvents(yourEventBuffer,0,-1,0)

Interestingly, I'm working on some architectural changes to make plugin programming a bit easier and I hadn't thought of this possibility, so thanks for getting it into my attention!

[Martin Spacek]

Thanks Aaron, I did exactly that, and it worked! No more jassert errors, and no more corruption. Here's the diff:

https://github.com/mspacek/plugin-GUI/commit/4d785c0380f76eeca455f2c81e62b8bde684e9ba

I'm not sure if I should clear my temporary event buffer (blixxEvents.clear()) after I'm done adding its contents to the main buffer. It seems to work fine either way, but maybe it should be done to prevent memory leaks?

[Aarón Cuevas]

Hi,

You won't exactly have memory leaks without clearing it, but I'm surprised you aren't getting duplicate events. As far as I know, when you add the events from one buffer to another they get copied, not moved, so if you don't clear your temporal buffer the old events should stay in there and get copied over and over again.

Just one question. Why did you code your own checkForEvents method instead of relying on the standard one and responding to events in the handleEvent method?

Aarón.

[Martin Spacek]

Hmm, I didn't seem to be getting duplicate events when I wasn't clearing the temporary event buffer. Isn't that because I that temporary buffer isn't being passed on to anything downstream?

I wrote my own checkForEvents() method because I originally wanted access to the source event buffer, so I could write back to it. I suppose now I could factor some of that out into a handleEvent method and put the rest of it in process(). But we've decided, as I think Nikolas Karalis suggested earlier, that we should simply save the full TTL word and its timestamp to disk on every bit change, and then do bitmasking offline to detect particular combinations of bits representing a successfully drawn frame. This will reduce the number of files we need to save, since we'll also use some of the remaining digital input lines for other stuff, like trial and optogenetic timing.

Writing this plugin was still a useful experience though!

[Aarón Cuevas]

I thought there might be duplicate events because you were copying the temporal buffer each time without clearing it. Perhaps I missed something and it's getting cleared somehow.

Anyway, we're working on an update of the event system since, as you've surely noticed, it was becoming sort of messy. The new system takes care of many of those low-level nuances, so it should be easier to develop plugins in the future, with not many modification being needed on most current plugins. We'll keep you updated!

Best,

Aarón

[Martin Spacek]

Sounds great. I'm eager to test it out! I've pretty much thrown out the plugin I wrote. Instead, I've been mostly modifying BinaryRecording.cpp for our own purposes. I'm now saving all TTL events to their own dedicated headerless .din file, and all analog data to a single big .dat file. No experiment numbers any more, just recording numbers that are appended to a simple base filename in case the continuous recording is interrupted for some reason. Experiment indexing for us will happen on the stimulus side. We want one big .dat file for continuity across experiments for spike sorting.

By the way, after leaving it recording for about 20 h, the .dat file hit 128.0 GiB, and then open-ephys segfaulted. I wasn't there to see it when it happened, and unfortunately I wasn't running it in gdb, but I will next time. Any idea offhand if I've hit some kind of indexing limit within open-ephy? ext4's filesize limit is apparently 16 TB.

[Martin Spacek]

I've opened up a Github issue for this:

https://github.com/open-ephys/plugin-GUI/issues/122