Open Ephys crash

[André Bastos]

Hey Guys,

I believe a similar bug was recently reported, but I thought I would post this anyway to build more awareness of this problem and potential solutions

I tested the open Ephys system today, recording with a single 32 channel headstage, and both analog inputs and digital inputs connected.

The system would freeze after about 5 to 10 minutes of recording. The LFP viewer would basically stop, but the system would indicate the recordings were still on. However, the data files would not increase in size, so apparently the system was already crashed but some aspects of the GUI appeared to be active. I tried multiple times, having only the Rhythms FPGA + LFP viewer active, but still got the problem about 5 times. Finally, I tried unplugging the digital inputs, and this seemed to solve the problem - the system recorded without problems for an hour after that.

The error message is :

JUCE Assertion failure in juce_AudioSampleBuffer.cpp:531

My system specifications are:
Memory 3.7Gigs
Processor: Intel Core i5 2400 @ 3.1 Ghz x 4
Graphics: Intel Sandybridge Desktop
OS: 64 but
Disk: 500 Gb

I am using the build which was committed on July 3rd 2015, commit number a52dbdb2 ...

It looks like Francisco Battalgia has also encountered this problem and suggested a code fix.

Has anyone else tried Francisco's recommended workaround and had success? Is there any explanation for the system only crashing when both analog and digital inputs are coming in? Would anyone else having this problem be able to replicate this behavior?

Thanks in advance for any input.

Andre

[Aarón Cuevas]

Francisco's patch seems to prevent the crash in most cases, but I think the issue might be elsewhere. Did you notice any weird TTL activity prior to the crashes? In some cases we've observed oscillation in the digital inputs when the IO board is plugged but no signal is connected to the BNCs. We've also observed crashes when there is a very high number of TTL events in a short time. Maybe your issue is related to this.

Aarón.

[André Bastos]

Hi Aaron,

To answer your questions:

There is no real weird TTL activity, I never experienced TTL pulses coming through that I didn't send myself (to my knowledge). Also haven't seen any of the oscillations in TTL pulses.

Also, what would you consider to be a large number of TTL events? I sometimes do send ~1-4 codes per second during certain moments of the task - would this be in the range that might crash the system?

I haven't yet installed Francisco's patch, but will report back on that.

Thanks,
Andre

[Francesco Battaglia]

What OS are you on? The problem I've fixed was Linux (perhaps Mac OS X) specific (depending on different behaviors of the C compiler)

[André Bastos]

My system specifications are:
Memory 3.7Gigs
Processor: Intel Core i5 2400 @ 3.1 Ghz x 4
Graphics: Intel Sandybridge Desktop
OS: 64 but
Disk: 500 Gb

Thanks,

Andre

[Alik Widge]

Hi folks,

Following up on this thread, just tested with Andre, and something interesting has happened -- his FPGA is now making a high-pitched whine (fan underneath the heat sink?) that was not happening 1wk ago (I've heard his box before, I'm certain) and now it appears that the Rhythm interface cannot be loaded or interfaced to USB.

Bad FPGA? Perhaps also the source of the trigger-crash problem? Wondering if anyone's seen this before?

[Aarón Cuevas]

The high TTL counts that are known to sometimes crash the GUI are in the order of dozens a second, so I doubt that's the case.

The noise issue looks quite bad. Our boards ship without a fan, so that noise is probably very bad news for the FPGA. Something might have broken on the board.

Aarón.

[Christian Tatarau]

Hi Aarón,
I am experiencing a very similar behavior (GUI crashes, timer goes on, LFP viewer freezes, data acquisition stops with 8 channels recorded out of 32 @2 kHz) on Linux Mint on a decent computer. We don't have TTL inputs but are planing to synchronise 2 cameras with m,ax 40 fps each by using the TTL inputs. Sounds like bad news because that will add more crashes. Is the TTL problem solved?

[Aarón Cuevas]

Hi Christian,

The high-count ttl crashes when recording were caused by some deficiencies on the recording subsystem, which has been extensively improved in the last year. I cannot completely confirm that there might be no issue, though, since such high frequency ttl signals are not common so we don't have many use cases for them, but in the case there were still some issues with those I believe they can be easily spotted and fixed. I'll try to run some tests, though.

The freezes you're experiencing now are a bit more worrisome, Which version of the GUI and which frontPanel dll (the default usb2-only one or the optional usb3-capable one) are you using? Is there any special console output when the freezes happen?

Best,

Aarón

[Christian Tatarau]

Hi Aarón,
we'll try out the cameras with TTL frame exposure signal soon and I'll give feedback.
About the crashing. I compiled the newest master version from Github with the standard USB2 driver. I can try running from console, I did not do that before. Anyway the freezing occurs rather rarely so it is hard to get our hands on it. The problem is that we can't stare at the monitors for hours to restart when it freezes... What is your advice for finding out what is causing that?

[Christian Tatarau]

I see in the makefile the CFLAGS -g -ggdb which means the GUI has been compiled with debug symbols, is that correct?

[Aarón Cuevas]

Yes, on Linux even the release build is compiled con debug symbols. The GUI is quite multithreaded so trying to get info from gdb could be tricky.
The first step would be to try to run the GUI from the console. We output a fair amount of debug messages so it might help to see if the GUI informs of anything weird when it freezes.

[Christian Tatarau]

The GUI freezes mostly during measurements with animals, although there is no movement or short circuit (under anesthesia for now). In saline solution with the same wires and electrodes it did not happen during a 1 week continuous measurement. Anyway after finishing the week long recording it did crash with the message

error usb control message: No such device

And LEDs were off.

[Christian Tatarau]

Update: after a couple of days of continuous measurement the acquisition box itself does not work any more. The GUI does not find it anymore and when connected to power it will take 2,0 A @ 5,0V because that is maximum current output I set on the benchtop power supply. There was no short-circuit, one electrode in saline solution, headstage was safe outside.

Did anything like that ever happen before? How can we find out now if the power circuitry or the FPGA is broken?

[Christian Tatarau]

Update1: I took off the FPGA board from the black PCB acquisition board. When the FPGA board allone is connected to power and USB it gets recognized by the GUI. So this one is fine.
On the black PCB there is a short circuit between power socket (central pin) and GND. How can that happen? So I guess I need an new black PCB. But first we need to find what went wrong.

[Andrew McKinstry-Wu]

I'm having a near-identical problem with a freezing of the feed with failure to write using both OS X (10.9, 10.10, and 10.11) and Ubuntu Linux. I've recreated the problem with multiple computers, multiple acquisition boards (3 different), using bench top and other protected power supplies, and am at a loss as to what is causing this. This has happened with 32, 64, 96 channel recordings at 1 kHz, with and without an A to D board, and can occur any time from 10 minutes after the start of recording to 4+ hours later, seemingly at random. Does anyone have any suggestions on how to pinpoint the source of the problem? 

Thanks!

Andrew

[Josh Siegle]

Maybe it’s an issue specific to 1 kHz acquisition? Have you tested it at higher sampling rates?

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at https://groups.google.com/group/open-ephys.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/c5f8a5d6-1aee-4305-a28c-13ea665f1e63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew McKinstry-Wu]

Yes, I've reproduced it at 2k and 30k.

On Tue, Feb 7, 2017 at 3:57 PM, Josh Siegle <josh....@gmail.com> wrote:

Maybe it’s an issue specific to 1 kHz acquisition? Have you tested it at higher sampling rates?

On Feb 7, 2017, at 11:45 AM, Andrew McKinstry-Wu <mckinst...@gmail.com> wrote:

I'm having a near-identical problem with a freezing of the feed with failure to write using both OS X (10.9, 10.10, and 10.11) and Ubuntu Linux. I've recreated the problem with multiple computers, multiple acquisition boards (3 different), using bench top and other protected power supplies, and am at a loss as to what is causing this. This has happened with 32, 64, 96 channel recordings at 1 kHz, with and without an A to D board, and can occur any time from 10 minutes after the start of recording to 4+ hours later, seemingly at random. Does anyone have any suggestions on how to pinpoint the source of the problem? 

Thanks!

Andrew

On Tuesday, January 24, 2017 at 9:54:25 AM UTC-5, Christian Tatarau wrote:

Update1: I took off the FPGA board from the black PCB acquisition board. When the FPGA board allone is connected to power and USB it gets recognized by the GUI. So this one is fine.
On the black PCB there is a short circuit between power socket (central pin) and GND. How can that happen? So I guess I need an new black PCB. But first we need to find what went wrong.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.

To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

[Aarón Cuevas]

Andrew,

Can you run the software with the debug console opened? There might be messages there that could help shed a light on the issue.

Christian, 

Do you usually plug the power supply directly to the FPGA board or through the black board's connector? A short circuit between +5v and gnd in the power socket could range from a simple soldering issue somewhere to a busted IC, it might be complicated to find out.

[Christian Tatarau]

Hi all,
On the broken acquisition board I was using the black motherboards connector. I'll soon try to find the problem on it.

I run the program from the console and that is the output during the crash. BTW, it happens to me at 2 and 30 kHz too. I'd like to contribute to finding the problem because it became a major issue.

*Start program*

Loading Plugin: FilterNode... Loaded with 1 plugins
Loading Plugin: PhaseDetector... Loaded with 1 plugins
Loading Plugin: PulsePalOutput... Loaded with 1 plugins
Loading Plugin: NetworkEvents... Loaded with 1 plugins
Loading Plugin: LfpDisplayNode... Loaded with 1 plugins
Loading Plugin: Rectifier... Loaded with 1 plugins
Loading Plugin: SpikeSorter... Loaded with 1 plugins
Loading Plugin: SerialInput... Loaded with 1 plugins
Loading Plugin: ArduinoOutput... Loaded with 1 plugins
Loading Plugin: CAR... Loaded with 1 plugins
Loading Plugin: LfpDisplayNodeBeta... Loaded with 1 plugins
Loading Plugin: ChannelMappingNode... Loaded with 1 plugins
Loading Plugin: BasicSpikeDisplay... Loaded with 2 plugins
Loading Plugin: RecordControl... Loaded with 1 plugins
Loading Plugin: EventBroadcaster... Loaded with 1 plugins

Loading window bounds.

Item dropped at insertion point 0
Creating from description...
Sources::Rhythm FPGA (0-0)
/home/neurophys/plugin-GUI-master/Builds/Linux/build
---- Intan Technologies ---- Rhythm RHD2000 Controller v1.41 ----


FrontPanel DLL loaded.  Built: Feb  1 2012  17:34:05

Scanning USB for Opal Kelly devices...

Found 1 Opal Kelly device connected:
  Device #1: Opal Kelly XEM6010LX45 with serial number 1447000A3C

FPGA system clock: 100 MHz
Opal Kelly device firmware version: 3.1
Opal Kelly device serial number: 1447000A3C���
Opal Kelly device ID string: Opal Kelly XEM6010

Rhythm configuration file successfully loaded.  Rhythm version number: 1

Initializing acquisition board.
Sample rate set to 30000
Sample rate set to 30000
Number of enabled data streams: 8
Checking for connected amplifier chips...
Number of enabled data streams: 1
Sample rate set to 30000
  Adding node to graph with ID number 100


Rhythm FPGA updating settings.
Rhythm FPGA setting num outputs to 35
Setting buffer address to 0x1b901b0
Input buffer address is 0x1b901b0
Sample rate set to 2000
Setting sample rate to index 3
Rhythm FPGA updating settings.
Rhythm FPGA setting num outputs to 35
Setting Lower Bandwidth to 0.1
Actual Lower Bandwidth:  0.0945291
Setting Upper Bandwidth to 500
Actual Upper Bandwidth:  499.939
Setting DSP Cutoff Freq to 0.1
Actual DSP Cutoff Freq:  0.0777219
Item dropped at insertion point 1
Creating from description...
Sinks::LFP Viewer (1-4)
  Adding node to graph with ID number 101


Rhythm FPGA updating settings.
Rhythm FPGA setting num outputs to 35
LFP Viewer updating settings.
Setting num inputs on LfpDisplayNode to 35
Found 1 event channels.
Adding channel 35 for event source node 100
Drawer button clicked
Updating connections:


Signal chain: 0

Source node: Rhythm FPGA.
     Connecting to audio and record nodes.
     Connecting Rhythm FPGA 100 to LFP Viewer 101

Source node: LFP Viewer.
     NOT connecting to audio and record nodes.
     No dest node.

Enabling processors...
Source node received enable signal
Expecting 35 channels.
Number of 16-bit words in FIFO: 0
Is eval board running: 0
Starting acquisition.
Flushing FIFO.
Expecting blocksize of 15600 for 1 streams
Resizing buffer. Samples: 10000, Inputs: 35
   Enabling VisualizerEditor
Filter Viewport disabled.
DAC

Adding audio callback.
Setting num inputs on LfpDisplayCanvas to 35
Setting displayBufferSize on LfpDisplayCanvas to 10000
Beginning animation.
Adding tab with index 3

Removing audio callback.
Disabling processors...
Disabling Record Node
STOP RECORDING.
Disabling Audio Node
Disabling Message Center
Disabling Rhythm FPGA
Source node received disable signal
RHD2000 data thread stopping acquisition.
Thread exited.
Flushing FIFO.
Number of 16-bit words in FIFO: 0
SourceNode returning true.
Disabling LFP Viewer
Ending animation.
Filter Viewport enabled.
Updating connections:


Signal chain: 0

Source node: Rhythm FPGA.
     Connecting to audio and record nodes.
     Connecting Rhythm FPGA 100 to LFP Viewer 101

Source node: LFP Viewer.
     NOT connecting to audio and record nodes.
     No dest node.

Enabling processors...
Source node received enable signal
Expecting 35 channels.
Number of 16-bit words in FIFO: 0
Is eval board running: 0
Starting acquisition.
Flushing FIFO.
Expecting blocksize of 15600 for 1 streams
Resizing buffer. Samples: 10000, Inputs: 35
   Enabling VisualizerEditor
Beginning animation.
Filter Viewport disabled.

Adding audio callback.
Creating new directory.
Sources/Rhythm FPGA
Create subnodes with parameters
  Moving forward along signal chain.
Sinks/LFP Viewer
Create subnodes with parameters
  Moving forward along signal chain.
  No processor found.
  End of chain.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/2017-02-10_18-51-38_178-DAT-baseline/all_channels.events
Writing header.
File ID: 0x7f9f64000a00, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/2017-02-10_18-51-38_178-DAT-baseline/messages.events
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/2017-02-10_18-51-38_178-DAT-baseline/100_CH1.continuous
Writing header.
File ID: 0x7f9f64000f80, number of bytes: 1024
Wrote header.

*snip*
2 min later:

OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/2017-02-10_18-51-38_178-DAT-baseline/100_CH31.continuous
Writing header.
File ID: 0x7f9f640025e0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/2017-02-10_18-51-38_178-DAT-baseline/100_CH32.continuous
Writing header.
File ID: 0x7f9f64004cb0, number of bytes: 1024
Wrote header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.

[Christian Tatarau]

I got this error message with our second functioning acquisition box.

[Christian Tatarau]

Second crash today. I am posting just the end because the console log doesn't go so far back. The program just keeps printing the error message.

Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.

                                                                                                       <***********I am closing the GUI now*******
Removing audio callback.

Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.

Disabling processors...
Disabling Record Node
STOP RECORDING.

Exiting record thread
Closing files

Disabling Audio Node
Disabling Message Center
Disabling Rhythm FPGA
Source node received disable signal
RHD2000 data thread stopping acquisition.
Thread exited.
Flushing FIFO.
Number of 16-bit words in FIFO: 0
SourceNode returning true.
Disabling LFP Viewer
Ending animation.
Filter Viewport enabled.

Saving window bounds.

DISABLING DATAVIEWPORT CONNECTION

Sources/Rhythm FPGA
Create subnodes with parameters
  Moving forward along signal chain.
Sinks/LFP Viewer
Create subnodes with parameters
  Moving forward along signal chain.
  No processor found.
  End of chain.

RHD2000 interface destroyed.

[Josh Siegle]

Important question: is the crash always preceded by the GUI opening additional files (e.g., CH31 and CH32 in this example)? Can you check and see if all of the expected files are present immediately after recording begins?

This will make a big difference in where we need to debug things.

[Christian Tatarau]

Hi Josh,
yes, all files are there from beginning on. CH31 and CH32 are not additional files, I am recording channels [1-6] and [29-32] so they should be there and they have been there too.

[Josh Siegle]

Ok, that would seem to imply that the problem is with the acquisition board, not the software. Aarón, can we get any more information besides “incorrect header” when it fails?

It looks like you’re using the Opal Kelly XEM6010 (USB 2.0 FPGA). Do you know if it’s plugged into a USB 2.0 or 3.0 port? And have you tried any acquisition boards with the XEM6310 (USB 3.0 version)? That might solve the problem if it’s an issue with data transmission being interrupted somehow.

On Feb 10, 2017, at 11:02 AM, Christian Tatarau <ctat...@gmail.com> wrote:

Hi Josh,
yes, all files are there from beginning on. CH31 and CH32 are not additional files, I am recording channels [1-6] and [29-32] so they should be there and they have been there too.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.

To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

To post to this group, send email to open-...@googlegroups.com.
Visit this group at https://groups.google.com/group/open-ephys.

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/50f496ed-0d7d-4f88-8d5d-50946e4ab4d6%40googlegroups.com.

[Christian Tatarau]

We got only USB 2.0 ports. No I haven't tried XEM6310.

[Aarón Cuevas]

The incorrect header error usually indicates USB transmission issues. This usually means that the USB thread isn't acquiring data fast enough so the internal hardware buffer ends overflowing and the data gets corrupted and out of sync, but can indicate other kinds of usb transmission issues. We do not have any checks in place to detect that yet, so the GUI just keeps running but not actual data is being sent.

What are the specs of your computer? For 32ch any decent computer should just work, but to have that in mind.

Also, have you tried replacing the libokFrontPanel.so file for the one in Resources/DLLs/Linux-USB3 ? That version of the opal kelly driver has some Linux performance improvements, even for USB2 boards. (You might encounter some library issues doing that, though, that's why we don't set it as the default one. Look at the "Linux users and USB3 boards section of https://open-ephys.atlassian.net/wiki/display/OEW/Rhythm+FPGA for more info)

To help discover other possible causes, can you please try adding a line like:

if (res < 0) std::cout << "USB error code " << res << std::endl;

at the end of the Rhd2000EvalBoard::readRawDataBlock method in the Source/Processors/DataThreads/RhythmNode/rhythm-api/rhd2000evalboard.cpp file? Just before the last return true statement.(Line 1481). Also, when the system freezes, can you run a dmesg and check if there are any usb related messages there?

Best,

Aarón.

[Christian Tatarau]

Aarón, thanks a lot for the suggestions, I'll try that out all that. The PC has an i7 CPU and I'm writing data to the SSD. After recording I'm moving the data to the larger mechanical disk drive. For the exact specs I need to go to the lab.

Apart from all that, would it make sense to get a USB PCI board? Still, the PC is about 3 years old and I can't imagine the USB to be too slow.

[Aarón Cuevas]

An i7 should be plenty enough for what the GUI does, specially on simple acquisition of just 32 channels. I don't see the physical USB port being an issue, unless it were defective. 

[Christian Tatarau]

Hi all! Here the crash output. I added the line Aaron suggested in rhd2000evalboard.cpp in all 3 functions

readRawDataBlock    if (res < 0) std::cout << "Rhd2000EvalBoard::readRawDataBlock USB error code " << res << std::endl;
readDataBlocks         if (res < 0) std::cout << "Rhd2000EvalBoard::readDataBlocks USB error code " << res << std::endl;
readDataBlock           if (res < 0) std::cout << "Rhd2000EvalBoard::readDataBlock USB error code " << res << std::endl;

right before return true; As you can see in the output, there is no USB error, at least not at this level. Additionally, I changed RHD2000Thread::updateBuffer and added a header print:
bool RHD2000Thread::updateBuffer()
{
......................
            if (!Rhd2000DataBlock::checkUsbHeader(bufferPtr, index))
            {
                cerr << "Error in Rhd2000EvalBoard::readDataBlock: Incorrect header." << endl;
                cout << "RHD2000Thread::updateBuffer " << Rhd2000DataBlock::printUsbHeader(bufferPtr, index);
                break;
            }
....................

Rhd2000DataBlock::printUsbHeader is based on checkUsbHeader and does just that:

// Check first 64 bits of USB header against the fixed Rhythm "magic number" to verify data sync.
bool Rhd2000DataBlock::printUsbHeader(unsigned char usbBuffer[], int index)
{
    unsigned long long x1, x2, x3, x4, x5, x6, x7, x8;
    unsigned long long header;
    char hex[8];
   
    x1 = usbBuffer[index];
    x2 = usbBuffer[index + 1];
    x3 = usbBuffer[index + 2];
    x4 = usbBuffer[index + 3];
    x5 = usbBuffer[index + 4];
    x6 = usbBuffer[index + 5];
    x7 = usbBuffer[index + 6];
    x8 = usbBuffer[index + 7];

    header = (x8 << 56) + (x7 << 48) + (x6 << 40) + (x5 << 32) + (x4 << 24) + (x3 << 16) + (x2 << 8) + (x1 << 0);
    for (int i=0; i < 8; i++){
        sprintf(hex, "%02x", (unsigned int) usbBuffer[i]);
        std::cout << hex;
    }
    std::cout << endl;
    return (header == RHD2000_HEADER_MAGIC_NUMBER);
}

I also inserted a similar line in Rhd2000DataBlock::fillFromUsbBuffer but this one did not fire:
void Rhd2000DataBlock::fillFromUsbBuffer(unsigned char usbBuffer[], int blockIndex, int numDataStreams, int nSamples)
{
......................
        if (!checkUsbHeader(usbBuffer, index)) {
            cerr << "Error in Rhd2000EvalBoard::readDataBlock: Incorrect header." << endl;
            cout << "Rhd2000DataBlock::fillFromUsbBuffer " << printUsbHeader(usbBuffer, index);
            break;
        }
...................

Source node: Rhythm FPGA.
     Connecting to audio and record nodes.

     Connecting Rhythm FPGA 100 to Bandpass Filter 102

Source node: Bandpass Filter.

     Connecting to audio and record nodes.

     Connecting Bandpass Filter 102 to LFP Viewer 101

Source node: LFP Viewer.
     NOT connecting to audio and record nodes.
     No dest node.

Enabling processors...
Source node received enable signal
Expecting 35 channels.
Number of 16-bit words in FIFO: 0
Is eval board running: 0
Starting acquisition.
Flushing FIFO.
Expecting blocksize of 15600 for 1 streams
Resizing buffer. Samples: 10000, Inputs: 35
   Enabling VisualizerEditor
Beginning animation.
Filter Viewport disabled.
                                               
Adding audio callback.

Threshold set to 2                                  <----------------------- me playing around with the GUI
transfer_B set to 0.870551
Threshold set to 6
transfer_B set to 0.698827
Threshold set to 12
transfer_B set to 0.608364
Threshold set to 17
transfer_B set to 0.567427
Threshold set to 22
transfer_B set to 0.538909
Threshold set to 28
transfer_B set to 0.513533
Threshold set to 33
transfer_B set to 0.496932
Threshold set to 38
transfer_B set to 0.483107
Threshold set to 40
transfer_B set to 0.478176
Threshold set to 44
transfer_B set to 0.469147
Threshold set to 47
transfer_B set to 0.462999
Threshold set to 49
transfer_B set to 0.459156
Threshold set to 50
transfer_B set to 0.457305
Threshold set to 52
transfer_B set to 0.453732
Threshold set to 55
transfer_B set to 0.44867
Threshold set to 58
transfer_B set to 0.44393
Threshold set to 61
transfer_B set to 0.439475
Threshold set to 64
transfer_B set to 0.435275
Threshold set to 68
transfer_B set to 0.430029
Threshold set to 72
transfer_B set to 0.425141
Threshold set to 75
transfer_B set to 0.421685
Threshold set to 78
transfer_B set to 0.41839
Threshold set to 81
transfer_B set to 0.415244
Threshold set to 82
transfer_B set to 0.414226
Threshold set to 83
transfer_B set to 0.413223
Threshold set to 85
transfer_B set to 0.41126
Threshold set to 87
transfer_B set to 0.409351
Threshold set to 89
transfer_B set to 0.407495
Threshold set to 91
transfer_B set to 0.405688
Threshold set to 94
transfer_B set to 0.403064
Threshold set to 97
transfer_B set to 0.40054
Threshold set to 100
transfer_B set to 0.398107
Threshold set to 98
transfer_B set to 0.399719
Threshold set to 94
transfer_B set to 0.403064
Threshold set to 82
transfer_B set to 0.414226
Threshold set to 72
transfer_B set to 0.425141
Threshold set to 62
transfer_B set to 0.438048
Threshold set to 50
transfer_B set to 0.457305
Threshold set to 39
transfer_B set to 0.480604
Threshold set to 28
transfer_B set to 0.513533
Threshold set to 20
transfer_B set to 0.54928
Threshold set to 16
transfer_B set to 0.574349
Threshold set to 13
transfer_B set to 0.598703
Threshold set to 10
transfer_B set to 0.630957
Threshold set to 8
transfer_B set to 0.659754
Threshold set to 7
transfer_B set to 0.677611
Threshold set to 6
transfer_B set to 0.698827
Threshold set to 3
transfer_B set to 0.802742
Threshold set to 0
transfer_B set to inf
Threshold set to 1
transfer_B set to 1
Threshold set to 0
transfer_B set to inf

Sources/Rhythm FPGA
Create subnodes with parameters
  Moving forward along signal chain.

Filters/Bandpass Filter

Create subnodes with parameters
  Moving forward along signal chain.
Sinks/LFP Viewer
Create subnodes with parameters
  Moving forward along signal chain.
  No processor found.
  End of chain.

OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/all_channels_2.events
Writing header.
File ID: 0x7f31bc001bb0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/messages_2.events
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH1_2.continuous
Writing header.
File ID: 0x7f31bc0022f0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH2_2.continuous
Writing header.
File ID: 0x7f31bc002530, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH3_2.continuous
Writing header.
File ID: 0x7f31bc0008e0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH4_2.continuous
Writing header.
File ID: 0x7f31bc000b20, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH5_2.continuous
Writing header.
File ID: 0x7f31bc000d60, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH6_2.continuous
Writing header.
File ID: 0x7f31bc000fe0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH7_2.continuous
Writing header.
File ID: 0x7f31bc001280, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH8_2.continuous
Writing header.
File ID: 0x7f31bc001560, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH9_2.continuous
Writing header.
File ID: 0x7f31bc004be0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH10_2.continuous
Writing header.
File ID: 0x7f31bc004e20, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH11_2.continuous
Writing header.
File ID: 0x7f31bc005060, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH12_2.continuous
Writing header.
File ID: 0x7f31bc0052a0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH13_2.continuous
Writing header.
File ID: 0x7f31bc0054e0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH14_2.continuous
Writing header.
File ID: 0x7f31bc005750, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH15_2.continuous
Writing header.
File ID: 0x7f31bc005a00, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH16_2.continuous
Writing header.
File ID: 0x7f31bc005cb0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH29_2.continuous
Writing header.
File ID: 0x7f31bc005f60, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH30_2.continuous
Writing header.
File ID: 0x7f31bc006370, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH31_2.continuous
Writing header.
File ID: 0x7f31bc006620, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_CH32_2.continuous
Writing header.
File ID: 0x7f31bc0068d0, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_AUX1_2.continuous
Writing header.
File ID: 0x7f31bc006b80, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_AUX2_2.continuous
Writing header.
File ID: 0x7f31bc006e30, number of bytes: 1024
Wrote header.
OPENING FILE: /home/neurophys/Desktop/TEMP RECORDINGS/test/2017-02-17_21-49-46/100_AUX3_2.continuous
Writing header.
File ID: 0x7f31bc0070e0, number of bytes: 1024

Wrote header.
Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.

42190227991991c6
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
5f7fcb7fa07f5d80
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
cd7e387f587fa27f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
487f737f997f4080
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
647ffa7f01805c80
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
137f307fec7ef27e
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
dd7fb47f517f6c7f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
b47f517fd77f917f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
aa7fa77fd77f8a7f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
0180e37fae7fb47f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
567f0c7ff07edb7e
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
8e7fd87fde7ebc7e
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
d37f1180897fcc7f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
ee7f1380dc7fb07f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
9c7f5f7fd77e957e
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
777f5d7fb77eb77e
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
bc7ff97fbe7fd47f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
4e7fbf7f487f677f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
997fb37f477f407f
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
d07f4b80d57f5b80
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
d77f2b80d97f0b80
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
4980a5808580f080
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
64809f800e808780
RHD2000Thread::updateBuffer 0Error in Rhd2000EvalBoard::readDataBlock: Incorrect header.
c97f32805980b97f
                                                              <------------------------------- goes forever if not closed

[Christian Tatarau]

Hm, I've just noticed I might have made a mistake in printUsbHeader but I'm not sure.
I declare char hex[8]; and use it as if it was single char here

    for (int i=0; i < 8; i++){
        sprintf(hex, "%02x", (unsigned int) usbBuffer[i]);
        std::cout << hex;

Not sure whether it prints the correct header.

[Aarón Cuevas]

Hi,

Well, independent of the way you're printing it, the fact that the header is different in each call is definitely an indicator of a synchronization error, since the header should be the same for each block. Since the opal kelly library is not complaining, we can the issue is probably somewhere else.

From what I see, definitely something is making the usb transmission go real slow, making the internal buffer in the FPGA to overflow and, thus, data getting completely out of sync.

Have you looked at the dmesg logs to see if there are any low-level USB errors? Also, have you tried with the newer libOkFrontpanel.so (the one in the usb3 folder, like I explained in a previous mail)?

Also, as a really naive possibility, do you have any bandwidth-consuming usb device connected to a port adjacent to the one you're using? Usually for each two external usb ports, the computer has actually just one internal port with an internal hub, so adjacent ports share bandwidth. Have you tried if the freezes happen in other usb ports?

Aarón.

[Christian Tatarau]

That is a part of the dmesg output:

[ 5295.543369] usb 2-1.7: new high-speed USB device number 3 using ehci-pci
[ 5295.636978] usb 2-1.7: New USB device found, idVendor=151f, idProduct=002d
[ 5295.636983] usb 2-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5295.636986] usb 2-1.7: Product: Opal Kelly XEM6010-LX45
[ 5295.636988] usb 2-1.7: Manufacturer: Opal Kelly
[ 5295.636990] usb 2-1.7: SerialNumber: 1447000A3C
[ 5306.360060] systemd-hostnamed[4774]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[ 9742.325745] perf interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[11326.929139] systemd-hostnamed[5360]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[12207.979245] snd_hda_intel 0000:01:00.1: IRQ timing workaround is activated for card #1. Suggest a bigger bdl_pos_adj.
[12290.004647] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.008889] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.013146] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.017415] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.021676] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.025920] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -71
[12290.029923] usb 2-1.7: USB disconnect, device number 3
[12290.030169] usb 2-1.7: usbfs: USBDEVFS_CONTROL failed cmd Data Thread rqt 192 rq 181 len 64 ret -108
[12298.529247] usb 2-1.7: new high-speed USB device number 4 using ehci-pci
[12298.622209] usb 2-1.7: New USB device found, idVendor=151f, idProduct=002d
[12298.622211] usb 2-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[12298.622213] usb 2-1.7: Product: Opal Kelly XEM6010-LX45
[12298.622214] usb 2-1.7: Manufacturer: Opal Kelly
[12298.622215] usb 2-1.7: SerialNumber: 1447000A3C
[14784.769273] usb 2-1.7: USB disconnect, device number 4
[262584.781061] usb 2-1.7: new high-speed USB device number 5 using ehci-pci
[262584.874766] usb 2-1.7: New USB device found, idVendor=151f, idProduct=002d
[262584.874780] usb 2-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[262584.874783] usb 2-1.7: Product: Opal Kelly XEM6010-LX45
[262584.874785] usb 2-1.7: Manufacturer: Opal Kelly
[262584.874787] usb 2-1.7: SerialNumber: 1447000A3C
[269029.228084] systemd-hostnamed[11281]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[517025.922607] systemd-hostnamed[15923]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[526291.311307] usb 2-1.7: USB disconnect, device number 5


The newer libOkFrontpanel is the next thing I'm trying.

[Christian Tatarau]

I tried on a different machine with the same OS Linux Mint 17.3. The newer driver for USB3 only works when the acquisition box is connected to the USB3 port (the other better computer has no USB3) but still freezes. It does not recognize the acquisition box when connected to a USB2 port.

Asking the other way around: on which Linux OS does the GUI work without freezing with USB2? Anybody here using it without problems?

[Christian Tatarau]

I propose a more pragmatic fix. Is it possible to change the FPGA behavior so that when the buffer can't take an entire block because it is almost full and data has not been read out by the GUI then the entire buffer is dumped and new blocks are being written to it. The GUI should then notice that some blocks are missing but never be out of sync. Lost blocks are then filled with 0 by the GUI and written to disk. Given the very rare occurence of such errors it should not make any difference if some 100 ms are set to 0 out of hours of recordings. The alternative of unnoticed freezing leads to huge data losses.

Or we just switch to an OS, whatever it might be, where it works flawlessly.

[Christian Tatarau]

Hi! What do you think about the bug-fix I am proposing? With a bit of pointing in the right direction I can try to work on the GUI. What is the size of the FPGA buffer, in case I want to read it out and empty it?

[Aarón Cuevas]

Hi,

I don't think it's possible to do so just un software. While emptying the buffer is easy (there is a function for flushing the buffer in the API) the issue is doing it in the right moment. The FPGA state machine does not send whole packets to the internal FIFO, but it fills it with the data whenever it gets read. For this to work using a software-only solution the flush would need to be done at the same exact time a new packet is starting to get send into the FIFO. For 32 channels ar 30KS/s the internal FIFO holds around 40 seconds, so just flushing and zeroing everything would not be ideal, though.

Nevertheless, we definitely could, and should, modify the firmware to at the very least avoid getting stuck in an out-of-sync situation.

Going back to why are you having this issue, which by all means you shouldn't, the fact that with the newer .so it does not work when plugged into a USB2 port worries me quite a lot, since that should not be the case. Perhaps there is some kind of issue between mint's version of libusb and the frontpanel .so file? We need to check that...

Best,

Aarón

[Christian Tatarau]

IF it holds 40s I doubt there is an overflow. Which Linux OS and kernel is best for the GUI?

[Christian Tatarau]

Which OS are you using for development and testing?

[Aarón Cuevas]

It might still be an overflow if, for some reason, the usb transmission is working ever so slightly slower than the acquisition rate (that might explain the apparent randomness). it definitely shouldn't be the case, though. For 32 channels the required data rate is not high at all (Around 15 Mb/s), but I cannot think of another explanation at this moment. I need to investigate the issue further.

Most of our linux tests have been done under Ubuntu 14.