Issues with analog inputs on Measurement Computing cards

[mata...@gmail.com]

I have computers running Ubuntu 14 with Measurement Computing PCI-DAS 1602/16 cards, and they've worked well for me with comedi. I have C-code that accesses the comedi libraries, and I compile that into Matlab MEX executables. Recently, I got some PCIe-DAS1602/16 cards and installed them and used them, and I found that the cards overall worked, but the analog input values I got seemed "off". That is, I could send out digital outputs, and read from the analog inputs, and at first everything seemed to work, but further inspection showed the analog inputs were scaled in a way that didn't make sense. I was able to tell that by sending in a +/- 1V max sinusoidal signal and reading it. 

At first I thought the issue was the new cards, but now after quite a bit of testing I've found that if I take the same computer and same hardware, and install a new version of Ubuntu (and presumably therefore comedi) the error happens. It goes away when I reinstall Ubuntu 14. I know that Ubuntu 18 & 20 show this problem, but 14 doesn't. I'm doing some testing to see if 16 also has the issue. I wonder if you have any insight into what might have been introduced in comedi over this time that could show this error with these cards? I can give you more details on the error, but essentially what I see is a signal that should be a +/- 1V sinusoid showing up as a much smaller value, but still a clear sinusoid. 

Let me know if you have any thoughts. I've tested enough to be sure that the computer and cards are working, it's definitely a software issue and not hardware issue.

[mata...@gmail.com]

To add just a little more info, here is a plot in Matlab of a +/- ~2V input into the same card on the top in Ubuntu 14, and on the bottom in Ubuntu 18. Same software in each case, in terms of the Matlab code that's running, the compiled Mex code and therefore C-code to access the comedi libraries, etc. And, same hardware so it's not a card or computer hardware issue.

https://ibb.co/5FC7WCv

[Ian Abbott]

It's probably a bug that crept in during some tidy-ups a few years ago.

If you have Linux kernel version numbers, that would help to narrow it down.

--
-=( Ian Abbott <abb...@mev.co.uk> || MEV Ltd. is a company )=-
-=( registered in England & Wales. Regd. number: 02862268. )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

[mata...@gmail.com]

OK, we've narrowed it down a bit. We have a machine with Ubuntu 16 that has a leftover kernel from Ubuntu 14 (it was upgraded from Ubuntu 14). We found that with the 3.13.0-170-lowlatency kernel everything is fine. However same hardware and matlab/mex/c-code accessing comedi, and with kernel 4.4.0-193-lowlatency (also 4.4.0-193-generic) we get the weird scaled output. Does that help narrow it down? 

[Ian Abbott]

On 20/10/2020 20:13, mata...@gmail.com wrote:
> OK, we've narrowed it down a bit. We have a machine with Ubuntu 16 that
> has a leftover kernel from Ubuntu 14 (it was upgraded from Ubuntu 14).
> We found that with the 3.13.0-170-lowlatency kernel everything is fine.
> However same hardware and matlab/mex/c-code accessing comedi, and with
> kernel 4.4.0-193-lowlatency (also 4.4.0-193-generic) we get the weird
> scaled output. Does that help narrow it down?

It does a bit. There were 90 commits to to the cb_pcidas driver during
those kernel versions. I have looked at them all. The only change that
looked like it _might_ possibly have had some sort of effect is a
refactoring of the code that writes to the calibration trim-pots that is
called during board initialization. It seems to have removed a 1
microsecond delay before one of the register writes. Here is a link to
the commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eddd2a4c675c95f19da79b5adbf46851b2441212

(That commit occurred between kernel versions 4.3 and 4.4.)

There ought to be a call to 'udelay(1);' before the 'outw' call in the
'if (trimpot) {' section of the 'cb_pcidas_calib_write()' function to
match the behaviour of the original code.

I don't know how much effect that has, but the function is called twice
in a loop during board initialization to write to the 'ADC Gain Cal' and
'ADC Offset Cal' trimmer channels.

One way to test if that is the problem would be to build the comedilib
demo programs and use the 'outp' demo to write to the trim-pots:

./outp -f /dev/comedi0 -s 5 -c 0 128
./outp -f /dev/comedi0 -s 5 -c 1 128
./outp -f /dev/comedi0 -s 5 -c 0 127
./outp -f /dev/comedi0 -s 5 -c 1 127

-f /dev/comedi0 selects the device, -s 5 selects the trim-pot subdevice,
-c 0 and -c 1 select the gain and offset trimmer channels respectively,
and 128 and 127 are the data values. Two different values are written
to each channel because the driver does nothing if the value written
matches the previous value. The trim-pot channel values for
PCI-DAC1602/16 are set to 127 during initialization, so the above
sequence of 'outp' commands changes the values slightly and then changes
them back. The missing 'udelay(1);' call mentioned earlier shouldn't
have any effect because it will be swamped out by other delays running
the commands.

As I mentioned, I don't know if this is really the cause of the problem,
but it is something to try. From the plots you posted, it looked more
like different input ranges had been selected somehow, but I haven't
spotted anything in the code commits to explain that.

Best regards,
Ian

> <http://www.mev.co.uk> )=-
>
> --
> You received this message because you are subscribed to the Google
> Groups "Comedi: Linux Control and Measurement Device Interface" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to comedi_list...@googlegroups.com
> <mailto:comedi_list...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/comedi_list/67e4c557-2598-4cee-ae2b-cb0ce144bc33n%40googlegroups.com
> <https://groups.google.com/d/msgid/comedi_list/67e4c557-2598-4cee-ae2b-cb0ce144bc33n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[mata...@gmail.com]

OK, we've now gotten a chance to do a bit more testing. We tried the outp option that you specified but had a few problems (more on that later). But we also tried installing a different kernel version to try to narrow down the search. We were able to install 4.15, and found that the error was still there in that kernel version. So maybe that points to the change you found between 4.3 and 4.4 as not the likely culprit?

Also, we tried to run 'outp', but when we ran it there was an error that said "/dev/comedi0: Invalid subdevice".

Running comedi_board_info produced the results below. I feel like I need to change the argument on outp, is it clear to you from what you see in comedi_board info what I would change it to? But then again, maybe this isn't relevant given the kernel info I've given you above.

Thanks again for the help!

OUTPUT OF COMEDI_BOARD_INFO:

overall info:

  version code: 0x00074c

  driver name: cb_pcimdas

  board name: cb_pcimdas

  number of subdevices: 3

subdevice 0:

  type: 1 (analog input)

  flags: 0x00110000

  number of channels: 16

  max data value: 65535

  ranges:

    all chans: [0,1]

  command:

    not supported

subdevice 1:

  type: 2 (analog output)

  flags: 0x00020000

  number of channels: 2

  max data value: 4095

  ranges:

    all chans: [0,1]

  command:

    not supported

subdevice 2:

  type: 5 (digital I/O)

  flags: 0x00030000

  number of channels: 24

  max data value: 1

  ranges:

    all chans: [0 V,5 V]

  command:

    not supported

[Ian Abbott]

On 30/10/2020 16:12, mata...@gmail.com wrote:
> OK, we've now gotten a chance to do a bit more testing. We tried the
> outp option that you specified but had a few problems (more on that
> later). But we also tried installing a different kernel version to try
> to narrow down the search. We were able to install 4.15, and found that
> the error was still there in that kernel version. So maybe that points
> to the change you found between 4.3 and 4.4 as not the likely culprit?

The suspicious change found between 4.3 and 4.4 is currently present in
all kernels from 4.4 onwards.

I submitted a patch to add the removed delay back, and tagged the patch
to be applied to "stable" kernel series, so it should get applied back
into distro kernel updates eventually.

> Also, we tried to run 'outp', but when we ran it there was an error that
> said "/dev/comedi0: Invalid subdevice".
>
> Running comedi_board_info produced the results below. I feel like I need
> to change the argument on outp, is it clear to you from what you see in
> comedi_board info what I would change it to? But then again, maybe this
> isn't relevant given the kernel info I've given you above.
>
> Thanks again for the help!
>
> OUTPUT OF COMEDI_BOARD_INFO:
> overall info:
>   version code: 0x00074c
>   driver name: cb_pcimdas
>   board name: cb_pcimdas
>   number of subdevices: 3

Oops. It looks like I've been looking at the wrong driver. I was
looking at changes to the cb_pcidas driver that is used for PCI-1602/16.
I need to look at changes to the cb_pcimdas driver too, because that
is the one used for the PCIe-1602/16!

[Ian Abbott]

OK, one mystery puzzling me is that the card was reported working on
Ubuntu 14 with a leftover 3.13.0-170-lowlatency kernel. However, in the
mainline Linux kernel, PCIe-DAS1602/16 was not supported in the
cb_pcimdas driver until kernel version 3.19.

The PCI device IDs recognized by the the cb_pcimdas driver since kernel
version 3.19 are 1307:0056 (for PCIM-DAS1602/16) and 1307:0115 (for
PCIe-DAS1602/16). Before kernel version 3.19, only 1306:0056 (for
PCIM-DAS1602/16) was recognized.

[Ian Abbott]

Those "ranges:" values for the subdevice 0 and 1 are interesting.

Before kernel version 4.1, the "cb_pcimdas" driver only supported a
single range for the subdevices. The range was from 0.0 to 1.0 with no
units. This was actually a lazy cop-out in the original version of the
driver.

The "comedi_board_info" output above appears to be for a kernel before 4.1.

In kernel version 4.1, the range support for analog input was enhanced
to allow either 4 different unipolar ranges or 4 different bipolar
ranges depending on the setting of the "Analog Input Polarity Switch" on
the board. The driver reads the board's "ADC Channel Status and Switch
Settings Register" during initialization in order to determine whether
the analog inputs are in unipolar or bipolar mode, and also whether they
are in single-ended or differential mode (which also sets the number of
channels to 16 or 8).

The unipolar or bipolar choice is set by the switch, but the 4 different
ranges are set by software.

Since kernel version 4.1, the supported ranges for analog input are:

Bipolar range 0: [-10V, +10V]
Bipolar range 1: [-5V, +5V]
Bipolar range 2: [-2.5V, +2.5V]
Bipolar range 3: [-1.25V, +1,25V]

Unipolar range 0: [0V, +10V]
Unipolar range 1: [0V, +5V]
Unipolar range 2: [0V, +2.5V]
Unipolar range 3: [0V, +1.25V]

(For the analog output subdevice, the range is set by switches and is
not software programmable. The driver allows the available ranges for
the analog output subdevice to be queried, but any attempt to select an
analog output range will be ignored.)

Could your strange scaling issue for analog inputs be due to the
software selecting a different input range now that more supported
ranges are reported since kernel version 4.1? Before kernel version
4.1, the hardware was always programmed to use the 10V range (which
would be either the bipolar or unipolar 10V range according to the
Analog Input Polarity Switch setting).

All the above is relavent for PCIe-DAS1602/16 and PCIM-DAS1602/16 (which
use the cb_pcimdas driver), but not for PCI-DAS1602/16 (which uses the
cb_pcidas driver).

[Matthew Smith]

Thanks for the debugging! OK, some additional background info. We have several machines with the PCIM-DAS1602/16 card. Those all worked fine, as of Ubuntu 14, which was when we were doing our original development. Then, several months ago we bought some PCIe-DAS1602/16 cards, and installed them in Ubuntu 18, and noticed these problems. We were convinced that it was a bad card, but eventually were able to determine the card is OK by putting it in a machine that uses Windows and use the Measurement Computing Instacal software and seeing the right input (sent in with a function generator). This is how we eventually learned that whether we use the PCIe or PCIM card, both produce the same (incorrect) results for us in the newer versions of Ubuntu. Just to add on to that, we don't have any PCI-DAS versions of the card (which use the cb_pcidas driver).

The machine I'm testing now has the PCIM card, but to my knowledge the exact problems happen also with PCIe. In any case, it's possible we somehow messed up in our testing of different kernel versions. The only reason we even tried that is that we updated the machine to Ubuntu 16, saw there was a problem there, and then tried booting back into the old kernel and saw the problem went away. And since we're using the PCIM card in that machine, I think that solves the question you had about kernel 3.19 - since this is a PCIM card it is one that should have worked on that old kernel too. But again, I get the same problems in both the PCIM and PCIe card, we just are using the PCIM card in the testing machine right now.

The analog input range could be in, potentially. We did a lot of testing with the DIP switchings when we first saw this problem. I'm confident those are set the same, and since I can boot into a different kernel and see the problem come and go then it shouldn't be a DIP switch. I've attached the C/MEX code we use to read from the device. Maybe that might help? You can see I use the comedi_get_range function.

Thanks for the help on this!

[Ian Abbott]

The PCIM card would have worked in kernel 3.13 so you were probably
correct about the kernel version used in the earlier report. The
cb_pcimdas driver treats the PCIM-DAS1602/16 and PCIe-DAS1602/16 as
essentially the same type of device - the only difference is that they
match different entries in a table of supported PCI device IDs.

> The analog input range could be in, potentially. We did a lot of testing
> with the DIP switchings when we first saw this problem. I'm confident
> those are set the same, and since I can boot into a different kernel and
> see the problem come and go then it shouldn't be a DIP switch. I've
> attached the C/MEX code we use to read from the device. Maybe that might
> help? You can see I use the comedi_get_range function.

That probably explains it. The code uses range number 0. In the old
kernel (version 4.0 or earlier) the raw value range [0, 65535] would map
to the physical range [0.0, 1.0] with unspecified units. But in the new
kernel (version 4.1 or later), the raw value range [0, 65535] would map
to a physical range of either [-10.0, +10.0] or [0.0, +10.0] volts
(depending on the Analog Input Polarity Switch setting).

In effect, the code is scaling the voltages twice when using the new kernel.

The attached, modified version of "unixGetEyes.c" uses the range
reported by the driver unless it appears to be the old, bogus range, in
which case it uses a replacement -10V to 10V range. The changed code
should be backwards compatible as long as the Analog Input Polarity
Switch is set to Bipolar mode.

[Matthew Smith]

Thanks!!!!! We'll give it a try and report back.

[Matthew Smith]

Testing complete. This works, thank you so much for the help on this. This was vexing us for quite a time as we were exploring whether the card was bad in some way, or the computer. Only once we figured out it was the Ubuntu version that caused the problem did we start to get on the right track.