Is Comedi limited to 32768 samples?

[John L]

I have a DAS4020 board running under Debian Squeeze.  It seems that Comedi will collect no more than 32,768 samples.  For example:

$ ./cmd -F 10000000 -N40000 -n1 | wc
command before testing:
subdevice:      0
start:      now      0
scan_begin: timer    100
convert:    now      0
scan_end:   count    1
stop:       count    40000
first test returned 0 (success)
subdevice:      0
start:      now      0
scan_begin: timer    100
convert:    now      0
scan_end:   count    1
stop:       count    40000
second test returned 0 (success)
start time: 1334965465.212570
read: Broken pipe
end time: 1334965465.236414
time: 0.023844
  32768   32768  196608

I can change the parameters but the 32768 sample limit remains consistent.  For example:

$ ./cmd -F 100000 -N80000 -n4 | wc
command before testing:
subdevice:      0
start:      now      0
scan_begin: timer    10000
convert:    now      0
scan_end:   count    4
stop:       count    80000
first test returned 0 (success)
subdevice:      0
start:      now      0
scan_begin: timer    10000
convert:    now      0
scan_end:   count    4
stop:       count    80000
second test returned 0 (success)
start time: 1334965703.839333
read: Broken pipe
end time: 1334965704.021100
time: 0.181767
   8192   32768  172032

Any suggestions?

Thanks,

John

[Ian Abbott]

Comedi only allocates a 64KiB buffer by default, which is only enough
for 32768 16-bit samples. There is a maximum read buffer size, a
maximum write buffer size, a current read buffer size and a current
write buffer size, but these are all 64 KiB by default.

There are various ways to increase these sizes.

An application can use the comedilib function comedi_set_buffer_size()
to set the current buffer size for a subdevice, but it cannot be set
larger than the maximum buffer size.

A superuser can change the maximum and current read or write buffer size
for a device using the 'comedi_config' command, e.g.:

comedi_config /dev/comedi0 --read-buffer-size 10240

would set both the maximum and current read buffer sizes for
/dev/comedi0 to 10240 KiB (or 10 MiB).

Alternatively, there are a some sysfs attribute files that can be
written to to set the maximum read or write buffer size in KiB. For
example:

echo 10240 > /sys/class/comedi/comedi0/max_read_buffer_kb

sets the maximum read buffer size for /dev/comedi0 to 10240 KiB (or 10 MiB)

echo 10240 > /sys/class/comedi/comedi0/read_buffer_kb

sets the current read buffer size in the same way.

--
-=( Ian Abbott @ MEV Ltd. E-mail: <abb...@mev.co.uk> )=-
-=( Tel: +44 (0)161 477 1898 FAX: +44 (0)161 718 3587 )=-

[Bernd Porr]

The buffer size shouldn't be a problem as long as you read out the data
regularly with the read command so that comedi's ring buffer stays
empty. Have a look at cmd.c: the main loop does reads as quick as
possible. There should never be a buffer overflow in kernel space. I
suspect that you either have a slow computer, heavy load or something
else which limits performance. Try setting cmd->stop_src = TRIG_NONE;
which makes cmd running endlessly until you press ctrl-C. Pipe the data
to dev/null and see if it runs forever. If it crashes at some point then
you know that your computer is even too slow just to read out the ring
buffer.

/Bernd

www: http://www.linux-usb-daq.co.uk/
http://www.berndporr.me.uk/
http://www.imdb.com/name/nm3293421/
Mobile: +44 (0)7840 340069
Work: +44 (0)141 330 5237
University of Glasgow
School of Engineering
72 Oakfield Avenue (Rankine Building for deliveries)
Glasgow, G12 8LT

[John Lawless]

Bernd and Ian, thanks much!

I increased the buffer as Ian suggested and that helped a lot.

My machine is very lightly loaded (X not running, 'cmd' run from
console) 3 GHz PC. If I open a second console and run top, it
shows that cmd is taking 100% of CPU with no other process
accounting for more than 0.1%. Despite being lightly loaded, data
collection is reliable only for data rates of 3 MHz or less.

3 GHz CPU speed / 3 MHz data rate = 1,000 clock cycles per data point

That seems like a lot of overhead to me. It might be because cmd
converts data to ASCII and then prints to stdout. Next, I may try
writing binary data directly to file.

Thanks,

Jack

> --
> You received this message because you are subscribed to the Google Groups "Comedi: Linux Control and Measurement Device Interface" group.
> To post to this group, send email to comed...@googlegroups.com.
> To unsubscribe from this group, send email to comedi_list...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/comedi_list?hl=en.

[Bernd Porr]

Hi Jack,

3MHz to ASCII is indeed pretty tricky. I've tried that with comedirecord
and the USBDUXfast board which also saves in ASCII and it's just too
demanding on most computers. As you said the ASCII conversion takes a
lot of clock cycles and then of course it creates about 10 times more data.

/Bernd

[Bernd Porr]

Talking about binary data formats. I'm also a strong beliver in ASCII
data for recording. However for higher sampling rates it needs to be
binary. Is anybody using a library which stores binary data in a
reasonably portable format. For me the problem with most of these
libraries is that they are just horribly documented and hard to compile.
I was playing with the library coming with the sigviewer (biomed
signals) but gave up after a couple of hours trying to even compile it.
Any people out there loving their favourite binary format? In the
meantime I stay with ASCII...

/Bernd

[W. Trevor King]

On Fri, May 11, 2012 at 11:22:30AM +0100, Bernd Porr wrote:
> Is anybody using a library which stores binary data in a
> reasonably portable format.

I haven't used the C interface directly, but HDF5 works pretty well
for me [1]. There are really nice Python bindings [2], which you can
use for analysis later.

[1]: http://www.hdfgroup.org/HDF5/
[2]: http://code.google.com/p/h5py/

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[Bernd Porr]

thanks! Time to add this to comedirecord. Shame that gnuplot won't like
binaries...

/Bernd

W. Trevor King wrote:
> On Fri, May 11, 2012 at 11:22:30AM +0100, Bernd Porr wrote:
>> Is anybody using a library which stores binary data in a
>> reasonably portable format.
>
> I haven't used the C interface directly, but HDF5 works pretty well
> for me [1]. There are really nice Python bindings [2], which you can
> use for analysis later.
>
> [1]: http://www.hdfgroup.org/HDF5/
> [2]: http://code.google.com/p/h5py/
>

--

www: http://www.berndporr.me.uk/
http://www.linux-usb-daq.co.uk/

Mobile: +44 (0)7840 340069
Work: +44 (0)141 330 5237
University of Glasgow
School of Engineering

72 Oakfield Avenue (for deliveries: Rankine Building)
Glasgow, G12 8LT

[Risa]

Our RTXI (www.rtxi.org) project was converted over to HDF5 a couple years ago and we love it for its portability and ability to store a bunch of metadata alongside.

-Risa

[W. Trevor King]

On Fri, May 11, 2012 at 01:02:37PM +0100, Bernd Porr wrote:
> thanks! Time to add this to comedirecord. Shame that gnuplot won't like
> binaries...

You can always do

gnuplot>>> splot '< h5totxt -s " " -d name' using 1:2:3

Which I'm stealing shamelessly from [1].

[1] http://groups.google.com/group/comp.graphics.apps.gnuplot/browse_thread/thread/988523a24195b125

[Bernd Porr]

Thanks for stealing that. ;) My PhD student Graeme (hello!) also
educated me recently about piping data into gnuplot even from the shell.

/Bernd

W. Trevor King wrote:

> On Fri, May 11, 2012 at 01:02:37PM +0100, Bernd Porr wrote:
>> thanks! Time to add this to comedirecord. Shame that gnuplot won't like
>> binaries...
>
> You can always do
>
> gnuplot>>> splot '< h5totxt -s " " -d name' using 1:2:3
>
> Which I'm stealing shamelessly from [1].
>
> [1] http://groups.google.com/group/comp.graphics.apps.gnuplot/browse_thread/thread/988523a24195b125
>

--

www: http://www.linux-usb-daq.co.uk/
http://www.berndporr.me.uk/
http://www.imdb.com/name/nm3293421/

Mobile: +44 (0)7840 340069
Work: +44 (0)141 330 5237
University of Glasgow
School of Engineering

[John Lawless]

To follow-up, I tried eliminating the ASCII conversion from cmd.c
and just writing out binary data. With this change, a sample rate
of 15 MHz, five times faster than ASCII output allowed, seemed to
work.

Jack

On Fri, May 11, 2012 at 11:22:30AM +0100, Bernd Porr wrote:

[Bernd Porr]

Hi Trevor,

h5totxt doesn't work with incremental datasets but here's the
appropriate replacement (always fun using shell commands):

plot "< h5ls -d -S ggg.hdf | tail -n +3" using 2 with lines

Or in case you need it in a file first:
h5ls -d -S ggg.hdf | tail -n +3 > /tmp/ggg.txt

Annoyingly the header cannot be switched off so that a tail command is
needed.

That makes it also compatible for the octave load command which fails
with the hdf files (MATLAB can handle them though, sigh!).

/Bernd



W. Trevor King wrote:

> On Fri, May 11, 2012 at 01:02:37PM +0100, Bernd Porr wrote:
>> thanks! Time to add this to comedirecord. Shame that gnuplot won't like
>> binaries...
>
> You can always do
>
> gnuplot>>> splot '< h5totxt -s " " -d name' using 1:2:3
>
> Which I'm stealing shamelessly from [1].
>
> [1] http://groups.google.com/group/comp.graphics.apps.gnuplot/browse_thread/thread/988523a24195b125
>

--

www: http://www.linux-usb-daq.co.uk/
http://www.berndporr.me.uk/
http://www.imdb.com/name/nm3293421/

Mobile: +44 (0)7840 340069
Work: +44 (0)141 330 5237
University of Glasgow
School of Engineering