USBSerial: Incoming data truncates

[Peter van der Walt]

Hi, first of all. Love droidscript! Only discovered a couple weeks ago, but its amazing.
Secondly, I have a working implementation using USBSerial. But, I have a little problem.

1) My connected device sends a string (Could be between 30-60 characters).
- The string is nicely formatted as "<datahere>\n" (So data is enclosed between Lessthansign and Greaterthansign, and each response is terminated with a Newline - should make parsing easier)

2) My callback

usb.SetOnReceive( function(e){
  if ($('#log div').length > 5) {
    // remove oldest if already at 300 lines
    $('#log div').first().remove();
  }
  $('#log').append($('<div/>').text(e));
});

Using this simple debug to log out the serial Data, I saw it often truncates the line before it gets to the \n

So I'd get something like


<Idle,10,10,10,0,0,0>\n
<Idle,10,10,10,0,0,0>\n
<Idle,
10,10,10,0,0,0>\n
<Idle,10,10,10,0,0,0>\n
<Idle,10,10,10,0,0,0>\n
<Idle,10,10,10,0,0,0>\n

<Idle,10,10,10,0,0,0>\n

<Idle,10,10,
<Idle,10,10,10,0,0,0>\n


(Looks like either the lines get truncated, or theres an extra Newline injected? ) 


I am porting this project to Mobile, from a NodeJS original. in NodeJS we use the Node-serialport module, and there, we setup the port like this:

port = new SerialPort('COM1', {
  parser: serialport.parsers.readline('\n'),
  baudRate: 115200
});

(Setup the Module's parser to only call the callback once it hits a newline) 

I am wondering if Droidscript's USBSerial maybe calls the callback once a buffer is full? Or what? 

PS: Is the source for USBSerial available? I'd like to take a look to understand it better, and i also want to look at adding some events (Disconnect so app can know when the USB maybe gets unplugged with the app running, etc) 

[Andrew Graham]

I once did a lot of work on writing low-level Android USB serial Java libraries, particularly for parsing GPS NMEA strings, although that was many years ago and the memory is a bit hazy :(

However you cannot assume that what is sent as one message will arrive as one transfer. I would implement the messages, as you are doing, with a unique header and terminator character, read the incoming characters into a circular buffer and when I saw the terminator character work backwards to find a header character, if any, and cut the message out if found.

Alternatively look for an incoming header character, start saving subsequent characters in a buffer, scan them for the terminator, and if found cut the message and check any remaining characters for a header and if so promote them to the start of the buffer : rinse and repeat for each transfer. There are various ways of skinning this cat.

Depending upon how sophisticated you want to get, and how important each message is  and the serial speeds you are using, you may need to implement various checks for missed messages caused by buffer overruns or corrupted characters but start simple and if possible make your protocol robust. Sometimes missing messages are not that important, in GPS work there's always another message coming down the line in one seconds time so missing one is no big deal.

[Peter van der Walt]

Thanks Andrew

Thats what I suspected.   With that confirmed, time to handle it as a buffer, and not in the callback directly (:

Have a great one and thanks for your help!

[Dave]

Hi guys,

A while ago I used to work on US AES emergency alert kit for TV stations which was serial controlled via a PC.  I used to have fun pretending an earthquake or nuclear strike had happened and watching the machine noisily pump out ticker tape Domesday messages.. like something from a cold war era movie.

Anyway serial comms (and buffering) can be a bit fiddly as mentioned by Andrew... that's why I added the SetSplitMode method to make your lives easier than mine was back then :)

Try using this line of code right after creating the USB component.  It will collect the returning data until it sees a newline character and then fire the callback.

usb.SetSplitMode( "End", "\n" )

Hope that helps

[Andrew Graham]

Thanks for the reminder Dave. I'd forgotten about SplitMode.

I note down interesting tidbits that I see in the forums and I've got a record that SplitMode has three options

 SplitMode('Size', p2),  SplitMode('End', p2) and SplitMode('Start-End', p2, p3)

'Size' and 'End' are obvious but what are the parameter meanings for the 'Start-End' option?

Some things never die. I've been writing serial comms for nearly fifty years but RS232 just won't die as it's so damn convenient!

[Dave]

The Start-End option should allow you to pick out the payload between two delimiters.. for example html or xml tags... not tested that option much tho

[Dave]

Since we seem to have a little serial appreciation society going here... 

You might like to know that I am working on a Plugin for the FTDI FT311/2 chips at the moment.  I've got the UART mode working on a UMFT311EV board already :)  

This little chip is very useful as it acts like your PC when connected to an Android tablet and allows you to charge a tablet at the same time as getting serial IO from the micro USB port.  Good for Android tablet based kiosks / machine control screens etc.  (most decent tablets don't have separate power inputs these days)

(Note: This will initially be for Premium users only)

[Andrew Graham]

That IS interesting. I've looked at the device datasheet and if I understand correctly it would let me interface an Android device to a load of devices with I2C or SPI interfaces that I have in a drawer left over from playing with Arduinos a couple of years ago.

Is that board commercially available or did you knock it up yourself?

[Andrew Graham]

Aha - I see Farnell have the UMFT311EV in stock for a not too unreasonable price.

If you plan to support the I2C and SPI modes I would be happy to act as a tester for the plugin if you need one.

[Dave]

If you look on ebay for "FT311D" you will find Chinese versions of this board which are cheaper and smaller too ;)

Yes I plan to support I2C and SPI modes... are you Premium?

[Andrew Graham]

Yes, I'm Premium :)

[Charles Wilt]

I think I have one of those dev boards lying around somewhere.  

[Charles Wilt]

On second thought....

Why would you use the FTDI311D instead of the espruino pico?

[Dave]

Because it can't power the device and the battery will eventually run flat!

[Dave]

However, you could use a Puck.js but you are relying on Bluetooth comms and suffer possible interference from wifi and motors etc + security issues.

[Charles Wilt]

I did not know you were powering the device.

That would explain it.

[Andrew Graham]

If you look on ebay for "FT311D" you will find Chinese versions of this board which are cheaper and smaller too ;)

I've got a board:)

I'm Premium

I'm ready when you are Dave if you need a tester!