Kivy Python to Android serial best approach/ practice ?

[Roger Morrell]

I have noted elsewhere I'm porting an existing Kivy App to Android

One of the 'to do" items is to port some serial I/O that uses a USB  to serial dongle. Thisis done with  the generally used PYSERIAL library. This was initially done on windows and works on both Linux and Raspbian as well with no changes needed

I have read that 'most' recent Android implementations will support this (USB to serial dongle) and have seen examples on how to do this in Java .

It seems that there are 3 ways I might do this :

make a recipe for the pyserial library, which I do not think has been done yet ? (or one of the other serial libraries on PYPI) 

use Plyer

use Pyjnius

Has anyone done this before ? 

or a suggestion from someone wiser on which might be the best way to do it ?

thanks

Roger

[QuanTech]

USB to serial is very popular and useful for electronics engineers. Implementing it with Android largely increases mobility and convenience.

The main hard bit is that Android does not have the drivers to these USB-to-serial devices built in. "Pyserial" is an excellent library for this job, but it still relies on the underlayer drivers that are lucky included in the main stream desktop OSs.

I'm just starting a simple library to make it easier. For the moment it just supports FTDI devices. More are coming when I get other devices.

https://github.com/jacklinquan/usbserial4a

Or refer to a previous post talking about CDC ACM driver:

https://groups.google.com/forum/#!topic/kivy-users/GcN15VS3NA0

[QuanTech]

Now it supports CDC ACM driver.

https://github.com/jacklinquan/usbserial4a

[Roger Morrell]

Quan Tech

I have run your latest test program against my USB dongle and it works !   as a FYI  mine is 'custom generic' CDC that is implemented using a PIC24FJ128GB202 mcu.  This model PIC has built in USB capability. The PIC reads the USB serial connection  from the PC/Linux/ and now Android (well almost) and my code does some conversion/translation needed to talk to other 'embedded' hardware components

I used a modified version of your 'example.py'  to have a 2 way conversation with my dongle and this proved that a 2 way conversation is possible.  My application has a more complex conversation and uses a few more pyserial capabilities so I have more work to do to get everything working.

I used your Pydriod 3 App and this is very convenient to do this kind of test. I do have a question . I tried to follow your instructions and used pip to get the usbserial4a . That  worked.  You say them to open the example.py code .... did that end up in pydroid environment as part of the pip load ? it was not in the samples you give.  I ended up by getting a copy on my PC and emailing as an attachment that I then downloaded to the tablets downloads directory and from there got it into Pydroid3 - sorry for this this being a little off topic .. but is there an easier way  I was supposed to do it ?

Roger

[QuanTech]

Hi Roger,

I'm very glad to hear this library works for your application!

As you found, this library installed with pip does not include the example. The example can be only downloaded from github(or just copy and paste the plain text from the web page since there are only a few lines). I copy the file by connecting my laptop with the android phone/tablet through a normal USB cable and transfer the file when the phone/tablet appears as a USB drive.

I personally prefer Pydroid(the python 2 app) to Pydroid3(the python 3 app) because "python3" + "python for android" is not yet a very mature combination. Some modules does not work well with Pydroid3 or needs workaround. Anyway if you don't come across any problem, it's just fine.

Thanks for using it!

Quan

[Roger Morrell]

Quan

I have moved ahead with my application and have encountered something I don't understand

in my application the host (Android)  reads chunk on data from the embedded system. These can vary in size.  The way I do it is use the readinto() into a buffer that is big enough to hold the max abount of data .  I set a timeout that can vary depending on the amount of data  . it is typically between 2 and 6 seconds.   When I tried to do this with usbserial4a it never worked and I never got the full amount of data .  I thought the problem might be caused by the fact I was changing the time and that function was not supported, so I removed the timerout comments and set the timer at 10 seconds at the opening of the serial port.  I was getting about 6 character in the buffer.

This did not make any difference

in most cases my application data is in JSON format and ends with a } that does not appear anywhere else.  So I did a simple loop reading the serial line one character one at a time and testing for the ending }. This worked and the chunk of data I read was 471 bytes - what I was expecting.

It appears to me that timeout may be coming in too quickly.  I looked at the usbserial4a code and I noticed that the calculation was an int.  In the pyserial documentation it says the timeout is a float. Could this be a reason for the issue ?  Looking at the time stamps on the log it seems to be timeing out id about half a second

I have some code I can show you  and am willing  to do any testing that might help.

In a separate issue in your demo example and in mine it asks for permission  to access the USB port , that that be set as a buildozer/p4a  --permission. I can't seem to find it's name?

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[QuanTech]

Hi Roger,

Thank you for testing the library and sharing your result.

About the timeout parameter, the implementation of usbserial4a directly pass it to the usb communication interface of the hardware. As you found, in the code there is an int timeout. It is required to be int and passed to a java function that is wrapped by pyjnius, and the unit is "milli-second" instead of "second". So the behavior of timeout largely depends on the hardware(in your case the dongle implemented with a PIC micro). In the software, you can set whatever value you want but the hardware might not accept it. I'm not sure how the drivers on the desktop OS cope with variations of the hardware and implement this function, but they could do it in a more sophisticated way. What I can imagine is a loop in another dedicated IO thread keeping reading from the hardware and queue the received data in a large enough buffer. And all the timeout behavior apply to accessing the buffer.

As for the read method of usbserial4a, if the application requires continuous reading, I suggest putting it in a loop in another thread. So the timeout can be a small value and does not matter very much.

About the USB host permission, it's a very special one on Android. And this is also the reason it is somewhat complicated to set up. It anyway can not be set with the normal permissions. For Android 4.0+(probably can down to 3.1+) you have to go through the readme of  https://github.com/jacklinquan/usbserial4a to get permission when you plug the usb device to your Android device. For Android 6.0+, a good news is the USB permission can be required on the fly and there are some java APIs wrapped by pyjnius to do that. In usbserial4a, this happens automatically when you try to open a serial port or try to call "get_serial_port". But by the time the system ask you for the permission, the "open" method already returned and the serial port status is still NOT open. So after you accept the permission you need to call "open" again to use it. There's a better way to do it with the usb4a library(usbserial4a depends on it). When I get enough time, I can come up with another example with UI and permission handling for Android 6+.

I hope this answered your question.

Quan

[Roger Morrell]

The way the timeout works with pyserial is independent of the USB to serial converter.  I set the timeout - say 2 seconds  and I issue a readinto a buffer of say 600 bytes.  This is a non-blocking read. It reads until either the 600 byte buffer is full or 2  seconds have elapsed.  It tells you how many characters have been read.   All of that work is done on the host i.e. the Android side.  It works like this for both Windows and Linux and they do not know what kind of USB hardware I'm using . You are probably correct that operating system is doing a parallel thread or similar that manages the time out, lets you do a non-blocking read  and nables the feature that lets the application see how many character are still in the input and output buffers.

about the current implementation

Firstly one capability that is very good is the way you can find the address of the USB port.  If you did not you the end user would have to try and figure it which on Android is next to impossible.  On a PC for example under  you can use the Device manager to find the com port number and enter it. 

can I issue a port.timeout function ? or do I have to set the time out when the port is opened ?

How does the timeout work ? for example if I set the timeout to zero - and issue a read , should the read come back right away if there is no character to be read ?

I'm not clear from looking at the code if a want a a 2 second timeout is I should set it to 2 and it is multiplied by 1000 inside the function ? or I should put 2000 ?   

What happened in my testing is that the code would lock up and my app would be unresponsive . Looking at the Android log my application was  stopped with an Android On_Pause, I assume because it issued a read and no character had arrived yet - then it said it would not start because I did not have the driver for the Android On_Resume.  In the end I was able to get around most obstacles because the App can knows when it has read the last character it was expecting. 

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[QuanTech]

Hi Roger,

Timeout can be set at any time even after opening the serial port. Like: "serialport.timeout = 2" This will set the read timeout to 2 seconds.

But with usbserial4a, do not trust this timeout function. Unlike the USB serial drivers on desktop OSs that adds another logic layer to make it hardware independent, usbserial4a directly feed this timeout value(*1000) to the hardware interface. And whatever you set, the hardware could accept, cap, or even reject it depends on the hardware implementation.

I'm writing another example with UI that hopefully can help you with your application.  

[Roger Morrell]

This is some background information  that might be useful

As you talked about right at the beginning there is sometimes the need to have a "nice" UI to talk to an embedded system to do configuration or read out special information.  Adding a screen and perhaps a 'keyboard' is not practical for a  number of reasons, like size, cost, complexity, ...  Years ago, about 20 in my case I did this with a Palm Pilot as it a number of other people, these had a serial interface that would connect easily with an embedded system.  And there has been a steady  market in old Palm Pilots to  keep on doing this.  Now the challenge is to connect to a legacy embedded system with a more modern device that the Palm. The problem is that most modern portable platforms don't have a serial interface.  This can be done using a using a USB to serial converter.  This requires the portable platform to be a USB host and support USB CDC communications.  Most laptops and tablets do this.  The ones that don't are iOS.  

In my case the serial connection on the embedded system is at logic level with a special connector so I made a USB to serial dongle to do this.  In this case and typical of embedded system I have control of the serial end so I can set it , speed, number of bits, flow control, etc at the values required.  This is why there is no need to pass this  information through from the USB side to the serial side because the serial side knows it already..

Going forward with a new embedded device I would (and am working on it ) use some form of wireless connection, typically bluetooth.  So what's attractive about Kivy  in general and P4a for androids is that we can support both a 'traditional' serial connection and 'modern' bluetooth with hopefully close to  the same code.   There may be some embedded systems  where a wireless connection is not practical for environment reasons.  The add-on complete  bluetooth modules that can be integrated into an embedded system are very small and cheap so in most cases size and cost is not a problem.

But I still look forward to able to control communications timeouts better on the Android side, simliar to what's done in Windows and Linux.

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[QuanTech]

Hi Roger,

Thank you for sharing your experience and explaining your needs. Now I understand better on your case and realise it is really important and necessary to make the timeout behaviour more compatible with the implementation of pyserial. I got a rough idea to make the timeout function behave similarly without introducing another thread. I'll try to make it happen as soon as possible after holiday. I don't have those devices at home for testing and debugging. I'll update my progress when I get a chance.

Quan

[Quan Lin]

Hi Roger,

The latest version(0.1.5) of usbserial4a is ready. Now the timeout behavior is similar to serial.Serial.

It's tested with MCP2200 for the CDC ACM driver. Please test it with your application.

Good luck!

Quan

On Wednesday, December 26, 2018 at 9:52:18 AM UTC+11, Roger Morrell wrote:

[Quan Lin]

An example with UI is added for usbserial4a(https://github.com/jacklinquan/usbserial4a)

https://github.com/jacklinquan/usbserial4a/tree/master/example

This example works on Android and desktop OS(Windows, Linux and OSX)

[Roger Morrell]

Quan

I have been traveling but am about to do some testing and will keep you posted

Roger

Sent from my iPhone

--
You received this message because you are subscribed to a topic in the Google Groups "Kivy users support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kivy-users/PYZ5iZ2iRqk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kivy-users+...@googlegroups.com.
To post to this group, send email to kivy-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kivy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/a7d412c7-fd36-4464-87d9-0754c2047edd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yves Surrel]

Hi Quan

I will test on my side, too (maybe not immediately however).

Yves

You received this message because you are subscribed to the Google Groups "Kivy users support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kivy-users+...@googlegroups.com.

To post to this group, send email to kivy-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kivy-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/8749CFC0-0257-4B40-84D4-68D694232556%40gmail.com.

[Roger Morrell]

I have done some testing.  Basically it works but sometimes I get some dropped characters, it may be to do with the data block sizes .  I will investigate further. But the readinto and timeouts word the same way as windows and linux.  The android tablet probably has less memory for buffers.

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Quan Lin]

Thank you Roger and Yves for spending time testing this library!

I really appreciate that.

Quan

[Quan Lin]

Hi Roger,

That's very important information. I experienced once or twice dropped byte during my tests with MCP2200. The wiring condition of my prototype is not very neat, so I thought the reason could be bad connection. Now since you can repeat this problem, I think there may be something I can improve in the CDC ACM driver.

If you are doing more tests, could you help estimate how frequently it happens? Let's say how many bytes dropped in a long message(in your application 400~500 bytes in a single message), and how many messages are affected among several messages(e.g. 1 in 10 messages or 1 in 100 messages) 

Thanks a lot!

Quan

[Roger Morrell]

To update you I have done a number of tests 

I use the readinto(bytearray) function of pyserial

The byte array size varies in size from 10 to 2500 bytes 

It works very well when the block size is small.  How ever with bigger blocks, it drops characters.  I have experimenting with the timeout and it does not seem to make any difference.  It appears that the number of dropped characters gets worse as more data is read.

I did look at the code and noted that the DEFAULT_READ_BUFFER_SIZE is set at 1K.  I tried making this bigger but this made no difference.  I did an experiment of instead of doing one 2K readinto , doing 2 by 1K reads and then the first 1K was read Ok but the second dropped 20 or 30 characters.

What I am going to try is reading in smaller blocks and see if that makes a difference

What is surprising ? is that if I do not use the readinto () function but read the characters one  at a time looking for the end character no characters are lost.

Looking at this behavior my assumption is that there is more overhead in using the readinto (with possible timeout) than the individual reads.  This character  loss does not happen on Windows or Linux, including on a RaspberryPi.  The application not designed to always have a terminating charcater that could be tested for and the timeout provided a recoverable catch in some error situations.

I will do some testing as see if I can get around this problem.

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Roger Morrell]

Here is some additional information.

It is maybe more than you need to know

I will refer to the 2 sides of the USB link as the Android and the Dongle.

The Dongle is an embedded system with a PIC24 that has builtin USB capability.  The Android application is used to configure the embedded application that runs on the PIC and collect information from it.

 

Both sides of the USB connection – Android and Dongle are set to speed of 9600

Under many cases the Android App sends  a command to the Dongle and the Dongle responds quickly. These work with no problems.  The amount of data transmitted is 3 to 10 bytes.

The problems occur when larger amounts of information are sent back and forth.  These are configuration files and are in JSON format.  They are between 400 and 2500 bytes in length.  These files are sent in both direction. There does not appear to be an issue when the file are sent from the Android to the Dongle.

The Android sends a command that means “send me your config”,  and then issues a readinto for the max amount of data it might get back.

Dongle spends some time collecting this information, during that time it sends some  “progress indicators”  back to the Android.  The time between each progress indicator is Less that the timeout period but greater than the inter character gap at 9600 baud. I am approximating the time to send 1 character at 9600 baud  is 1 millisecond.  The delay between progress indicators is about 0-2 seconds and the time out set at 3-5 seconds.  Then when all the data is collected by the dongle it is sent back as a character stream.  The config data is in a binary table on the dongle and the program steps through the data item by item genetating a JSON formatted item and sending it.  For example if one item was temperature with a value of 21  The dongle generates a line that says in ASCII

“Temperature” : 21 \n\r

And sends it . Essentially keyword value pair.  This happens faster or close to the inter character gap at 9600 baud.  None of data is lost

Then there is a second case where the there are multiple value items for the data item and look like this,

"Step 4": [0,1,90,"J",110,10,70,2],\n\r

 

In generating this line in ASCII there are 8 binary to ASCII numeric conversions so this kind of line takes longer to generate that the simpler line.  I do not know how much longer.  There are about 20 lines like this and data is always lost on some of the lines. I think that the time to generate this line is longer than the inter character gap at 9600 baud.

 

Details of a specific case

 

Original situation (works on windows and linux)

 

Do a

buff = bytearry(2500)

count = readinto(buff) – expecting just under 2k with a buffer size of 2500.

 

This returns a count of 34 and stops during the preliminary “progress indicators”

The amount expected was 2020

 

 

The second test is splitting the above into 2 separate readinto

 

buff1 = bytearray(1020)

count1 = readinto(buff1)

 

buff2 = bytearray(1020)

count1 = readinto(buff2)

 

there is no delay between the 2 pairs of instruction.

 

In this case count1 = 1020 and all 36 of the preliminary progress indicators are read plus JASON data. In this part of the JSON data there are no multivalued items and no data is lost.

 

Then count2 = 930 , I was expecting 1000 and 70 characters were missing, scattered though out the data .  The last data item in the JSON stream is a single value item and was present.

 

 

Conclusion – two slight different kinds of error

 

First case – when the inter character gap is greater than 9600 baud and the block read is very large, perhaps bigger than the DEFAULT_READ_BUFFER_SIZE ? then the read times out soon on one of the inter character gaps

 

Second case – when the inter character gap is greater than 9600 baud and the block read is smaller than the DEFAULT_READ_BUFFER_SIZE ? then one block is read OK

 

But when the second block is read some char again with a probably more big inter character gaps than the first  block is read until the end but there are characters missing from the middle of the block.

 

Question – about how the USB CDC protocol works – when you set the speed to 9600 does it really run at 9600 or is that there to control the speed con the serial side coming out of the USB and is something that is used by the USB to serial converter ?

Question Does my Android tablet – a Lenovo  TB-8504F , Android 8.1.0  RAM 2GB , ROM 16GB have enough horsepower hardware and software to handle this data rate . Should do because for example internet access via WiFi and other datatransfers vis USB seem good. – The is more over head running python .

 

Question in looking at the serial4a code I noticed that the hardware read  , i.e. _read()  has a time out of 1 millisecond, about the same as the time for one character ? is this relevant?

 

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Quan Lin]

Hi Roger,

Thank you very much for your very detailed information on what's happening with your application.

To answer your question first:

1, About the baud rate, this is applied to "USB to serial" converter. Between Android and the dongle, that's the USB protocol with USB speed which is much faster than the serial baud rate. Android sends a command to the dongle and sets the baud rate, number of bits, parity and number of stop bits. This configuration is for the dongle to communicate with other devices on the serial line. In your case since it's a micro-controller, it can decide to take the configuration sent by Android or whatever else hard coded.

2, I'm quite sure your Android tablet overkill this application. 9600 is a very basic baud rate and even a much-lower-end device can handle it very well. 

3, Yes, USB_READ_TIMEOUT_MILLIS in the hardware read method(_read) is the one I suspect the most.

When I was doing some tests with FT230X(FTDI driver) and MCP2200(CDC ACM driver), they behaved very differently on this parameter.

Briefly, USB_READ_TIMEOUT_MILLIS is used as hardware read timeout.

What the driver does is to read from the hardware in a loop and queue the data in a software buffer, until all the required data is received or the software timeout expired.

You can try to set USB_READ_TIMEOUT_MILLIS to a larger value after you create the serial port like this:

serial_port = serial4a.get_serial_port('serial port name', 9600, 8, 'N', 1, timeout=the_timeout_you_need)

serial_port.USB_READ_TIMEOUT_MILLIS = 10

And then try to read a large block to see if there is any improvement.

I will also do some experiments with my devices to try some different value tomorrow.

Quan

[Roger Morrell]

Further update

I increased the USB_READ_TIMEOUT_MILLIS = 10

and this made it worse. Lost about twice as many character, almost10 percent 

I did go and look at the code on the sending side.  The way it works is a little more complicated than I remembered.  The output buffer is 64 bytes. The lines where it is dropping characters are longer than 64 so these are generated in  segment.  The code generates the line and sends it.  The send function check the output buffer in a loop until it is empty, then sends the block.  Even with splitting up the lines into smaller segments there is more  work to generate each smaller segment because of the printf statement that do the binary to ascii number conversion .

Also during the test process , although not when I did the USB_READ_TIMEOUT_MILLIS = 10 change as it is not relevant - I go back and test the Windows version to make sure it still runs and I did not inadvertently break  something.

roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Quan Lin]

Hi Roger,

I did some experiments with my MCP2200 device. The data block I'm trying is more than 3000 bytes. Sometimes it works fine, but sometimes it is just like what you described, dropping some data.

It looks like in the reading loop sometimes the USB data transfer function failed.

I tried some ways to improve it. The most effective one is to introduce a little bit delay in _read() method. So in the reading loop the USB data transfer function will not be called back to back. The PIC might not like too quick repeated USB data transfer.

Just for your information, MCP2200 is actually a PIC micro-controller programmed with a firmware that make it behave like a USB to serial bridge. 

Please help try this solution:

1, set USB_READ_TIMEOUT_MILLIS = 10

2, add some code in _read() method like below (the highlighted part is what I added):

def _read(self):

        '''Hardware dependent read function.

        

        Returns:

            read (bytes): data bytes read from the serial port.

        '''

        if not self.is_open:

            raise portNotOpenError

        if not self._read_endpoint:

            raise SerialException("Read endpoint does not exist!")

        

        # Introduce a little delay for stable read operation.

        delay = Timeout(self.USB_READ_TIMEOUT_MILLIS/1000.0)

        while not delay.expired():

            pass

        

        # Get raw data from hardware.

        buf = bytearray(self.DEFAULT_READ_BUFFER_SIZE)

        totalBytesRead = self._connection.bulkTransfer(

            self._read_endpoint, 

            buf, 

            self.DEFAULT_READ_BUFFER_SIZE, 

            self.USB_READ_TIMEOUT_MILLIS)

        if totalBytesRead < 0:

            # Read timeout. Set totalBytesRead to 0.

            totalBytesRead = 0

    

        read = buf[:totalBytesRead]

        return bytes(read)

Please update with your test result on this modification. If it's successful, I will include it in the next version and upgrade it very soon.

Good luck!!

Quan

[Roger Morrell]

This is just a quick note to say I tried inserting the delay and it did not work.    I need to do more testing, which I will do soon.

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Roger Morrell]

Results of testing with the delay on _read.

High level – My code did not work.  What happened initially is that it would not get past the “progress indicators that precede the main data transmission.   I ‘fixed’ this by changing the frequency that they were issued. In the standard pyserial this can adjusted with the intercharacter gap, but I have never had to do this.  I did not try using this with the Android.  Then when transmitting the main blocks of data some was lost, this was the same as happened before the delay was put on the _read.  In my case this seemed worse than before. Some was lost in a smaller data block of about 500 bytes that previously never lost data.  So I do not think it is solved yet.

Even higher level comment.  Quan’s comment about engineers wanting to user serial communication to talk to embedded equipment,  some specialized equipment and do data capture is very valid.  The way pyserial works on Windows and Linux does this quite well. It provides quite a number of options and is very forgiving.  Basically my app – sends a command that might get 2 to 3K of data back. I know this is going to take 2-5 seconds . I don’t know exactly how much data nor it format nor exactly how long it will take.  My app uses  port.readinto() combined with a port. Timeout() that says this and it works. It does not mind if the data flow is slightly irregular in rate , exactly how many or how long and no data is lost.

Where we are at this point is that if the data comes at a slightly irregular rate, it times out and does not seem to recover.  And when the data stream is flowing it drops characters.  It does work consistently with smaller data blocks, approximately up to 500 characters when the data flow is regular.

These days people expect to do many things with their phone or tablet. These devices  are not well equipped to connect to some external devices.  In doing something new from scratch on both sides of the connection probably the best way to go is Bluetooth or similar.  But there are lots of serial devices we have to communicate with.

In the interests of full disclosure – I have been involved in IT since the 1960s, I worked for startups and been the CIO of a  large company. I am retired.  I have written code either for work or other activites all of that time and still do some development. I have some systems , communications, applications and embedded software experience.  I’m more comfortable in C than other languages but some I do these days is in Python/Kivy for portability.  I’ve worked with Microchip MCU as a hobby activity for over 20 years. I’m ok with Windows and Linux but not so familiar with the insides of Android.

Roger

On Thursday, December 13, 2018 at 4:39:17 PM UTC+13, Roger Morrell wrote:

[Quan Lin]

Hi Roger,

Thank you for the test result and sharing your experience!

You are one of the very early batch of people who work on IT and electronics engineering. I'm sure you are definitely far more experienced than I am. I hope I can keep programming and electronics my life long hobby.

I work for a company as electronics engineer on various projects like pump control, heater control, communication, automation and testing. So I'm involved in all of hardware, firmware and software.

Python + Serial is a very convenient combination for me to do testing. Implementing it on Android will make my life easier.

I tried more on reading large data block with the CDC ACM driver.

Another idea to improve it is to increase the size of the hardware reading buffer and allow more time to read from the dongle each time in the loop. This can reduce the number of hardware reads, so the chance of USB transfer failure will reduce.

I tried this solution on more than 3000 bytes data block with the following configuration:

serial_port = serial4a.get_serial_port('serial port name', 9600, 8, 'N', 1, timeout=10)

serial_port.DEFAULT_READ_BUFFER_SIZE = 16 * 1024

serial_port.USB_READ_TIMEOUT_MILLIS = 100

And the code to introduce delay is totally removed.

With MCP2200 I read more than 3000 bytes each time and repeated for more than 10 times, no data lost.

Please have a try with your application.

Good luck!

Quan

[Roger Morrell]

Quan

I tried the 

suggestion

serial_port = serial4a.get_serial_port('serial port name', 9600, 8, 'N', 1, timeout=10)

serial_port.DEFAULT_READ_BUFFER_SIZE = 16 * 1024

serial_port.USB_READ_TIMEOUT_MILLIS = 100

and it works I need to check a bit more just make sure.

That's great . You would probably tune the DEFAULT_READ_BUFFER_SIZE depending on the max size if expected data block.

It did not fix the irregular gap between the 'progress indicators' that precede the main data all together in one block.   I will investigate that a bit further and pit a little more instrumentation on my side.

[Quan Lin]

Hi Roger,

After further experiments, I think the key to keep read operation stable is to allow enough time for the USB data transfer operation. Some time margin is necessary.

For DEFAULT_READ_BUFFER_SIZE = 1024, the safe USB_READ_TIMEOUT_MILLIS is about 30.

You can try to confirm my finding with this configuration:

serial_port = serial4a.get_serial_port('serial port name', 9600, 8, 'N', 1, timeout=10)

serial_port.DEFAULT_READ_BUFFER_SIZE = 1024

serial_port.USB_READ_TIMEOUT_MILLIS = 30

It's interesting that for DEFAULT_READ_BUFFER_SIZE = 16 * 1024, USB_READ_TIMEOUT_MILLIS = 100 is enough. There could be some other fixed overhead in each USB data transfer operation.

Quan

[Quan Lin]

The latest version 0.1.7 is ready. 

https://github.com/jacklinquan/usbserial4a
The CDC ACM driver reading problem is fixed.

[Quan Lin]

Version 0.1.8 is ready now.

CP210x serial driver is added.