Scheduler and Serial Port Problems

[Richard Brown]

I have used Web2py to create a Home Heating Control application for Raspberry Pi with a web front end to define the and I am using the Web2py Scheduler to perform the control tasks every 30 seconds. I have also designed a radio linked system for temperature measurement and boiler control based around a USB dongle plugged into the Pi.

  

The problem I have is how to initialise and open the /dev/ttyUSB0 port once at start up, but not reopen it on every subsequent Scheduler run as this resets the dongle via the RTS line as the dongle is based on the Arduino architecture. I have some simple test code running, this section is called by the Scheduler as follows:

 

In file 'poll.py:'

def ch_scheduler():

    from utilities import ch_control

    import time

    t = time.ctime()

    ch_control()

    return t

In file 'utilites.py':

from control import *

def ch_control():

    RelCtrl(1,0)

    time.sleep(0.1)

    RelCtrl(2,1)

    time.sleep(0.2)

    RelCtrl(2,0)

    time.sleep(0.1)

    RelCtrl(1,1)

    return

In file 'control.py':

import serial

ser=serial.Serial('/dev/ttyUSB0', 57600, timeout=1, xonxoff=0, rtscts=0)

def RelCtrl(in1,in2):

# convert relay parameters to integers

    relay = int(in1)

# 1: ON, 0: OFF

    relon = int(in2)

    if relay > 0 :

        if relon == 1 :

            command = 's1 ' + str(relay - 1) + '\x0d\x0a'

        else :

            command = 'c1 ' + str(relay - 1) + '\x0d\x0a'

        ser.write(command)

# wait for reply, read characters but ignore

        time.sleep(0.1)

        characters = ser.inWaiting()

        if characters > 0 :

            reply = ser.read(characters)

    return

Unsurprisingly this opens the port (and resets my dongle) each time the scheduler runs ch_control(). However, and bizarrely, if I save 'control.py' without changing its contents from the Web2py Admin Interface, or access my Site via a browser then the port does not get reopened on every Scheduler run and the system operates perfectly. This is of no help in practice as the background tasks have to operate autonomously from power on, but may offer a clue as how to make it work.

I have attached oscilloscope traces of the behaviour which show the transmitted commands to the dongle from each of the four calls to RelCtrl() (Yellow upper trace) and received data from the dongle (Cyan lower trace). 'Not Working from Boot.bmp' shows the port being reopened, the dongle being reset and not responding until the third command. 'Working after File Save.bmp' show what happens after 'control.py' was saved from within Web2py, and shows what I need to happen all the time.

 

So my question is: how can I configure and open the serial port the first time the Scheduler runs, and then access it without reopening it from then on? I'm not aware of any way to preserve 'ser' from run to run, nor of generating it on each run without reopening the port. I can test if a port is open, but only if I can reference it - which forces it to be reopened.

Thanks,

Richard

 

 

  

[Richard Brown]

An update to this - I have done some more investigation and discovered that in the 'Not Working' scenario the RTS output is mostly high but goes low whilst the port is open, and in the 'Working' scenario the RTS output has somehow been forced low and so stays low when the port is opened. The Dongle gets reset by a high-to-low transition of RTS, so no falling edge, no Reset and everything works. For reference my Dongle uses an FTDI device (FT230X) and hence the FTDI driver supplied with the latest Raspbian (Jessie) release.

The question now, therefore, is how Web2py forces RTS low - either by the saving of a file or when I access the index page of my Site, and How I can replicate this in the Scheduler code! Using 'touch' to update the file does not work, it has to be saved from within Web2py.

Richard

[Dave S]

On Friday, November 25, 2016 at 4:50:34 AM UTC-8, Richard Brown wrote:

An update to this - I have done some more investigation and discovered that in the 'Not Working' scenario the RTS output is mostly high but goes low whilst the port is open, and in the 'Working' scenario the RTS output has somehow been forced low and so stays low when the port is opened. The Dongle gets reset by a high-to-low transition of RTS, so no falling edge, no Reset and everything works. For reference my Dongle uses an FTDI device (FT230X) and hence the FTDI driver supplied with the latest Raspbian (Jessie) release.

The question now, therefore, is how Web2py forces RTS low - either by the saving of a file or when I access the index page of my Site, and How I can replicate this in the Scheduler code! Using 'touch' to update the file does not work, it has to be saved from within Web2py.

Richard

I can't imagine how saving the file causes RTS to go low.  Web2py should not have any reason to do anything to the serial port on its own, and probably doesn't even know the serial port exists (it's only concerned with the console). (I haven't done any serial port handling under Ubuntu, nor on the RaspberryPi, so I can't tell you about any special tricks such ports need.)

You may want to investigate having a separate program that gets started on system startup, runs forever, and holds the serial port open all the time.  It can check for a file existing, write the current value to the file, and snooze again.  Your scheduled task would the create the file, wait briefly, and then read the file to get the current value. 

I do know someone with a RaspberryPi, and he's a bit of hacker, but it will be next week before I see him again.

/dps

[Richard Brown]

I've now had chance to do some more investigation and research since my last posting and found the following. The reason that saving files in Web2py causes RTS to be asserted and makes my application work is that my file 'control.py', which contains all my serial functions, includes the lines:

import serial

ser=serial.Serial('/dev/ttyUSB0', 57600, timeout=1, xonxoff=0, rtscts=0)

If I remove/comment out the line 'ser=serial.Serial... ' then the file save has no effect. Presumably Wb2py must be executing this line somehow as part of it's parsing process, and saving my other file 'utilities.py' has the same effect because it contains 'import * from control.py'.

I have also been playing with the Pyserial functions ser.setRTS() and ser.rts, which should force the state of RTS. I have been running these from Python scripts in my user account and found that if reboot the Pi, save one of the files in Web2py to force RTS low, then execute ser.rts or ser.setRTS using my scripts, I can set RTS high or low and that state persists. However, if I reboot the Pi and run my script without saving the files in Web2py then run my scripts it only cause RTS to pulse low momentarily, presumably while the script is running. I've tried including a similar script within the Web2py 'modules' area, I've tried including the set.rts commands within my serial functions, I've tried a separate function just to include that command but nothing has done anything other than pulse RTS until the script/function ends. I haven't tried running a routine once at startup yet, but I'm not sure how this will be different from running scripts 'by hand', except as far as I am aware I cannot run scripts as the 'www-data' user as there is no login/password for that account, and I have a strange feeling that there must be some sort of ownership issue here. I have written/hacked scripts to run at startup to start the Web2py Scheduler and GPIOSever but this is all a bit close to the limits of my knowledge and ability!

In summary, I am further forward in the sense that I understand why, if not how, Web2py file saves make my application work, although not why RTS gets set permanently low by this mechanism and not when the serial port is opened at any other time, and more information on what is happening here would be useful. I also know that there is a mechanism for changing the state of RTS, which it must be possible to do somehow, but have no real idea how to do it. I also now believe that keeping the port permanently open is not necessary, provided RTS can be forced low.

So, any clues as to how I might be able to set RTS at startup will be gratefully received!

Richard

Richard

[José Luis Redrejo]

Hi Richard, maybe I'm wrong, but I think this is  not related to web2py at all.

Arduino boards are build to make a reset whenever the port is open/reopen so they change to "programming" state.

This is made on purpose, but it's trivial to avoid it. If using an Arduino UNO board, just connecting a 10μF capacitor between RESET and GND will avoid those resets. Just remember to remove the capacitor if you need to re-program your board.

Hope this was the issue.

José L.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Brown]

Hi José,

Thank you for your post. You are correct that it is a feature of the Arduino architecture that a transition on RTS resets the target microcontroller to enable automatic programming, and it is possible to disable this feature, either as you describe or by removing the capacitor which couples RTS to Reset. However, as I intend to make my Web2py application available to others in the future, and they may choose to use standard Arduino products to implement the radio transmitter then I was keen to avoid any hardware modifications, especially as these will inevitably lead to support questions! Also my USB radio dongle is enclosed in a plastic case which would have to be opened to reprogram the code.

I also agree that it is not strictly a Web2py 'problem', except that I am using Web2py and the problem is as a result of using the serial port in a background process running under its Scheduler - and the observation that when Web2py saves and parses files it does something to the serial port which overcomes the problem and I was hoping to find out what that was and make use of it. However, I have since found a solution using a script, installed during the boot sequence which holds the serial port open. I created a file 'OpenPort.py' which contains the following code:

import serial

import time

　

ser=serial.Serial('/dev/ttyUSB0', 57600, timeout=1, xonxoff=0, rtscts=0)

ser.rts=True

while 1:

time.sleep(0.0001)

　

This is called from a bash script installed using 'sudo update-rc.d scheduler-openport defaults' which contains the lines:

dir="/home/www-data/web2py"

user="www-data"

cmd="sudo nohup python applications/Heating_Controller/utilities/OpenPort.py &"

　

I put the file in a new 'utilities' directory under my application as I wanted it to be part of the application's files but it cannot be in a directory that Web2py recognises (such as Models or Modules) because OpenPort.py remains open which prevents the Admin interface from working.

This has solved my problem and doesn't have any hardware implications.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.