python3 with weewx4 and interceptor 0.54

249 views
Skip to first unread message

Max Gutberlet

unread,
Apr 28, 2020, 9:50:01 AM4/28/20
to weewx-user
Gents, did upgrade to Ubuntu 20.04 which is now python3 based so it broke my python2 based weewx/interceptor config.

I did a clean install using python3-weewx_4.0.0b18-1_all.deb which AFAIK is the only one supporting python3. Some hickups
around necessary packages (python3-usb ; -serial and -obj) but nothing major.

Then installed latest weewx-interceptor - again some hickups to get the pcap running but again nothing major. A quick pip install
solved that. I am doing wu interception.

Extension Name    Version   Description
interceptor       0.54      Capture weather data from HTTP requests

Now the service is running, but I am not seeing any packets being captured:

Apr 28 15:27:52 server weewx[597240] message repeated 4 times: [ DEBUG user.interceptor: empty queue]
Apr 28 15:28:02 server weewx[597240] DEBUG user.interceptor: empty queue
Apr 28 15:28:12 server weewx[597240] DEBUG user.interceptor: empty queue
Apr 28 15:29:12 server weewx[597240] message repeated 6 times: [ DEBUG user.interceptor: empty queue]
Apr 28 15:29:22 server weewx[597240] DEBUG user.interceptor: empty queue
Apr 28 15:29:32 server weewx[597240] DEBUG user.interceptor: empty queue
Apr 28 15:29:42 server weewx[597240] DEBUG user.interceptor: empty queue

Just to be 200% I used wireshark to see if the device is still sending packages and it is fine, sending a packet
every minute up to WU.

My interceptor config:

[Interceptor]
    driver = user.interceptor
    device_type = wu-client
    mode = sniff
    iface = eth0
    pcap_filter = src 192.168.0.68 and dst port 80
    promiscuous = True

I tried both promiscuous True and False - no change.

Any ideas ? Thanks in advance !
CU
Max


G Hammer

unread,
Apr 28, 2020, 1:59:46 PM4/28/20
to weewx-user
Are you certain the upgrade left your interface as eth0

Max Gutberlet

unread,
Apr 28, 2020, 2:08:55 PM4/28/20
to weewx-user
LOL - literally the first thing I checked :-) Yep, the packets (in wireshark) are still comming in on eth0.

I did some more investigation, this is starting to look like a python3.8 with interceptor problem. My station actually talks ecowitt as well, so I configured
it to send ecowitt to my server and started interceptor in listen mode für ecowitt as a standalone using

PYTHONPATH=bin python3 bin/user/interceptor.py --device=ecowitt-client --mode=listen --port=9988

same "no data received" behaviour and you have to actually kill the process, a control-c does not stop it which also hints at something "hanging".

I am currently playing with an alternative ecowitt implementation https://github.com/iz0qwm/ecowitt_http_gateway but that also gives me a lot of 
python errors on identation :-( 

Any help greatly appreciated ! I still hope I am "just" misconfiguring something.....
Thx
Max

G Hammer

unread,
Apr 28, 2020, 3:10:30 PM4/28/20
to weewx-user
This is what is in my weewx.conf, seems a little different to yours.
[Interceptor]
    # This section is for the network traffic interceptor driver.
    
    # The driver to use:
    driver = user.interceptor
    
    # Specify the hardware device to capture.  Options include:
    #   acurite-bridge - acurite internet bridge
    #   observer - fine offset WH2600/HP1000/HP1003, aka 'observer'
    #   lw30x - oregon scientific LW301/LW302
    #   lacrosse-bridge - lacrosse GW1000U/C84612 internet bridge
    device_type = fineoffset-bridge
    port = 8000
    [[sensor_map_extensions]]
        windBatteryStatus = wh68_battery
        txBatteryStatus = wh26_battery
        supplyVoltage = wh40_battery

Message has been deleted

Max Gutberlet

unread,
Apr 28, 2020, 5:15:01 PM4/28/20
to weewx-user
Thx -  just tried mirroring exactly your config, but I still get "empty queue".

Here is the debug log from the run (as you can see, I have updated to 0.54b and even included your little fix you just posted on GIT):

Apr 28 22:52:20 server systemd[1]: Started LSB: weewx weather system.
Apr 28 22:52:20 server weewx[661236] INFO user.interceptor: driver version is 0.54b
Apr 28 22:52:20 server weewx[661236] INFO user.interceptor: device type: fineoffset-bridge
Apr 28 22:52:20 server weewx[661236] INFO user.interceptor: mode is listen
Apr 28 22:52:20 server weewx[661236] INFO user.interceptor: listen on :8000
Apr 28 22:52:20 server weewx[661236] INFO user.interceptor: sensor map: {'pressure': 'pressure', 'barometer': 'barometer', 'outHumidity': 'humidity_out', 'inHumidity': 'humidity_in', 'outTemp': 'temperature_out', 'inTemp': 'temperature_in', 'windSpeed': 'wind_speed', 'windGust': 'wind_gust', 'windDir': 'wind_dir', 'windGustDir': 'wind_gust_dir', 'radiation': 'solar_radiation', 'dewpoint': 'dewpoint', 'windchill': 'windchill', 'rain': 'rain', 'rainRate': 'rain_rate', 'UV': 'uv', 'txBatteryStatus': 'wh26_battery', 'extraTemp1': 'temperature_1', 'extraTemp2': 'temperature_2', 'extraTemp3': 'temperature_3', 'extraHumid1': 'humidity_1', 'extraHumid2': 'humidity_2', 'soilTemp1': 'soil_temperature_1', 'soilTemp2': 'soil_temperature_2', 'soilMoist1': 'soil_moisture_1', 'soilMoist2': 'soil_moisture_2', 'soilMoist3': 'soil_moisture_3', 'soilMoist4': 'soil_moisture_4', 'leafWet1': 'leafwetness_1', 'leafWet2': 'leafwetness_2', 'pm2_5': 'pm2_5', 'extraTemp4': 'temperature_4', 'extraTemp5': 'temperature_5', 'extraTemp6': 'temperature_6', 'extraTemp7': 'temperature_7', 'extraTemp8': 'temperature_8', 'extraHumid3': 'humidity_3', 'extraHumid4': 'humidity_4', 'extraHumid5': 'humidity_5', 'extraHumid6': 'humidity_6', 'extraHumid7': 'humidity_7', 'extraHumid8': 'humidity_8', 'soilTemp3': 'soil_temperature_3', 'soilTemp4': 'soil_temperature_4', 'windBatteryStatus': 'wh68_battery', 'supplyVoltage': 'wh40_battery'}
Apr 28 22:52:20 server weewx[661236] DEBUG user.interceptor: start tcp server
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdTimeSynch
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdTimeSynch
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdConvert
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: StdConvert target unit is 0x1
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdConvert
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdCalibrate
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdCalibrate
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdQC
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdQC
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.wxservices.StdWXCalculate
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.manager: Daily summary version is 1.0
Apr 28 22:52:20 server weewx[661236] INFO weewx.wxservices: The following values will be calculated: pressure=prefer_hardware, barometer=prefer_hardware, altimeter=prefer_hardware, windchill=hardware, heatindex=hardware, dewpoint=prefer_hardware, inDewpoint=prefer_hardware, rainRate=hardware, maxSolarRad=prefer_hardware, cloudbase=prefer_hardware, humidex=prefer_hardware, appTemp=prefer_hardware, ET=prefer_hardware, windrun=prefer_hardware
Apr 28 22:52:20 server weewx[661236] INFO weewx.wxservices: The following algorithms will be used for calculations: altimeter=aaASOS, maxSolarRad=RS
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.wxservices.StdWXCalculate
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdArchive
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: Archive will use data binding wx_binding
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: Record generation will be attempted in 'hardware'
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: Using archive interval of 300 seconds (specified in weewx configuration)
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Use LOOP data in hi/low calculations: 1
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdArchive
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdStationRegistry
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: StationRegistry: Station will be registered.
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdStationRegistry
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdWunderground
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.restx: WU essentials: {}
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: Wunderground-PWS: Data for station IHESSENR12 will be posted
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdWunderground
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdPWSweather
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: PWSweather: Posting not enabled.
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdPWSweather
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdCWOP
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: CWOP: Posting not enabled.
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdCWOP
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdWOW
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: WOW: Posting not enabled.
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdWOW
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.restx.StdAWEKAS
Apr 28 22:52:20 server weewx[661236] INFO weewx.restx: AWEKAS: Data will be uploaded for user maxgutberlet
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.restx.StdAWEKAS
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdPrint
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdPrint
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Loading service weewx.engine.StdReport
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Finished loading service weewx.engine.StdReport
Apr 28 22:52:20 server weewx[661236] INFO __main__: Starting up weewx version 4.0.0b18
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.engine: Station does not support reading the time
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Apr 28 22:52:20 server weewx[661236] INFO weewx.manager: Starting backfill of daily summaries
Apr 28 22:52:20 server weewx[661236] INFO weewx.engine: Starting main packet loop.
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.manager: Daily summary version is 1.0
Apr 28 22:52:20 server weewx[661236] DEBUG weewx.manager: Daily summary version is 1.0
Apr 28 22:52:30 server weewx[661236] DEBUG user.interceptor: empty queue
Apr 28 22:53:30 server weewx[661236] message repeated 6 times: [ DEBUG user.interceptor: empty queue]
Apr 28 22:53:40 server weewx[661236] DEBUG user.interceptor: empty queue
Apr 28 22:53:50 server weewx[661236] DEBUG user.interceptor: empty queue
Apr 28 22:55:00 server weewx[661236] message repeated 7 times: [ DEBUG user.interceptor: empty queue]
Apr 28 22:55:10 server weewx[661236] DEBUG user.interceptor: empty queue

And here is the tcpdump at the same time that clearly shows the unit connecting to port 8000:

tcpdump -i eth0 "src 192.168.0.68 and dst port 8000"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:54:15.110857 IP 192.168.0.68.6929 > server.gutberlet.org.8000: Flags [S], seq 19845874, win 5840, options [mss 1460], length 0
22:54:15.118857 IP 192.168.0.68.6929 > server.gutberlet.org.8000: Flags [P.], seq 19845875:19846415, ack 4034944965, win 5840, length 540
22:54:15.127088 IP 192.168.0.68.6929 > server.gutberlet.org.8000: Flags [F.], seq 540, ack 504, win 5337, length 0
22:55:19.115660 IP 192.168.0.68.22747 > server.gutberlet.org.8000: Flags [S], seq 19957218, win 5840, options [mss 1460], length 0
22:55:19.141821 IP 192.168.0.68.22747 > server.gutberlet.org.8000: Flags [P.], seq 19957219:19957759, ack 732863350, win 5840, length 540
22:55:19.149284 IP 192.168.0.68.22747 > server.gutberlet.org.8000: Flags [F.], seq 540, ack 504, win 5337, length 0
22:56:23.112251 IP 192.168.0.68.40696 > server.gutberlet.org.8000: Flags [S], seq 20072022, win 5840, options [mss 1460], length 0
22:56:23.120399 IP 192.168.0.68.40696 > server.gutberlet.org.8000: Flags [P.], seq 20072023:20072563, ack 292068405, win 5840, length 540
22:56:23.128518 IP 192.168.0.68.40696 > server.gutberlet.org.8000: Flags [F.], seq 540, ack 504, win 5337, length 0

Same things happen when I do wu-client sniff - I clearly see the packages going to WU in the tcpdump but nothing ever pops up in interceptor.

Looks like something is broken when actually queuing the packets - I was already suspecting something like apparmor to interfere but I am not seeing
any blocks there either. 

What version of python are you running ? 
CU
Max

G Hammer

unread,
Apr 29, 2020, 11:22:05 AM4/29/20
to weewx-user
I'm running Python 3.6.9 on Linux Mint 19.3 kernel 5.3.0-46



G Hammer

unread,
Apr 29, 2020, 11:25:42 AM4/29/20
to weewx-user
You may wish to install using the setup.py method. 
I think that the deb install is not complete as yet. Plus, it looks like it is -3 vs your -1 at this time.

G Hammer

unread,
Apr 29, 2020, 1:53:55 PM4/29/20
to weewx-user
I ran debug logging for the interceptor driver and have attached it here for you to compare.
BTW- It is a nice feature of WeeWX 4 to have the flexible logging available.

My example:
# Set logging for Interceptor to debug
[Logging]
  [[loggers]]
    [[[user.interceptor]]]
      level = DEBUG
      handlers = syslog,
      propagate = 0
interceptor debug.txt

Max Gutberlet

unread,
May 1, 2020, 10:02:21 AM5/1/20
to weewx-user
Reporting success(ish)....

Spend another 8 hours playing with this, got it to work but not without caveats:

Did a squeaky clean install based on Matts new 4.0 deb package and using the 0.54b interceptor with  the latest GIT commit on the PASSKEY.

- listen mode works using ecowitt-client - that was a RTFM error where the /data/report/ path for the web-call on the station was changed - setting
  this back to the standard path made this work - so happily logging data again. Btw, my system is running EasyWeather1.4.9 firmware.

- sniff mode is broken - no success with pylibpcap nor with pypcap. I had to change the source code to make current pylibpcap - which is available for
  ubuntu as package python3-pcapy work at all as syntax has changed but I am not seeing any packages on the sniffer interface - which leads me to 
  believe that there is a more fundamental problem with python3 and pcap in general. Just to test out this theory, I installed pyshark, which is another
  capture package and that one fails as well as soon as it tries to open an interface (goes into timeout). I am going to continue to dig here for a while
  but will likely post a bug on the Ubuntu 20.04 forum as soon as I have enough evidence gathered. 

Thanks for all your help
Max

G Hammer

unread,
May 1, 2020, 10:20:27 AM5/1/20
to weewx-user


On Friday, May 1, 2020 at 10:02:21 AM UTC-4, Max Gutberlet wrote:
Reporting success(ish)....

Spend another 8 hours playing with this, got it to work but not without caveats:

Did a squeaky clean install based on Matts new 4.0 deb package and using the 0.54b interceptor with  the latest GIT commit on the PASSKEY.

- listen mode works using ecowitt-client - that was a RTFM error where the /data/report/ path for the web-call on the station was changed - setting
  this back to the standard path made this work - so happily logging data again. Btw, my system is running EasyWeather1.4.9 firmware.


Glad to hear it works as ecowitt-client, I was stumped.

As for the sniff, Ubuntu 20 has many functions supplied by a snap. Are you installing via apt or via pip? May (or may not) make a difference.
Take a look at how Tom installs to localenv;

S R

unread,
Jan 22, 2021, 11:00:36 AM1/22/21
to weewx-user
can you please clarify precisely what you setting you are referring to
"error where the /data/report/ path for the web-call on the station was changed - setting this back to the standard path"

aside from a couple of initial connections after weewx is restarted, my log then becomes full with this message and the db is not updated at all.
Reply all
Reply to author
Forward
0 new messages