FileParse not functioning

[Kenny Stier]

I have weewx running on a Raspberry Pi 3 using the RTL-SDR to get info from my Acurite 5-in-1 station. Only problem is, this station doesn't have a barometer built in, it is inside of the Acurite Access.

So I have setup a separate VM with a fresh install of Ubuntu 18.04 and weewx. I created a script to scrape the pressure from the Access and use cron to keep it up to date.

My hope is to later have the Pi get the pressure data from the VM, but I need the VM to work first.

#!/bin/bash

pressure=$(curl -fs http://192.168.0.150/ | grep -o -P '(?<=PRESSURE: ).*(?= Inch</p><p>)')

echo pressure=$pressure > /var/tmp/pressure.txt

This works great, and outputs pressure.txt

pressure=29.06

I setup this VM with FileParse to utilize this data. All reports run, I get no errors, but the reports are empty.

index.html

<td class="label">Barometer</td>

<td class="data">   N/A (   N/A)</td>

syslog

Jan  6 01:56:44 weewx systemd[1]: Started LSB: weewx weather system.

Jan  6 01:56:44 weewx weewx[4911]: engine: Using configuration file /etc/weewx/weewx.conf

Jan  6 01:56:44 weewx weewx[4911]: engine: Loading station type FileParse (user.fileparse)

Jan  6 01:56:44 weewx weewx[4911]: fileparse: data file is /var/tmp/pressure.txt

Jan  6 01:56:44 weewx weewx[4911]: fileparse: polling interval is 300.0

Jan  6 01:56:44 weewx weewx[4911]: fileparse: label map is {}

Jan  6 01:56:44 weewx weewx[4911]: engine: StdConvert target unit is 0x1

Jan  6 01:56:44 weewx weewx[4911]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware

Jan  6 01:56:44 weewx weewx[4911]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS

Jan  6 01:56:44 weewx weewx[4911]: engine: Archive will use data binding wx_binding

Jan  6 01:56:44 weewx weewx[4911]: engine: Record generation will be attempted in 'hardware'

Jan  6 01:56:44 weewx weewx[4911]: engine: Using archive interval of 300 seconds (specified in weewx configuration)

Jan  6 01:56:44 weewx weewx[4911]: engine: Using binding 'wx_binding' to database 'weewx'

Jan  6 01:56:44 weewx weewx[4911]: manager: Starting backfill of daily summaries

Jan  6 01:56:44 weewx weewx[4911]: restx: StationRegistry: Registration not requested.

Jan  6 01:56:44 weewx weewx[4911]: restx: Wunderground: Posting not enabled.

Jan  6 01:56:44 weewx weewx[4911]: restx: PWSweather: Posting not enabled.

Jan  6 01:56:44 weewx weewx[4911]: restx: CWOP: Posting not enabled.

Jan  6 01:56:44 weewx weewx[4911]: restx: WOW: Posting not enabled.

Jan  6 01:56:44 weewx weewx[4911]: restx: AWEKAS: Posting not enabled.

Jan  6 01:56:44 weewx weewx[4911]: engine: Starting up weewx version 3.9.2

Jan  6 01:56:44 weewx weewx[4911]: engine: Starting main packet loop.

Jan  6 02:00:01 weewx CRON[4923]: (mt-dev) CMD (/home/mt-dev/pressure.sh)

Jan  6 02:01:44 weewx weewx[4911]: manager: Added record 2020-01-06 02:00:00 UTC (1578276000) to database 'weewx'

Jan  6 02:01:44 weewx weewx[4911]: manager: Added record 2020-01-06 02:00:00 UTC (1578276000) to daily summary in 'weewx'

Jan  6 02:01:45 weewx weewx[4911]: cheetahgenerator: Generated 8 files for report SeasonsReport in 0.77 seconds

Jan  6 02:01:46 weewx weewx[4911]: imagegenerator: Generated 28 images for SeasonsReport in 1.14 seconds

Jan  6 02:01:46 weewx weewx[4911]: copygenerator: copied 5 files to /var/www/html/weewx

weewx.conf snippets

[Station]

    station_type = FileParse

[FileParse]

    driver = user.fileparse

    path = /var/tmp/pressure.txt

    poll_interval = 300

[DataBindings]

    [[wx_binding]]

        database = archive_mysql

    [[MySQL]]

        driver = weedb.mysql

        host = localhost

        user = weewx

        password = ...

All help is appreciated!

[gjr80]

Hi,

Since the end result is not what you expect you really want to follow your data from start to finish to see where it is being lost. You have looked at the file being generated by your bash script and it appears to have valid data. The next step is to look at what is being emitted by the fileparse driver. There are a couple of ways you can do this, since you already have WeeWX installed you could run WeeWX directly. This will show you the WeeWX augmented loop packets being emitted by the fileparse driver on the console. It will also show you the archive records being synthesised by WeeWX, the archive records are what are used to populate your reports. Alternatively, you could run the fileparse driver directly, refer to the comments in the last few lines of fileparse.py for how to do this, something like this should work (untested):

$ PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/fileparse.py This will show you only the loop packets emitted by the fileparse driver, this is exactly what fileparse passes to WeeWX when WeeWX is run. If correct data is being emitted by the fileparse driver then you need to look at WeeWX itself, if incorrect or no data is being emitted then you have an issue with the driver and it's processing of/acces to the raw data. Gary

[Kenny Stier]

Thanks for that, Gary

Running fileparse.py directly works

mt-dev@weewx:~$ PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/fileparse.py

2020-01-06 20:32:35 EST (1578360755) {'pressure': 29.17, 'usUnits': 1, 'dateTime': 1578360755}

When I run weewx directly, the pressure data is collected

mt-dev@weewx:~$ sudo weewxd /etc/weewx/weewx.conf

LOOP:   2020-01-06 20:18:09 EST (1578359889) altimeter: 30.1414615854, dateTime: 1578359889, maxSolarRad: None, pressure: 29.17, rainRate: 0, usUnits: 1

LOOP:   2020-01-06 20:23:09 EST (1578360189) altimeter: 30.1414615854, dateTime: 1578360189, maxSolarRad: None, pressure: 29.17, rainRate: 0, usUnits: 1

REC:    2020-01-06 20:20:00 EST (1578360000) altimeter: 30.1414615854, dateTime: 1578360000.0, ET: None, interval: 5, maxSolarRad: None, pressure: 29.17, rainRate: 0.0, usUnits: 1

LOOP:   2020-01-06 20:23:09 EST (1578360189) altimeter: 30.1414615854, dateTime: 1578360189, maxSolarRad: None, pressure: 29.17, rainRate: 0, usUnits: 1

My reports are still blank, however.

I don't believe it is a permissions issue, as the files are being modified

-rw-r--r-- 1 root root  17K Jan  6 20:23 index.html

[gjr80]

Ok, now the cause is clear. WeeWX uses three 'pressures'; pressure which is station or absolute pressure, altimeter which is pressure adjusted for altitude and barometer which is pressure adjusted for altitude, temperature and humidity. When one or more of these three are missing WeeWX will attempt to calculate it. In your case you have pressure but no barometer or altimeter - that is clear when you see what is coming from the driver when run directly. When you look at the LOOP packets when WeeWX is run directly you will notice pressure and altimeter but no barometer, that is because altimeter is simply a function of pressure and station altitude and WeeWX was able to calculate that but barometer requires (amongst other things) temperature data and you don't have any. So WeeWX cannot calculate barometer.

Now cut to your reports, your report will be trying to display barometer with (likely) the tag $current.barometer. There is no barometer data so WeeWX displays N/A. If you change barometer to pressure or altimeter it will work fine (albeit displaying pressure or altimeter not barometer). When/if you integrate this with a station that has temperature and humidity data barometer will appear.

So the upshot is that everything is working as expected. The trick in getting your pressure data into your RPi will be getting your SDR 'arrangement' working with the fileparse 'arrangement' as each WeeWX instance only supports a single driver. There are a few options, one would be to run a single WeeWX instance using the SDR driver and adapt the fileparse driver to be a service that augments loop packets with pressure; WeeWX with then calculate altimeter and barometer and save to archive and generate reports as required. Another option is the reverse, a single WeeWX instance with fileparse driver and SDR as a service; I suspect the former would be easier to implement since the fileparse driver is very basic in operation. Another option is to run two instances of WeeWX each running a single driver though this will be tricky because one instance will require temperature and humidity data that only the other instance has. There is not a huge difference between a driver and a service that augments loop packets so I expect the first option will be very easy to do.

Gary

[Kenny Stier]

Running two instances was my plan, but I can see how that would be an issue.

Is there some documentation somewhere that I can use to adapt the fileparse driver to be a service?

Is this similar here? https://github.com/bdwilson/acurite/blob/master/pond.py

[gjr80]

Adding a second data source (http://weewx.com/docs/customizing.htm#Adding_2nd_source) in the Customization Guide is about all the documentation I am aware of. Other than that it is learn by example. The link you provided looks pretty good, one point for consideration is whether to bind to NEW_ARCHIVE_RECORD or NEW_LOOP_PACKET, the former will augment each archive record whereas the latter will augment each loop packet (which is then aggregated by WeeWX to produce the archive record value). Either will work but I prefer to have a number of samples per archive record rather than just one.

Gary