Wind gust data suddenly not appearing

Ron Walker

unread,

Jul 2, 2019, 3:34:14 PM7/2/19

to weewx-user

Hi and thanks in advance for any help!

I have an Arduino sending weather data to a Raspberry Pi. I'm using WeeWX with the FileParse driver and the Seasons skin. Up until sometime yesterday, I was able to see wind gust data in the web page and in the graphs. Suddenly, for reasons unknown, I am unable to see the data in the charts and the web page shows no entry at all for wind gust and direction. NA NA ( NA) is what is displayed in it's place. I can see the values for windGust in the incoming datastream. I've looked everywhere that I know, but can't seem to find the reason. I've started a new database, but that had no effect.

Weewx.conf:

# WEEWX CONFIGURATION FILE

#

# See the file LICENSE.txt for your rights.

##############################################################################

# Set to type of station hardware. There must be a corresponding stanza

# in this file with a 'driver' parameter indicating the driver to be used.

station_type = FileParse

# If you have a website, you may specify an URL

#station_url = http://www.example.com

station_url = localhost/weewx

# The start of the rain year (1=January; 10=October, etc.). This is

# downloaded from the station if the hardware supports it.

rain_year_start = 1

# Start of week (0=Monday, 6=Sunday)

week_start = 6

#############################################################################

[FileParse]

poll_interval = 10

path = /var/tmp/data.csv

driver = user.fileparse

loop_interval = 10

[[Label_Map]]

Time = dateTime

pressure = pressure

outTemp = outTemp

outHumidity = outHumidity

windSpeed = windSpeed

windDir = windDir

windGust = windGust

windGustDir = windGustDir

supplyVoltage = supplyVoltage

lightLevel = radiation

rainRate = rainRate

rain = rain

##############################################################################

# This section specifies what reports, using which skins, to generate.

[StdReport]

[[SeasonsReport]]

# The SeasonsReport uses the 'Seasons' skin, which contains the

# images, templates and plots for the report.

skin = Seasons

enable = true

# Various options for customizing your reports.

[[Defaults]]

[[[Units]]]

# The following section sets what unit to use for each unit group.

# NB: The unit is always in the singular. I.e., 'mile_per_hour',

# NOT 'miles_per_hour'

[[[[Groups]]]]

group_altitude = foot # Options are 'foot' or 'meter'

group_degree_day = degree_F_day # Options are 'degree_F_day' or 'degree_C_day'

group_pressure = inHg # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'

group_rain = inch # Options are 'inch', 'cm', or 'mm'

group_rainrate = inch_per_hour # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'

group_speed = mile_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = mile_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

group_temperature = degree_F # Options are 'degree_F' or 'degree_C'

# The following section sets the formatting for each type of unit.

[[[[StringFormats]]]]

centibar = %.0f

cm = %.2f

cm_per_hour = %.2f

degree_C = %.1f

degree_F = %.1f

degree_compass = %.0f

foot = %.0f

hPa = %.1f

hour = %.1f

inHg = %.3f

inch = %.2f

inch_per_hour = %.2f

km_per_hour = %.0f

km_per_hour2 = %.1f

knot = %.0f

knot2 = %.1f

mbar = %.1f

meter = %.0f

meter_per_second = %.1f

meter_per_second2 = %.1f

mile_per_hour = %.0f

mile_per_hour2 = %.1f

mm = %.1f

mmHg = %.1f

mm_per_hour = %.1f

percent = %.0f

second = %.0f

uv_index = %.1f

volt = %.1f

watt_per_meter_squared = %.0f

NONE = " N/A"

# The following section sets the label for each type of unit

[[[[Labels]]]]

day = " day", " days"

hour = " hour", " hours"

minute = " minute", " minutes"

second = " second", " seconds"

NONE = ""

# The following section sets the format for each time scale.

# The values below will work in every locale, but they may not look

# particularly attractive.

[[[[TimeFormats]]]]

hour = %H:%M

day = %X

week = %X (%A)

month = %x %X

year = %x %X

rainyear = %x %X

current = %x %X

ephem_day = %X

ephem_year = %x %X

[[[[Ordinates]]]]

# Ordinal directions. The last one is for no wind direction

directions = N, NNE, NE, ENE, E, ESE, SE, SSE, S, SSW, SW, WSW, W, WNW, NW, NNW, N/A

# The following section sets the base temperatures used for the

# calculation of heating and cooling degree-days.

[[[[[DegreeDays]]]]]

# Base temperature for heating days, with unit:

heating_base = 65, degree_F

# Base temperature for cooling days, with unit:

cooling_base = 65, degree_F

# A trend takes a difference across a time period. The following

# section sets the time period, and how big an error is allowed to

# still be counted as the start or end of a period.

[[[[[Trend]]]]]

time_delta = 10800 # 3 hours

time_grace = 300 # 5 minutes

# The labels to be used for each observation type

[[[Labels]]]

# Set to hemisphere abbreviations suitable for your location:

hemispheres = N, S, E, W

# Formats to be used for latitude whole degrees, longitude whole

# degrees, and minutes:

latlon_formats = %02d, %03d, %05.2f

# Generic labels, keyed by an observation type.

[[[[Generic]]]]

barometer = Barometer

dewpoint = Dew Point

ET = ET

heatindex = Heat Index

inHumidity = Inside Humidity

inTemp = Inside Temperature

outHumidity = Humidity

outTemp = Outside Temperature

radiation = Radiation

rain = Rain Today

rainRate = Rain Last Hour

UV = UV Index

windDir = Wind Direction

windGust = Gust Speed

windGustDir = Gust Direction

windSpeed = Wind Speed

windchill = Wind Chill

windgustvec = Gust Vector

windvec = Wind Vector

extraTemp1 = Temperature1

extraTemp2 = Temperature2

extraTemp3 = Temperature3

# Sensor status indicators

rxCheckPercent = Signal Quality

txBatteryStatus = Transmitter Battery

windBatteryStatus = Wind Battery

rainBatteryStatus = Rain Battery

outTempBatteryStatus = Outside Temperature Battery

inTempBatteryStatus = Inside Temperature Battery

consBatteryVoltage = Console Battery

heatingVoltage = Heating Battery

supplyVoltage = Supply Voltage

referenceVoltage = Reference Voltage

##############################################################################

# This section controls the origin of derived values.

[StdWXCalculate]

[[Calculations]]

# How to calculate derived quantities. Possible values are:

# hardware - use the value provided by hardware

# software - use the value calculated by weewx

# prefer_hardware - use value provide by hardware if available,

# otherwise use value calculated by weewx

pressure = hardware

barometer = software

altimeter = software

windchill = software

heatindex = software

dewpoint = software

inDewpoint = software

rainRate = hardware

##############################################################################

# This section is for configuring the archive service.

[StdArchive]

# If the station hardware supports data logging then the archive interval

# will be downloaded from the station. Otherwise, specify it (in seconds).

archive_interval = 300

# If possible, new archive records are downloaded from the station

# hardware. If the hardware does not support this, then new archive

# records will be generated in software.

# Set the following to "software" to force software record generation.

record_generation = hardware

# Whether to include LOOP data in hi/low statistics

loop_hilo = False

# The data binding used to save archive records

data_binding = wx_binding

##############################################################################

# This section binds a data store to a database.

[DataBindings]

[[wx_binding]]

# The database must match one of the sections in [Databases].

# This is likely to be the only option you would want to change.

database = archive_sqlite

# The name of the table within the database

table_name = archive

# The manager handles aggregation of data for historical summaries

manager = weewx.wxmanager.WXDaySummaryManager

# The schema defines the structure of the database.

# It is *only* used when the database is created.

schema = schemas.wview.schema

##############################################################################

# This section defines various databases.

[Databases]

# A SQLite database is simply a single file

[[archive_sqlite]]

database_name = weewx.sdb

database_type = SQLite

#############################################################################

gjr80

unread,

Jul 2, 2019, 5:39:49 PM7/2/19

to weewx-user

Hi,

A couple of suggestions. You say “I can see the values for windGust in the incoming datastream” - is this on the Arduino, the transfer file or elsewhere on the RPi? You might like to edit weewx.conf, set debug = 1, save weewx.conf and then restart WeeWX. Let WeeWX run for at least 10 minutes then post a log extract from when you started WeeWX through until the 10 minutes elapsed, don’t edit anything and be sure to capture the entire startup. Post the log extract here, that will show us how fileparse is configured. I suspect that will show everything is OK as the portion of weewx.conf you posted looks OK, but worth a check.

Secondly, stop WeeWX and then run it directly (http://weewx.com/docs/usersguide.htm#Running_directly) this will display loop packets and archive records on the console. What do you see in the loop packets? Is there any windGust? How does that tally with what is in the fileparse source data file? You can post the screen capture. I am guessing that windGust in the loop packets and archive records will be None which makes me suspect that windGust in your fileparse source data file will be missing, blank or otherwise invalid.

Gary

Ron Walker

unread,

Jul 2, 2019, 8:32:24 PM7/2/19

to weewx-user

Hi Gary,

Thanks for the reply! While trying to collect this data, I entered the 'sudo /etc/init.d/weewx stop' command, but WeeWX reported an error saying that the term failed! Once the cursor came back I tried entering the command again and it was completely ignored. See screenshots 'Stop_fail.png' and Stop_fail1.txt'. I finally rebooted the Pi and then tried again. The stop command worked this time.

I grabbed the data that you requested. The syslog from startup and through two archive cycles is called 'Start_up.txt'. I have also attached screenshots of data going into the file (data.csv) and the loop data. I notice an badly formed 'ndGustDir10avg' in the loop data, but did not see that in the input data.

Finally, Input_file.png is a screenshot of the data going into the file that FileParse reads, 'Loop_Error.png'. Show an error.

Thanks!

Stop_fail.txt

Start_up.txt

Input_file_Screenshot.png

Loop_andError.png

gjr80

unread,

Jul 2, 2019, 9:37:40 PM7/2/19

to weewx-user

Can’t comment on the error when stopping WeeWX, I guess it’s possible something was messed up that prevented the ‘stop’ script running successfully. I’d give it the benefit of the doubt for now.

I note from your previous thread I mentioned there was no need for a label map (field map) as your label names were the same as WeeWX field names. This is not the case, radiation being a case in hand. That being said fileparse expects a label map to be under a [label_map] stanza but you have a [Label_Map] stanza. Case matters here so you might want to change that. This won’t fix your windGust problem though.

Other than the label map case issue the startup log seems to indicate no config issues.

The malformed field name could be a sign you still have some issues with your data.csv. Appreciate the data going into data.csv may be correct but the fileparse driver uses what is actually in the file; the two could be different. You might want to try some additional logging in the fileparse driver. Adding a loginf line like the following in genLoopPackets() will log each line read from data.csv:

with open(self.path) as f:
for line in f:
loginf(“line=%s” % line)
eq_index = line.find('=')
name = line[:eq_index].strip()
value = line[eq_index + 1:].strip()

It’s going to log a lot but at least we can see the raw data WeeWX is reading from data.csv.

It’s also interesting that windGust is coming through loop packets as 0.0 but is accumulated in an archive record as None; I would have expected it to come through as 0.0.

Gary

Ron Walker

unread,

Jul 3, 2019, 8:47:17 AM7/3/19

to weewx-user

Hi Gary,

Two things! First, you were quite right about the amount of logging that would result! Is it possible to upload a file of greater that 15 mb here? The second thing is a new issue that started occurring as I was try to gather data. I started seeing the following in the syslog:

xisting report thread still running

Jul 3 08:44:55 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Jul 3 08:45:05 WeatherPi weewx[627]: manager: Added record 2019-07-03 08:45:00 EDT (1562157900) to database 'weewx.sdb'

Jul 3 08:45:05 WeatherPi weewx[627]: manager: Added record 2019-07-03 08:45:00 EDT (1562157900) to daily summary in 'weewx.sdb'

Jul 3 08:45:05 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Jul 3 08:45:15 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Jul 3 08:45:25 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Jul 3 08:45:29 WeatherPi weewx[627]: cheetahgenerator: Generated 8 files for report SeasonsReport in 51.84 seconds

Jul 3 08:45:30 WeatherPi weewx[627]: manager: Daily summary version is 2.0

Jul 3 08:45:35 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Jul 3 08:45:45 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Is there a way to stop this 'existing thread'? I've tried stopping and restarting and rebooting the PI. Nothing seems to stop it!

Ron

Ron Walker

unread,

Jul 3, 2019, 3:10:14 PM7/3/19

to weewx...@googlegroups.com

Hi Gary,

I decided to start fresh! Clean OS and WeeWX installs! Too many weird issues occurring. I did look at the output of the logging that I did. The data was coming over clean. I had started a new db and a new data file, but the wind gust issue remained! Thanks, as always, for your help!

Sent from Yahoo Mail for iPhone

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/aef77bfd-536a-475a-a751-39d1bb99bc00%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

mwall

unread,

Jul 4, 2019, 12:23:04 AM7/4/19

to weewx-user

On Wednesday, July 3, 2019 at 8:47:17 AM UTC-4, Ron Walker wrote:

The second thing is a new issue that started occurring as I was try to gather data. I started seeing the following in the syslog:

xisting report thread still running
Jul 3 08:44:55 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running
Jul 3 08:45:05 WeatherPi weewx[627]: manager: Added record 2019-07-03 08:45:00 EDT (1562157900) to database 'weewx.sdb'
Jul 3 08:45:05 WeatherPi weewx[627]: manager: Added record 2019-07-03 08:45:00 EDT (1562157900) to daily summary in 'weewx.sdb'
Jul 3 08:45:05 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running
Jul 3 08:45:15 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running
Jul 3 08:45:25 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running
Jul 3 08:45:29 WeatherPi weewx[627]: cheetahgenerator: Generated 8 files for report SeasonsReport in 51.84 seconds
Jul 3 08:45:30 WeatherPi weewx[627]: manager: Daily summary version is 2.0
Jul 3 08:45:35 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running
Jul 3 08:45:45 WeatherPi weewx[627]: engine: Launch of report thread aborted: existing report thread still running

Is there a way to stop this 'existing thread'? I've tried stopping and restarting and rebooting the PI. Nothing seems to stop it!

it looks like you are trying to generate a report every 10 seconds, and the hardware on which you are running weewx is unable to complete all of the reports that you have requested in that time.

change your archive interval to something more reasonable such as 5 minutes (300 seconds)

if you need real-time display, consider uploading to influxdb or an MQTT broker, with something like grafana attached for the display.

m

Ron Walker

unread,

Jul 4, 2019, 8:45:25 AM7/4/19

to weewx-user

Hi Gary,

After a fresh install of OS and WeeWX, the problem persists. I took your advice and inserted the logging line into fileparse. I am seeing windGust data in the incoming data stream, I see it in the LOOP packets and I see it in the output of the logging command. What I see in the REC is "None" for windGust. How is it possible that with a new database and a new input file it is interpreting None for a 0.0 value? I have even changed around the order of the input stream to see if that would affect anything. Nothing changed. If the issue is my input data stream, why would it be consistently THIS observation and not affect any other? Previous issues with the data file showed fileparse errors and inconsistent values, etc. I can send in the logging file, but it quite large. I will up load it if you need to see it. Is there anywhere else I can look?

Ron

On Wednesday, July 3, 2019 at 8:47:17 AM UTC-4, Ron Walker wrote:

Ron Walker

unread,

Jul 4, 2019, 8:58:10 AM7/4/19

to weewx...@googlegroups.com

I forgot to ask, is there anyway to change the records in the database to ‘0.0’ instead of ‘NONE’?

Ron

Sent from Yahoo Mail for iPhone

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3a37c19d-83a3-4e4b-adf9-302705a47105%40googlegroups.com.

Ron Walker

unread,

Jul 4, 2019, 9:12:58 AM7/4/19

to weewx...@googlegroups.com

Gary,

I also forgot to add that the exact same thing is happening with the observation ‘windGustDir’. Data shows up everywhere but in the REC packet, where the value is ‘NONE’.

Ron

Sent from Yahoo Mail for iPhone

On Tuesday, July 2, 2019, 9:37 PM, gjr80 <gjrod...@gmail.com> wrote:

Ron Walker

unread,

Jul 4, 2019, 10:18:26 AM7/4/19

to weewx-user

Hi,

I definitely did not have an interval of 10 seconds set anywhere. This was just another of the weird issues that suddenly appeared after the unit had been working well over the last couple of weeks. I did a clean install of the OS and WeeWX and that problem is no longer an issue.

The lack of windGust data being displayed, unfortunately, has not.

Andrew Milner

unread,

Jul 4, 2019, 10:57:11 AM7/4/19

to weewx-user

Looking at a log file you have posted earlier I see loads of undervoltage reports from your kernel. You should not be getting these at all. Combining that with what I assume to be bad data transfers from the arduino over possibly usb (ustWindDir10avg for example instead of what I think should have been GustWindDir10avg) and other similar instances.

I think you need to go step by step, getting full logs and debug info along the way (regardless of the file sizes). Your input data appears to have loads of different wind speeds and gust speeds available - do you need all these??

Ron Walker

unread,

Jul 4, 2019, 11:41:47 AM7/4/19

to weewx-user

Hi Andrew,

Thanks for your reply. The undervoltage reports resulted from me using a Pi UpTime UPS as power backup. I have removed it from the equation and the undervoltage reports have ceased.

The reason for the multiple wind observation in the input file was two-fold. First, the averages are of more use to us in our application because we don't have real-time information available. If we just used windGust, we might miss gusts between loop intervals. The second was as a test of using observations that were not in the database on the web page. I have tested with those included and removed. It made absolutely no difference.

On the subject of the input data, I will do the logging as you suggest and post the results. One thing that bothers the snot out of me is the consistency of this. No matter what changes are made, new database, new input file, changes to the order of the observations in the input file, the result is consistently the same. I find that very strange.

To get around this in the short term, I've added two observations in the input file. 'mywindGust' is mapped in the input file to my 'windGust' variable and 'mywindGustDir' is mapped to 'windGustDir'. Doing this, I am able to display the actual values for 'windGust' and 'windGustDir' on the web page. As an aside, these values display with many decimal places. Is there a way to change that so that they display the windGust would?

Ron Walker

unread,

Jul 4, 2019, 1:41:50 PM7/4/19

to weewx-user

Hi Andrew,

I've posted logging for two archive intervals with startup here. I see no anomalies in the Loop data, the raw data, or the data file. I'm at a loss. Any input would be appreciated.

Ron

loop_REC_data.txt

syslog.txt

data.csv

Andrew Milner

unread,

Jul 4, 2019, 2:15:34 PM7/4/19

to weewx-user

sadly i do not use seasons or the latest version of weewx. however looking at the fileparse example on an earlier version it looks as though the fileparse driver uses the contents of a file to create a loop packet. presumably the arduino is changing this file at regular intervals to generate a stream of loop packets. I then assume that at every archive interval weewx is generating the rec packet for archival purposes ie record generation is software. I am pretty sure that somewhere along the line I saw record generation is hardware in your weewx.conf. I am wondering if there is a mismatch between hardware and software archive creation and hardware/software preferred values and the use or otherwise of loop data in hi/lo storage? just a thought.

for the formatting just add a formatting string on the web page is the simplest answer

http://weewx.com/docs/customizing.htm#Formatting_examples

gjr80

unread,

Jul 4, 2019, 4:44:03 PM7/4/19

to weewx-user

Ron,

My apologies for being quiet, travelling yesterday and 4 July today.

It would seem that the current state of play is that you have stable loop packets including all expected obs and correct values but windGust data in each archive record is always None. Andrew is correct in that the fileparse driver produces loop packets only and thus WeeWX must use software record generation; however, if WeeWX is set to use hardware record generation WeeWX will automatically fall back to software record generation if the driver does not support hardware record generation. In essence, in your case it does not matter how record_generation is set.

I am at a bit of a loss at present. Others obs are being correctly accumulated and extracted but windGust is not.

If I understand correctly you have reinstalled your OS and WeeWX, could you post a current copy of weewx.conf. That can easily be done by posting the output from a wee_debug report (http://weewx.com/docs/utilities.htm#wee_debug_utility) just make sure you check the output for sensitive info before posting. wee_debug should obfuscate such info but it’s not effect.

Gary

Ron Walker

unread,

Jul 5, 2019, 12:27:08 PM7/5/19

to weewx-user

Hi Gary and All who posted replies to this,

The issue is resolved. I did a complete uninstall of Weewx. I then did a new install and manually edited the weewx.conf, putting in the necessary basic info to get my station info, driver, label map, etc. No other changes were made to the default settings. I can only conclude that the issue lies in the weewx.conf file from the problem install.

Thank you all for lending ideas. I learned much from your insights and suggestions!

Ron

Reply all

Reply to author

Forward