wind loop and console data calculation

[Simon Davies]

Is there a way to determine how the wind (gust and speed) LOOP data is converted into the console data and the archive data.  Below is a small table of data showing LOOP, CONSOLE and ARCHIVE for the same/similar time.  

Timestamp	Wind Gust	Wind Speed	Source
2020-02-27 18:27:24 GMT	windGust: 0.683510010315	windSpeed: 0.310686368325	LOOP DATA
2020-02-27 18:27:32 GMT	windGust: 0.683510010315	windSpeed: 0.310686368325	LOOP DATA
2020-02-27 18:27:40 GMT	windGust: 0.683510010315	windSpeed: 0.49709818932	LOOP DATA
2020-02-27 18:27:56 GMT	windGust: 0.683510010315	windSpeed: 0.062137273665	LOOP DATA
2020-02-27 18:28:12 GMT	windGust: 0.683510010315	windSpeed: 0.12427454733	LOOP DATA
2020-02-27 18:28:27 GMT	windGust: 0.683510010315	windSpeed: 0.12427454733	LOOP DATA
27/02/2020  18:29 GMT	4	0.4	CONSOLE
27/02/2020  18:28:00 GMT	1.100002734	0.400000994	ARCHIVE

Weather station (Maplin N23DQ,  fineoffset)

The reason for asking is that the display(archive) data is different to the console data and I'm trying to understand at what point the error is occurring. e.g. between LOOP and console or between console and archive.

[gjr80]

Hi,

Going to need a little more info, in particular units. The LOOP DATA I presume is taken from the screen when running WeeWX directly? If so what units are being used, what is the usUnits value? 1, 16 or 17? Likewise the archive data, what is the usUnits value? If the archive data  was obtained by querying the database the units will be dependent on your database unit system and may not be the same units displayed on the console or reported by the weather station driver. Finally what units is your console displaying?

Gary

[Simon Davies]

Hi Gary,

yes the LOOP DATA is taken when running weewx directly.  (normally I use the daemon process, I don't seem to be able to run the daemon process and the weewx directly at the same time).

For the units I've added an extra column to the table,  as there are some variations.

Also I've added an extra row for the "REC:" records taken while running weewx directly.  The archive data is not from querying the database directly.  I have a report that produces this file.

Timestamp

Wind Gust

Wind Speed

Source

units	recorded

2020-02-27 18:27:24 GMT

windGust: 0.683510010315

windSpeed: 0.310686368325

LOOP DATA

usUnits: 1	weewx direct (stdout)

2020-02-27 18:27:32 GMT

windGust: 0.683510010315

windSpeed: 0.310686368325

LOOP DATA

usUnits: 1	weewx direct (stdout)

2020-02-27 18:27:40 GMT

windGust: 0.683510010315

windSpeed: 0.49709818932

LOOP DATA

usUnits: 1	weewx direct (stdout)

2020-02-27 18:27:56 GMT

windGust: 0.683510010315

windSpeed: 0.062137273665

LOOP DATA

usUnits: 1	weewx direct (stdout)

2020-02-27 18:28:12 GMT

windGust: 0.683510010315

windSpeed: 0.12427454733

LOOP DATA

usUnits: 1	weewx direct (stdout)

2020-02-27 18:28:27 GMT

windGust: 0.683510010315

windSpeed: 0.12427454733

LOOP DATA

usUnits: 1	weewx direct (stdout)

27/02/2020  18:29 GMT

4

0.4

CONSOLE

km/h	visually

27/02/2020  18:28:00 GMT

1.100002734

0.400000994

ARCHIVE

km_per_hour (weewx.conf)	weewx reports
2020-02-27 18:28:00 GMT	windGust: 0.683510010315	windSpeed: 0.295152049909	REC	usUnits: 1	weewx direct (stdout)

Simon

[Simon Davies]

Here is the last line (on the previous post it was truncated) 

2020-02-27 18:28:00 GMT

windGust: 0.683510010315

windSpeed: 0.295152049909

REC

usUnits: 1

weewx direct (stdout)

[gjr80]

Hi Simon,

Easy one first; you cannot run WeeWX both as a daemon and directly with a single station, it's one or the other.

As for the data, a few observations. All of the speeds on rows where usUnits == 1 are in miles per hour. The console data is of course km/h. The ARCHIVE and REC rows are km/h and miles per hour respectively and they should be the same speeds but in different units, windGust is but windSpeed is not. There is a large discrepancy between the console data and any other data though the console data is timestamped at least one minute after any of the other data.

I guess there is still a lot of variables. In an archive record sense WeeWX considers windSpeed to be the average wind speed seen over an archive period, windGust is seen as the highest wind speed seen over an archive period. When you start looking at loop data it comes down to what the station and driver output. is the loop windSpeed the average over the loop period or a point in time value. What is windGust in a loop context, the highest wind speed seen over a loop period (given the data above I don't think that is the case) or the highest value seen over some unknown period. You have not said what driver you are using (I believe there is more than one that supports your station). Also what tags are you using in the report that provides the ARCHIVE row? Finally, what does the console display values mean, particularly the wind gust value. You might find this wiki entry interesting.

Sorry, no real answers and just more questions.

Gary

[Simon Davies]

Hi Gary,  

The console data is timestamped with odd minutes ,  but the other data is timestamped with even.  I'd prefer it if they were the same,  but I'm not sure how to sync them.  I checked the console clock and its correct.

The driver I'm using is

[Interceptor]

    # This section is for the network traffic interceptor driver.

    # The driver to use:

    driver = user.interceptor

The tags in the report i'm using are

$_span.windSpeed.max.raw,   and   $_span.windGust.max.raw

I've read the wiki entry page and I'm now suspicious that the tags I'm using are wrong.  May be this will explain the discrepancy with the values shown on the console.  

Thanks Simon

[gjr80]

Hi Simon,

That is odd that the console timestamp is always odd minutes and the other is always even. What archive interval are you using, if the 'other' is always even is your archive interval two (or some multiple of two) minutes? How often does the console update, every 2 minutes (seems an awfully long period if it does) and perhaps it just started on odd minutes. Not familiar at all with the station/console I am afraid.

Since you are using interceptor you can actually get the interceptor driver to log when and what it is seeing from the console as well as what it is sending to WeeWX. Just edit weewx.conf and set debug = 1, save weewx.conf then restart WeeWX. You can tail the WeeWX log in a terminal window if you wish to transcribe data while watching the console. Only thing is to watch what units are being displayed.

Tag wise $span (assume the _ is a typo/cut-paster artefact) should work, $span was developed to provide aggregates across a span in time, for example $span($week_delta=2).outTemp.max would display the max outside temp over the last two weeks. Omitting the $xxxx_delta parameters should give you the current value of the observation since the span of values that are considered is just the current value. However, it it was me I would just use $current.windSpeed.raw and $current.windGust.raw to display the average wind speed over the archive period and the highest wind speed seen during the archive period respectively.

Gary

[Simon Davies]

Hi Gary,

I've changed the console interval & the archive interval to 1 minute which is the minimum for the console.  This at least has made it slightly easier to match the readings.

I tried setting the debug level to 1 and viewed the syslog.  The units displayed in the log are usunits=16 (which I assume is meters per second?).  I could not understand how these values were used to be stored in the archive database.  See the table below,  for example 1.1 mps  becomes 1.36 mph.  it should be 2.46 mph using my calculations.  Do you have any ideas why the archive DB has this value?

		time	debug data wind gust (mps)	debug data usunits	console minutes	console wind gust km/h	DB (mph)
Mar	8	22:40:57	1.1,	16	40	4	1.36702
Mar	8	22:42:49	0.0,	16	42	0	0.68351
Mar	8	22:44:57	1.1,	16	44	4	2.05053

For the report,  I removed all of the $span and used $record = $day.records and now the output figures match the archive DB.  at least this one is now clear - thanks.

Regards Simon

[gjr80]

Hi Simon,

I tried setting the debug level to 1 and viewed the syslog.  The units displayed in the log are usunits=16 (which I assume is meters per second?). 

The usUnits decodes are 1 (or 0x01) is US (aka US customary) and uses miles per hour for wind speeds, 16 (or 0x10) is Metric with wind speeds in km per hour and 17 (or 0x11) is known as MetricWX with wind speeds in metres per second.

 

I could not understand how these values were used to be stored in the archive database. 

A little background on how units are used within WeeWX. First up the database, in weewx.conf under [StdConvert] you have a setting target_unit that will either be US, METRIC or METRICWX. This sets the database unit system used throughout the database. The default is US. Some folks insist on having target_unit set to the units they use natively, there really is little value in doing this unless you have specific requirements as WeeWX automatically converts to/from the units required (in fact I have seen it cause issues variously where database units are used (for example specifying QC limits without units), folks forget their database units are different to the normal and they can get unexpected results). Data coming from drivers, whether they be loop packets or archive records can be in any of the three unit systems, WeeWX automatically converts any data being saved to database to the target_unit unit system. Likewise any archive records or loop packets being processed elsewhere by WeeWX could be in any of the three unit systems. Data being output in a Cheetah report or plotted by the image generator is automatically converted to the units specified by the tag, skin.conf or weewx.conf or the defaults as applicable. Bottom line is when analysing WeeWX data never assume any units, always check the usUnits value.

As mentioned above you usUnits=16 data will be km/hr rather than m/s. That still doesn't help things tally. So what is 'DB', is that windSpeed or windGust?

Appreciate the table may make things easy to read but perhaps it might help more if you provide the actual loop/archive record output when WeeWX is run directly, a copy of the log (with debug set) over the same period and a query from the database again covering the same period and your recorded observations of the console gain over the same period.

So when there is some wind around let's run WeeWX directly for about 5 minutes. At the same time WeeWX is running directly note your wind speed values on the station display. Capture the console output (LOOP: and REC: data) for the entire period. Take a log extract that covers the entire period that WeeWX was run directly.

Now query the database after finishing the direct WeeWX run. Am assuming you are using SQLite as your DBMS. Note you may need to install the sqlite3 application using sudo apt-get install sqlite3 if not already installed. Also assuming your database is named weewx.sdb and it is located in /home/weewx/archive (it may be located in /var/lib/weewx depending on your WeeWX install method - adjust following command as necessary):

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite> .headers on
sqlite> SELECT datetime,datetime(dateTime,'unixepoch','localtime'),usUnits,windSpeed,windGust FROM archive ORDER BY dateTime DESC LIMIT 20;

that will display the timestamp, human readable date-time, usUnits, windSpeed and windGust values for the last 20 minutes (use .quit to exit sqlite3). If run within 15 minutes of completing your direct run we should have some overlapping data to compare from (1) the loop packets and archive records seen by WeeWX (2) the raw data seen by the SDR driver (3) your visual observations of the station display and (4) the database. Post these four pieces of data.

One final question, is your loop_hilo setting under [StdArchive] in weewx.conf set to True or something else?

Gary

[Simon Davies]

Hi Gary,

From my previous table "DB" showed the windGust value.

As requested,  I've taken readings from the weewx direct LOOP/REC data,  debug data,  extract from the DB and I've taken a couple of screenshots from the console.  

Time period would be best to take from 21:03 for  approximately 15 minutes,  although the logs cover a larger period.  I've attached 5 files with this information.  I hope they contain the information you have requested.

I've not checked all of the values myself,  I'll will do after posting here.

loop_hilo is set to True.

Many Thanks Simon

[Season Ticket]

Hi Gary,  I've spent some time looking at the log files and focusing on the windgust, still unresolved unfortunately. 

I’m using the interceptor driver,  I looked at the comments in the interceptor driver code which says

"The gust measure has resolution of 1.1 m/s (2.46 mph) - one revolution of the anemometer." 

However the lowest gust value in the LOOP logs is “0.683510010315”.  There must be a units problem here,  I understand that “0.683510010315” usunits=1=mph, displayed in the each log record, and will convert to 1.1 km/h,  but the console shows 4 km/h,  so I’m assuming that 1.1 is m/s which converts to 3.96 km/h, very confusing,  but I can get the figures to match even if they don’t make sense to me. Is there a simple explanation about these figures?

 

I notice that the LOOP & debug data for windgust is consistently the same, however debug data does not show the REC: records,  these are only shown from the weewx direct logs. I'm unsure if the interceptor driver is sending the REC: records,  do you know?

I assume it is as weewx.conf says

“[StdArchive]

….

# Set the following to "software" to force software record generation.

    record_generation = hardware”

[gjr80]

Hi Simon,

I am terribly sorry, I completely forgot about your problem, I put your previous post aside to sit down and do the maths but never got back to it.

Does sound like there is a problem, going to need to refresh my memory on the interceptor/observer operation. Unfortunately, it will be a tomorrow job, late here, won't forget it this time. As for archive records (REC:), the interceptor driver only knows how to emit loop packets, it does not know how to emit archive records. WeeWX accumulates the loop packets to produce an archive record (known as software record generation) which is why you will see nothing about archive records in the interceptor debug output. You are right in that setting record_generation = hardware under [StdArchive] will force WeeWX to obtain archive records from the driver; however, if the driver cannot emit archive records WeeWX effectively falls back to software record generation and archive records are synthesised by WeeWX from the accumulated loop data.

Gary