Data Precision for Dewpoint

[satwa...@gmail.com]

Hi, 

I searched the group for this subject and couldn't find anything so I am posting.

I have the Vantage Pro 2, serial interface to Weewx 3.9.2 run on an Ubuntu server.

Below is a sample line from the archive database (broke into individual lines due to width) using the command: 

   sqlite3 -header -csv /var/lib/weewx/weewx.sdb "select * from archive;"

One line from the output:

1597612740,1,1,29.617,28.0238601652778,29.7814033867068,83.7,115.4,

38.0,18.0,0.0,,1.0,225.0,0.0,0.0,61.1737982063856,115.4,

115.4,0.0,,,,,,,,,,,,,,,,,,,,93.9583333333333,0.0,5.31,,,,,,,,,,

Note the temperature is reported as 115.4 degs (ugh...this is Arizona).  The dewpoint is reported as 61.1737982063856.  All dewpoint values in the database have a decimal precision of 13 places, clearly outside the sensor quality range. The sensor documentation indicates that it provides Relative Humidity values so apparently the console software derives the dewpoint value as it can be displayed on the console. I write a lot of external programs and have to repeatedly reduce the precision for this variable.

Is there an easy way to modify my Weewx to store only 1 decimal place for dewpoint or other variables?  I did check /etc/weewx/weewx.conf and could not find any obvious setting for this.

Thanks for any assistance,

Ken

[Tom Keffer]

Archive records are what are stored in the database. A Vantage station produces its own archive record (unless record_generation is set to 'software'). However, for whatever reason, Vantage stations do not include dewpoint in the hardware generated record, so WeeWX must calculate it. By default, WeeWX calculates the average dewpoint seen over the archive interval. If dewpoint changes over the interval, the average is not necessarily an even number.

One way around this is to have WeeWX calculate the last value seen, instead of the average. Add this to the end of your weewx.conf file:

[Accumulator]

  [[dewpoint]]

    extractor = last

Alternatively, what sort of external programs are you writing? Does the language you are using provide a way to format decimal output?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/20d1b87c-afd9-4f1c-b00c-88f9011057d8n%40googlegroups.com.

[Ken Waters]

Tom,

Thanks for your useful response.  

Here in the desert (and especially during the southwest monsoon) dewpoints are scrutinized carefully for trends showing surface moisture surges. 

I write mostly in Python and Perl and do additional analyses such as average daily dewpoint (not shown in the standard NOAA-format tables produced with Weewx) as well as 24-hour trend charts using SQL calls and Gnuplot (example: http://satwatcher.us/wxstation/trend/).  I do use standard sprintf calls to format the output but I just always felt it strange to have to do that when the temperature was nnn.n format.  Also, as a scientist it just rattles me every time I see a number with too many significant figures.  :-)  Lastly, I'm interested in the resulting reduction of the size of the database file for efficiency.

On a related note, I do see a lot of variation in the dewpoint trace, often jumping 1 or 2 degrees back and forth.  As a meteorologist I do understand that dewpoint typically has a higher variability than temperature but I don't think it's that much, with the sole exception of a sudden wind shift or gust of wind.  Now you got me thinking if maybe making this change will actually lead to a more stable value for dewpoint.

I will try this change and let you know what happens.

Thanks,

Ken

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBuz-FwYAt2z6YT4mZ39WipiBhLny4zwnbd3FZYkRci1Q%40mail.gmail.com.

[Paul R Anderson]

Alternative method would be to set Vantage driver loop_request to 3 so that you would get loop1 and loop2 data. Vantage loop2 provides dewpoint to 1 decimal point . Please review User Guide and be sure you understand how to set it up , further changes are required.

On a system here that is set up to alternate loop1 and loop2 here is a database query for dewpoint:

sqlite> select datetime(dateTime,'unixepoch','localtime'),dewpoint from archive ORDER BY datetime desc limit 10;
2020-08-16 20:00:00|62.0
2020-08-16 19:55:00|62.0
2020-08-16 19:50:00|62.0
2020-08-16 19:45:00|62.0
2020-08-16 19:40:00|62.0
2020-08-16 19:35:00|62.0
2020-08-16 19:30:00|62.0
2020-08-16 19:25:00|62.0
2020-08-16 19:20:00|62.0
2020-08-16 19:15:00|62.0

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBuz-FwYAt2z6YT4mZ39WipiBhLny4zwnbd3FZYkRci1Q%40mail.gmail.com.

[Paul R Anderson]

For more info on loop2 the davis portal document is the definitive word on the fields and their definition. 

Forgot that I have a little test file that shows what you get from Loop2:

ALL LOOP2 DATA
trendIcon -20.000000
barometer 29.913 inHg
inTemp 73.8F
inHumidity 63%
outTemp 65.1F
windSpeed 1 mph
windDir 338
windSpeed10 2 mph
windSpeed2 2.000000
windGust10 0.900000
windGustDir10 67.000000
dewpoint 62.0F
outHumidity 89%
heatindex 66.0F
windchill 65.0F
THSW 64.0F
rainRate 0.00 in/h
UV 0.0
radiation 0 W/m
stormRain 0.00 in
stormStart ?'stormStart'?
dayRain 0.00 in
rain15 0.000000
hourRain 0.00 in
dayET 0.038000
rain24 0.00 in
bar_reduction 0.002000
bar_offset 0.028000
bar_calibration 17.606000
pressure_raw 29.716000
pressure 29.744 inHg
altimeter 29.918 inHg

Paul

[Tom Keffer]

I get what you're saying. By training, I am also an earth-scientist (physical oceanography).

In sqlite, all floating point numbers are stored as a big-endian IEEE 754-2008 64-bit floating point number. Doesn't matter whether it is a nice "round" number or not ---- they all take the same amount of space.

One can be led astray by the formatting algorithms used by sqlite3. When you see "61.2", what you're actually seeing is a shorthand for 61.2000000000000, which is just as absurd in its precision claims as 61.1737982063856. The only difference is that sqlite3 chose to truncate all those zeros but, inside the computer, they are both first-class, 64 bit numbers, albeit with slightly different bit patterns. The difference is in the formatting convention that is used when the number ends in a bunch of zeroes, versus when it does not.

I would stick with sprintf() and forget about what's inside the computer!

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CACZLRFphCbvBFDwGJm-ahMaWz%3D9SdXc9LJ6nQdFPqyZ%3DVhm2GQ%40mail.gmail.com.

[Greg Troxel]

Paul R Anderson <pa...@pauland.net> writes:

> Alternative method would be to set Vantage driver loop_request to 3 so that
> you would get loop1 and loop2 data. Vantage loop2 provides dewpoint to 1
> decimal point . Please review User Guide

> <http://www.weewx.com/docs/usersguide.htm#[Vantage]> and be sure you

> understand how to set it up , further changes are required.

That makes sense, but if there is consensus that dewpoint from davis,
calculated by davis or weewx, is only good to one decimal point (0.1
degree), then it seems that the calculation process should round to that
0.1 degree. Given what I see, even 0.1 degree is a bit aspirational,
but I would agree with those who say rounding to 1 is throwing away
probably data.

[Ken Waters]

Paul and Tom,

(1) I did the addition that Tom suggested to weewx.conf and it did not change the number of digits after doing a stop/start.

(2) I did the suggestion that Paul had and it did not change the number of digits after doing a stop/start.

I added the following to the end of the [Vantage] section of wx.conf:

    # Changing the loop request from 1 (default) to 3
    loop_request = 3

This is the result after making both changes to weewx.conf (last number is dewpoint):

1597623840,"2020-08-16 17:24:00",107.7,48.2011679891737
1597623780,"2020-08-16 17:23:00",107.8,48.2789309796842
1597623720,"2020-08-16 17:22:00",107.9,50.2068320569903
1597623660,"2020-08-16 17:21:00",107.9,51.9515297553893
1597623600,"2020-08-16 17:20:00",108.1,52.1095650274203

I don't completely understand what Loop1, Loop2, and Loop3 are so I guess I'll have to spend some time poring over the Davis document that Paul referred to.  I do understand what Tom is saying about real format storage and those extra digits just being artifacts of the storage method.

In the meantime.....maybe I'll just cut my losses and do as Tom suggests and just keep using sprintf.

Interesting discussion....

Thanks,

Ken

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAOAVAecvOtnCNZA37B9H37qhUTj%3D3MeejoMG1mCDh_9ouXDrTA%40mail.gmail.com.

[Tom Keffer]

(1) I did the addition that Tom suggested to weewx.conf and it did not change the number of digits after doing a stop/start.

Did you go back to the original value of loop_request? That is, 1?

I do understand what Tom is saying about real format storage and those extra digits just being artifacts of the storage method.

Sorry, but my point was just the opposite: it's an artifact of the rounding convention used by sqlite3.

-tk