$archive[1].wind.gustdir.format: N/A

[Vetti52]

Since update to version 4.2.0, I have a curious result in season skin. Although $current.windDir is properly displayed and continuously collected, there are no longer windDir data in the hi/lo section. Wind Max shows speed correctly, but no direction. I checked hilo.inc. There are no changes, compared to the original version (hilo.inc.dpkg-dist dated from 2020-04-30).

Where should I start to trace the bug?

[Tom Keffer]

What do you mean by, "there are no longer windDir data in the hi/lo section"? Is there a placeholder? Or, white space? Or, N/A?

It might be worth seeing what is in your database. This assumes your database is located at /var/lib/weewx/weewx.sdb. If you used the setup.py install method, it would be at /home/weewx/archive/weewx.sdb.

sqlite3 /var/lib/weewx/weewx.sdb

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), * from archive_day_wind order by dateTime desc limit 20;

sqlite> .quit

-tk

On Tue, Dec 8, 2020 at 8:15 AM Vetti52 <drv...@gmail.com> wrote:

Since update to version 4.2.0, I have a curious result in season skin. Although $current.windDir is properly displayed and continuously collected, there are no longer windDir data in the hi/lo section. Wind Max shows speed correctly, but no direction. I checked hilo.inc. There are no changes, compared to the original version (hilo.inc.dpkg-dist dated from 2020-04-30).

Where should I start to trace the bug?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com.

[Vetti52]

It says "N/A".

The output of sqlite:

2020-12-08 00:00:00|1607382000|0.0|1607382032|10.29|1607442892|148.553777777778|211|148.553777777778|211||141.25403836784|8.54431223906981|211|307.861736888889|307.861736888889
2020-12-07 00:00:00|1607295600|0.0|1607295628|18.34|1607348466|969.075666666667|288|969.075666666667|288||563.975677916051|-355.104682689556|288|4027.90666114815|4027.90666114815
2020-12-06 00:00:00|1607209200|0.0|1607209233|9.17|1607219761|77.5104166666667|286|77.5104166666667|286||69.480956552117|2.39263945299658|286|132.33436038966|132.33436038966
2020-12-05 00:00:00|1607122800|0.0|1607123329|12.53|1607140258|550.433333333334|288|550.433333333334|288||357.29020463216|-182.82480166047|288|1638.89117222222|1638.89117222222
2020-12-04 00:00:00|1607036400|0.0|1607039337|20.58|1607050956|806.906472222223|288|806.906472222223|288||609.110049598142|-419.170693241216|288|2757.25666137114|2757.25666137114
2020-12-03 00:00:00|1606950000|0.0|1606950381|14.76|1606982936|938.132555555555|286|161542.062333333|52611|136.0|63684.8752720733|-143160.707316072|52611|3367.86934245679|550406.753590012
2020-12-02 00:00:00|1606863600|0.0|1606863628|10.29|1606933960|298.527702380952|288|89558.3107142857|86400|97.0|42277.4937890549|-72643.1316507023|61200|647.727487337254|194318.246201176
2020-12-01 00:00:00|1606777200|0.0|1606778615|9.17|1606778252|265.736888888889|288|79721.0666666667|86400|194.0|42931.7123928354|-43844.9965062139|69900|434.78745508642|130436.236525926
2020-11-30 00:00:00|1606690800|0.0|1606690818|12.53|1606753726|458.85391053391|287|137656.173160173|86100|138.0|-43283.5066028223|-120674.014788181|77100|1142.32670378797|342698.01113639
2020-11-29 00:00:00|1606604400|0.0|1606604418|5.82|1606650983|50.8892222222223|288|15266.7666666667|86400|309.0|-11596.0986434007|7995.68851878374|24600|54.2518490740741|16275.5547222222
2020-11-28 00:00:00|1606518000|0.0|1606518015|11.41|1606565426|165.474444444445|287|49642.3333333333|86100|63.0|47302.3988525243|1887.80479760903|28200|423.739858888889|127121.957666667
2020-11-27 00:00:00|1606431600|0.0|1606431616|4.47|1606483066|9.17855555555556|288|2753.56666666667|86400|131.0|2261.0873478704|-539.506810163124|4200|10.6575301851852|3197.25905555556
2020-11-26 00:00:00|1606345200|0.0|1606345204|8.05|1606394153|107.788222222222|287|32336.4666666667|86100|259.0|-25803.5675497035|-15488.3625339286|42000|134.601425382716|40380.4276148148
2020-11-25 00:00:00|1606258800|0.0|1606258804|9.17|1606302300|337.383111111111|288|101214.933333333|86400|189.0|-19895.5174799498|-97556.2211722166|76500|636.16824345679|190850.473037037
2020-11-24 00:00:00|1606172400|0.0|1606172999|10.29|1606187816|577.891555555556|288|173367.466666667|86400|213.0|-90246.0852000086|-146738.507905923|86400|1319.29265822222|395787.797466666
2020-11-23 00:00:00|1606086000|0.0|1606086005|10.29|1606137950|368.728555555556|288|110618.566666667|86400|261.0|-97191.5602413715|-50554.3362971601|72300|718.446958555555|215534.087566667
2020-11-22 00:00:00|1605999600|0.0|1606008451|14.76|1606041552|645.880888888889|288|193764.266666666|86400|245.0|-171272.270757009|-84245.3229881982|73500|2109.85465123457|632956.39537037
2020-11-21 00:00:00|1605913200|0.0|1605913439|15.88|1605942546|1114.47377777778|288|334342.133333333|86400|191.0|-175428.251363656|-273150.682819638|86400|4553.4892182716|1366046.76548148
2020-11-20 00:00:00|1605826800|0.0|1605826872|9.17|1605833902|169.842222222222|288|50952.6666666667|86400|238.0|-42329.2825046772|-24698.2615292903|59400|247.763276197531|74328.9828592593
2020-11-19 00:00:00|1605740400|0.0|1605798095|22.82|1605788229|919.519666666666|288|275855.9|86400|255.0|-208559.830853793|-91789.2604135498|82800|3926.4786627037|1177943.59881111

[Tom Keffer]

Thanks.

I assume you updated on 2020-12-04? There is no value for "max_dir" (which is where "gust_dir" comes from) in your daily summaries since that date.

Looking through your daily summaries, you are suffering from issue #623 (as are most V4.2 users). I am working on a fix for this. However, as far as I know, this problem should not be affecting max_dir, but I may be missing something. 

A few questions:

1. What kind of hardware?

2. What is your setting for option ignore_zero_wind, if any?

3. What is your setting for option record_generation?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com.

[Vetti52]

tha update was on 2020-12-04, right.

at 1.

I am running Weewx on a Raspi4 under buster, installed with apt-get install weewx. My station is a EFWS2500, which is a clone of Ecowitt 2500. The data are collected by a Froggit DS2500, which is a clone of GW1000. Weewx still collects the data from interceptor.py ecowitt-client. Although user.gw1000  is already implemented as a service, I still have not yet replaced interceptor.py.

at 2.

 didn't detect ignore_zero_wind in weewx.conf.

at 3.

    record_generation = software

HTH

Thanks

-ph

[Vetti52]

BTW, there are some values not null in the database after update, as seen, when filtering the archive for windGustDir <> null:
dateTime            windGustDir

2020-12-08 10:50:00 89.0
2020-12-08 05:35:00 95.0
2020-12-08 01:25:00 148.0
2020-12-07 22:20:00 168.0
2020-12-05 01:35:00 120.0
2020-12-04 22:00:00 101.0

[Vetti52]

So, I am  still not sure, if wee_database --update or --reweight should solve the problem. Or do I have to wait for a special fix?

[Tom Keffer]

I do not think your problem is related to the "weighting" problem. It affected only the daily summaries and, even then, only fields that are weighted by the archive length. Wind direction is not one of them.

Unfortunately, I am not very familiar with any of the drivers and services you are using. If I understand correctly, the data in question was collected by interceptor.py. I would assume that's where the problem is. Does the driver emit windGustDir on every LOOP packet? Or, does it rely on software record generation to provide it at the end of an archive period?

Sorry I cannot be of more help.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com.

[Vetti52]

Here a comparison of three outputs, first as GW1000.service , then as gw1000 driver and finally the current working ecowitt-client by the interceptor driver:

# PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-driver

Using configuration file /etc/weewx/weewx.conf
Interrogating GW1000 at 192.168.100.150:45000
2020-12-12 17:55:00 CET (1607792100): UV: 0, dateTime: 1607792100, dayRain: 0.0, daymaxwind: 7.7, inHumidity: 59, inTemp: 17.7, luminosity: 0.0, monthRain: 5.7, outHumidity: 93, outTemp: 2.7, pressure: 999.0, rain: None, rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 17, uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 70, windGust: 2.6, windSpeed: 1.0, yearRain: 729.9

# PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-service
Using configuration file /etc/weewx/weewx.conf
Interrogating GW1000 at 192.168.100.150:45000
LOOP:   2020-12-12 17:56:02 CET (1607792162) UV: 0, dateTime: 1607792162, dayRain: 0.0, daymaxwind: 7.7, dummyTemp: 96.3, inHumidity: 59, inTemp: 63.86, luminosity: 0.0, monthRain: 0.22440944881889766, outHumidity: 94, outTemp: 36.86, pressure: 29.5004575125, rain: None, rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 1, uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 77, windGust: 4.473883703878609, windSpeed: 2.6843302223271652, yearRain: 28.736220472440944
LOOP:   2020-12-12 17:56:12 CET (1607792172) dateTime: 1607792172, dummyTemp: 96.3, usUnits: 1

# PYTHONPATH=/usr/share/weewx python3 user/interceptor.py  --device=ecowitt-client --mode=listen --port=9000
mapped packet: {'dateTime': 1607792207, 'usUnits': 1, 'pressure': 29.492, 'barometer': 29.613, 'outHumidity': 94.0, 'inHumidity': 58.0, 'outTemp': 36.9, 'inTemp': 63.9, 'windSpeed': 2.46, 'windGust': 5.82, 'windDir': 108.0, 'radiation': 0.0, 'rain': None, 'rainRate': 0.0, 'rainEvent': 0.0, 'UV': 0.0, 'txBatteryStatus': 0.0}

The windGustDir does not seem to be emitted. There are two other issues: "rainEvent" and "txBatteryStatus" are missing in the GW1000 data or are not mapped correctly. Thus, Weewx shows no results for Rain Event, and indicates a low transmitter battery status. I would like to move to the GW1000 driver (or better use GW1000.service?), if it would work flawlessly.  But it would be nice, to know, how to proceed now.

I had GW1000.service activated in parallel to the interceptor driver, although this is somewhat nonsense, as both data come from the same sensors, either "translated" to Ecowitt style by the GW1000, or pulled by the GW1000.service directly. This I did just for a short time to see, if there were any errors occuring. Although the raw data seem not to be comparable, I could not see any effect. Actually I went back to the interceptor driver, so at least rainEvent and the battery status are ok again.

[gjr80]

If I can add a little regard the GW1000. The GW1000 API provides four wind related obs; these are labelled Wind Direction, Wind Speed, Gust Speed and 'Day max wind'. There is no other description/detail on what they mean though I think their broad meaning is self evident. By default the GW1000 driver (whether operated as a driver or as a service) maps Wind Direction, Wind Speed and Gust Speed to WeeWX fields windDir, windSpeed and windGust respectively. 'Day max wind' is mapped to WeeWX field daymaxwind. In the default GW1000 driver mapping nothing is mapped to windGustDir, as you would expect as the GW1000 API does not provide such data and it is not the place of the driver to derive such data.

I have not seen any form of specification for the 'Ecowitt format' data emitted by the GW1000 when uploading to a custom server in Ecowitt format but given the fields available via the API I would suspect/expect that a wind gust direction field would not be included. In other words, I would not expect the interceptor driver to emit WeeWX field windGustDir. I would expect that any non-None windGustDir data in the archive will have been derived by WeeWX from  accumulated loop data.

If I digress a little to the rain and battery state issues raised in the previous post. The GW1000 driver uses a field map to map data obtained from the GW1000 API to WeeWX fields. The driver uses a default map that maps GW1000 API data to fields in the WeeWX extended schema where an appropriate equivalent field exists; for example, the previously mentioned Wind Direction data is mapped to WeeWX field WindDir. Where no such equivalent exists the GW1000 data is mapped to a lowercase self evident field; for example, the 24 hour average PM2.5 data from the fourth WH41/WH43 PM2.5 sensor is mapped to WeeWX field pm2_54_24hav. In the case of the rain event data stormRain is an equivalent field in the WeeWX extended schema and the default GW1000 driver mapping maps rain event to stormRain. If you look in the sample GW1000 driver output data you posted you will see field stormRain. If this does not suit you can simply override the default field mapping and map the rain event data to WeeWX field rainEvent (or any other field name you wish).

It is a similar case with battery state data. The GW1000 is capable of operating with many sensors. These sensors are individual powered and the GW1000 can provide battery state data for each sensor. In such a situation the use of the general txBatteryStatus would be confusing (what sensor does it refer to?). The GW1000 driver maps battery state information to WeeWX fields that detail the sensor type/model, and where applicable channel number; for example wh57_batt or wh41_ch1_batt. Again the user has the ability to override the default mapping and map any battery field to txBatteryStatus. If you look at the sample GW1000 driver output data you will see field wh65_batt.

Gary

[Tom Keffer]

That tells us that the driver is not emitting windGustDir, but it's not telling us why it cannot be extracted out of the accumulators by WeeWX. 

Attached is an instrumented version of accum.py. Swap the version you have with it. It should be in /usr/share/weewx/weewx/accum.py.

Set debug=1, restart weewx, then let it run through the first reporting cycle. Post the log.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com.

[Vetti52]

Here is the result

Thanks

-ph

BTW, I had a look at the most recent thread, presenting a greek version (https://groups.google.com/g/weewx-user/c/h7nEzNRpNhM/m/tCXM0sgYAgAJ). There seems to be the same problem with N/A for windGustDir.

[Vetti52]

Just to complete the log file, I restarted again using GW1000 driver, which I would like to use, as sson as the missing . Looks different.

HTH

--ph

[Tom Keffer]

OK, looks like the gust direction is not getting added to the accumulators. Same drill: replace your copy of accum.py with this one.

Make sure you let it run through the first reporting cycle. In your first run, you stopped it a bit too early.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/83d99b6d-863c-4424-afd6-74c119fd25f7n%40googlegroups.com.

[Vetti52]

ok, this time, I think, I got several loop in. Sorry...

[Tom Keffer]

I see the problem. It's a subtle bug. Try this version of accum.py.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b260aeab-db22-4a65-90a0-20cfa61bf408n%40googlegroups.com.

[Vetti52]

ok, here we are:

[Tom Keffer]

That last version was not an instrumented version. It was intended to fix the problem. Do you now see values for $month.wind.gustdir, and so on?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f242a4ec-4759-4eb0-bb84-5374aa9530c9n%40googlegroups.com.

[Vetti52]

Thanks, Gary for the explanation!

I would like to compare the data, which are provided by the ecowitt-client of interceptor.py to those of the GW1000 driver with respect to the two parameters mentioned:

The sensors are declared in interceptor.py either in the stanza:

    DEFAULT_SENSOR_MAP = {
        'pressure': 'pressure',
        'barometer': 'barometer',
        'outHumidity': 'humidity_out',
        'inHumidity': 'humidity_in',
        'outTemp': 'temperature_out',
        'inTemp': 'temperature_in',
        'windSpeed': 'wind_speed',
        'windGust': 'wind_gust',
        'windDir': 'wind_dir',
        'windGustDir': 'wind_gust_dir',
        'radiation': 'solar_radiation',
        'dewpoint': 'dewpoint',
        'windchill': 'windchill',
        'rain': 'rain',
        'rainRate': 'rain_rate',
        'rainEvent': 'rain_event',
        'UV': 'uv',

...

}

and for customer's mapping in weewx.conf in the stanza

[Interceptor]
    # This section is for the network traffic interceptor driver.

    # The driver to use:
    driver = user.interceptor
    device_type = ecowitt-client
    mode = listen
    port = 9000
    [[sensor_map_extensions]]
        txBatteryStatus = wh65_battery

which, to my understanding, map the output of the ecowitt-client data to the appropriate weewx database fields.

When using the GW1000 driver, the mapping seems to be established in weewx.conf in the stanza

# Options for extension 'GW1000'
[Accumulator]
    [[daymaxwind]]
        extractor = last

    [[lightning_distance]]
        extractor = last
    [[lightning_strike_count]]
        extractor = sum
    [[lightning_last_det_time]]
        extractor = last
    [[stormRain]]
        extractor = last
  ...

If the mapping needs to be modified for the GW1000 driver, I do not know, how to adopt, in my case stormRain to rainEvent or wh65_battery to wh65_batt.

Obviously these two are not mapped correctly in my case. Unfortunately I am not smart enough to detect it by myself. So, please give me some simple guide.

In case, this is more complicated than I assume, it might be helpful to cut this topic out into a separate thread.

Thanks

--ph

[Vetti52]

No, unfortunately still "N/A" in all of December values.

[pmcg...@gmail.com]

Tom,

I was having the same issue using the gw1000 driver, I now see an entry for every archive in the database in windGustDir.  I suspect this will fix the problem if any of the new entries exceed the max for the month..  

I guess I could back fill the NULLs from windDir

[Tom Keffer]

pmcgeorge: I take it that you swapped out your version of accum.py for the one I posted?

Vetti52: The new version of accum.py is working in my tests. Are you sure the page you are looking at was regenerated, and not the old one?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9ea1b485-057d-45c7-9158-d6b03a714b39n%40googlegroups.com.

[pmcg...@gmail.com]

Yes I did.

[Tom Keffer]

Thanks. Good to know that the patch is working for you.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1fa7d142-932e-4935-9ee5-5180279a4e60n%40googlegroups.com.

[Vetti52]

I had a look at it overnight, now it looks fine to me also!!

Thanks a lot. Perfect work!

--ph