Vantage Pro2 - missing consBatteryVoltage and txBatteryStatus (among others) when using custom data service

[halol goog]

Hello,

I just upgraded to 5.1.0, and took this opportunity to do some maintenance on my station. This is when I noticed that my REC packets are missing some data. As a result, my database is not populated with all the values I am expecting.
In particular, consBatteryVoltage and txBatteryStatus are missing from the REC packets.

I have a data service, that is adding info from a second station to the packet, as described here:
https://weewx.com/docs/5.1/custom/service-engine/#add-data-source

When I disable it, I am getting values for consBatteryVoltage and txBatteryStatus. So, I must be doing something wrong. But all my code should be doing, really, is to add to the record.
I tried to print the record I am getting with 'event.record', and it doesn't contain the information either. This is what I get:

2024-10-20 19:35:16 weewxd[1]: INFO user.ws3000DataService: initial data:{'dateTime': 1729445700, 'usUnits': 1, 'interval': 5, 'rxCheckPercent': 99.9375, 'outTemp': 65.1, 'highOutTemp': 65.4, 'lowOutTemp': 65.1, 'rain': 0.0, 'rainRate': 0.0, 'barometer': 30.196, 'radiation': 0.0, 'wind_samples': 117.0, 'inTemp': 86.4, 'inHumidity': 52.0, 'outHumidity': 79.0, 'windSpeed': 1.0, 'windGust': 4.0, 'windGustDir': 337.5, 'windDir': 337.5, 'ET': 0.0, 'highRadiation': 0.0, 'forecastRule': 1}

So I am wondering: does using a custom data service 'stops' a subsequent process from happening and from adding the missing information to the REC (missing values sound like they are computed by weewx, like alarms, dayRain, yearRain, etc.)?

Thanks!

[Tom Keffer]

A custom data service does not "stop" and replace data  in a record. Written properly, it should just augment it.

But, it's hard to say what's happening without seeing what you're doing. Make sure you're adding to the even record, and not replacing it. Also, make sure that you're adding the service to the list of services to be run (under [Engine] // [[Services]] in weewx.conf) and not displacing another service.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6886fbd0-1646-44e5-9f93-0baf04ea4c2cn%40googlegroups.com.

[halol goog]

Thanks for the reply. I did more tests, but I still cannot figure out what's wrong.
So I prepared a new dummy data service (attached) to rule out an issue with my original one: I have the same issue. This new data service does nothing except print the record. It is based on the example provided in the documentation.

Here is what I get when I enable the data service with the following configuration in weex.conf:

[Engine]
  [[Services]]
    prep_services = weewx.engine.StdTimeSynch
    process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
    xtype_services = weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta
    archive_services = weewx.engine.StdArchive
    restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.mqtt.MQTT
    report_services = weewx.engine.StdPrint, weewx.engine.StdReport
    data_services = user.dummyDataService.PrintPacket

The output produced is:
2024-10-26 17:55:02 weewxd[1]: DEBUG user.dummyDataService: Dummy data service - printing archive record
2024-10-26 17:55:02 weewxd[1]: DEBUG user.dummyDataService: {'dateTime': 1729958100, 'usUnits': 1, 'interval': 5, 'rxCheckPercent': 99.9375, 'outTemp': 59.2, 'highOutTemp': 59.3, 'lowOutTemp': 59.2, 'rain': 0.0, 'rainRate': 0.0, 'barometer': 30.002, 'radiation': 6.0, 'wind_samples': 117.0, 'inTemp': 79.5, 'inHumidity': 54.0, 'outHumidity': 93.0, 'windSpeed': 1.0, 'windGust': 3.0, 'windGustDir': 270.0, 'windDir': 315.0, 'ET': 0.0, 'highRadiation': 9.0, 'forecastRule': 44}
2024-10-26 17:55:02 weewxd[1]: INFO weewx.manager: Added record 2024-10-26 17:55:00 CEST (1729958100) to database 'weewx'
2024-10-26 17:55:02 weewxd[1]: INFO weewx.manager: Added record 2024-10-26 17:55:00 CEST (1729958100) to daily summary in 'weewx'
REC: 2024-10-26 17:55:00 CEST (1729958100) 'altimeter': '1016.3862117251151', 'appTemp': '16.058246419654616', 'barometer': '1015.9841754081338', 'cloudbase': '397.1353776690074', 'dateTime': '1729958100', 'dewpoint': '13.986259038068189', 'ET': '0.0', 'forecastRule': '44', 'heatindex': '15.106111111111112', 'highOutTemp': '15.166666666666664', 'highRadiation': '9.0', 'humidex': '18.474908466486777', 'inDewpoint': '16.319400135449317', 'inHumidity': '54.0', 'inTemp': '26.38888888888889', 'interval': '5', 'lowOutTemp': '15.111111111111112', 'maxSolarRad': '15.956834013222045', 'outHumidity': '93.0', 'outTemp': '15.111111111111112', 'pressure': '985.3738011279798', 'radiation': '6.0', 'rain': '0.0', 'rainRate': '0.0', 'rxCheckPercent': '99.9375', 'usUnits': '17', 'wind_samples': '117.0', 'windchill': '15.111111111111112', 'windDir': '315.0', 'windGust': '1.34112', 'windGustDir': '270.0', 'windrun': '0.13411199999999998', 'windSpeed': '0.44704'
2024-10-26 17:55:02 weewxd[1]: INFO weewx.engine: Starting main packet loop.

Without the data service (I commented out the entire 'data_services' line in the configuration above):
REC: 2024-10-26 18:00:00 CEST (1729958400) 'altimeter': '1016.3868998672192', 'appTemp': '16.132600646010406', 'barometer': '1015.9841754081338', 'cloudbase': '397.1955242260177', 'consBatteryVoltage': '3.86', 'dateTime': '1729958400', 'dayET': '0.004', 'dayRain': '47.39999971086', 'dewpoint': '14.04132126433811', 'ET': '0.0', 'extraAlarm1': '0.0', 'extraAlarm2': '0.0', 'extraAlarm3': '0.0', 'extraAlarm4': '0.0', 'extraAlarm5': '0.0', 'extraAlarm6': '0.0', 'extraAlarm7': '0.0', 'extraAlarm8': '0.0', 'forecastIcon': '6.0', 'forecastRule': '44', 'heatindex': '15.167222222222223', 'highOutTemp': '15.166666666666664', 'highRadiation': '0.0', 'humidex': '18.562786577632735', 'inDewpoint': '16.319400135449317', 'inHumidity': '54.0', 'insideAlarm': '0.0', 'inTemp': '26.38888888888889', 'interval': '5', 'lowOutTemp': '15.111111111111112', 'maxSolarRad': '10.014834789005617', 'monthET': '1.11', 'monthRain': '330.19999798578', 'outHumidity': '93.0', 'outsideAlarm1': '0.0', 'outsideAlarm2': '0.0', 'outTemp': '15.166666666666664', 'pressure': '985.3744721896454', 'radiation': '0.0', 'rain': '0.0', 'rainAlarm': '0.0', 'rainRate': '0.0', 'rxCheckPercent': '99.9375', 'soilLeafAlarm1': '0.0', 'soilLeafAlarm2': '0.0', 'soilLeafAlarm3': '0.0', 'soilLeafAlarm4': '0.0', 'stormRain': '51.59999968523999', 'stormStart': '1729807200.0', 'sunrise': '1729922230.2439024', 'sunset': '1729960260.0', 'txBatteryStatus': '0', 'usUnits': '17', 'wind_samples': '117.0', 'windchill': '15.166666666666664', 'windDir': '270.0', 'windGust': '1.34112', 'windGustDir': '270.0', 'windrun': '0.13411199999999998', 'windSpeed': '0.44704', 'windSpeed10': '0.44704', 'yearET': '26.23', 'yearRain': '1369.999991643'

When the data service is not enable, the record contains a lot more information. From what I have seen, it appears that this information is added to what the Vantage is producing. It looks like enabling a data service causes this step to be skipped.

Any clues on how I can investigate this further?

Thanks!

[Tom Keffer]

Before starting, there's one thing to note: by default, the Vantage series emits its own archive records, but they don't contain everything that was in the LOOP packets. For example, a Vantage archive record does not contain "consBatteryVoltage", which is only in the LOOP packets. To include them, the WeeWX engine augments the record off the hardware device with everything that it can get out of the accumulators (which compile LOOP packets). This is done by service StdArchive.

The services listed under "data_services" actually run quite early: right after "prep_services", well before StdArchive. So if you put your dummy service under data_services, it won't see what will be added by StdArchive. Hence, it's missing things like consBatteryVoltage.

But, I can't explain why the augmentation does not happen when you include dummyDataService, but does when you omit it. I tried replicating on my own system, and it acted as expected. Here's what it looks like with dummyDataService:

Oct 26 12:15:15 NUC-black weewxd[84442]: DEBUG user.dummyDataService: {'dateTime': 1729970100, 'usUnits': 1, 'interval': 5, 'rxCheckPercent': 99.9375, 'outTemp': 54.8, 'highOutTemp': 55.0, 'lowOutTemp': 54.6, 'rain': 0.0, 'rainRate': 0.0, 'barometer': 30.086, 'radiation': 200.0, 'wind_samples': 117.0, 'inTemp': 68.7, 'inHumidity': 49.0, 'outHumidity': 66.0, 'windSpeed': 1.0, 'windGust': 3.0, 'windGustDir': 180.0, 'windDir': 180.0, 'UV': 1.3, 'ET': 0.0, 'highRadiation': 234.0, 'highUV': 1.3, 'forecastRule': 44}

and the corresponding REC:

REC:    2024-10-26 12:15:00 PDT (1729970100) 'altimeter': '30.08152038664648', 'appTemp': '52.77093282004806', 'barometer': '30.086', 'cloudbase': '3231.4521080878726', 'consBatteryVoltage': '4.7', 'dateTime': '1729970100', 'dayET': '0.015', 'dayRain': '0.0', 'dewpoint': '43.66161072441336', 'ET': '0.0', 'extraAlarm1': '0.0', 'extraAlarm2': '0.0', 'extraAlarm3': '0.0', 'extraAlarm4': '0.0', 'extraAlarm5': '0.0', 'extraAlarm6': '0.0', 'extraAlarm7': '0.0', 'extraAlarm8': '0.0', 'forecastIcon': '6.0', 'forecastRule': '44', 'heatindex': '53.081999999999994', 'highOutTemp': '55.0', 'highRadiation': '234.0', 'highUV': '1.3', 'humidex': '54.8', 'inDewpoint': '48.75937484652667', 'inHumidity': '49.0', 'insideAlarm': '0.0', 'inTemp': '68.7', 'interval': '5', 'lowOutTemp': '54.6', 'maxSolarRad': '479.7192813786106', 'monthET': '1.77', 'monthRain': '0.71', 'outHumidity': '66.0', 'outsideAlarm1': '0.0', 'outsideAlarm2': '0.0', 'outTemp': '54.8', 'pressure': '29.326441515975546', 'radiation': '200.0', 'rain': '0.0', 'rainAlarm': '0.0', 'rainRate': '0.0', 'rxCheckPercent': '99.9375', 'soilLeafAlarm1': '0.0', 'soilLeafAlarm2': '0.0', 'soilLeafAlarm3': '0.0', 'soilLeafAlarm4': '0.0', 'stormRain': '0.0', 'sunrise': '1729953540.0', 'sunset': '1729990800.0', 'txBatteryStatus': '0', 'usUnits': '1', 'UV': '1.3', 'wind_samples': '117.0', 'windchill': '54.8', 'windDir': '180.0', 'windGust': '3.0', 'windGustDir': '180.0', 'windrun': '0.08333333333333333', 'windSpeed': '1.0', 'windSpeed10': '1.0', 'yearET': '35.94', 'yearRain': '0.71'

As you can see, it contains consBatteryVoltage.

There's something else going on, but I'm not sure what it is. Are you working with a freshly installed version of WeeWX with no modifications? What about the custom data service that you mentioned in your first post? Are you using the copy of weewx.conf that you think you're using?

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/f62dcc08-6e96-45d2-96e4-6fde7526e300n%40googlegroups.com.

[halol goog]

Alright. I started again from scratch, with a brand new weex.conf. This time I managed to get this working correctly with the dummy data service (not sure what was wrong before though). But as soon as I configured the real data service, I again ran into the same problem.

After some investigation and testing, I came up with the following hypothesis to explain the problem, but since I don't know how the StdArchive is working, I am not sure if this really makes sense:

I have 2 stations configured: the Vantage one, which is my main station. I also have a WS3000 with 8 temperature sensors, and I am adding the data coming from the WS3000 to the records from the Vantage station.
This is all working fine (except for the augmentation done by StdArchive of course). But the sequence of event is the following:

1. The Vantage is producing LOOP packets as expected (containing consBatteryVoltage).
2. When a record is generated, my data service is invoked.
3. The data service pulls the information from the WS3000 station, and adds it to the record.
4. Then StdArchive if supposed to augment the record as usual.

The packet from the WS3000 station has a slightly different timestamp than the record from the Vantage. And when merging the data, I was simply looping all the keys from the WS3000 packet, and was overwritting the dateTime value from the Vantage.
If I skip the dateTime key when doing the merge, everything is then working fine and the StdArchive is able to augment the record with the consBatteryVoltage and all the other values.
So, it looks like the dateTime value from the record is critical for the augmentation to work.

In any case, thanks a lot for your help in resolving this!

[Karen K]

halol goog schrieb am Sonntag, 27. Oktober 2024 um 13:45:37 UTC+1:

So, it looks like the dateTime value from the record is critical for the augmentation to work.

Yes, it is. LOOP packets are accumulated in the accumulator. You can configure the accumulator in the Accumulators section of weewx.conf. At the end of the archive interval readings are extracted out of the accumulators, and they are marked with the timestamp of that end of archive interval time. So, if the timestamp of your record (in dateTime) does not correspond the the timestamp of the readings out of the accumulators, they cannot be merged.

If you have different timestamps I would suggest to save those readings to different files. You can easily have more than one database file by setting up bindings in weewx.conf. You could have your "dummy data service" save its readings directly to that separate file. There is no problem for the report generating modules to deal with files of different timestamps.

[Graham Eddy]

don’t change dateTime from a service - it is created by the driver with a purpose

⊣GE⊢

[halol goog]

OK, thanks for the explanation.

On Sunday, October 27, 2024 at 2:23:14 PM UTC+1 Karen K wrote:

If you have different timestamps I would suggest to save those readings to different files. You can easily have more than one database file by setting up bindings in weewx.conf. You could have your "dummy data service" save its readings directly to that separate file. There is no problem for the report generating modules to deal with files of different timestamps.

 
I don't need the timestamps coming from the WS3000. My goal was just to augment the Vantage record with data coming from another source, at the time the Vantage record is generated. It just turned out that I did merged the data using a simple loop:
for key in ws3000packet.keys():
  event.record[key] = ws3000packet[key]
Since the ws3000packet contained a dateTime slightly different from the Vantage one, this resulted in this issue. Lesson learned. Thanks again.

[Karen K]

halol goog schrieb am Montag, 28. Oktober 2024 um 09:15:44 UTC+1:

I don't need the timestamps coming from the WS3000. My goal was just to augment the Vantage record with data coming from another source, at the time the Vantage record is generated. It just turned out that I did merged the data using a simple loop:

for key in ws3000packet.keys():
  event.record[key] = ws3000packet[key]
Since the ws3000packet contained a dateTime slightly different from the Vantage one, this resulted in this issue. Lesson learned. Thanks again.

I would suggest:

del ws3000packet['dateTime']

del ws3000packet['interval']

del ws3000packet['usUnits']

event.record.update(ws3000packet)