GW1000 Gateway - sensor battery stats not entering onto DB

[Jon Fear]

Hi

I am currently working with Dave of the weewx-wdc skin fame create a sensor stats page for the skin.

Dave has a copy of my db to play with however he has noticed that the sensor values are not being recorded frequently. He notes that the values were recorded for a few days in January and then not until the end of March, then again only for a few days.

I do not think I have changed very much at either time, perhaps a restart here and there and obviously updated weewx itself but..

Any thoughts?

Thanks

Jon

[vince]

One thought would be to read and follow https://github.com/weewx/weewx/wiki/Help!-Posting-to-weewx-user  and https://github.com/weewx/weewx/wiki/faq-how-to-report-a-problem

[gjr80]

Further to what Vince said, if the problem is data not being saved to database then it will be worth while running WeeWX directly for at least two archive intervals to see what data appears in loop packets and archive records. This will see loop packet and archive record data sent to the console, take a (text) copy of this output and post along with your log extract/wee_debug output.

Gary

[Jon Fear]

Hi Gary

Sorry for the delay, other domestic "priorities" got in the way!

Attached are the files which I hope will help. Dave has now released the v3.2.0 of the weewx-wdc skin which has the sensors page. Currently it is not displaying any data. I suspect this is part of the bigger issue.

Please let me know if you need more information. FYI wslwx.co.uk is my site address.

Best wishes

Jon

[gjr80]

Jon,

No problems. The console output shows the Ecowitt gateway driver is populating loop packets with sensor status data, eg ws90_sig, wh57_batt etc. I assume the sensor data is complete for all sensors registered with your gateway device (I cannot tell this, only you can). WeeWX is then accumulating this data and it appears in WeeWX generated archive records.

Looking at the weewx-wdc GitHub page it appears to use a similar system to the Seasons skins for displaying sensor status data. The default fields to be displayed are set in sensor-status.html.tmpl and these can be overridden in skin.conf. The default appears to include fields such as rxCheckPercent, consBatteryVoltage, inTempBatteryStatus, rainBatteryStatus etc. None of the Ecowitt sensor state fields are included so I expect you need to override/augment the sensor state fields to be displayed in skin.conf. How you do this (ie which settings to adjust) is best discussed with the skin author.

On thing you will need to consider is whether or not to save the sensor state data to archive. It is not necessary to do so if only displaying current values (ie values from the current archive record), but if you wish to display any aggregates based on historical data you will need to do so. Fortunately it is easy to do, just modify your database schema using wee_database to include the desired fields and WeeWX will take care of the rest. Refer to Modifying an existing database in the Customization Guide.

Gary

[David Bätge]

Hi Gary and Jon,

I just got some time to look into this (and also made some notes for myself):

First of all, the GW1000 driver provides a lot of sensor information, the complete list is:

Batteries:
wh31_ch1_batt
wh31_ch2_batt
wh32_batt
wh40_batt
wh41_ch1_batt
wh51_ch1_batt
wh57_batt
ws80_batt
ws90_batt

Signals:
wh31_ch1_sig
wh31_ch2_sig
wh32_sig
wh40_sig
wh41_ch1_sig
wh51_ch1_sig
wh57_sig
ws80_sig
ws90_sig


Here is the complete LOOP package with relevant values:
{
   'wh31_ch1_batt': '0',
   'wh31_ch1_sig': '4',
   'wh31_ch2_batt': '0',
   'wh31_ch2_sig': '4',
   'wh32_batt': '0',
   'wh32_sig': '4',
   'wh40_batt': '1.39',
   'wh40_sig': '4',
   'wh41_ch1_batt': '5',
   'wh41_ch1_sig': '4',
   'wh51_ch1_batt': '1.3',
   'wh51_ch1_sig': '4',
   'wh57_batt': '4',
   'wh57_sig': '4',
   'ws80_batt': 'None',
   'ws80_sig': '0',
   'ws90_batt': '2.86',
   'ws90_sig': '4',
}

To be honest, I don't quite understand all of the fields, but I will try to give an explanation and how to configure these fields in weewx-wdc:

Observations that show something like "Battery OK" or "Battery LOW" (where 0 is OK and 1 is LOW): `wh31_ch1_batt, wh31_ch2_batt, wh32_batt`, and `wh41_ch1_batt, wh57_batt` (see https://github.com/gjr80/weewx-gw1000/wiki/Sensor-battery-states). This kind of observation needs to be added to [DisplayOptions]sensor_battery_status in skin.conf.

Note to me: wh41_ch1_batt, wh57_batt using another representation of Battery status, see https://github.com/gjr80/weewx-gw1000/blob/master/supplementary/skins/Seasons/sensors.inc and https://github.com/gjr80/weewx-gw1000/wiki/Adapting-the-Seasons-skin-to-display-battery-states (There are a few tweaks needed to support these fields)

Signal level observations: All *_sig observations like, wh31_ch1_sig, wh31_ch2_sig, wh32_sig. I don't understand these: What is the signal level here? Values seem to be `0` or `4` in that LOOP packet (looks like battery state fields?).

Battery Voltages observations: wh40_batt, wh51_ch1_batt, ws80_batt, ws90_batt. These should be added to any of [DisplayOptions]sensor_stat_tile_observations, [DisplayOptions]sensor_diagram_observations or [DisplayOptions]sensor_table_observations in skin.conf.

So the problem with all these observations is basically (as far as I understand, please correct me if I am wrong): WeeWX does not know about them and does not save them to the database (but you will need this data to show plots of the battery voltages for the last week for example). There should be two ways to overcome this:

1. Add the fields provided by the GW1000 driver to the database, see http://weewx.com/docs/latest/customizing.htm#Modifying_an_existing_database
2. Map the GW1000 fields to WeeWX schema fields (see https://github.com/weewx/weewx/blob/master/bin/schemas/wview_extended.py#L16) I assume that was somehow done for a short period of time because e.g. rxCheckPercent or consBaterryVoltage are showing/having data for a few days!?)

@Gary I don't quite understand what you have written about that:

It is not necessary to do so if only displaying current values (ie values from the current archive record)

How would you show fields like wh41_ch1_batt from the current archive record when the data is not saved? Am I missing something? 

[Rainer Lang]

Hi David,

the sentence "It is not necessary to do so if only displaying current values (ie values from the current archive record)"  is maybe a bit trickily phrased - at the time the report is generated weewx includes values from the database and the current values of all observations in the current values table, also from those which have not been archived but are still available in memory from what has been provided from the driver (here Ecowitt Gateway). That's what you address by $current.
They will be discarded/replaced by new values coming in - and won't be accessible later unless also archived by one of the methods described (new columns/fields or repurposing of existing schema fields) while the archived values are of course still accessible in the database.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c2ce950d-c9f5-4b3c-8349-f13045679c9fn%40googlegroups.com.

[gjr80]

David,

Some answers/comments below.

On Tuesday, 27 June 2023 at 01:16:51 UTC+1 david....@gmail.com wrote:

Signal level observations: All *_sig observations like, wh31_ch1_sig, wh31_ch2_sig, wh32_sig. I don't understand these: What is the signal level here? Values seem to be `0` or `4` in that LOOP packet (looks like battery state fields?).

 

Ecowitt gateway signal levels are a pseudo-signal levels at best; the number represents the number of the last four data packets from the sensor that were successfully received/decoded. So most of the time, if the sensors have a reliable connection to the gateway device, you will see a 4, on startup you will see 0 and it will ramp up to 4 if all sensors connect successfully and reliably. The number bears no reference to the sensor signal level received by the gateway device or emitted by the sensor.

 

So the problem with all these observations is basically (as far as I understand, please correct me if I am wrong): WeeWX does not know about them and does not save them to the database (but you will need this data to show plots of the battery voltages for the last week for example). There should be two ways to overcome this:

1. Add the fields provided by the GW1000 driver to the database, see http://weewx.com/docs/latest/customizing.htm#Modifying_an_existing_database
2. Map the GW1000 fields to WeeWX schema fields (see https://github.com/weewx/weewx/blob/master/bin/schemas/wview_extended.py#L16) I assume that was somehow done for a short period of time because e.g. rxCheckPercent or consBaterryVoltage are showing/having data for a few days!?)

@Gary I don't quite understand what you have written about that:

It is not necessary to do so if only displaying current values (ie values from the current archive record)

 

A quick and simplified WeeWX data flow 101. A WeeWX driver gathers data from a station/sensors and emits loop packets on a regular and frequent basis. WeeWX accumulates these loop packets and at the end of an archive period WeeWX synthesises an archive record from the accumulated loop packets, this record is known as the current archive record. WeeWX saves the current archive record to database, but only those fields that are common to the WeeWX database schema and the current archive record. WeeWX then generates reports. Note, there are usually a number of additional WeeWX services that 'do things' (eg quality control, calculate derived values etc) to the loop packets and archive records before they are accumulated/saved to database.

When WeeWX generates reports all fields in the current archive record are available for use in WeeWX generated reports via the $current tag, irrespective of whether the field was saved to database or not. So if you wish to include only the current value of a field in a WeeWX report there is no need to save that field to the database, but if you wish to display any sort of historical data (eg an aggregate in a report or plot historical values) for a field you need to save the field to database. For example, generally folks want to display max/min values of outside temperature or plot outside temperature over some period so outTemp is usually saved to database. However, you may only wish to display the current signal level value for a given sensor using a $current tag so you may chose not to save the sensor signal level to database.

So in terms of WeeWX 'knowing about a field' there are two things to consider. (1) does the WeeWX schema need to include this field, if it does this can be achieved by mapping to an already existing database field or adding a new field to the database schema (or a combination of both) and (2) do any enabled reports need to know about the field, this may entail a simple config change in a skin config file or modification/creation of a template.

 

How would you show fields like wh41_ch1_batt from the current archive record when the data is not saved? Am I missing something?

 

As outlined above, in such a case you can display the current wh41_ch1_batt value but will not be able to display any historical values or aggregates nor will you be able to plot wh41_ch1_batt. If you wish to display any historical values or aggregates or plot wh41_ch1_batt you need to add the field wh41_ch1_batt to the database or map field wh41_ch1_batt to an existing, unused database field.  

 

[David Bätge]

Hi Rainer and Gary,

thank you both very much for the detailed explanation! I was not aware that the $current tag also provides access to ALL observations of the current record, regardless if they are archived or not. 

Is the $current tag the only tag that provides this access? I assume the $latest tag also does? 

The $span tag would not be able to get values from those fields, since it's an aggregate, right? Just asking because I have added a check for displaying the battery states, like eg. #if $getattr($span($day_delta=1, boundary='midnight'), 'wh31_ch1_batt').has_data which does not really make sense after understanding the internals, it should test against $current.

[...] weewx includes values from the database and the current values of all observations in the current values table, also from those which have not been archived but are still available in memory from what has been provided from the driver [...]

Where is this data actually stored if not in the database? Is there any way to test this out, besides actually using the said driver?

Sorry for a load of questions, please point me to the right documentation if I missed something in the guides. Thanks in advance and thanks again for enlighten me about this :) 

[gjr80]

Comments below.

Gary

On Wednesday, 28 June 2023 at 00:19:33 UTC+1 david....@gmail.com wrote:

Is the $current tag the only tag that provides this access? I assume the $latest tag also does? 

 

Under most circumstances $latest will return fields from the current archive record irrespective of whether they were saved to database or not. There are a few exceptions, eg if a binding is specified by the $latest tag. Off the top of my head I cannot think of any other tags that make non-saved fields available.

 

The $span tag would not be able to get values from those fields, since it's an aggregate, right?

 

Correct.

 

Just asking because I have added a check for displaying the battery states, like eg. #if $getattr($span($day_delta=1, boundary='midnight'), 'wh31_ch1_batt').has_data which does not really make sense after understanding the internals, it should test against $current.

 

Depends what you are testing and for what purpose, but your '$span' test it will require wh31_ch1_batt to be saved to database, testing against $current will not have such a requirement. If it is a simple test to determine whether you have battery state data to display then I would expect testing against $current would be appropriate.

[...] weewx includes values from the database and the current values of all observations in the current values table, also from those which have not been archived but are still available in memory from what has been provided from the driver [...]

Where is this data actually stored if not in the database? Is there any way to test this out, besides actually using the said driver?

 

WeeWX keeps the current loop packet and current archive record in two python dictionaries. WeeWX services have access to and can modify these dictionaries via a property of the event object that is passed to each WeeWX service. In terms of WeeWX generated reports, access to the current archive record is via the $current tag. There is no access to the current loop packet. 

Not sure what you mean by 'testing this out', the dictionaries are there and used routinely throughout the WeeWX code base.

 

[Rainer Lang]

"Where is this data actually stored if not in the database? Is there any way to test this out, besides actually using the said driver?"

The data is not stored on a medium (e.g. disk) but just kept in memory during the runtime of the program - and even there
only as long as the program logic allows.

Maybe the use of other terms will make it clearer if you are not familiar with the Python programming language and its jargon.
A dictionary in Python is a defined (potentially complex) data structure of the same or different data types.

These structures have different names in different programming languages. They exist during the runtime of the program and are used
to store data during runtime and also use this data during runtime. In older programming languages like COBOL they were just a so-called
record, a structured description of a memory area  (like a line in a spreadsheet with different columns).

On a low level they are (from the point of view of the computer program structured)
areas in memory which can be accessed and manipulated by the program code.

If you want to do testing here, whatever are your ideas here and whatever you want to do and for what purpose, you would have to create your
own Python code which does what you want to do (your "testing").
If you want to see the definition of these structures (Python dictionaries),
you will have to find the place where they are defined in the weewx coding.

As Python is an interpreter language, the program code is available on your computer.

I'm not exactly clear on what you want to "test" here and what for ...

- certain observations are stored or not stored, depending on what you want and define (e.g. in weewx.conf)
you want some observation to be stored, do it via the one of the two approaches mentioned earlier,
some are predefined by the database schema used, for others this would need to be defined
- you can also define what type of accumulation of this data you want (also in weewx.conf)
- when displaying the data you can also define (in skin.conf) what type of depiction your want

so much for the technical aspect - storing or not storing (storing here in the sense of archiving into a database)

as for the functional/logical aspect, you have to be clear on what the information you want to archive means
like in the example of the Ecowitt sensor "signal" observation which is an indicator of successfully received and processed data packages
i.e. rather a signal quality indicator but not a signal strength indicator. Even though on an Ecowitt console or app you can see "reception bars",
they cannot be compared to the signal reception bars of your mobile phone (where they are an indicator of signal strength).

If you want to enter more deeply into how Ecowitt consoles and sensors work, I recommend to read the (whole !) wxforum.net thread
https://www.wxforum.net/index.php?topic=40730.0

In the field of battery and signal values there is unfortunately (historically grown) not a consistent logic, especially with the batteries.
You have to know which values each sensor provides and what they mean

weewx is only using the output from these consoles through whichever driver you are using to do so (Ecowitt Gateway, Interceptor, RTL433).
It's up to you to make sense of it and what you choose to depict and how to depict.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/abf92ed8-282f-45f3-b367-c9ee084eee74n%40googlegroups.com.

[David Bätge]

Thanks again to both, this got me an idea about how things are working!

I was asking about testing because my Station/Driver does not provide any "extra-fields" which are not saved, just fields included in the schema. I should have said that I am the Dave from the weewx-wdc Skin, whom John mentioned earlier, so I wanted to test this out when implementing it into the skin.

 I had a quick look at the Simulator driver provided by WeeWX. Looks like it could be easily modified to emit some extra fields for testing.