GW1000 output and database

[Timothy Buchanan]

I installed GW1000 as a service and the test-service command gives this output:

dateTime: 1597861672, dummyTemp: 96.3, extraHumid1: 26, extraTemp1: 85.1, inHumidity: 33, inTemp: 75.74, pressure: 22.2360805875, relbarometer: 753.0, soilMoist1: 15, usUnits: 1, wh31_ch1_batt: 0, wh51_ch1_batt: 0

In weewx.conf, I put this list of options:

[GW1000]

    driver = user.gw1000

inTemp = extraTemp2

inHumidity = extraHumid2

extraTemp1 = extraTemp1

extraHumid1 = extraHumid1

soilMoist1 = soilMoist1

pressure = ""

I chose these because inTemp is being used by another service. When I tried to use the data in my skin (modified exfoliation), I found that extraTemp2 and extraHumid2 are shown as NA. The other values are displayed. What did I do wrong here? Thanks for any help.

[gjr80]

Hi,

I think we might need a bit of clarification as to what it is you are trying to do. You say "inTemp is being used by another service", do you mean that WeeWX field inTemp is being provided by another service/driver (which has consequences for running the  GW1000 driver as as service) or do you mean WeeWX field inTemp is being used by another service (which has no consequences for the GW1000 driver when run as a service). Does the same apply for inHumidity? When you ran the GW1000 driver directly with --test-service there was no extraTemp2 or extraHumid2 fields and your field map does not map anything to these fields so I would expect extraTemp2 and extraHumid2 to show as N/A in any reports.

Perhaps you have a misunderstanding re the field map. The field map maps GW000 'fields' to WeeWX fields. The GW1000 driver uses a default field map. The default field map can be altered in two ways; you can replace the entire default field map by defining your own field map using a [[field_map]] stanza, or you can alter the default field map by listing changes to the default field map in a [[field_map_extensions]] stanza. It is not clear if you were attempting one of these approaches as you have no [[ ]] level stanza in the config extra you posted. Also when defining a field map entry the format is:

    WeeWX field name = GW1000 field name

    where:

        GW1000 field name is the name of the GW1000 'field' that you wish to map

        WeeWX field name is the WeeWX field name you wish to contain the data from the GW1000 'field' concerned. The WeeWX field name is the the field name you will use in reports etc

You can see what fields your GW1000 emits by running the GW1000 driver directly with the --live-data command line option (the fields shown when running the driver directly using the --test-service or --test-driver command line options are WeeWX field names, ie they have been mapped using the field map, fields shown using --live-data are GW1000 'fields' ie they are unmapped).

Gary

[Timothy Buchanan]

Thanks for the detailed reply; I'll try to be more clear. I have two services running, as indicated by this line:

data_services = user.bme280wx.Bme280wx, user.gw1000.Gw1000Service

The first service is an i2c sensor providing temp/humidity at the raspberry pi. It's options are so:

[Bme280wx]

    temperature_must_have = ""

    humidityKeys = inHumidity

    pressureKeys = ""

    pressure_must_have = ""

    i2c_port = 1

    humidity_must_have = ""

    i2c_address = 0x76

    usUnits = US

    temperatureKeys = inTemp

The sensor data are humidityKeys and temperatureKeys and they are mapped to inHumidity and inTemp. This service runs and reports properly; where my skin uses inTemp the data from the sensor is displayed.

The GW1000 options are currently set so, following the instructions in your reply:

[GW1000]

    driver = user.gw1000

[[field map]]

extraTemp2  = inTemp

extraHumid2  = inHumidity

extraTemp1 = extraTemp1

extraHumid1 = extraHumid1

soilMoist1 = soilMoist1

Three of the fields provided by the GW1000 are the same in the database (the latter three), and they are reporting correctly. I want the GW1000 fields inTemp and inHumidity to map to the weewx database fields extraTemp2 and extraHumid2. That is not working with the field map as shown. Have I missed something?

[gjr80]

Thanks, now I understand. You have the right approach but unfortunately the execution is not quite correct. Did you run the driver directly with the —live-data command line option? The output will detail all of the available GW1000 ‘fields’ and their current values. Those field names are what should be used on the right hand side of the = in the field map entries. You will notice the GW1000 ‘fields’ are all in lowercase. So you should be using:

    extraTemp2 = intemp

Likewise extraHumid2 and soilMoist1. Your extraTemp1 mapping should be:

    extraTemp1 = temp1

Likewise extraHumid1.

Gary

[Timothy Buchanan]

OK, I placed this at the end of weewx.conf:

# Options for extension 'GW1000'

[GW1000]

    driver = user.gw1000

[[field map]]

extraTemp2  = intemp

extraHumid2  = inhumid

extraTemp1 = temp1

extraHumid1 = humid1

soilMoist1 = soilmoist1

Result is the same as before; the first two lines produce null values in the database, and the latter three appear in the correct fields. In fact, it doesn't seem to matter what I put in this file. For example:

# Options for extension 'GW1000'

[GW1000]

    driver = user.gw1000

[[field map]]

extraTemp2  = intemp

extraHumid2  = inhumid

extraTemp1 = foobar1

extraHumid1 = foobar2

soilMoist1 = foobar3

produces the same results! The first two lines give nulls; foobar lines work. Is this the correct format and location for a field map?

[gjr80]

try [[field_map]] instead of [[field map]]. If that does not work edit weewx.conf, set debug = 1, save weewx.conf and then restart WeeWX. Let WeeWX run for five minutes or so then post a log extract from when you restarted WeeWX through until the five minutes elapsed. post the log extract here.

Gary

[Timothy Buchanan]

[[field_map]] makes all values null, including the ones that worked before. Syslog follows

[gjr80]

The highlighted line tells me the default field map is being used and not [[field_map]]:

Aug 20 20:01:29 raspberrypi weewx[31556] DEBUG weewx.engine: Loading service user.gw1000.Gw1000Service
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: GW1000 was found at 192.168.1.6:45000
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: field map is {'UV': 'uvi', 'dateTime': 'datetime', 'dayRain': 'rainday', 'daymaxwind': 'daymaxwind', 'dewpoint': 'dewpoint', 'extraHumid1': 'humid1', 'extraHumid2': 'humid2', 'extraHumid3': 'humid3', 'extraHumid4': 'humid4', 'extraHumid5': 'humid5', 'extraHumid6': 'humid6', 'extraHumid7': 'humid7', 'extraHumid8': 'humid8', 'extraTemp1': 'temp1', 'extraTemp2': 'temp2', 'extraTemp3': 'temp3', 'extraTemp4': 'temp4', 'extraTemp5': 'temp5', 'extraTemp6': 'temp6', 'extraTemp7': 'temp7', 'extraTemp8': 'temp8', 'heatindex': 'heatindex', 'hourRain': 'rainhour', 'inHumidity': 'inhumid', 'inTemp': 'intemp', 'leak1': 'leak1', 'leak2': 'leak2', 'leak3': 'leak3', 'leak4': 'leak4', 'lightning_distance': 'lightningdist', 'lightning_last_det_time': 'lightningdettime', 'lightning_strike_count': 'lightning_strike_count', 'luminosity': 'light', 'monthRain': 'rainmonth', 'outHumidity': 'outhumid', 'outTemp': 'outtemp', 'pm2_5': 'pm251', 'pm2_51_24hav': 'pm251_24hav', 'pm2_52': 'pm252', 'pm2_52_24hav': 'pm252_24hav', 'pm2_53': 'pm253', 'pm2_53_24hav': 'pm253_24hav', 'pm2_54': 'pm254', 'pm2_54_24hav': 'pm254_24hav', 'pressure': 'absbarometer', 'rain': 'rain', 'rainRate': 'rainrate', 'relbarometer': 'relbarometer', 'soilMoist1': 'soilmoist1', 'soilMoist2': 'soilmoist2', 'soilMoist3': 'soilmoist3', 'soilMoist4': 'soilmoist4', 'soilMoist5': 'soilmoist5', 'soilMoist6': 'soilmoist6', 'soilMoist7': 'soilmoist7', 'soilMoist8': 'soilmoist8', 'soilMoist9': 'soilmoist9', 'soilMoist10': 'soilmoist10', 'soilMoist11': 'soilmoist11', 'soilMoist12': 'soilmoist12', 'soilMoist13': 'soilmoist13', 'soilMoist14': 'soilmoist14', 'soilMoist15': 'soilmoist15', 'soilMoist16': 'soilmoist16', 'soilTemp1': 'soiltemp1', 'soilTemp2': 'soiltemp2', 'soilTemp3': 'soiltemp3', 'soilTemp4': 'soiltemp4', 'soilTemp5': 'soiltemp5', 'soilTemp6': 'soiltemp6', 'soilTemp7': 'soiltemp7', 'soilTemp8': 'soiltemp8', 'soilTemp9': 'soiltemp9', 'soilTemp10': 'soiltemp10', 'soilTemp11': 'soiltemp11', 'soilTemp12': 'soiltemp12', 'soilTemp13': 'soiltemp13', 'soilTemp14': 'soiltemp14', 'soilTemp15': 'soiltemp15', 'soilTemp16': 'soiltemp16', 'stormRain': 'rainevent', 'totalRain': 'raintotals', 'uvradiation': 'uv', 'weekRain': 'rainweek', 'wh25_batt': 'wh25_batt', 'wh26_batt': 'wh26_batt', 'wh31_ch1_batt': 'wh31_ch1_batt', 'wh31_ch2_batt': 'wh31_ch2_batt', 'wh31_ch3_batt': 'wh31_ch3_batt', 'wh31_ch4_batt': 'wh31_ch4_batt', 'wh31_ch5_batt': 'wh31_ch5_batt', 'wh31_ch6_batt': 'wh31_ch6_batt', 'wh31_ch7_batt': 'wh31_ch7_batt', 'wh31_ch8_batt': 'wh31_ch8_batt', 'wh40_batt': 'wh40_batt', 'wh41_ch1_batt': 'wh41_ch1_batt', 'wh41_ch2_batt': 'wh41_ch2_batt', 'wh41_ch3_batt': 'wh41_ch3_batt', 'wh41_ch4_batt': 'wh41_ch4_batt', 'wh51_ch1_batt': 'wh51_ch1_batt', 'wh51_ch2_batt': 'wh51_ch2_batt', 'wh51_ch3_batt': 'wh51_ch3_batt', 'wh51_ch4_batt': 'wh51_ch4_batt', 'wh51_ch5_batt': 'wh51_ch5_batt', 'wh51_ch6_batt': 'wh51_ch6_batt', 'wh51_ch7_batt': 'wh51_ch7_batt', 'wh51_ch8_batt': 'wh51_ch8_batt', 'wh51_ch9_batt': 'wh51_ch9_batt', 'wh51_ch10_batt': 'wh51_ch10_batt', 'wh51_ch11_batt': 'wh51_ch11_batt', 'wh51_ch12_batt': 'wh51_ch12_batt', 'wh51_ch13_batt': 'wh51_ch13_batt', 'wh51_ch14_batt': 'wh51_ch14_batt', 'wh51_ch15_batt': 'wh51_ch15_batt', 'wh51_ch16_batt': 'wh51_ch16_batt', 'wh55_ch1_batt': 'wh55_ch1_batt', 'wh55_ch2_batt': 'wh55_ch2_batt', 'wh55_ch3_batt': 'wh55_ch3_batt', 'wh55_ch4_batt': 'wh55_ch4_batt', 'wh57_batt': 'wh57_batt', 'wh65_batt': 'wh65_batt', 'wh68_batt': 'wh68_batt', 'windDir': 'winddir', 'windGust': 'gustspeed', 'windSpeed': 'windspeed', 'windchill': 'windchill', 'ws80_batt': 'ws80_batt', 'yearRain': 'rainyear'}
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: version is 0.1.0b11
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: GW1000 IP address not specified, attempting to discover GW1000...
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: GW1000 address is 192.168.1.6:45000
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: poll interval is 60 seconds
Aug 20 20:01:32 raspberrypi weewx[31556] DEBUG gw1000: user.gw1000: max tries is 3, retry wait time is 10 seconds
Aug 20 20:01:32 raspberrypi weewx[31556] DEBUG gw1000: user.gw1000: broadcast address 255.255.255.255:46000, socket timeout is 2 seconds
Aug 20 20:01:32 raspberrypi weewx[31556] INFO gw1000: user.gw1000: max age of API data to be used is 60 seconds
Aug 20 20:01:32 raspberrypi weewx[31556] DEBUG weewx.engine: Finished loading service user.gw1000.Gw1000Service

What was your [GW1000] stanza when this was log was taken, if you did not have [[field_map]] then we need to see the log extract with [[field_map]].

This also tells me that whilst the GW1000 was found twice at startup contact was subsequently lost with the GW1000 and it looks like contact was not re-established:

Aug 20 20:02:58 raspberrypi weewx[31556] DEBUG gw1000: user.gw1000: Failed attempt 3 to send command 'CMD_GW1000_LIVEDATA': timed out
Aug 20 20:02:58 raspberrypi weewx[31556] ERROR gw1000: user.gw1000: Failed to send command 'CMD_GW1000_LIVEDATA' after 3 attempts
Aug 20 20:02:58 raspberrypi weewx[31556] INFO gw1000: user.gw1000: Attempting to re-discover GW1000...

Would certainly explain the lack of GW1000 data. Have you noticed these type of entries before? Is your GW1000 being allocated a fixed/static IP?

Gary

[Timothy Buchanan]

Yes, the GW1000 disconnected and I moved it close to the router for testing, where I can see it. the router reserves an ip for it. I decided to start over. I uninstalled the extension, went back to an earlier weewx.conf then reinstalled the extension. I modified weewx.conf as shown in the github page, and placed this in weewx.conf:

# Options for extension 'GW1000'

[GW1000]

    driver = user.gw1000

[[field map]]

extraTemp2  = intemp

extraHumid2  = inhumid

extraTemp1 = temp1

extraHumid1 = humid1

soilMoist1 = soilmoist1

I've attached a syslog extract. I tried using [[field_map]], and that made all values null. Using [[field map]] gives the last three values. Maybe you could post a working GW Options field, and I will paste that into my weewx.conf and see what happens.

[gjr80]

Sorry if you think I am harping on the [[field_map]] issue but I need to understand if we are trying to track down a field map issue or something else. Is the log extract you just posted with [[field map]] or [[field_map]]? because again it shows the setting was ignored.

The reason that [[field map]] partially works is that it is completely ignored and the default map used, this results in GW1000 fields intemp and in humid being discarded because their mapped locations (WeeWX fields inTemp and inHumidity) are already populated. The GW1000 driver is coded to use two field map type stanzas; [[field_map]] and [[field_map_extensions]]. As far as the field map is concerned anything else is ignored.

Gary

[Timothy Buchanan]

I think you're being helpful, not harping, and I appreciate it. I just restarted weewx with [[field_map]] and found the following line in syslog:

Aug 21 19:51:10 raspberrypi weewx[22309] INFO gw1000: user.gw1000: field map is {'extraHumid1': 'humid1', 'extraHumid2': 'inhumid',

 'extraTemp1': 'temp1', 'extraTemp2': 'intemp', 'soilMoist1': 'soilmoist1'}

so yes, it is mapping correctly but something is preventing these values from being placed in the database, where they are all null. Incidentally, if I use [[field_map_extension]] it is ignored and I get the latter three values as before. So what could be keeping the mapped GW1000 service from the database?

[Graham Eddy]

when you run weewx in foreground (not as daemon) so you can see the weewx records, looking at the REC records (not LOOP records), what are the fields being offered to archive service? then looking in weewx db, which of these fields are actally being populated in db?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/890276d2-116b-43f8-b51c-57bec5a1166dn%40googlegroups.com.

[gjr80]

Actually, now that we have the GW1000 field mapping set I suspect the answer may be in the loop packets. Tim has three sources contributing to loop packets, and on top of that I believe the Weatherflow driver emits partial packets which further complicates the process. that being said it should be possible to have all three work co-operatively, perhaps with some judicious config settings. As Graham said you need to run WeeWX directly, once it is running directly let it run for at least a full archive interval and then post the entire console output making sure you include at least one REC: line.

Gary

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Timothy Buchanan]

Here is the output from one archive period; the REC loop is near the end.

Timothy

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[gjr80]

The GW1000 data is being picked up every 60 seconds as it should but it is sans extraHumid2/extraTemp2. Time to look a little closer at what the GW1000 is picking up. Can you run the GW1000 driver directly with the --live-data command line option. For a setup.py install something like:

$ PYTHONPATH=/home/weewx/bin python -m user.gw1000 --live-data

or for a package install:

$ PYTHONPATH=/usr/share/weewx python -m user.gw1000 --live-data

This will show us what data is available for use from the GW1000 and its connected sensors.

Gary

[Timothy Buchanan]

Here is live data:

GW1000 live sensor data: absbarometer: 753.4, datetime: 1598189179, humid1: 42, inhumid: 30, intemp: 21.7,

 relbarometer: 753.4, soilmoist1: 26, temp1: 12.8, wh31_ch1_batt: 0, wh51_ch1_batt: 0

This morning, I deleted the other service and restarted weewx. GW1000 not reports intemp and inhumid, but mapped to inTemp and inHumidity. Here is the field_map:

[[field map]]

        extraTemp2 = intemp

        extraHumid2 = inhumid

        extraTemp1 = temp1

        extraHumid1 = humid1

        soilMoist1 = soilmoist1

GW1000 intemp is mapped to extraTemp2, but is reported in the database as inTemp, as if it is hard coded to do so. 

Timothy

[Graham Eddy]

this is still incorrect. it should be [[field_map]] ← note the underscore

[Timothy Buchanan]

With bme280wx uninstalled and GW1000 set to [[field_map]], I get all nulls in the database.

With bme280wx uninstalled and GW1000 set to [[field map]], I get all GW values, but intemp and inhumid are sent to inTemp and inHumidity. 

I have switched back and forth repeatedly and this is what I get.

With bme280wx installed, I get values from the bme280, but not from GW1000, no matter what settings I use. So it seems that bme280wx and GW1000 are incompatible. But maybe someone who is running a bme280 and GW1000 successfully will chime in here.

Timothy

[gjr80]

Please don’t use [[field map]], it is ignored by the GW1000 driver/service and will result in the default field map being used which will map intemp to inTemp; it definitely will not map intemp to extraTemp2. Plus we are left trying to work out if you made a typo or not.

So we know the field map is being read and used (when [[field_map]] is used) and the GW1000 is emitting intemp and inhumidity. Now we need to look closer at how the data is processed through the GW1000 service. Can you please edit weewx.conf, set debug = 2 and make sure the [GW1000] stanza has [[field_map]] and that it is set to:

[[field_map]]

        extraTemp2 = intemp

        extraHumid2 = inhumid

        extraTemp1 = temp1

        extraHumid1 = humid1

        soilMoist1 = soilmoist1

Then save weewx.conf and restart WeeWX. Let WeeWX run for 10 minutes then take a log extract from when WeeWX was restarted through until 10 minutes elapsed and post the extract here. 

Can you also run wee_debug and post the wee_debug output, do check the wee_debug out for sensitive info such as usernames, passwords and api keys etc, wee_debug should obfuscate such entries but it is not perfect.

Gary

[Timothy Buchanan]

debug=2 GW options so (pasted):

# Options for extension 'GW1000'

[GW1000]

    driver = user.gw1000

    [[field_map]]

        extraTemp2 = intemp

        extraHumid2 = inhumid

        extraTemp1 = temp1

        extraHumid1 = humid1

        soilMoist1 = soilmoist1

Syslog extract and debug.txt attached. Output from this is null. Thanks again.

[gjr80]

Thanks, from the log it looks like the GW1000 service thread has silently died (which it shouldn't, it should die loudly if it dies!). Let's step the debug level up, leaving everything else as it is edit weewx.conf and set debug = 3, save weewx.conf and restart WeeWX. Let WeeWX run for about five minutes and again take a log extract from when you just started WeeWX. Post the log here.

Gary

[Timothy Buchanan]

field_map used with debug 3.

[Graham Eddy]

it will be interesting to see what gary concludes.

it looks to me like the fields are correctly identified and mapped (i presume; the data is there but i don't see the mapped result) but not inserted into packet by augmentation. one reason for not inserting the data would be if it is stale - do the timestamps of the weatherflowudp driver (datetime in packet) line up with the gw1000 service (datetime in mapped_data) within age toleration?

[Graham Eddy]

try adding ‘dateTime’ to the field_map. i think augmentations are skipped (gw1000 fields are not added) because on its absence the service assumes the offered data is infinitely old

[Timothy Buchanan]

How would I add dateTime to the field_map?

[gjr80]

I doubt adding dateTime will do anything, fields that already exist in the loop packet are ignored. Given me some time to work through the log extract and the driver code.

Gary

[Graham Eddy]

add ‘dateTime = datetime’ back to field_map.

worth a go while gary works on it

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fb279140-dd76-4767-92b8-1cdb780a2e30n%40googlegroups.com.

[gjr80]

Actually, looking at the code I can see that timestamps of the GW1000 data are handled differently when operated as a driver and as a service. In this case adding dateTime = datetime to [[field_map]] will get things operating as expected.

I need to revisit the handling of GW1000 data timestamps, the need for dateTime = datetime may or may not be the final solution. Either way the final version will be backwards compatible with a dateTime = datetime entry in the field map.This issue has also highlighted the need for some better debug output around the processing of loop packets.

Gary

[Timothy Buchanan]

It is working fully now. Thanks, guys, for your efforts and patience.

[Paul Ward]

Thanks Gary and others on this thread - I'm using the GW1000 API driver mode and have re-purposed one of my internal multi-channel WH31 sensors for main outdoor temp/humidity use temporarily (whilst I await return of WS80). Very simple once I'd figured out I needed to use the [[field_map_extensions]] and get the correct field names from the driver output. Is it documented somewhere what the entire list of fields your API driver/service can handle is and default mapping into weewx? I couldn't see a link in the GitHub notes.

[gjr80]

Yes, documentation is a work in progress. There is a driver wiki page that I have started work on that covers the field map and its use. There is a table at the bottom of that page that summarises the default field map. The column GW1000 field name gives a list of the GW1000 fields that are available for mapping by the driver. The corresponding entry in the WeeWX field name column in that table is the WeeWX field that the GW1000 field is mapped to by default. Note of course that some fields are dependent on the sensors you have and some fields are catered for within the API but are not yet used by Ecowitt. The other content on that page is very much a work in progress and to be truthful if I hadn't accidentally published the page I would have kept it hidden from public view.

Gary