Proper/Best Way to Add Additional Ecowitt Sensors

[Blaze]

I think this is really a multipart question, but I am trying to understand the best/proper way to add Ecowitt soil sensors to my reports. Keeping in mind that I want to be able to upgrade to future releases as they come available with the least amount of post upgrade effort. 

Setup

Ubuntu 20.04.1 LTS

WeeWx v4.1.1 using setup.py

gw1000-0.1.0b11

sqlite db

Seasons skin

Ecowitt GW1000 WiFi Gateway

1 Outdoor Temp Sensor

1 Indoor Temp Sensor

5 Soil Moisture Sensors (temporarily 5th sensor is broken while waiting for replacement)  

Out of the box everything is showing up in the Seasons' report except my soil moisture sensors. I do have data for the first 4 soil moisture sensors in the sqlite db, but nothing for 5th one.  And I can see there is no entry in the db for SoilMoisture sensors 5-8. So begins my adventure of learning more about this awesome project called WeeWx.

So I am thinking I have at least 3 tasks.

First I need to extend the db to accommodate SoilMoisture sensors 5-8?

For this task do I extend the schema by modifying the wview_extended.py and the wview.py to include these extra sensors in the db? Or is the better approach to use something like adding a new type to the database as described in the Customization Guide?

Second modify the reporting mechanism to show SoilMoisture in my Seasons' report.

I'm really uncertain for this portion. I did a grep in the /home/weewx/bin/ directory, and I can see several files where "soil " is referenced. Do I just need to add entries in these files for extra sensors?

Command used: 

    grep -iHR soil /home/weewx/bin/*

Relevant files (or so I think) referencing the word "soil".

    /home/weewx/bin/schemas/wview.py

    /home/weewx/bin/schemas/wview_extended.py

    /home/weewx/bin/user/gw1000.py (this one seems to already have all the sensors)

    /home/weewx/bin/weewx/units.py

    /home/weewx/bin/weewx/restx.py

And would the last task be to update the appropriate files in the /home/weewx/skins/Seasons/ to actually generate the graphs I want to have displayed in my report?

Thanks in advance for any help or guidance.

Rob

[gjr80]

Hi Rob,

To take the second part of your question first, I am currently putting together a page to go in the GW1000 driver wiki on how to extend the Seasons skin to include additional observations such as provided by the GW1000 driver. This will also cover adding plots and will complement the page re displaying GW1000 sensor battery states in Seasons.

Now for the db shema. As a general rule of thumb you should avoid modifying any of the WeeWX files except for weewx.conf, files in /home/weewx/bin/user (or /usr/share/weewx/user for package installs) and files in the skins directory. Those files/locations are preserved during WeeWX upgrades whereas other WeeWX files are liable to be overwritten during an upgrade meaning your changes may be lost, or at best may need to be re-applied. However, what you want to do can easily be done without running foul of future upgrades.

There are two basic ways you can add a new observation to your database, first you can re-purpose an existing field. For example, someone with a second rain gauge might re-purpose the field hailRate as the rain rate field for the second rain gauge. Most easily done if the re-purposed field uses the same units as the data you wish to store in the re-purposed field but you can essentially use any field. The upside is you don't need to change your schema, the downside is it may not be obvious from the field name as to what observation is stored in that field.

The second approach is to modify your database schema by adding a new field. This is more involved, thought the process is straight forward. Adding another field will not substantially increase the size of the database or adversely affect performance and I think the benefits of self-evident field names count for a lot, especially if you are going to be making changes to skins etc to display your data. At the end of the day the choice is yours.

I will assume you are going down the second path. The process you need to follow is not so much 'modifying wview_extended.py' (remember the rule of thumb I mentioned) but rather creating your own schema based on the schema in wview_extended.py which you then extend with your new fields. The process is covered in Adding a new type to the database in the Customization Guide though you will need to make a couple of slight changes. The first change you need to make is instead of inserting the lines of code at step 1 in the process in the file user/electricity.py (the user directory is in /home/weewx/bin or /usr/share/weewx depending on your WeeWX install type), you should add the lines to the end of the special file users/extensions.py, something like this (untested):

import schemas.wview_extended

my_schema = { 
    'table': schemas.wview_extended.table + [('soilMoist5', 'REAL')],
    'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5', 'SCALAR')]
}

The second change is that you will need to tell WeeWX what unit group your new field belongs to so WeeWX knows what units to use and how to format the field in reports. This is done by adding the following to the end of user/extensions.py (untested):

import weewx.units

weewx.units.obs_group_dict['soilMoist5'] = 'group_moisture'

Otherwise follow the steps as is and once complete you should find your database now includes a field soilMoist5.

Gary

[Richard Horobin]

(I'm providing comments I think fill in context and gaps.)

I'm using GW1000 and have a WH51 soil sensor, too.

GJR80 (Gary), excellent advice. Thanks.

Blaze (Rob), I chose to use the Ecowitt web site until I can read and use the new GW1000 driver in weewx. Its advantage is that Ecowitt tells me all the sensors it finds and is therefore a very useful testing tool. The GW1000 seems like an SDR dongle and a wifi router, with a temperature, humidity, pressure sensor attached. There's some calculation and memory, too. The RTL-433 and some of the weewx functions seem to be at the Ecowitt web site.

So the GW1000 is also a sensor-aggregator. Its manual says it will recognise up to eight (8) WH51 moisture meters. The manual says:

1. Additional/optional sensors:
    One WH32 outdoor temperature and humidity sensor.  possibly because this is not a multi-channel device - see WH31 below.

   •   One WH40 self-emptying rain gauge sensor

   •   One WS68 wireless anemometer

   •   One WH57 lightning sensor

   •   Up to 8 WH31 multi-channel temperature and humidity sensors

   •   Up to 8 WH51 soil moisture sensors

   •   Up to 4 WH41 PM2.5 air quality sensors

   •   Up to 4 WH55 Water leak sensors

 Future sensors (to be developed), such as: water temp, soil temp 

[robc...@gmail.com]

Hi Gary,

Thank you again for your response. And you are correct in that I am looking at going down the second path. I think I understand what you are explaining. If I am planning to use the maximum 8 soil moisture sensors, would I duplicate each of these lines you have provided for each sensor, or can I string them together in some order? We've had really good results with these sensors,and I am thinking of adding more. So if I am going to extend the schema, why not do it for the rest of them. Looking at the "wview_extended.py", I see that the first part is repeated, ('soilMoist1', 'REAL').

Maybe something like this?

import schemas.wview_extended

my_schema = { 
    'table': schemas.wview_extended.table + [
      ('soilMoist5', 'REAL')

      ('soilMoist6', 'REAL')
      ('soilMoist7', 'REAL')
      ('soilMoist8', 'REAL')
],
    'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5','soilMoist6','soilMosit7','soilMoist8', 'SCALAR')]
}

import weewx.units

weewx.units.obs_group_dict['soilMoist5','soilMoist6','soilMost7,'soilMoist8'] = 'group_moisture'

Thanks for all your help and guidance!

Rob

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8dbd5209-bc5e-4f76-9457-c04369eefea6o%40googlegroups.com.

[Graham Eddy]

i am presently running off a GW1000: 1 x wh40, 1 x wh57, 2 x wh51, 2 x wh41, 1 x ws80.

i also have river temp running off a VP2

and several custom controllers such as river depth

the point is that i am merging from multiple controllers, you don't need “one that rules all”

[Graham Eddy]

more like (and watch your commas, semicolons and case):

import schemas.wview_extended

my_schema = { 
    'table': schemas.wview_extended.table + 

[
      ('soilMoist5', 'REAL’),
      ('soilMoist6', 'REAL’),
      ('soilMoist7', 'REAL’),

      ('soilMoist8', 'REAL')
],

    'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5’,’scalar’),(‘soilMoist6’,’scalar’),(‘soilMoist7’,’scalar’),('soilMoist8', ’scalar')]
}

import weewx.units

weewx.units.obs_group_dict[‘soilMoist5’] = ‘group_moisture’ weewx.units.obs_group_dict[‘soilMoist6’] = ‘group_moisture’ weewx.units.obs_group_dict[‘soilMoist7’] = ‘group_moisture’ weewx.units.obs_group_dict['soilMoist8'] = 'group_moisture'

[robc...@gmail.com]

Thank you both Graham and Gary! Would either of you or someone else mind reviewing my proposed steps?

In reading the Adding a new type to the database, and what you have both provided, I think I have a good understanding except for the 4th step. Below are the steps I think I need to follow. 

Step 1.

Stop WeeWX

/etc/init.d/weewx stop

Step 2.

Make a backup of the sqlite database

cp /home/weewx/archive/weewx.sdb /home/weewx/archive/weewx.sdb.bkup08232020a

Step 3.

Edit the "/home/weewx/bin/user/extensions.py" file.

#

#    Copyright (c) 2009-2015 Tom Keffer <tke...@gmail.com>

#

#    See the file LICENSE.txt for your full rights.

#

"""User extensions module

This module is imported from the main executable, so anything put here will be

executed before anything else happens. This makes it a good place to put user

extensions.

"""

my_schema = {

                'table': schemas.wview_extended.table + [

                    ('soilMoist5', 'REAL'),

                    ('soilMoist6', 'REAL'),

                    ('soilMoist7', 'REAL'),

                    ('soilMoist8', 'REAL')

                    ],

                'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5','scalar'),('soilMoist6','scalar'),('soilMoist7','scalar'),('soilMoist8', 'scalar')]

import weewx.units

                weewx.units.obs_group_dict['soilMoist5'] = 'group_moisture' weewx.units.obs_group_dict['soilMoist6'] = 'group_moisture' weewx.units.obs_group_dict['soilMoist7'] = 'group_moisture' weewx.units.obs_group_dict['soilMoist8'] = 'group_moisture'

import locale

# This will use the locale specified by the environment variable 'LANG'

# Other options are possible. See:

# http://docs.python.org/2/library/locale.html#locale.setlocale

locale.setlocale(locale.LC_ALL, '')

Step 4.

Change the line "schema = schemas.wview_extended.schema" to "schema = user.extensions.my_schema" for the [[wx_binding]] option under the [DataBindings] in my "/home/weewx/weewx.conf" file.

[DataBindings]

    [[wx_binding]]

        # The database must match one of the sections in [Databases].

        # This is likely to be the only option you would want to change.

        database = archive_sqlite

        # The name of the table within the database

        table_name = archive

        # The manager handles aggregation of data for historical summaries

        manager = weewx.manager.DaySummaryManager

        # The schema defines the structure of the database.

        # It is *only* used when the database is created.

        #schema = schemas.wview_extended.

schema = user.extensions.my_schema

    [[forecast_binding]]

        manager = weewx.manager.Manager

        schema = user.forecast.schema

        table_name = archive

        database = forecast_sqlite

Step 5.

Run the following command to create and populate the new database.

/home/weewx/bin/wee_database /home/weewx/weewx.conf --reconfigure

Step 6.

Replace the old database with the one created in Step 5.

mv /home/weewx/archive/weewx.sdb_new /home/weewx/archive/weewx.sdb

Step 7.

Start WeeWX

/etc/init.d/weewx start

Thanks again!

Rob

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7069D2C1-0BEE-4008-8C71-3D282A27220A%40gmail.com.

[gjr80]

Looks good though you have missed an import:

import schemas.wview_extended

needs to be included in extensions.py before you create my_schema. Not sure if it’s a typo/formatting issue or not but the group assignment statements need to be on separate lines, python will complain otherwise.

Also, when running wee_database you may or may not need to prefix wee_database with sudo, it will depend on you system/WeeWX install. Try it without, if you get an error try it with.

Gary

[robc...@gmail.com]

Thanks Gary, and yes you are correct I did miss the "import schemas.wview_extended".

I gave it a try, but I had to make a few adjustments to the "/home/weewx/bin/user/extensions.py" file. Below is the one that seemed to work for me. Or at least I can see my database has been extended! Plus when the report runs, I'm not seeing any errors in the "/var/log/syslog". My extra sensors should arrive in the next day or two, and I will be able to confirm the database is being updated. Is there a way to tell if my sensors belong to "group_moisture", or if what I have done is successful?

#

#    Copyright (c) 2009-2015 Tom Keffer <tke...@gmail.com>

#

#    See the file LICENSE.txt for your full rights.

#

"""User extensions module

This module is imported from the main executable, so anything put here will be

executed before anything else happens. This makes it a good place to put user

extensions.

"""

import locale

# This will use the locale specified by the environment variable 'LANG'

# Other options are possible. See:

# http://docs.python.org/2/library/locale.html#locale.setlocale

locale.setlocale(locale.LC_ALL, '')

import schemas.wview_extended

my_schema = {

                'table': schemas.wview_extended.table + [('soilMoist5', 'REAL'),('soilMoist6', 'REAL'),('soilMoist7', 'REAL'),('soilMoist8', 'REAL')],

                'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5','SCALAR'),('soilMoist6','SCALAR'),('soilMoist7','SCALAR'),('soilMoist8','SCALAR')]

                }

import weewx.units

weewx.units.obs_group_dict['soilMoist5','soilMoist6','soilMoist7','soilMoist8'] = 'group_moisture'

Below is the output to my screen when I ran the --reconfigure command.

root@myweewx:~# /home/weewx/bin/wee_database /home/weewx/weewx.conf --reconfigure

Using configuration file /home/weewx/weewx.conf

Using database binding 'wx_binding', which is bound to database 'archive_sqlite'

Copying database 'weewx.sdb' to 'weewx.sdb_new'

The new database will use the same unit system as the old ('US').

Are you sure you wish to proceed (y/n)? y

Database 'weewx.sdb' copied to 'weewx.sdb_new' in 2.92 seconds.

And below is my query for the extra sensors in the database.

root@myweewx:~# sqlite3 /home/weewx/archive/weewx.sdb_new

SQLite version 3.31.1 2020-01-27 19:55:54

Enter ".help" for usage hints.

sqlite> .headers on

sqlite> SELECT dateTime,soilMoist4 FROM archive ORDER BY dateTime ASC LIMIT 5;

dateTime|soilMoist4

1596909900|60.0

1596910200|60.0

1596910500|60.0

1596910800|60.0

1596911100|60.0

sqlite> SELECT dateTime,soilMoist5 FROM archive ORDER BY dateTime ASC LIMIT 5;

dateTime|soilMoist5

1596909900|

1596910200|

1596910500|

1596910800|

1596911100|

sqlite> SELECT dateTime,soilMoist6 FROM archive ORDER BY dateTime ASC LIMIT 5;

dateTime|soilMoist6

1596909900|

1596910200|

1596910500|

1596910800|

1596911100|

sqlite> SELECT dateTime,soilMoist7 FROM archive ORDER BY dateTime ASC LIMIT 5;

dateTime|soilMoist7

1596909900|

1596910200|

1596910500|

1596910800|

1596911100|

sqlite> SELECT dateTime,soilMoist8 FROM archive ORDER BY dateTime ASC LIMIT 5;

dateTime|soilMoist8

1596909900|

1596910200|

1596910500|

1596910800|

1596911100|

Thank you for all your help!

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1c827370-5291-4b7e-aa13-8d51b4ee737cn%40googlegroups.com.

[gjr80]

Rob,

weewx.units.obs_group_dict['soilMoist5','soilMoist6','soilMoist7','soilMoist8'] = 'group_moisture'

may have been accepted by python but it is not correct, it needs to be:

weewx.units.obs_group_dict['soilMoist5'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist6'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist7'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist8'] = 'group_moisture'

There is no need to re-run the wee_database commands, just make the change to user/extensions.py and restart WeeWX before you use soilMoist5 to soilMoist8 in any reports.

Gary

[robc...@gmail.com]

Thanks Gary, I ran back through the process to confirm everything would work as expected. Below is my complete "/home/weewx/bin/user/extensions.py" for reference.

#

#    Copyright (c) 2009-2015 Tom Keffer <tke...@gmail.com>

#

#    See the file LICENSE.txt for your full rights.

#

"""User extensions module

This module is imported from the main executable, so anything put here will be

executed before anything else happens. This makes it a good place to put user

extensions.

"""

import locale

# This will use the locale specified by the environment variable 'LANG'

# Other options are possible. See:

# http://docs.python.org/2/library/locale.html#locale.setlocale

locale.setlocale(locale.LC_ALL, '')

import schemas.wview_extended

my_schema = {

                'table': schemas.wview_extended.table + [('soilMoist5', 'REAL'),('soilMoist6', 'REAL'),('soilMoist7', 'REAL'),('soilMoist8', 'REAL')],

                'day_summaries' : schemas.wview_extended.day_summaries + [('soilMoist5','SCALAR'),('soilMoist6','SCALAR'),('soilMoist7','SCALAR'),('soilMoist8','SCALAR')]

                                                        }

import weewx.units

weewx.units.obs_group_dict['soilMoist5'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist6'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist7'] = 'group_moisture'

weewx.units.obs_group_dict['soilMoist8'] = 'group_moisture'

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5b542d8e-4eaf-4b59-a329-2d7e72649f48n%40googlegroups.com.

[Blaze]

Hi Gary,

My new sensors arrived today, and all seems to be in working order as I now have data in my database for my 2 new sensors. Thank you for your help!  

sqlite> SELECT dateTime,soilMoist5,soilMoist6 FROM archive ORDER BY dateTime DESC LIMIT 2;

dateTime|soilMoist5|soilMoist6

1598393700|99.2|0.0

1598393400|64.25|0.0

When you get to a point where you would like someone to test the Seasons skin for additional observations, please count me in. I'm not a Python developer, but I am pretty comfortable navigating around Linux. I would love to contribute to the WeeWX project, and your work as a tester if that helps any.

Thanks!

Rob

[gjr80]

Rob,

Took a bit longer than I hoped but have put together some instructions on how to add a new field to the Seasons skin. If you are still interested you will find the instructions in the GW1000 driver wiki here. Please let me know if you find any issues.

Gary

[Larry]

I think there is an error in the wiki

or for package installs:

$ cp /etc/weewx/skins/Seasons/current.inc /home/weewx/skins/Seasons/current_orig.inc

should be 

or for package installs:

$ cp /etc/weewx/skins/Seasons/current.inc /etc/weewx/skins/Seasons/current_orig.inc





Larry

[gjr80]

Thanks. Fixed.

Gary

[Larry]

I am not the best Linux user but happy when I can figure out why things don't work right !

By the way I believe there are other instructions in the wiki  for package  with the same issue  - maybe it was in the battery sensors wiki  section I saw that.

Thanks

Larry

[gjr80]

Thanks again, fixed those too. Hopefully that is all.

Gary