Developing a WeeWX-Driver for custom weather station hardware (adding custom sensors)

[Henry Denston]

Hello everyone!

I'm currently developing a custom weather station (a microcontroller collecting data from custom sensors and forwarding it to a RaspberryPi).

I already read the customization guide multiple times and started to develop a custom weewx-driver for my custom weather station.

The customization guide offers some very valuable insights. However, I still have some questions and misunderstandings I guess, so I hope some experienced users/developers can help me out. :)

As already stated I started developing my own driver, I also edited the weewx.conf and my custom driver is loaded when I start weewx with: sudo ./bin/weewxd weewx.conf (it yields/prints packets)

I checked the sql table structures in the documentation and now I'm not sure how to save my custom sensors values.

For example: I have a windsensor (measuring windspeed only), a temperature sensor and 4 pyranometer sensors of different kind measuring the sun irradiation.

Now in my custon driver I know that I can store/assign the windspeed and temperature like this for example:

data = dict()

data['dateTime'] = int(data_array[0])  # unix timestamp
data['usUnits'] = weewx.METRIC
data['windSpeed'] = float(data_array[1])  # Anemometer measuring windspeed only
data['outTemp'] = float(data_array[0])  # PT100 measuring temperature outside

yield data  

So this should be straight forward if I understand correctly.

But how do I store my irradiation values from the various pyranomerters?

It seems that in the expected database structure there is only one column for irradiation available.

How can I store it so WeeWx can use it in its reports and archives?

I have smth. like this in mind:

data['radiation'] = float(dataset_array_sensor[2])  # pyranometer 1
data['radiation1'] = float(dataset_array_sensor[3])  # pyranometer 2
data['radiation2'] = float(dataset_array_sensor[4])  # pyranometer 3
data['radiation3'] = float(dataset_array_sensor[5])  # pyranometer 4

Would this work?
Do I have to change the sql database structure for it to work? (I plan to use sqlite for it as its easier to use it out of the box)
If so, how would I do it?

I did read the customization guide multiple times but I'm really confused regarding this.

I hope you can give me a hint into the right direction (I think I'm probably way overthinking this and just missing a small detail)

Thank you very much in advance! :)

Best regards, Henry.

[Thomas Keffer]

Yes, your strategy would work, and your names (radiation1, radiation2, and radiation3) are sensible names to give the extra sensors.

You would have to change the database schema to accommodate the extra sensors. 

Do you have an existing database, to which you want to add the new observation types? If so, then follow the instructions in the Customizing Guide, section Adding a new type to the database. If there is something that is confusing about the instructions, let us know what the problem is so we can clarify and, perhaps, update the documents and make it clearer.

If you do not have an existing database, then it is slightly easier. To the bottom of the file user/extensions.py, add the following: 

import schemas.wview

radiation_schema = schemas.wview.schema + [

    ('radiation1'', 'REAL'),

    ('radiation2'', 'REAL'),

    ('radiation3'', 'REAL'),

]

Then go into your weewx.conf and change the [DataBindings] and [Databases] section so they look like this:

[DataBindings]

    [[wx_binding]]
        # The database must match one of the sections in [Databases].
        # This is likely to be the only option you would want to change.
        database = archive_sqlite
        # The name of the table within the database
        table_name = archive
        # The manager handles aggregation of data for historical summaries
        manager = weewx.wxmanager.WXDaySummaryManager
        # The schema defines the structure of the database.
        # It is *only* used when the database is created.
        schema = user.extensions.radiation_schema

##############################################################################

#   This section defines various databases.

[Databases]

    # A SQLite database is simply a single file
    [[archive_sqlite]]
        database_name = radiation.sdb
        database_type = SQLite

    # MySQL
    [[archive_mysql]]
        database_name = weewx
        database_type = MySQL

Restart WeeWX.

WeeWX will now use the new database radiation.sdb, which will have the new SQL columns.

-tk

[Henry Denston]

Hi Tom,

thank you very much for your great reply! :)

I do not have an existing database yet.

I already read the  Adding a new type to the database section in the Customizing Guide, however was confused as I thought this only applies in case one wants to add a new observation type that is not present in weewx yet (as in the example electricity).

I wanted to add additional radiation sources (and there is already the radiation observation type implemented in weewx), so I thought by using an already existing observation type I might not need to adjust the database structure.

Your answer however, clearly shows how to make the required changes, thank you very much! (Maybe add this little example to the docs, right next to the electricity eample? :))

Another question:

By using the file user/extensions.py additional sensors can be added to weewx as you have presented.

Is it best practice to just append additionally required sensors to the standard weewx database structure or should I (make some additional changes to) 'rebuild' the used database structure for my system?

For example in case I intend to use only about 8 sensors/values in my current system (e.g.: dateTime, usUnits, windSpeed, outTemp, radiation, radiation1, radiation2, radiation3) I could remove all additional values in the database that I will not use (e.g. barometer, pressure, altimeter, inTemp, etc....) ? Would love to hear your opinion on that.

Thank you very much, Henry.

[Thomas Keffer]

If you're going to all the trouble of creating a custom database schema, you can certainly make additional changes to suit your needs. Just be careful not to drop something you'll need for your reports.

However, you'll find removing unused observation types does not save as much space as you think. A major part of the database is the index, which will still be there.

As far as what's the best practice? I'd say, do what Google does: save everything. Storage is cheap.

-tk

[Henry Denston]

Got it, thank you.

I'm testing my driver right now and I noticed some strange behavior.

My driver works like this:

It queries an external MySQL database and selects all datasets (rows, and one row contains a timestamp, the value of sensor1, sensor2 etc.) that have not been processed by WeeWx yet.

The driver yields data once every 5 minutes. Everytime the driver runs, it has to yield about 60 packets.

My question:

Is this too much for WeeWx to handle?

Yielding about 60 data packets at once in a loop in the genLoopPackets(self) method?

I'm asking because when I run WeeWx (using: sudo ./bin/weewxd weewx.conf), only about 16 LOOP messages appear in the console window while I'm expecting about 60.

Hope you can point me into the right direction again, thank you very much! :)

Kind regards, Henry.

[Thomas Keffer]

WeeWX uses a blocking call when it processes LOOP packets. So, nothing happens until the hardware returns a packet.

As the packet is processed, the engine looks at the timestamp. If/when the timestamp exceeds the end of an archive period (plus a slight delay, usually about 15 seconds, called the archive_delay), the engine starts the process of generating a new record and doing the reporting.

One theory: 60 packets over 5 minutes is 12 packets a minute, or one every 5 seconds. If your archive_interval is one minute, add the 15 second archive_delay, that would be 75 seconds, or about 15 packets --- close to what you're seeing.

So, if you have set your archive interval to a very short one minute, but it's trying to process 5 minutes worth of data, then I could see this happening. If this is the case, it really should be the other way around: poll your database every minute, set record generation to every 5 minutes.

But, without seeing the log, it's impossible to know what's going on for sure.

WeeWX really wasn't designed to be used this way. It uses the LOOP packets as a sort of processing clock. There is no internal "select" statement, nor any asynchronous processing.  Everything is governed by how often the LOOP packets arrive. Limited, yes, but also very simple and reliable.

-tk

[Henry Denston]

After experimenting it makes much more sense now, thank you very much!

Starting to like it, great project! :)

Can you confirm that this instruction in the docs is still up to date to the latest apache version?:

http://www.weewx.com/docs/usersguide.htm#integrating_with_webserver
I followed the mentioned steps but can not get it to work (reach it from the browser via localhost. Opening the index.html in the public_report folder works great)

I installed WeeWx using the setup.py. I installed it to opt/MyApp/weewx

Then I copied the weewx.conf file from this folder:

(weewx_install_folder)/util/apache/conf.d/weewx.conf

into this folder:

/etc/apache2/conf.d

The contents of this file:

Alias /weewx /opt/MyApp/weewx/public_html
<Directory /opt/MyApp/weewx/public_html>
  Options FollowSymlinks
  AllowOverride None
# for apache 2.2
  Order allow,deny
  Allow from all
# for apache 2.4
#  Require all granted
</Directory>

But still I get the 404 error in the browser when trying to reach it via /localhost/weewx.

Thank you.

Henry.

[Henry Denston]

Tom, in case I would like to create my own database schema, is it enough to just remove the cloumns from the dictionary in this file: .\weewx\bin\schemas\wview.py'' ?

Seems like weewx still tries to write to removed database columns when trying to save a record.

Would really appreciate it in case you could name at least some files off the cuff that I have to look into.

Thanks :)

[Thomas Keffer]

When a new record arrives, WeeWX tries to write all the types it can into the database. If a type is not in the database schema, it is skipped. 

If you're interested, the logic is in function manager._addSingleRecord().

-tk

[Henry Denston]

Thank you very much Tom.

One thing I can not figure out is: 

In the docs it says that everytime an archive period is over (every time a new record is done) all the newly generated images (plots) get copied into the public_html folder.

What I noticed for me is that this is not the case. (actually it seems to be a lot longer until they are updated).

When triggering a report manually using the wee_reports command the same behavior can be observed.

However, when I delete the .png images (plots) from the public_html folder the fresh images get copied into the public_html right away when the next record is done.

Can you tell me how to change this so the newest plot images are always copied to the public_html folder everytime a new record is done?

I tried to set copy_always flag in the skin.conf in the copy_generator section but no change.

Also cannot seem to find anything in the docs that can help me.

Thank you very much, Henry.

I have no stale_age or cronjob defined for report generation.

[Henry Denston]

Ok, after hours of trial and error I guess the lable 'aggregate_interval' is responsible for my issue.

I set it to 60 (so Images/Plots will get copied when older than a minute everytime a record runs).

Sorry for the spam.

But can you answer me the question if there are limitations regarding the color formatting in the skin.conf?

E.g. I can not set  the color '0xD17729' in the ImageGenerator Section for the label 'axis_label_font_color'. E.g. 'white' or 'yellow' works great.

When I try for example 'axis_label_font_color = 0xD17729' the axis label font color just appears as blue.

So are there limits what color codes I can use?

Any thoughts?

Thank you, Henry.

[mwall]

On Thursday, February 21, 2019 at 1:59:25 PM UTC-5, Henry Denston wrote:

But can you answer me the question if there are limitations regarding the color formatting in the skin.conf?

E.g. I can not set  the color '0xD17729' in the ImageGenerator Section for the label 'axis_label_font_color'. E.g. 'white' or 'yellow' works great.

When I try for example 'axis_label_font_color = 0xD17729' the axis label font color just appears as blue.

So are there limits what color codes I can use?

Any thoughts?

its not in the weewx docs, but there is a comment about this in the skin.conf:

    # Colors can be specified any of three ways:                                

    #   1. Notation 0xBBGGRR;                                                   

    #   2. Notation #RRGGBB; or                                                 

    #   3. Using an English name, such as 'yellow', or 'blue'.                  

    # So, 0xff0000, #0000ff, or 'blue' would all specify a pure blue color.     

it sounds like you want:

axis_label_font_color = 0x2977D1

m 

[gjr80]

Hi,

Perhaps a small insight into the operation of the image generator might help. Yes it is true that at the end of each archive period (during the report generation cycle) the plots are copied to public_html (or wherever you have set them to be copied). However, it is not quite as simple as that, what is really going on is only those plots that have been generated are copied. As you have found out the aggregate_interval has a role in how often/when images are generated, in fact all other things being equal plots are generated every aggregate_interval seconds. If no aggregate interval is specified then they are generated every report cycle.

So you might say well why not have a very small or zero aggregate interval for each plot, you can certainly do that but there is a penalty to pay. When WeeWX generates a plot it queries the archive and obtains all of the data points for the observations required over the period of the plot. In the case of a day plot the period covered is 27 hours so assuming a 5 minute archive interval WeeWX would use 27x12=324 data points per observation. These points are then plotted. For a week plot we now have 7x24x12=2016 points, a month we have (approx) 31x24*12=8928 and for a year we have 105120 data points. Now WeeWX will quite happily plot 324 data points, as it will 2016 but when we get to 8928 and 105120 it takes WeeWX a very long time to generate the plots (remember there are numerous plots each with (usually) multiple obs each with 8928/105120 data points. So the way WeeWX mitigates this load is to reduce the number of points being plotted by using an aggregation, say the average over 1 hour (for month plots) and over 1 day for year plots. This brings us down to around 744 points for a month and 365 points for a year, much more manageable and comparable to day and week plots. The load on the processor is further reduced by only re-generating the plots after their aggregate_interval has passed.

So what, well as I said you can certainly reduce the aggregate interval to 60 and all plots will then be generated every report cycle. But if you have a look at your logs you will probably find the time taken to produce your reports has increased significantly. This may or may not be a problem depending on your archive interval and your processor. But if your reports are taking close to your archive period to complete you may find your system will become unstable or some report cycles may be skipped - quite the contrary to what you are trying to achieve. Also remember that on a 1 year plot with say 105120 points I don't think you will really be able to see the difference 1 extra point makes so really you are gaining very little by running a long period report every report cycle with a very small or zero aggregate interval.

I hope that helps explain what your are seeing, why you are seeing it and the possible consequences of your changes.

Gary

[Henry Denston]

Thank you both of you for the replies, they helped me a lot! :)

Also thank you very much for the great explanation and insight on the image generation, that will prevent bad surprises for sure!

As far as I can understand the python package 'python-pil' generates the plots, right?

Again, thank you very much.

Henry.

[gjr80]

python-pil is used by the WeeWX image generator to create/manipulate the image file (ie create the image file, draw points, lines, labels etc). The logic for obtaining the data to be plotted, working out what goes where and what colours are used etc are all contained in various WeeWX .py files (imagegenerator.py, weeplot/genplot.py etc).

Gary

[Henry Denston]

Thank you.

Another question that I would like to ask:

Is there any downside just adding (executing) the  sudo ./bin/weewxd weewx.conf script to the /etc/rc.locatl file?

So my WeeWx installation gets started everytime on startup of my system.

In the Docs this instruction is given:

cd /home/weewx
sudo cp util/init.d/weewx.debian /etc/init.d/weewx
sudo chmod +x /etc/init.d/weewx
sudo update-rc.d weewx defaults 98
sudo /etc/init.d/weewx start

But is there any disadvantage using the method I described above?


Thank you, Henry.

[Tim Urberg]

One you could use screen, which allows a totally separate screen to be running in the background.

https://linuxize.com/post/how-to-use-linux-screen/

Here's what I have in /etc/rc.local (on a Raspberry Pi):

screen -S weather_station -d -m sh -c 'weewxd /etc/weewx/weewx.conf; exec bash'

Then when you log in run: "sudo screen -r" which will re-attach to the screen session.  Once finished type Ctrl+A and then D to detach.

That's one way to do it.

Virus-free. www.avg.com

[Tim Urberg]

One pro to doing it this way is you can watch the LOOP data as it is coming in.  A good way to debug if nothing else.

[Henry Denston]

Ok, thanks Tim, good advice! :)

So basically there is no need to use the file from util/init.d/weewx.debian like the DOCs advice.

So there is no downside by not using the instruction from the documentary and just execute the ./bin/weewxd weewx.conf file with the weewx.conf as first parameter from the /etc/rc.local file?

Did I get that right?

Regards, Henry.

[Tim Urberg]

Yep, either way works.  Using screen is like a poor mans service, so to speak.  I use it in that way for other things too.  I was using WeewWX with an SDR (software defined radio) with an acurite for a while and it would only work that way.

[Henry Denston]

I was a bit confused as the Docs advice to use this file: sudo cp util/init.d/weewx.debian.

But great to know that this is nothing special and one can also just use the /weewxd file and put it into the /etc/rc.local for example.

Thank you very much! :)

[vince...@gmail.com]

On Sunday, February 24, 2019 at 11:42:22 AM UTC-8, Henry Denston wrote:

Ok, thanks Tim, good advice! :)

So basically there is no need to use the file from util/init.d/weewx.debian like the DOCs advice.

So there is no downside by not using the instruction from the documentary and just execute the ./bin/weewxd weewx.conf file with the weewx.conf as first parameter from the /etc/rc.local file?

The downside is that if you start/stop weewx via some custom mechanism, we will have a difficult time helping you for future questions....and you will have a difficult time updating weewx to future versions (maybe).

• If you are running a systemd-based operating system, start weewx with a systemd-based startup file
  
• If you are running an init.d-based operating system, start weewx with an init.d-based startup file
  

But 'technically' weewx does not care how you start it up.   You can do it any way you want.

 

[Thomas Keffer]

The init.d scripts files do more than just start weewx. They also make sure essential services are up and running before attempting the startup.

So, let me flip the question around: is there any reason not to use the init.d script? Is there something you need to work around that attracts you to putting the start up in the rc.local file?

-tk

[Henry Denston]

I'm using WeeWX on a RPi but there are also many other systems running simultaneously. I'm using a single bash script used by rc.local file and crontab that checks and manages all systems. Implementing WeeWx into my system like that just is a convenient way for me to not 'break' my system architecture and keeping things simple.

So far I did not encounter issues, so I'm happy :)

Again Tom, thank you very much for this great project and your efforts!

[Henry Denston]

Sorry, in case I create too much spam, but I'm trying to dig deeper into WeeWx and while I was checking out the database structure I can not fully understand how the day tables are managed.

I'm using mysql so I checked the weewx/manager.py (the DaySummaryManager classs) and all mysql.py files.

The daily tables look like this (e.g for my temperature):

I first assumed that every observation type (like radiation, temperature, rain etc.) gets it's own daily table (and that is right).

Then every daily table contained one row for every day the station was up and running and every row will have a min and max value of that day (for the specific observation type).

I could not test the station over a longer period of time so far but to me it seems that a daily table will always only hold one single row?

I'm confused as as you can see in my picture posted above, the dateTime is from the day before (02.03.2019) and the min/max timestamps are from another day, the day after (03.03.2019).

How will weewx get the hi/lo values for every single day in a month for example if the daily tables only hold one single row?

It's been a long day so maybe I'm just being dumb right now but I'm confused and I don't like being confused ;).

I hope someone will be so kind to clear this up for me.

Thank you very much, henry.

[Thomas Keffer]

The daily summaries hold one row per day. 

Each row is timestamped with the start of the day, local time. In your time zone (CET +1, I presume), they are all in the same day, 3-March-2019.

If you have only one row in your daily summaries and WeeWX has been running for more than one day, then there is something wrong and we will need to see the logs.

-tk

[Henry Denston]

Thank you for the clarification tom.

So I assume there are not going to be more tables added (e.g. for week, month etc) but only the daily hi/lows, right?

Regards, henry.

[Thomas Keffer]

That's right.

[Henry Denston]

Tom, can you tell me what WeeWX will do in case it gets a package with the same unix timestamp a package already had before?

As dateTime is a unique key I'm curious if WeeWX automatically 'filters' the data/packages or if some kind of exception is thrown (that may have an impact on system operation)?

Thank you!

[Thomas Keffer]

You get a "duplicate key" error from the database, which is logged, but ignored.