Looking for advice on driver to target for replacement ObserverIP

[Sean Begley]

Hello everyone,

I'm new to weewx but am in the process of, essentially, creating a replacement for the ObserverIP.

I own an Ambient Weather WS-1401-IP (outdoor Fineoffset WH24 weather station, indoor Fineoffset WH32B temp/humidity/pressure sensor, 8 optional indoor Ambient Weather WH31E temp/humidity sensors).

My setup includes a RaspberryPi (or any linux machine) with an RTL-SDR based receiver attached.  Using rtl_433 I can receive all data from the sensor units.  I'm in the process of writing a python script to transform the JSON output of rtl_433 to a format that weewx likes.  I, basically, have the flexibility to target any driver.  I was considering the observer driver as well as the wu-client driver.

Any recommendations on which driver to target?  The advantage that I have over traditional ObserverIP units is that I can upload everything available, not just what Wunderground uses.  Thus I can include more data (such as the extra WH31E indoor units' data)

Here is a sample JSON output from the WH24 outdoor unit

{"time" : "2019-05-16 10:42:36", "model" : "Fineoffset-WH24", "id" : 188, "temperature_C" : 24.500, "humidity" : 50, "wind_dir_deg" : 320, "wind_avg_m_s" : 0.000, "wind_max_m_s" : 0.000, "rain_mm" : 0.900, "uv" : 0, "uvi" : 0, "light_lux" : 0.000, "battery_ok" : 1, "mic" : "CRC"}

Here is a sample JSON output from the WH32B primary indoor unit

{"time" : "2019-05-16 10:42:47", "model" : "Fineoffset-WH32B", "id" : 8, "temperature_C" : 49.800, "humidity" : 113, "pressure_hPa" : 1412.900, "battery_ok" : 0, "mic" : "CHECKSUM"}

Here is a sample JSON output from one of the eight WH31E extra indoor units (channel will go from 1 to 8)

{"time" : "2019-05-16 10:42:47", "model" : "AmbientWeather-WH31E", "id" : 139, "channel" : 1, "battery_ok" : 1, "temperature_C" : 22.500, "humidity" : 55, "data" : "1600000000", "mic" : "CRC"}

Once I'm done I'll make a RaspberryPi SD Card image available and anyone with a Pi and an RTL-SDR dongle should be able to replace their ObserverIP unit.

Thanks,

Sean

[mwall]

On Thursday, May 16, 2019 at 11:46:51 AM UTC-4, Sean Begley wrote:

My setup includes a RaspberryPi (or any linux machine) with an RTL-SDR based receiver attached.  Using rtl_433 I can receive all data from the sensor units.  I'm in the process of writing a python script to transform the JSON output of rtl_433 to a format that weewx likes.  I, basically, have the flexibility to target any driver.  I was considering the observer driver as well as the wu-client driver.

you might want to look at the weewx-sdr driver.  it already does this. 

https://github.com/matthewwall/weewx-sdr

m

[Sean Begley]

Thank you.  This is a good resource.  This doesn't *quite* do what I want it to do because the SDR would be attached to the weewx machine. I run weewx in a container along side a number of other services on a Dell R510 Server in a network closet.

I'd like to have a small, stand alone device (basically just like the ObserverIP only not locked down to ambientweather.net and wunderground.com) that can catch the 915MHz from a physically ideal location and send the data over the network to weewx.

However running it directly or modifying it/borrowing from it might do the trick.

[mwall]

On Thursday, May 16, 2019 at 11:05:17 PM UTC-4, Sean Begley wrote:

I'd like to have a small, stand alone device (basically just like the ObserverIP only not locked down to ambientweather.net and wunderground.com) that can catch the 915MHz from a physically ideal location and send the data over the network to weewx.

thank you for clarifying.

you might consider creating a generic weewx driver that listens for data in json format.  that driver would receive any json data and use a sensor_map (defined in the configuration weewx file) to determine which json fields should be recognized and how those should be associated with database fields.

you would also have to decide on the protocol - do you listen for udp broadcasts (like the weatherflow stations send out), do you listen for http(s) connections (as if weewx were wu or any other online service), or something else?

there have been some attempts to come up with such a generic, universal system, but as far as i know no one has done it yet.

my impression is that the coding is not that difficult, but figuring out the right combination of components and avoiding feature creep requires a bit of genius.

m

[Sean Begley]

On Friday, May 17, 2019 at 5:17:23 AM UTC-4, mwall wrote:

you might consider creating a generic weewx driver that listens for data in json format.  that driver would receive any json data and use a sensor_map (defined in the configuration weewx file) to determine which json fields should be recognized and how those should be associated with database fields.

Thanks.  This answers my primary questions which was "Is there a driver that is ideal for the weewx receiver or only drivers that are ideal for the sending hardware?" and it sounds like there is no driver that targets all of the data that weewx wants in a format ideal for weewx.

My plan is to have my device (what I'm currently calling ObserverPi) receive asynchronous messages from 915MHz sensors (could be extended to other frequencies and sensors), populate a data structure and then have another thread periodically send that data to weewx.  Thus weewx will receive an http(s) message.

I'm planning on starting out targeting the wu-client, observer, and/or weewx-socketlogger drivers.  Later I might try to write a driver that is optimized for weewx.  I think it would be nice for there to be a driver that acts as a standard input mechanism for people trying to build tools to send data to weewx.

[mwall]

On Saturday, May 18, 2019 at 1:26:23 AM UTC-4, Sean Begley wrote:

My plan is to have my device (what I'm currently calling ObserverPi) receive asynchronous messages from 915MHz sensors (could be extended to other frequencies and sensors), populate a data structure and then have another thread periodically send that data to weewx.  Thus weewx will receive an http(s) message.

if you use http(s), i would suggest that you use POST, not GET

 

I'm planning on starting out targeting the wu-client, observer, and/or weewx-socketlogger drivers.  Later I might try to write a driver that is optimized for weewx.  I think it would be nice for there to be a driver that acts as a standard input mechanism for people trying to build tools to send data to weewx.

i know of three generic protocols that you will see in weewx drivers and uploaders:

- http(s)

- udp broadcast

- mqtt

as for the payload encoding, there are at least:

- json

- name/value pairs (e.g., uri encodings or just plain text)

if you use json you should consider:

- how to handle grouping of observations (sensor_map can probably take care of this)

- how to handle partial packets (one observation per message vs many observations in a message vs 'all' observations in a message)

- dealing with None/NULL

there are many other proprietary protocols and message encodings (some openly published, many reverse-engineered).

when i wrote the weewx-interceptor i imagined that i could write a single driver that would work with any weather station that sends data via tcp/ip.  it turns out that while that might be possible, there are huge variations in protocols (not to mention firmware bugs) that make it difficult to do that generically.  the lacrosse gw1000 is a case in point - totally different from any other tcp/ip device.  the os lw30x are another example - they use http like many others, but their encoding is different.  and the acurite hardware is yet another fine example - it started using proprietary encodings with http GET, then added 'standard' wu encodings, then changed the proprietary encodings to sorta look like wu encodings.

you will probably have to say "this is the way to encode data using json, and this is the way to use the http/https/mqtt/udp protocol.  if your sensor emits data in this way, then this weewx driver can receive it"

but hopefully you can design that so simply that anyone would be foolish to not use your design

m

[Sean Begley]

On Saturday, May 18, 2019 at 6:45:48 AM UTC-4, mwall wrote:

you will probably have to say "this is the way to encode data using json, and this is the way to use the http/https/mqtt/udp protocol.  if your sensor emits data in this way, then this weewx driver can receive it"

but hopefully you can design that so simply that anyone would be foolish to not use your design

m

This is what I'm thinking.  If/when I write a driver it will be a driver that accepts data in the convention that weewx likes and will be optimized for weewx ingestion.  Basically it will encompass all of the database fields weewx uses and will use weewx naming conventions.  Hopefully it will support a couple protocols as time goes on.

The purpose of the driver will be for use by people rolling their own software to send data too weewx (whether that be a custom sensor built on an arduino, or a piece of software that compiles data from other sensor, etc).

Thanks for you enthusiastic help so far.

Once I've got everything up and running I intend to make a Raspberry Pi image available for download.  This should allow anyone to replace their ObserverIP.  With a bit of work it could basically be extended to work with anything that sends out unencrypted data over ISM bands.