WeeWX and CoCoRaHS

[Joshua Myles]

Hi WeeWX folks,

tl;dr: I want to get my CoCoRaHS data into WeeWX.

I take daily manual rain and snow readings as a CoCoRaHS (https://cocorahs.org/) observer and would like to get that data into WeeWX so that I can do reporting on it, have a long term archive of it, etc. Here's an example of a daily report:

https://cocorahs.org/ViewData/ViewDailyPrecipReport.aspx?DailyPrecipReportID=c4f97a37-bed3-415d-a782-b3907d401f86

I can think of two ways to do this. The first is to write a driver that pulls my data down from the CoCoRaHS API after I submit it (something like the AmbientWeather API driver). I'd still use the normal form submission on cocorahs.org to input my daily observations.

The second would be to write an extension for WeeWX to POST data to CoCoRaHS instead. This method would need some kind of WeeWX input interface (or using an existing extension like FilePile).

I have other concerns about custom types, handling/reporting on once-a-day data, etc. but I think those can be addressed by me reading the documentation. I mostly wanted to throw this out there in case anyone else is doing this or has thoughts on a reasonable direction to head in.

Josh

[gjr80]

Hi,

Short answer, I am not aware of anyone doing this. Though I qualify that by saying I am not a CoCoRaHS user. Also bear in mind this is the development group where the audience is much smaller than the user group. You may have more luck in the user group.

Ii seems to me that if you do enter CoCoRaHS data into WeeWX it may well be a lot of work for little gain/value. Before I go into this further perhaps a quick 101 on WeeWX operation/data storage. WeeWX records data in a database table known as the archive. The archive consists of timestamped records that contain observation data as of that timestamp. Each record contains data covering the period since the previous record, eg for temperature, humidity etc this is typically (or is assumed to be) the average value since the last record. For observations such as rain and snow this is the amount of rainfall or snowfall since the last record (ie it is not a cumulative value). WeeWX usually (it may not) also use a number of additional database tables (one per observation type in the archive) known as the daily summaries, these tables are derived from the archive data and each contains one record for each day with min/max/sum/times etc for the observation concerned. The daily summaries are an optimisation system for speeding up reporting of day/week/month etc aggregate reporting. The daily summaries are updated each time a new archive record is added - key point the daily summaries are somewhat dynamic and it is not safe to manually edit them.

Presumably you intend to enter a 'gauge' reading of rain or snow along with some sort of date-time/timestamp. At first glance you may think that the obvious place to enter such data is the rain/snow daily summary. You can certainly do that, though this will need to be a manual update of the database as WeeWX provides no mechanism for updating a single field in a daily summary with external data. The problem you will likely eventually encounter is that something will cause WeeWX to rebuild the daily summaries (perhaps all, perhaps just the rain daily summary) and your data will be lost. The alternative is to record the data in the archive, again this will need to be a manual update of the archive table (either updating an existing record or adding a new record). Normally the daily summary tables would be updated when a new record is added to the archive, but since you made a manual change to the archive you will need to rebuild the daily summaries for that day for the daily summaries to reflect you data (you could write a short stand alone utility to do your data entry and daily summary update for you - the WeeWX utility wee_database has a lot of the necessary code/API calls that would be used). Also, if you do add your data to the archive remember that each archive record contains data covering the period since the previous record, so adding your daily (?) CoCoRaHS data will be breaking this condition. No matter, as long as you observe some strict rules in how you use/report the data it should not matter too much. Finally, on the topic of data entry, you don't say if you are using WeeWX for an existing station, ie is the database field 'rain' already being used. If it is you could easily add a new field, say 'cocorahs' to your database using the wee_database utility.

As I understand the CoCoRaHS reporting/data you will have some WeeWX reporting challenges as well. You will certainly be able to report/plot your data but I would question the value or accuracy of such reports. WeeWX works on a midnight to midnight 'day'. So when WeeWX reports there was 25mmm of rain on the 23rd November this means that the sum of the rain fields in each of the archive records covering the period from midnight at the start of 23rd November to midnight at the end of 23rd November is 25mm. In your case (if I understand CoCoRaHS data correctly) the CoCoRaHS data does not cover a midnight to midnight day so any WeeWX reports for a given day may well have some rain from the previous day. Likewise for any report covering week, month or year where it rained on the first or last day of the period concerned. At best you could extract a simple table showing the time of your reading and its value (handling days with zero rain/snow would need some extra processing to pick up the correct time of reading).

Likewise plots, though perhaps anything showing daily values may be more meaningful (sort of the plot equivalent of a tabular report of daily readings).

If it was me I think I would manually record my data is a separate spreadsheet to give me the plotting/reporting/archival capabilities needed.

But of course it is entirely up to the user/you as to what you want to do. At the end of the day if you do add CoCoRaHS data to WeeWX you need to understand/carefully interpret any reporting you do as the standard generated reports may be misleading with such coarse (daily) data.

Gary

[Vince Skahan]

Several year CoCoRaHS user here.   I use the mobile app.  Muuuuuuuch easier than using their website.

FWIW - they ask for readings 'at' 7am localtime for their 7am-7am rain day, although they'll take any readings they can get.  You can even enter multi-day readings (which would be particularly hard to get into weewx), and alter the time-of-day you took the reading to line up with when you actually did so.  They also support you adding readings after the fact anytime you want, as well as editing old entries to clean up errors etc.

Regardless, I'd suggest the o.p. look into just writing a standalone secondary sqlite3 db first before worrying anything weewx initially.   It wouldn't be too hard to programmatically generate the sqlite3 insert statement once you knew the date+time and value to use there.  Schema would be super-simple, just do the minimum weewx would expect.   

Perhaps something like the following....

schema = [
    ('dateTime', 'INTEGER NOT NULL PRIMARY KEY'),
    ('usUnits', 'INTEGER NOT NULL'),
    ('interval', 'INTEGER NOT NULL'),
    ('cocorahs','INTEGER'),
    ]

There are lots of sqlite3 examples out there but (this one) is pretty simple for starters.

Once you have a db with data in it in a weewx-expected format, do a simple skin reading from that secondary db and see what the graphs look like for day/week/month/year etc. to see if it's worth connecting the two.

Heck, use bogus data.  You can always delete your cocorahs.sdb and start over clean.  That's part of why a standalone db for this might help.

[Vince Skahan]

And of course I fat-fingered it.  The cocorahs should be a REAL of course, not an integer.  Doh.

[Joshua Myles]

Gary, Vince, thanks for the replies. Somehow I was able to subscribe to this group with a non-Google email address (and in fact got a copy of my own post) but never got any of the replies and can't even figure out how to reply to this thread with my original non-Google account. So I didn't see the replies until I checked with the web interface just now. Sorry!

I did anticipate some issues with CoCoRaHS data being on a much different schedule than typically WeeWX data, so thanks for providing thoughts on that. I'm not as interested in inserting multi-day reports, only the daily reports which are a bit more regular (ideally 7:00 AM). But I do get what you're saying about archive records and daily summaries.

One of my goals is to make my CoCoRaHS data available on my weather site alongside my other WeeWX data. I'm currently testing the Belchertown skin, which has very nice graphing capabilities, and wanted to avoid writing something completely separate for snow data. Vince, I think your suggestion of creating a separate WeeWX compatible database to start with is a good one; it seems like I could get a skin to use this second datasource when I get that far along.

At any rate, what I have right now is a few simple scripts that run from cron to pull down my daily CoCoRaHS data, transform the XML a little bit, and do some basic aggregation and reporting on it. So that part is good enough to build on to get the data into a database (I use MySQL with WeeWX so will just create another schema there). I currently use the CoCoRaHS website to submit my observations since the Android app is broken for me (password not being escaped correctly).

[Vince Skahan]

On Sunday, December 19, 2021 at 6:49:36 AM UTC-8 jam...@gmail.com wrote:

At any rate, what I have right now is a few simple scripts that run from cron to pull down my daily CoCoRaHS data, transform the XML a little bit, and do some basic aggregation and reporting on it.

It would be nice if you'd share the code that does the grabbing part in a github repo or the like...

I was thinking that something like a summarized NOAA chart with just day vs. rain for a month would be interesting.  I know the start time of the day differs for weewx and CoCoRaHS but over a month it would be close enough to compare them.

 

 the Android app is broken for me (password not being escaped correctly).

Ummm - change your password to something it accepts ?