Real-time Steel Series gauges

[Luc]

The weewx skin 'weewx-WD' generates the required file(s) to drive the Steel Series gauges, Weather Display Live (WDL) and the Carter Lake/Saratoga HTML templates. The data is updated each archive interval (typical 300 s).

'mesoWx' is a real-time HTML front-end for visualizing personal weather station data. The data is updated each loop interval (typical 10 s).

I have written an addition to mesoWx which updates each loop all weewx-WD Steel Series gauges data (except WindRoseData).

A test program show both weewx-WD and 'mesoWx-ss' data side by side. See http://www.lucdesign.nl/weewx/WD/showdata.php and attached picture.

The real-time Steel Series gauges can be found here: http://www.lucdesign.nl/gauges-ss/gauges-ss.htm

Luc

[Lloyd Adams]

Nice work Luc.

I've produced the same result, but taken a slightly different approach.  I've taken the raw extension from mesoWX, and record in its database all sensor readings in a rolling 24 hour period.  Each archive interval (300 seconds), I generate a php file which contains the more slowly changing data, such as day max/min.  This php file is then used by the steel gauges to fetch the realtime data every 15secs (my fastest sensor only updates every 14 seconds) using sql queries into the raw database.  My thinking behind this approach were that it would involve minimal changes to weewx, and would not be reliant on mesoWx. The raw extension is a standalone piece of code.  (After doing it this way I think mesoWx dropped support for sqlite which is what I am using, but I may be mistaken on this point.)  I suspect if my site had a large number of hits then my approach were eventually break, due to contention between raw database reads and writes, but I'm a long way from ever being there.

This is the result. I have also tried to use Bootstrap with this page for dynamic layout depending upon the rendering device. This has been reasonably successful, except for the position of some of the pop up graphs.

Lloyd

[Luc]

Hello Lloyd, nice work too!

On Sunday, 22 June 2014 06:53:30 UTC-3, Lloyd Adams wrote:

This php file is then used by the steel gauges to fetch the realtime data every 15secs ... 

Can you explain a bit more about your php file used by steel gauges fetching real time data every 15 seconds?

My WS produces data at intervals of 13 (temp), 17 (wind) and 19 (rain) seconds.

I've set weewx to read the data every 10 seconds and also mesoWx syncronises the data every 10 seconds.

In worst case a (rain) reading is 19 s (station) + 10 s (ws28xx driver) + 10 s (meso sync) + 10 s (Steel Series update) = 49 seconds behind...

Note: To get more real-time presentations you have to synchronise the data flow or increase the individual intervals.   

 

This is the result....  

Hmm... nice weather.  (But no idea, which part of the world the weather data comes from!)

I have also tried to use Bootstrap with this page for dynamic layout depending upon the rendering device. 

I didn't look further at Bootstrap because the graphs don't fascinate me.

Luc

[gjr80]

I have taken a different approach again. Inspired by Matthew's crt.py service and some other things I had been playing with, I have written a service that binds to the arrival of a loop packet and produces customclientraw.txt on every xth loop packet. Currently running on every 4th loop packet which gives me roughly a 10 sec update time. What I call 'slow moving data' such as all time barometer records, windrose data, forecast text etc is produced every archive period in json format through a normal Weewx report. The customclientraw service sucks in the json data when it needs it. Seems to work OK and the RPi does not seem unduly stressed. It's at http://www.therodericks.id.au/saratoga/wxssgauges.php, not much to see that is different to Luc's and Lloyd's, one set of updating steel series gauges looks much the same as another :-) Have had a few dynamic dns issues so apologies if the link is on the blink!

Gary

[John Harvey]

And I did something similar since our requirement as a sailing club was for the wind to be current. So I wrote a service that pushes the wind information out on every loop packet as a cut down customclientraw.txt and then left the full one being generated every minute on the archive interval.

This is displayed locally in the clubhouse and we have limited bandwidth to our external web server from there so I’ve extended the normal upload service so different jobs can use cron like control for when they run. So I upload the cutdown customclientraw every minute and the full one and graphs every 15 minutes and then upload a backup of the archive database once a day.

So I cant send you a link to the local site but the external one which combines the steel series gauges with the normal weewx graphs & data is at http://thornburysc.org.uk/index.php/about-the-club/live-weather/weewx/ . Doing this I have managed to avoid needing any server side code as well.

 

John

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gjr80]

Good stuff, seems there are a number of ways to skin the cat. As an aside re update frequency, I did want to update my gauges more often and was initially concerned whether generating a customclientraw every 2.5 sec would be too much of a load on the RPi. Interestingly, the RPi could handle the load but updating the gauges every 2.5 sec meant the scrolling forecast on the page at times did not have time to scroll fully before an update. Changing the scroll speed is a change to the steel series source code rather than changing a config setting. A job for another day.

Gary

[Luc Heijst]

On Wednesday, 25 June 2014 00:42:32 UTC-3, gjr80 wrote:

not much to see that is different to Luc's and Lloyd's, one set of updating steel series gauges looks much the same as another :-) 

Gary

Well, you should see any difference!  When you hoover over over the Steel Series wind gauge you will notice 10 min wind, gust (and bearing) averages.

I think the values of weewx-WD in customclientraw.txt are not 10 min averages.

Luc

[gjr80]

It was not until just now that I noticed the (10min) after the the 'Average wind speed' on the hover, most everywhere else the term 'average' is used with no time qualifier, in Weewx-WD we took that to be average over the archive period, so all 'average' winds are over the archive period (unless they are the day average). Clearly the average wind speed needs to be fixed. Further compounding the issue is when generating a customclientraw as a Weewx report you are essentially tied to archive data with the only loop input being the extent to which Weewx updates its accumulators with loop data. Not such a big issue per se, but the real issue from this is one of time granularity where, with a 5 minute archive period, you can be (worst case) 5 to 6 or even 7 minutes behind with your data which is a significant period compared to a 10 minute window for an observation that is as variable as wind.

At least the approaches here all make direct use of loop data and generate customclientraw or equivalent with much more time granularity.

Gary

[L.J.M. Heijst]

The dimensions of Steel Series gauges are all not that clear. Look at the temperature and pressure hovers. The presented plus/minus trend values are values per hour, so again weewx-WD differs from what SS expects. BTW I noticed the 10 min average fields in the language file where the parameter names have '10min' in their name field.

Luc

Verstuurd vanaf mijn iPad

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/NSxqel5kbZU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[gjr80]

Agree temptrend is per hour, mistake in our skin.conf which should have 'time_delta= 3600' or 1 hour not the default 3 hours. Beg to differ on pressure though, display is a per hour figure but if you look at the steel series code in gauges.js at lines 1662-1663 you will see that when working with Weather Display, Steel Series expects pressure as a 3 hour trend. It reduces it to a per hour value if your are using Weather Display - when used with Weewx-WD as far as Steel Series is concerned it is being driven by Weather Display. If you are telling Steel Series that you are using some other weather software (via 'weatherProgram' setting in gauges.js) then, yes you should be using a 1 hour pressure trend.

I work in English so had no need to look at language files :-)

Gary

[L.J.M. Heijst]

Most of the time I work in English too. But to know the right Dutch terms for wind chill, apparent temperature and heat index I searched all over the internet and thanks to the English terms I now know they are called gevoelstemperatuur, warmtegevoel and warmte index respectively!

Luc

Verstuurd vanaf mijn iPad

--

[Lloyd Adams]

Luc,

My php file is simply requested by the steel gauges java script every 15 seconds, rather than the normal approach of requesting customclientdraw.txt.  This effectively generates the same response as when requesting customclientdraw.txt, but the output is generated dynamically, on the server.  So when the gauges js makes the request, the php file will run sql queries against the raw database for the rapidly changing data. So, for example, I can determine average wind over the last 10 minutes, max/min since midnight, windrose data over the past 24 hours etc.

The use of the raw table works really well for me, because I collect from each sensor individually when they transmit, so I have all readings to query over. My only limit is the slow update of some sensors (and does this really matter, its only wind that is truly dynamic?).

[Lloyd Adams]

when I implemented mine I struggled to find over what periods the trends etc. should work, so I ended up with average wind direction over 10 minutes, and the wind rose over 24 hours.  Temperature and pressure trends are both over 60 minutes. I use the Cumulus setting in weatherProgram.

[Luc Heijst]

On Sunday, 29 June 2014 06:58:23 UTC-3, Lloyd Adams wrote:

Luc,

My php file is simply requested by the steel gauges java script every 15 seconds, rather than the normal approach of requesting customclientdraw.txt.  This effectively generates the same response as when requesting customclientdraw.txt, but the output is generated dynamically, on the server.  So when the gauges js makes the request, the php file will run sql queries against the raw database for the rapidly changing data. So, for example, I can determine average wind over the last 10 minutes, max/min since midnight, windrose data over the past 24 hours etc.

The use of the raw table works really well for me, because I collect from each sensor individually when they transmit, so I have all readings to query over. My only limit is the slow update of some sensors (and does this really matter, its only wind that is truly dynamic?).

Lloyd,

I'm really interested  in your code. Especially the data base queries. Would you mind sharing this with us? I could not find a good syntax for getting both max/min values with their timestamps and the code for generating the windrose data. I guess the sqlite queries don't differ much from the mysql ones.

Luc

[Lloyd Adams]

I'll try and post the code tomorrow. I don't have access to my pc at the moment.

[Lloyd Adams]

This is the template file.  A few words of warning - I never polish my code, and I know very little php - so it could be terribly inefficient.  Anyway, any questions, please feel free to ask.

[Luc Heijst]

On Monday, 30 June 2014 16:55:23 UTC-3, Lloyd Adams wrote:

This is the template file.  A few words of warning - I never polish my code, and I know very little php - so it could be terribly inefficient.  Anyway, any questions, please feel free to ask.

Thanks Lloyd,

I got the idea how it's working. I had to change some code and db queries first before it worked without (fatal) errors; see attached sg_json.php.tmpl.

When the php file is called with arguments "?entity_id=weewx_raw&data=dateTime" I got the following data on my screen:

{"date":"05:38", "temp":"24.1", "wlatest":"0", "intemp":"24.1", "hum":"86", "bearing":"90", "wgust":"0", "press":"1015.85029615", "inhum":"86", "UV":" N/A", "heatindex":"24.1", "wchill":"24.1", "dew":"21.6058208395", "tempTL":"24.0", "tempTH":"24.3", "humTL":"86", "humTH":"87", "pressTL":"1016.1", "pressTH":"1016.4", "dewpointTL":"21.7", "dewpointTH":"21.9", "wchillTL":"24.0", "wgustTM":"6.48", "TwgustTM":"01:00", "bearingTM":"N/A", "wspeed":"0", "rfall":"0", "apptemp":"13.9", "apptempTL":"8.8", "apptempTH":"18.9", "heatindexTH":"24.3", "humidex":"0", "avgbearing":"0", "pressL":"1007.0", "pressH":"1019.7", "rrate":"0", "rrateTM":"0", "SensorContactLost":"0", "forecast":"0", "tempunit":"F", "windunit":"m/s", "pressunit":"inHg", "rainunit":"in", "temptrend":"24.1", "presstrendval":"1015.85029615", "TtempTL":"00:00", "TtempTH":"00:27", "TdewpointTL":"00:00", "TdewpointTH":"00:15", "TapptempTL":"00:00", "TapptempTH":"00:27", "TwchillTL":"00:00", "TheatindexTH":"00:27", "TrrateTM":"00:00", "ThourlyrainTH":"00:01", "LastRainTipISO":"00:01", "hourlyrainTH":"00:01", "ThumTL":"00:12", "ThumTH":"00:00", "TpressTL":"00:25", "TpressTH":"00:00", "Tbeaufort":"00:00", "windTM":"00:02", "timeUTC":"05:38 on 01 Jul 2014", "BearingRangeFrom10":"0", "BearingRangeTo10":"0", "UVTH":" N/A", "SolarRad":"0", "SolarTM":"00:00", "CurrentSolarMax":"0", "domwinddir":"0", "WindRoseData":[302.04000000000013,389.88000000000017,501.8400000000003,2740.680000000005,3529.4399999999982,2354.7599999999993,1524.6000000000013,1369.0799999999997,1050.1200000000078,732.2400000000026,235.08000000000013,70.2,98.99999999999996,81.35999999999999,139.31999999999994,245.8800000000004], "windrun":"0", "version":"2.6.2", "build":"2014-04-11", "ver":"11"}

Note: The time stamps of "date" and "timeUTC" are not correct yet. (The weather station has time-zone: UTC-3 and the web server: UTC+2.)

I plan to create all steelseries data on the web server.with queries on the local archive and raw databases  If done so, there will be no further need for a template for sg_json.php or the SteelGauges skin of Weewx-WD.

Luc

[Lloyd Adams]

Hi Luc

Glad you got it working.  I did try and do away with the template file, and query the two databases (I think there are some remnants left in the file), but I see to remember having an issue, and being concerned with how long the archive data might be locked for (and hence prevent weewx from writing to it).

Lloyd

[Luc Heijst]

On Tuesday, 1 July 2014 09:20:25 UTC-3, Lloyd Adams wrote:

Hi Luc

Glad you got it working.  I did try and do away with the template file, and query the two databases (I think there are some remnants left in the file), but I see to remember having an issue, and being concerned with how long the archive data might be locked for (and hence prevent weewx from writing to it).

Lloyd

Hi Lloyd,

I have version 0.2 of sg_json.php now operational. The php file itself is no longer changed. Right now I have not all queries operational. A few values are used from the customclientraw.txt file which weewx-WD generates each archive loop.

It's just a matter of writing the queries for the missing real-time data and then we don't need a real-time customclientraw.txt. I still might use a fixed-values 'customclientraw.txt' file for values which never change and to preset the order of the json outputs. Note the json data is filled by the queries in a 'random' order and the one line command at the end of sg_json.php outputs all json data to the screen in the same order as in customclientraw.txt.

Note the code for filling the WindRoseData. For smaller periods (you had no 'WHERE dateTime' clause), like 3 hours, your code gave nill values instead of zeros. Also the non nill-values could be (and mostly were) not written to the right wind sectors. See the attachment for my solution.

BTW. I've one query on archive (all time barometer low and high values) all other queries are done on raw data.

I do not know if there is a 'database (dead?) lock danger'. How do we test if a lock occurs and/or data is missed?

Luc 

[Lloyd Adams]

Hi Luc

Have you timed how long your queries are taking?  We have very different hardware, but would still be interested to know how long it takes.

My wind rose is intentionally over 24hrs, and as the raw database only holds 24hrs worth of data, then this query does not need a WHERE clause.   I'll take a look at the null values.

Lloyd

[Luc Heijst]

On Friday, 4 July 2014 08:42:44 UTC-3, Lloyd Adams wrote:

Hi Luc

Have you timed how long your queries are taking?  We have very different hardware, but would still be interested to know how long it takes.

Lloyd

Lloyd,

I have all queries now operational in my "mesoWx-ss.php"  version 0.3 (formerly called sg_json.php).

Total execution time (except for the json output) is typically between 0,1 and 0.3 s.

I have written the execution time on the version / build info line. Note the value is only updated after a refresh of the screen. see: http://www.lucdesign.nl/gauges-ss/gauges-ss.htm

Luc

[Luc Heijst]

Hi Lloyd

Just noticed the meso sync's stopped yesterday at 20:10 local time.. The log history is already overwritten, so I don't know the reason.

Started weewx again which triggers again the meso sync's. 

The database queries are now as low as 0.023 seconds...

Luc

[Luc Heijst]

On Friday, 4 July 2014 11:17:32 UTC-3, Luc Heijst wrote:

Hi Lloyd

Just noticed the meso sync's stopped yesterday at 20:10 local time.. The log history is already overwritten, so I don't know the reason.

I changed the number of saved log files from 4 to 99. With 'all debug loggging' set, good for two days history. 

The good news is that so far the meso sync's didn't stop.

I have now two programs running who produce steelseries data.

1. customclientrawgauges.txt.

2. customclientrawsync.txt

The first one is triggered by script gauges.js of the standard steelseries gauges of Mark Crossley.and produces synchronous data each presentation interval (which is set to:10 s).

The second one is triggered by mesoWX program 'updateData.php' which is executed each weewx loop interval (here: 10 s)

'Why two programs', you may think. Well, I wrote the first one to get a faster update of real data on the steelseries gauges page.

Later I implemented the Saratoga and Leuven templates on my site which use the second (non-synchronous) data..

Luc

.