Change archive interval for single report

[A.J. Burnett]

Hi everyone!

I'm producing a simple JSON report using weeWX, in addition to all my other reports. I'd like to keep the interval I'm currently using for everything, but I want to fire off the JSON report every 5-10 seconds, if possible. Can anyone recommend where I can achieve this, if it's even possible?

Thank you!

[gjr80]

Hi,

WeeWX reports can be run at multiples of your archive interval but they cannot be run any more frequent than the archive interval. If you want weeWX to do something more frequent than the archive interval then you need write a custom service to do what you want (in your case generate some output).

I am not sure of your understanding of weeWX' inner workings or your python knowledge so my apologies if I am explaining what you already know. It is fairly trivial to create a custom service that is tied to the 'loop interval' of your station (refer to Loop packets in the Customization Guide if you don't know about loop packets and their interval). When you write a custom service you lose a lot of the conveneince functions that come with the weeWX report engine (the ability to use tags and the associated aggregation, formatting and conversion abilities) so you need to do a lot of the data gathering, processing, formatting and output yourself. If you have a basic understanding of python you should be able to work your way through some examples and ultimately create your own. A lot of the functions you need to access data within weeWX already exist in the weeWX API, its mainly the working out what you want, producing your output and handling the corner cases (ie dealing with None values) that take up your time.

If you are still keen, Customizing the weeWX service engine in the Customization Guide has some basic examples. One you get a feel for things you can dive into some more advanced services that do a similar thing to what you want to do. The crt extension implements a loop based service that does something similar to what you want to do. Also the realtime gauge-data extension does a similar thing to the crt extension but it produces a JSON format file to drive the SteelSeries weather gauges.

Hope that helps and don't be afraid to ask if you don't understand.

Gary

[A.J. Burnett]

Hi Gary,

Thanks so much for your response! I'll admit, I'm very lost.

I have done some minor customization to templates and have some basic Python knowledge, but I'm not understanding the steps in your very detailed message (thanks again!)

Perhaps harnessing the loop packets is the way to go, since I'd love to display the data in a "rapid-fire" manner, similar to Wunderground's very frequent updates.

How do I go about accessing those and spitting it into JSON format?

A.J.

[A.J. Burnett]

UPDATE:

I stumbled across SteelSeries Gauges and saw that it produced a great JSON file (and has some cool gauges to go along with it too!). So I have a JSON file now, but I want SteelSeries to update every 2.5 seconds (near real-time). I can't find that interval anywhere...right now, it appears to be every 10 minutes.

Any ideas on how to get the 2.5-second interval going?

A.J.

[Andrew Milner]

Loop interval depends on your station type AFAIK.

Is 10 minutes your archive interval?

[A.J. Burnett]

Hi Andrew,

Station type is a VP2. My StdArchive interval is set to 300, which should be every 5 minutes, but the gauge-data.txt file only updates every 10 minutes. My web page only updates at that same 10-minute interval too. <scratching head>

[gjr80]

A.J.,

Exactly what did you stumble across as there is a skin and an extension that can produce the gauge-data.txt file used to drive the SteelSeries Weather Gauges. Firstly there is the SteelSeries skin that generates gauge-data.txt via a weeWX report and hence gauge-data.txt is generated every report cycle, eg every 5 minutes. Secondly, there is the Realtime gauge-data extension for weeWX that I linked in my first post, this extension produces a loop based  version of gauge-data.txt. The period between generation of this loop based file can be controlled using the min_interval config option (refer the the comments at the front of file rtgd.py included in the extension package).

[gjr80]

If your pages are updating at 10 minute intervals that sounds like the archive interval being used is 10 minutes (if you are using the SteelSeries skin that would further support this idea). With a Davis station any archive interval setting in the console will override the [StdArchive] archive_interval setting. To confirm what is happening go back and look at your log from when you last started weeWX, you will probably see something like this in the start up logging:

Feb 13 13:45:18 stretch22 weewx[1589]: engine: The archive interval in the configuration file (300) does not match the station hardware interval (600).
Feb 13 13:45:18 stretch22 weewx[1589]: engine: Using archive interval of 600 seconds

If you can't find it stop then start weeWX and have a look at the startup log.

To fix it, stop weeWX and then use wee_device to set the desired archive interval in the console.

Note, the only effect this will have on gauge-dat.txt is to generate it every 5 minutes rather than every 10 minutes. Refer to my previous post.

Gary

[A.J. Burnett]

So I've managed to get the JSON file to update every minute, but would love to get it every 2.5 seconds. I installed the SGRT skin, but still only get the file every minute. What should I be doing now?

Thank you all for all your help!

A.J.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TEahPOTTAVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Burnett Media Group "Giving you a voice." E-mail: a...@AJBurnettOnline.net Web: www.AJBurnettOnline.com Phone: 978-595-2329

[gjr80]

I think it is time you post weewx.conf and a log extract so we can see what is installed and what is going on (I have not heard of the 'SRGT skin'). Please post a copy of weewx.conf making sure you remove any sensitive info, passwords etc. Also, set debug=1 in weewx.conf, save weewx.conf, restart weeWX and let it run for 10 minutes. Take an extract/copy of your weeWX log (nominally /var/log/syslog) from when you restarted weeWX to the point 10 minutes after the restart and post the extract here.

Gary

[A.J. Burnett]

Sorry, meant RTSG (Real-time Steel Guage).

On Tue, Feb 13, 2018 at 6:30 PM, gjr80 <gjrod...@gmail.com> wrote:

I think it is time you post weewx.conf and a log extract so we can see what is installed and what is going on (I have not heard of the 'SRGT skin'). Please post a copy of weewx.conf making sure you remove any sensitive info, passwords etc. Also, set debug=1 in weewx.conf, save weewx.conf, restart weeWX and let it run for 10 minutes. Take an extract/copy of your weeWX log (nominally /var/log/syslog) from when you restarted weeWX to the point 10 minutes after the restart and post the extract here.

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TEahPOTTAVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gjr80]

As far as I am aware, and as mentioned in this post, there is the SteelSeries Gauges skin and the Realtime gauge-data extension. The former updates each archive period (not realtime), the latter updates each loop period (near realtime). No matter, if you post your sanitised config file and log extract it will show exactly what is installed, how it is configured and what needs to be done to get what you want.

Gary

On Wednesday, 14 February 2018 09:32:16 UTC+10, A.J. Burnett wrote:

Sorry, meant RTSG (Real-time Steel Guage).

On Tue, Feb 13, 2018 at 6:30 PM, gjr80 <gjrod...@gmail.com> wrote:

I think it is time you post weewx.conf and a log extract so we can see what is installed and what is going on (I have not heard of the 'SRGT skin'). Please post a copy of weewx.conf making sure you remove any sensitive info, passwords etc. Also, set debug=1 in weewx.conf, save weewx.conf, restart weeWX and let it run for 10 minutes. Take an extract/copy of your weeWX log (nominally /var/log/syslog) from when you restarted weeWX to the point 10 minutes after the restart and post the extract here.

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TEahPOTTAVI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[A.J. Burnett]

OK...I've installed both the SteelGauge skin and the Realtime gauge-data extension, but now, my "gauge-data.txt" file is not being updated. I'm attaching my weewx.conf, log, and gauges.js files...maybe you'll spot something I missed?

[gjr80]

Installing both has complicated matters a little; you now have two instances of gauge-data.txt being generated, one by the SteelSeries Gauges skin and one by the Realtime gauge-data.txt extension. The upshot of this is you/we need to pay careful attention to paths or you will get the two confused and we will end up chasing our tail. No matter lets work through each and see what is going on. Firstly, in terms of the syslog extract, I am going to ignore everything before Feb 14 16:47:06, there is just so many loop packet errors, config errors and ftpupload errors that make it impossible to decipher. I trust Feb 14 16:47:06 was the last time you stopped/started weeWX after installing both the skin and the extension.

Let's look at the SteelSeries Gauges skin (aka 'the skin'). If installed correctly it should have created a directory /home/weewx/public_html/ss on your weeWX machine and it should have written a number of files to that directory, one should be gauge-data.txt and it should be being updated every 1 minute given that your archive interval is 1 minute. Is /home/weewx/public_html/ss/gauge-data.txt being updated every minute? If so that suggests the skin is working correctly. Related to the skin but not part of it is the FTP report that is uploading files to your web server/host via FTP. If the /home/weewx/public_html/ss directory exists and gauge-data.txt is in this directory then the directory and gauge-data.txt should be being uploaded to you web host every 1 minute. Right now (12:08am 15 February your time) I can see your site and it appears to be updating fine. If I goto your SteelSeries Gauges page I can see your gauges but they are not displaying any data (no matter, will fix that later). This tells me the ss directory has been created on your web server. I should be able to view the gauge-data.txt that was generated by the skin at http://www.ajburnettonline.com/weather/ss/gauge-data.txt but when I look at this file it is dated 23:47 13 February 2018, so either gauge-data.txt is not being uploaded by FTP or it is not being generated by the skin.

When I look at your log extract throughout it I see various FTP errors, mostly timeouts but a number of permission errors as well. I don't know whether this is the cause, you appear to have had successful FTP uploads since 23:47 13 February 2018.

I said we would come back to the SteelSeries Gauges not displaying data. In ss/scripts/gauges.js there is a setting realTimeUrlWeewx (normally line 74) that tells the SteelSeries Gauges where on your web server (not necessarily weeWX server) to find gauge-data.txt. You presently have it set to /home/weewx/public_html/gauge-data.txt which is the location on your weeWX server and almost certainly not the location on you web server. Using relative paths is usually best, in this case the path is relative to the ss/index.html so just setting it to 'gauge-data.txt' should work. Now to make the change don't edit ss/gauges.js on your web server, it will be overwritten next time you start weeWX, but rather make the change on your weeWX server in the skins directory. Edit /home/weewx/skins/ss/scripts/gauges.js and make the change. Save the file then stop then start weeWX. The next FTP upload should see the updated gauges.js copied to your web server (monitor you log file to see that the FTP upload worked).

Now the Realtime gauge-data extension. There are no errors in the log so I would assume the extension is producing gauge-data.txt, to confirm have a look in the /home/weewx/public_html directory. Is there a gauge-data.txt there and is it being updated every loop period (should be roughly every 2.5 seconds for a Vantage station). If it is there and being updated have a look at the contents, does it appear correct (it will be somewhat cryptic but you should be able to make sense of it). If the file is not there we will need to have a look at a debug log to see what is going on. To get a debug log edit weewx.conf, set debug = 2, save weewx.conf then stop then start weeWX. Let weeWX run for a few minutes then take a log extract from when you just started weeWX until the few minutes were up and post the log extract. That should tell us what is or is not going on. Once we know that part of the extension is working we will worry about getting the data to your web server.

Gary

[WxManAJB]

So, I’ve got data being produced, but getting that near real time gauge-data.txt file to my server at a rapid fire interval is proving elusive. I’ve tried getting the rsync skin running by producing a key pair for ssh and using those credentials in the skin setting of Weewx.conf, but it’s not sending anything and nothing is showing up in my log reflecting an attempt to do so.

Any ideas?

Thanks again for all your help!

A.J.

[Liz]

have you tried rsync server and client setup?
the machines each have a file which includes the password secrets and
then rsync can copy without the credentials being in any open file.

There are some tutorials on the web
of which one example is https://everythinglinux.org/rsync/

Liz

[gjr80]

Sorry this has taken a couple of days, unfortunately there is more to life than weeWX :)

Some background for others who stumble on this.

If you are using the SteelSeries Gauges skin to produce gauge-data.txt then you can use the normal weeWX FTP or Rsync reports to upload gauge-data.txt; gauge-data.txt is only generated once per archive interval so there is no need to upload it any more freuently. If you are using the Realtime gauge-data (RTGD) extension which can generate gauge-data.txt as often as once per loop period, the weeWX FTP and Rsync reports can still be used but they will still only upload gauge-data.txt once per archive interval. If your web server is the same machine as your weeWX server the issue can be addressed by having the RTGD extension save gauge-data.txt to an approparite directory accessible by the web server. If your web server is remote to your weeWX machine one approach is to have the RTGD extension transfer gauge-data.txt to your web server via HTTP POST. You could also conceivably have some external method of transferring the file by FTP or RSYNC; CRON based transfers are of limited use as CRON has a 1 minute 'resolution'. WeeWX has no builtin mechanism for transferring files every loop period; it only has the report based transfer reports that work on an archive interval period.

To have the RTGD extension transfer gauge-data.txt to your web server via HTTP POST refer to the section Using HTTP POST to transfer gauge-data.txt to a remote server in the RTGD extension User's Guide. Bear in mind this was wrritten about 8 months agao, I have reviewed it quickly thismorning and believe it is still correct, though if you find any issues please let me know.

Gary

[A.J. Burnett]

Hi Gary...

I never did THANK YOU for all your help with this issue from months ago. I got it all to work very well, thanks to your assistance! It is much appreciated!

I'm working on another station/page for work now and have a similar but different issue that I'm hoping you can assist with.

I have installed the realtime gauge data extension on a RPi. The same Rpi is hosting the intranet page to display the data. I've set up the guages.js script to write to the directory I need gauge-data.txt to go to, but the file is not making it there.

weewx root = /home/weewx

skin = [weewx root]/skins/ss

HTML root = /var/www/html  // This is where I'm hoping to get the realtime gauge-data.txt file to, every 2.5 seconds or LOOP packet generation.

My syslog is attached as a .txt file (shows the file being generated, but no more info available).

Any thoughts?

Thank you once again!

A.J.

[Andrew Milner]

have you tried to find where it HAS put the file (if anywhere)??

sudo find / -iname "gauge*"

[gjr80]

Hi A.J.,

Glad I could help. The location where gauge-data.txt is saved depends on the rtgd_path config option in the [RealtimeGaugeData] stanza in weewx.conf. If the rtgd_path config option is not set or does not exist then gauge-data.txt will be written to /var/tmp. If rtgd_path is set and is an absolute directory path (ie starts with /) then gauge-data.txt is written to the directory specified by rtgd_path. If rtgd_path is a relative path (ie does not start with /) then gauge-data.txt is written to the rtgd_path directory but relative to HTML_ROOT. The default install will set rtgd_path to /home/weewx/public_html so unless you have changed rtgd_path you should find gauge-data.txt in /home/weewx/public_html. You should be able to have gauge-data.txt written to /var/www/html by setting rtgd_path to /var/www/html.

Having actually sat down to reply to you I realise this is probably a bit more complex than it needs to be. I will have a look at simplifying things a little. I should also add a log entry to give the full path and filename of the file that rtgd creates.

Just as an aside, you mention that gauges.js is set to write to the directory you need gauge-data.txt to go to. I presume you meant read instead of write as gauges.js only reads gauge-data.txt it does not write it.

Gary

[WxManAJB]

Brilliant! That solved all my issues! Much appreciated!