Belchertown Highcharts not updating

[rich T]

First of all, very nice skin design.  Everything thing is working correctly except the Highcharts are not updating.  I'm just updated 0.8 and still I have the same issues.  One of my conf files must be incorrect, but I can't see where. Attached are the current config files with sys_log.

Thanks

Rich

[Pat]

0.8 isn't released yet so I guess you've become a beta tester. :)

You have 2 options. You can downgrade to the supported 0.7 release, or we can try to figure out what's wrong together.

If you choose the latter, I notice your syslog doesn't show any glaring errors. However, I do see some spacing in your weewx.conf which may be the problem?

Check out this link for a good config example within weewx.conf - (e.g. removing the newline between skin = Belchertown and [[[Extras]]]

Restart weewx when done and see what happens.

[rich T]

Pat

I don't mind being a beta tester.  :)  I cleaned up the extra newlines within weewx.conf file and restarted weewx and allowed to run several archive periods, the charts did not updated.  I rebooted the RPI and allowed to run several archive periods, again the charts did not updated. What I do notice in the sys_log after the files for the Highcharts_Belchertown report are generated, zero files are being copied to the  /home/weewx/public_html/belchertown.    

Nov 25 21:40:40 raspberrypi weewx[546]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 7.62 seconds

Nov 25 21:40:40 raspberrypi weewx[546]: copygenerator: copied 0 files to /home/weewx/public_html/belchertown

In the [[StandardReport]] does the skin need to point Belchertown? Could it point to Highcharts_Belchertown?

Thanks

Rich

[Pat]

I noticed it said it moved 0 files too. Don't know why though - there is no other information. In my standard report I have this below as an example where both skins are copying data to the same html folder. 

I assume you have both skins outlined in weewx.conf similar to the below?

I'm still a little lost since there's not much data to go off of quite yet. 

    [[Belchertown]]
        skin = Belchertown
        HTML_ROOT = /var/www/html/belchertown
        [[[Extras]]]
            logo_image = https://belchertownweather.com/images/content/btownwx-logo-slim.png
            show_apptemp = 0
            show_windrun = 1
            forecast_enabled = 1
            darksky_secret_key = 123
            earthquake_enabled = 1

    [[Highcharts_Belchertown]]
        HTML_ROOT = /var/www/html/belchertown
        skin = Highcharts_Belchertown

[Pat]

I do see that it's trying to copy to /home/weewx/public_html/belchertown which tells me you've installed from the tar file (and not from apt-get). 

Most of the skin should be install agnostic, but maybe there's something getting gummed up with the tar install (I've seen it before). Let me do some testing on a vanilla system with the tar install and see how it goes. 

[rich T]

Pat

Don't know if this will help, but if I change the skin to point to Highcharts_Belchertown and include the extras, everything works except the 8 Day Forecast.

Rich

On Monday, November 26, 2018 at 4:39:25 PM UTC-5, Pat wrote:

[Pat]

Interesting. OK.

Also, the 8 day forecast is part of darksky, not highcharts. Does that work when Belchertown is the skin?

[rich T]

Darksky updates once an hour.

[gjr80]

Whoops, premature posting...

Be careful when interpreting 'how many files are copied'. The line

Nov 25 17:35:44 raspberrypi weewx[1619]: copygenerator: copied 0 files to /home/weewx/public_html/belchertown

is the result of the copy generator being run. The copy generator is frequently used to copy css, js or other type files usually not the files produced by the report generator. If you are properly using HTML_ROOT and other path related tools there is no need for copy generator to copy a generated report as the report generator will save the generated file to the required destination directory. Typically the copy generator copies files either once only on the first report cycle or on every report cycle. When files are copied and what files are copied is controlled by the [CopyGenerator] stanza in skin.conf. Looking at the Highcharts_Belchertown skin config file [CopyGenreator] stanza is set thus:

[CopyGenerator]
   
    # This section is used by the generator CopyGenerator

    # List of files to be copied only the first time the generator runs
    copy_once = favicon.ico, images/*, json/index.html, js/index.html, js/responsive-menu.js
   
    # List of files to be copied each time the generator runs
    copy_always = style.css

Since none of the copy_once or copy_always files/patterns exist for that skin no files will ever be copied. So seeing the log entry above for the Highcharts_Belchertown skin is exactly what is expected and is not an indicator of correct or otherwise oepration.

Gary

[Colin Larsen]

Or sooner if you set it to :)

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pat]

I think you've lost me a bit on this one. 

Your statement is true but the highcharts skin generates 4 json files on every archive interval. Which should be copied every time it runs I'd gather? This is the first I'm hearing of the skin ignoring the generated files?

The copy_once, and copy_always files are invalid for the Belchertown highcharts skins and I never cleaned them up when I split the skin into 2 parts. I'll do that shortly. 

[gjr80]

On Tuesday, 27 November 2018 11:17:49 UTC+10, Pat wrote:

I think you've lost me a bit on this one. 

Your statement is true but the highcharts skin generates 4 json files on every archive interval. Which should be copied every time it runs I'd gather?

No. I think we need to be clear here on what each of the generators do. The report generator generates one or more files, in this case 4 .json files,  in HTML_ROOT (of course if you happen to have .tmpl files in a directory under your skin then the respective generated files will appear in that same directory under HTML_ROOT). So if you set your HTML_ROOT to the final destination you want the generated files to appear then once the report generator is complete the generated files are there exactly where you want them. The copy generator normally runs after the file generator (depends on the generator order in skin.conf) and copies files as per its config settings (once/always and file spec). You could of course have the report generator put the file(s) in say /var/tmp and then use copy generator to copy them from /var/tmp to your final destination but why bother double processing.

 

This is the first I'm hearing of the skin ignoring the generated files?

Rich remarked on there being no files copied implying this meant that something was amiss. I was trying to point out that this is quite normal and expected behaviour for the Belchertown skins, it is not an indicator that something is wrong. I don't believe that any generator is ignoring any generated files based on the log extract provided.

The copy_once, and copy_always files are invalid for the Belchertown highcharts skins and I never cleaned them up when I split the skin into 2 parts. I'll do that shortly. 

Suspected this was the case but since none of the file specs are met the copy generator was not doing anything anyway, its just a few extra characters in a file and a few extra machine cycles taken to run copy generator.

Gary

[Pat]

Ok your explanation is my understanding as well. Understood on Rich's report of 0 files copied - which is why I'm lost as to why it's failing since it should have 4 files to copy. The Belchertown highcharts skin should create new json files every archive interval, which should be a fresh copy to the HTML_DIR. 

Unless there's a typo in a config, or something else, that's preventing that from happening. My comment of "first I'm hearing of it" is in regards to the 3 or so beta testers of 0.8 who haven't reported this issue. 

My next step is to setup a like-for-like environment as best I can (I have one already just need to get it updated) and see what happens. Then go from there. 

[gjr80]

On Tuesday, 27 November 2018 12:09:05 UTC+10, Pat wrote:

Ok your explanation is my understanding as well. Understood on Rich's report of 0 files copied - which is why I'm lost as to why it's failing since it should have 4 files to copy. The Belchertown highcharts skin should create new json files every archive interval, which should be a fresh copy to the HTML_DIR.

Now you are confusing me. You say 'it should have 4 files to copy', that implies to me you are expecting the report generator to generate 4 files then you are expecting them to be copied somewhere. Given the conversation centres on the copy generator I take it you are expecting the copy generator to copy these 4 .json files. The 4 files are in fact generated by the report generator, this is evidenced by:

Nov 25 17:35:44 raspberrypi weewx[1619]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 6.84 seconds

but the copy generator will never copy these files, the copy generator file specs in Highcharts_Belchertown skin config file do not match *.json files and as I said earlier a properly configured skin does not need such files to be copied.

I'm sorry if this seems to be pedantic but when you start talking about copying files and copy generators etc you need to quite specific.

 

Unless there's a typo in a config, or something else, that's preventing that from happening. My comment of "first I'm hearing of it" is in regards to the 3 or so beta testers of 0.8 who haven't reported this issue. 

In this case I think a bit of sleuthing might be needed on Rich' system to ensure the generated files content looks correct (do they have data), are saved (not copied) to the correct directory and then suitably made available (copied or via symlink) to the web server. Then make sure the JS is looking in the correct location for the .json files.

 

Gary

[Pat]

Gary, understood on all counts. My belief is that the copy generator will copy the newly generated files by the report generator. In this case, it's the 4 json files. This is how it's working in Belchertown 0.7, as well as in 0.8rc1,2,3. I don't think I need to explicitly specify copy_always = *.json. If this were the case, then the Standard skin would have copy_always = *.html - which is does not. Further, my skin would have been broken since 0.1. 

Aside from all of that, I did perform an install with the install.py, and my syslog did return an error of a missing file. It doesn't seem to be related, but still something I just fixed in a recent commit. 

Back to this issue: From an install using setup.py, I have installed the Belchertown skin using /home/weewx/bin/wee_extension --install. My weewx.conf is below:

    [[Highcharts_Belchertown]]
        HTML_ROOT = public_html/belchertown
        skin = Highcharts_Belchertown
    [[Belchertown]]
        HTML_ROOT = public_html/belchertown
        skin = Belchertown

Upon issuing a /home/weewx/bin/wee_reports, I get the following:

Nov 27 14:32:09 setuppy wee_reports[1473]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 2.52 seconds
Nov 27 14:32:09 setuppy wee_reports[1473]: copygenerator: copied 0 files to /home/weewx/public_html/belchertown

However, within /home/weewx/public_html/belchertown/json I have 4 json files with a timestamp of 14:32. So it seems it IS copying them, but it's saying it is NOT. 

The charts on the website are updated as well. 

And again at 9:40 (note my system is in UTC, so it shows 14:40):

Nov 27 14:40:18 setuppy weewx[1689]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 2.70 seconds
Nov 27 14:40:18 setuppy weewx[1689]: copygenerator: copied 0 files to /home/weewx/public_html/belchertown

root@setuppy:/home/weewx# ls -al /home/weewx/public_html/belchertown/json
total 3320
drwxr-xr-x  2 root root    4096 Nov 27 14:40 .
drwxr-xr-x 11 root root    4096 Nov 27 14:40 ..
-rw-r--r--  1 root root    4150 Nov 27 14:40 day.json
-rw-r--r--  1 root root    1035 Nov 27 14:32 earthquake.json
-rw-r--r--  1 root root      14 Nov 27 14:40 index.html
-rw-r--r--  1 root root    1961 Nov 27 14:40 month.json
-rw-r--r--  1 root root    2157 Nov 27 14:40 week.json
-rw-r--r--  1 root root    9995 Nov 27 14:40 weewx_data.json
-rw-r--r--  1 root root 3353388 Nov 27 14:40 year.json

And the charts have updated as well for 9:40. 

At this time I don't see anything wrong with Belchertown. Maybe a slight cosmetic bug with the output of the copygenerator?

Rich, when you see the archive interval run and it says 0 files copied, please confirm the timestamps on the files in bold above. Do they update?

If you have MQTT disabled on your Belchertown skin, you will need to refresh the page to see the updated charts. 

Let me know what happens. 

[Andrew Milner]

Pat

If I am understanding correctly it is the report generator which is putting the .json files in the correct place - not the copy generator.

[rich T]

Pat

Will check later this evening, when I get back home.

Rich

[Pat]

Andrew, you may be right - and that's why I mentioned I was confused earlier - quite a few generators that do different things. :) 

Still - I'm not sure where the issue lies with this particular issue yet. I don't think it's a CopyGenerator problem as initially reported. That was what my testing was trying to rule out. 

[rich T]

Pat

This is the WEEWX Config:

 [[Highcharts_Belchertown]]

        HTML_ROOT = public_html/belchertown

        skin = Highcharts_Belchertown

     [[Belchertown]]

        HTML_ROOT = public_html/belchertown

        skin = Belchertown

        #[[[Extras]]]

            #logo_image = ""

            #footer_copyright_text = Weather Site

            #forecast_enabled = 1

            #darksky_secret_key = Secret Key

            #earthquake_enabled = 1

            #mqtt_enabled = 1

            #mqtt_host = http://xxx.xxx.xxx.xxx

            #mqtt_port = 9001

            #mqtt_ssl = 0

            #mqtt_topic = "weather/#"

            #disconnect_live_website_visitor = 0

Generated files @19:05

Nov 27 19:05:31 raspberrypi weewx[1748]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 7.56 seconds

Nov 27 19:05:31 raspberrypi weewx[1748]: copygenerator: copied 0 files to /home/weewx/public_html/belchertown

Copied Files @ 19:05

pi@raspberrypi:/home/weewx $ ls -al /home/weewx/public_html/belchertown/json

total 516

drwxr-xr-x  2 root root   4096 Nov 27 19:05 .

drwxr-xr-x 11 root root   4096 Nov 27 19:05 ..

-rw-r--r--  1 root root  30706 Nov 27 18:51 darksky_forecast.json

-rw-r--r--  1 root root  72665 Nov 27 19:05 day.json

-rw-r--r--  1 root root   1037 Nov 27 18:52 earthquake.json

-rw-r--r--  1 root root     14 Nov 27 19:05 index.html

-rw-r--r--  1 root root 117604 Nov 27 19:05 month.json

-rw-r--r--  1 root root 155755 Nov 27 19:05 week.json

-rw-r--r--  1 root root   8056 Nov 27 19:05 weewx_data.json

-rw-r--r--  1 root root 116287 Nov 27 19:05 year.json

Cleared browser cache, closed and reopened browser.  The charts did not updated.

Rich

On Tuesday, November 27, 2018 at 9:44:52 AM UTC-5, Pat wrote:

[Pat]

Ok so the files are updated correctly. I'm thinking its a JavaScript load error now. 

Are the graphs blank or old data? 

Is your site public? 

In weewx, did you define the site_url option? The site_url option needs to match your website URL exactly. e.g. http://richt/weewx/belchertown or something. You can also keep site_url disabled, and use belchertown_root_url in place of site_url.

[rich T]

Pat

The charts contain day old data and for now this site is only on my local LAN. Currently, the site_url option is "belchertown_root_url = http://192.168.XXX.XXX/weewx". Let me try pointing to the belchertown folder and see what happens.

Rich

[Pat]

If you view the skin at http://192.168.XXX.XXX/weewx/belchertown, then yes, make the belchertown_root_url the same url - without the trailing slash. 

[rich T]

I can view the site @ http://192.168.XXX.XXX/weewx or http://192.168.XXX.XXX/weewx/belchertown and get the same results.  Now staring at the site, I can see it just rewrites the old data. I like I said before if I point to the Highcharts_Belchertown, the charts update every 5 minutes.

[Pat]

I still think this is a root URL issue and duplicated data since you have effectively two locations for belchertown. Let's try and clean it up, then regenerate from scratch:

Validation:

1. Stop weewx. Keep it stopped until you go through the whole process. 

2. Delete the day/week/month/year.json files from /home/weewx/public_html/json and from /home/weewx/public_html/belchertown/json

3. Run sudo /home/weewx/bin/wee_reports

4. Check your website. Are the graphs updated or are they blank?

If they are updated, then we're done. But my guess is they are blank, or old based on my findings which I outline below. 

Continuing on: 

1. Check your json folder again to see if the files timestamp are from the latest wee_reports run... ls -al /home/weewx/public_html/belchertown/json

2. Confirm there is nothing in /home/weewx/public_html/json

3. If these are the new files from wee_reports, then highcharts needs to look for this data in http://192.168.xxx.xxx/weewx/belchertown/json which is probably not what you want and why you're seeing strange things. 

Diving a little deeper: 

1. If you view source on your page, search the source for for "js/highcharts-dayplots.js". Then browse to that full location.

2. At the bottom of that file you'll see "jQuery.getJSON". What is the URL inside the parenthesis? It needs to match the location highcharts is looking for.

Putting it together:

1. If wee_reports is putting files in /home/weewx/public_html/belchertown/json, this would be http://192.168.xxx.xxx/weewx/belchertown/json

2 But if the jQuery.getJSON is looking for the file to be at http://192.168.xxx.xxx/weewx/json then this could be the data mismatch. 

There's quite a few weewx and skin settings that control where the data goes. 

The fix:

If you want http://192.168.xxx.xxx/weewx to be the Belchertown skin, and only the Belchertown skin.:

1. Delete everything in /home/weewx/public_html to give us a clean slate

2. Update weewx.conf where the primary skin is Belchertown (like what you have already), and the HTML_DIR under each Belchertown skin is just "public_html".

3. Make sure the [[[Extras]]] belchertown_root_url is correct for your site. 

    [[StandardReport]]
        skin = Belchertown


    [[Highcharts_Belchertown]]
        HTML_ROOT = public_html
        skin = Highcharts_Belchertown


    [[Belchertown]]
        HTML_ROOT = public_html
        skin = Belchertown
        [[[Extras]]]
           belchertown_root_url = "http://192.168.xxx.xxx/weewx"

3. Run sudo /home/weewx/bin/wee_reports again. The end result should be /home/weewx/public_html resembling the below (note there is NOT a subfolder belchertown).

3a. If you get a bug about belchertown_root_url not being defined - you can ignore it. I just came across it myself and will fix it shortly. 

root@setuppy:~# ll /home/weewx/public_html/
total 124
drwxr-xr-x 11 root root  4096 Nov 28 00:39 ./
drwxr-xr-x  9 root root  4096 Nov 27 14:30 ../
drwxr-xr-x  2 root root  4096 Nov 28 00:39 about/
-rw-r--r--  1 root root  3638 Nov 28 00:39 favicon.ico
drwxr-xr-x  2 root root  4096 Nov 28 00:39 graphs/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 images/
-rw-r--r--  1 root root 33656 Nov 28 00:39 index.html
drwxr-xr-x  2 root root  4096 Nov 28 00:39 js/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 json/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 NOAA/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 pi/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 records/
drwxr-xr-x  2 root root  4096 Nov 28 00:39 reports/
-rw-r--r--  1 root root 40464 Nov 28 00:39 style.css

4. Browsing to your site http://192.168.xxx.xxx/weewx should show everything updated. If your website looks like it's missing the formatting, then make sure the belchertown_root_url is correct in the Extras section.

If all is good, then you can restart weewx. The archive periods should continue to update the JSON files with the updated graphs. Check this upon every archive interval. The Belchertown skin does implement a delay of 30-60 seconds to allow chart processing

If all is bad - you can still restart weewx, but we'll need to dig into it further, however I just replicated everything above on my lab system and am hopeful it'll resolve it. 

[rich T]

Pat

You are correct, there was had duplicated data.  Performed your recommended steps and allowed to run over an hour.  The site now updates as expected.   Just a side note; I did not get any error about belchertown_root_url not being defined.

Thanks for your help

Rich

[Pat]

Great! I'll update the GitHub readme so it's a little clearer when making the skin the default

Thanks for the patience.