Belchertown skin for weewx now available!

[Pat]

The Belchertown skin for weewx is now available. This skin is modeled after https://belchertownweather.com 

The skin features

• Real time streaming updates on the front page without needing to reload the site (weewx-mqtt extension required)
  
• Forecast data updated every hour without needing to reload (a free DarkSky API key required)
  
• Information on your closest Earthquake updated automatically every 3 hours
• Weather station observation graphs which update without needing to reload
  
• Weather records for the current year, and for all time
  
• A mobile and iPad ready display. You could use an iPad as a "2nd console". 
  

There are dozens of options to configure within the skin. Please view the readme on GitHub for a detailed explanation. 

The GitHub page: https://github.com/poblabs/weewx-belchertown

Direct download of v0.1: https://github.com/poblabs/weewx-belchertown/archive/weewx-belchertown-0.1.tar.gz

Quick install:

wget https://github.com/poblabs/weewx-belchertown/archive/weewx-belchertown-0.1.tar.gz

sudo wee_extension --install weewx-belchertown-0.1.tar.gz

Configure weewx.conf with your options

Restart weewx

Out of the box, it is a very basic theme. Make sure to check out the readme for all options available!

This is my first skin, and I've done all I could to test it out in various environments to iron out any initial bugs. If you do find something that's a little off, or room for improvement, please let me know!

[Colin Larsen]

Brilliant! Will install it today and let you know how it goes.

Many thanks Pat

Colin

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Colin Larsen]

And I bought you coffee 😁

[Pat]

Thanks! Have you got the skin setup?

[Colin Larsen]

I'm down in New Zealand so a few hours out with you. Just created my Darksky account and reading about MQTT - so much to learn! Hopefully in the next few hours but will post my success (or otherwise).

Thanks again - really looking forward to getting this up and running. 

1st question, can Windy.com be set to display wind/cloud rather than the radar image? We don't get that down here in NZ yet

[Pat]

MQTT can be the tricky bit because some of the freebies out there have limitations which your weewx may hit pretty quickly. If you're thinking of trying to host your own MQTT, I'm working on a write up for how mine is setup.

I chose windy.com because it's free and you can do a fair amount of customization. Except for the map color - gray is the only option... Looks like you can only have wind, or cloud. The cloud option looks like it may show both actually. 

Change the lat/lon for your area, but the bit I changed (and got lucky with) was overlay=wind

https://embed.windy.com/embed2.html?lat=-26.195&lon=138.560&zoom=5&level=surface&overlay=wind&menu=&message=&marker=&calendar=&pressure=&type=map&location=coordinates&detail=&detailLat=-26.195&detailLon=138.560&metricWind=default&metricTemp=default&radarRange=-1

You're not married to windy though, if you find another radar you like, you can use that instead. Just use the full HTML of <img src="http://thesite/radar.gif"> or something and making sure the width and height matches the parameters for an easy swap out. 

[Pat]

Sorry, meant to update that to say overlay=clouds if you wanted clouds. 

[Colin Larsen]

CloudMQTT with 5 connections and 10Kbit/sec might not be enough then? I'll definitely look at setting up my own I think. Might even just whack it on a Pi zero or something to play with

--

[Pat]

Not sure. It might be. Worth a try and see how it goes?

[Colin Larsen]

Ok working in test :)

Couple of minor things with the Darksky forecast;

I need to set the units to metric with kph wind - ?units=ca at the end of the query string, is there a place to do that?

There doesn't appear to be a "windy" icon

Otherwise so far so good. I haven't delved into mqtt just yet.

Cheers

Colin

[Colin Larsen]

www.41south.net nz

[Colin Larsen]

Arrgh, typo - www.41south.net.nz

Also found this on the records page, note the change from metric back to US units

Highest Temperature	21.1 °C	August 19, 2018 at 9:56 AM	August 19, 2018 at 9:56 AM	21.1 °C
Lowest Temperature	-1.1 °C	August 18, 2018 at 4:58 PM	August 18, 2018 at 4:58 PM	-1.1 °C
Highest Heat Index	21.1 °C	August 19, 2018 at 9:56 AM	August 19, 2018 at 9:56 AM	21.1 °C
Lowest Wind Chill	21.1 °C	August 19, 2018 at 9:56 AM	August 19, 2018 at 9:56 AM	August 19, 2018 at 9:56 AM
Largest Daily Temperature Range	40.0 °F	August 19, 2018 (Min: 30.0°F - Max: 70.0°F )	August 19, 2018 (Min: 30.0°F - Max: 70.0°F )	40.0 °F
Smallest Daily Temperature Range	5.9 °F	August 18, 2018 (Min: 30.0°F - Max: 35.9°F )	August 18, 2018 (Min: 30.0°F - Max: 35.9°F )	5.9 °F

[Juan Antonio Mosquera]

Thanks, i will try it!

[Pat]

Most of it looks pretty good. Not bad for my first skin attempt :) 

I knew imperial to metric would be an issue, and I had tried my best to plan for it. I didn't take timezones into consideration with my testing though - for example noticed on your test site that the forecast row has wrong borders and it's showing the weather for yesterday (for your timezone). I'll take a look into that. 

I see my error with the records page conversion, so I'll have to issue an update on that one later today. 

For DarkSky, I'll have to add the "units" flag. They have 6 different unit types, so I'll have to update the setup guide too. Can't believe I missed that one! 

I'll let you all know when I issue an update. Possibly late today (US Eastern time)

[Philip Kutzenco]

Like Colin, I registered my DarkSky account and am reading up on MQTT. I notice though, that on your https:/belchertownweather.com site, the icons you have for sunrise and for sundown show next to readouts for the same time (8:02 PM). AccuWeather.com lists sunrise as 6:06 AM and sunset as 7:37 PM for Belchertown, MA. Am I misreading the meaning of those icons?

[Pat]

Interesting, and good catch. My website is still using my custom code and not the weewx skin. I have been developing the skin on a separate lab system where I can change units and databases without affecting my real data. No idea why my Belchertown site is wrong, will need to look into that after fixed the items from Colin's testing. 

Sunrise and sunset come from the built in weewx calculations.

[Colin Larsen]

All good, thanks Pat 

[Pat]

Just released 0.2. Give it a try when you can. 

https://github.com/poblabs/weewx-belchertown/releases/tag/weewx-belchertown-0.2

[Colin Larsen]

What's the best way to update Pat? 

[Pat O'Brien]

Hopefully all of your configurations are in weewx.conf. If so you can just download it and install it like before with wee_extension. 

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Colin Larsen]

Hmmm, they weren't but the only difference was the Darksky_units I think. I'll fix that later.

Windy icon - fixed

Records change of units - fixed

Darksky - still a problem. Neither "auto" or "ca" seem to work so far. I'm guessing that 'Visibility' also comes from Darksky?

Visibility

3.84 miles

[Colin Larsen]

And found a couple more records issues with changes of units in Rainfall - in -- mm

Highest Daily Rainfall	0.2 in	August 19, 2018	August 19, 2018 at 12:00 AM	0.2 in
Highest Daily Rainfall Rate	0.3 mm	August 19, 2018 at 12:00 AM	August 19, 2018 at 12:00 AM	0.3 mm
Month with Highest Total Rainfall	1.0 in	August	August, 2018	1.0 in
Total Rainfall for	2018	25.4 mm	2018	25.4 mm

[Colin Larsen]

Not sure but I think this line needs to change from

forecast_url = "https://api.darksky.net/forecast/%s/%s,%s?%s"

to

forecast_url = "https://api.darksky.net/forecast/%s/%s,%s?units=%s"

[Colin Larsen]

Have confirmed that this url with my API key returns the right data and timezone - using both auto and ca

https://api.darksky.net/forecast/xxxxxxxxxxxxxxxxxxxxxx/-41.36,174.05?units=ca

[Pat]

100% correct. That's my mistake. You can make that change for now and I'll update it in GitHub for the 0.3 version soon

The visibility comes from DarkSky, and they don't provide the units within the API output, so I I'll have to grab the DarkSky units from the config, and then change the unit within the skin based on that. 

And I'll fix the static unit in the records page. Thought I got them all!

Also, your map is still showing the East Coast of USA. Not sure if that's intended or not. 

[Colin Larsen]

Thanks Pat, I thought I made the change correctly in belchertown.py but it hasn't fixed it - not sure what's up that I missed?

forecast_url = "https://api.darksky.net/forecast/%s/%s,%s?units=%s" % ( darksky_secret_key, latitude, longitude, darksky_units )

It's the one in /usr/share/weewx/user ?

The map is correct here which is strange, it shows NZ front and centre. Is it picking up a location variable from somewhere other than the config co-ordinates

[Pat]

That URL looks right. Using that as a test URL, it's working for me. I've created another lab weewx system in metric and I think everything you've pointed out is looking good now. Give it a test and let me know. 

For the map: Looking at your page source, looks like you've omitted the lat/lon for your windy map. So I am guessing I am seeing the east coast because of a saved windy cookie. If you go back to this message, you can see the lat/lon in the URL... update that and add that to your config. Probably don't need the detailLat/detailLon. 

v0.3 is available to download. https://github.com/poblabs/weewx-belchertown/releases/tag/weewx-belchertown-0.3

[Colin Larsen]

Darksky is stubborn, still won't use the units.

There is no lat/lon in the Windy url, just location=coordinates so I've taken that out and hard coded the NZ lat/lon, see if that helps the map

Whoops ......

Visibility

5.44 $visibility_unit .

[Colin Larsen]

Forget the visibility tag problem, it's sorted :)

[vk3...@gmail.com]

I just tried the Colin's site and the map is centred on my lat/long (Melbourne, Australia).

Looking at the web page source, I can see it passing the Wellington lat/long to windy.com so it must be picking my location up from somewhere else.

Susan

[Colin Larsen]

Thanks for the info Susan.

Good news - Darksky has updated properly!!!! Just a minor thing that the wind speed is mph below each forecast icon

Progress, nearly there, great work Pat, I'm loving this skin :)

--

[Colin Larsen]

I used the Windy embed widget and now have this in radar_html space

<iframe width="650" height="360" src="https://embed.windy.com/embed2.html?lat=-41.179&lon=172.441&zoom=5&level=surface&overlay=clouds&menu=&message=&marker=&calendar=12&pressure=&type=map&location=coordinates&detail=&detailLat=47.267&detailLon=11.400&metricWind=km%2Fh&metricTemp=%C2%B0C&radarRange=-1" frameborder="0"></iframe>

[Colin Larsen]

Whoops - wrong link

I accidentally picked up the 12 hour forecast, have used this one. Let me know what it looks like from your locations. My suspicion is that there is still a location override going on.

<iframe width="650" height="360" src="https://embed.windy.com/embed2.html?lat=-41.079&lon=170.552&zoom=5&level=surface&overlay=clouds&menu=&message=&marker=&calendar=now&pressure=true&type=map&location=coordinates&detail=&detailLat=47.267&detailLon=11.400&metricWind=km%2Fh&metricTemp=%C2%B0C&radarRange=-1" frameborder="0"></iframe>

[Colin Larsen]

And just one last thing before it is sleep time here in NZ .... on the charts, particularly the baro but I guess that solar will do it too, there is a gap once numbers are over 999, so 1000 becomes 1 000 etc and then there are 3 decimal places after 

i.e. 1 023.345.

Can this be changed to 1 decimal place and get rid of the gap so 1 023.345 becomes simply 1023.3 ?

Cheers and thanks

Colin  

[Pat]

Your iframe map looks better now. I think adding the lat/lon to the URL helped making it focused onto that region. 

> Just a minor thing that the wind speed is mph below each forecast icon

This should be resolved now. I didn't realize that DarkSky tells you what units they use when you specify "auto", so this should be done now

> Can this be changed to 1 decimal place 

I found a bug with my barometer generator. For some reason it was using the humidity string format for the number of decimals to round. Which was 3. No idea why humidity, but I changed it so it'll use the barometer's string value which is 1 for metric and 3 for imperial it looks like. 

In any event, the generator is resolved, but I think there's a 2nd "problem". The JavaScript file that displays barometer is using a y-axis resolution of 0.01. Sounds like you want 0.1 for y-axis resolution? 

I just pushed a commit which fixes the issues above, but let me know on the JavaScript y-axis before I create another release. I might need to think about that one if a resolution of 0.1 is what you're looking for (how to flip flop between imperial and metric barometer units?).... Maybe you're all set now with the generator fix and the y-axis isn't a problem anymore?

[Juan Antonio Mosquera]

Hello, 

I have a problem generating...

Aug 23 17:45:25 meteomontaos weewx[12818]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator

Aug 23 17:45:25 meteomontaos weewx[12818]: ****  start time (1534975200) is greater than stop time (1420066800)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****  Traceback (most recent call last):

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      obj.start()

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      self.run()

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 158, in run

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      ngen += self.generate(section[subsection], gen_ts)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      ngen += self.generate(section[subsection], gen_ts)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 320, in generate

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      default_binding)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 372, in _getSearchList

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      searchList += obj.get_extension_list(timespan, db_lookup)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/user/belchertown_highchartsSearchX.py", line 88, in get_extension_list

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      (time_start_vt, time_stop_vt, outTemp_vt) = db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), 'outTemp')

Aug 23 17:45:25 meteomontaos weewx[12818]: ****    File "/usr/share/weewx/weeutil/weeutil.py", line 315, in __new__

Aug 23 17:45:25 meteomontaos weewx[12818]: ****      raise ValueError("start time (%d) is greater than stop time (%d)" % (args[0], args[1]))

Aug 23 17:45:25 meteomontaos weewx[12818]: ****  ValueError: start time (1534975200) is greater than stop time (1420066800)

Aug 23 17:45:25 meteomontaos weewx[12818]: ****  Generator terminated

I use weewx in Raspberry (raspbian)

Thanks.

El martes, 21 de agosto de 2018, 20:45:13 (UTC+2), Pat escribió:

[Pat]

Hello, is this a brand new installation of weewx? 

I think I saw this on my lab system when the database had not completed an archive interval yet. Try again in about 5 to 10 minutes?

[Juan Antonio Mosquera]

Yes, fail all times... timeout?¿

[Pat]

I'm not too sure on this one. Based on the timestamps it's showing your start time is August 22 (yesterday) but your stop time is December 31, 2014. This is within the DAY function, so I'm a little confused why it's trying to pull back to 2014 when it needs only 1 day. 

Did you make any other changes to weewx?

I'm open to anyone else who may have an opinion. 

[Juan Antonio Mosquera]

I upgrade weewx this week.... to 3.8.2.... 

3.8.0 - 3.8.2 exactly.

Other skin is generated OK.

2014?, why?. System date is OK.

I restart weewx again... but this problem persist... attach log.

a lots of charts for make¿?

Thanks for your help.

[Colin Larsen]

Thanks for the update. I think if you've got the baro code sorted to one decimal place the issue is probably resolved.

The difference between 1024.6 and 1 024.6 is somewhat cosmetic and is only as you scroll over the charts so I'm happy.

Great that the map issue is fixed, using the Windy embed generator makes that very easy!

At some stage I'll be looking to add an Air Quality page with graphs for dust and c02 ..... a project for further down the track when I have the station properly up and running :)

For now I need to finish getting all of the About page etc finished and drop in a logo I found.

Colin

[Pat]

Version 0.4's been released. 

https://github.com/poblabs/weewx-belchertown/releases/tag/weewx-belchertown-0.4

[Colin Larsen]

All fixed - thanks

Very very very accurate rainfall :)

[Colin Larsen]

That super accuracy only appears in the week/month/year graphs

Also there is maybe some weird averaging going on with the Month baro chart and the month solar chart doesn't exist. Same with the year charts.

Thanks for all the work Pat

Cheers

Colin

[steep...@gmail.com]

I am getting a similar problem Juan Antonio

Aug 24 00:51:48 wxnuc weewx[10832]:         ****      default_binding)
Aug 24 00:51:48 wxnuc weewx[10832]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 372, in _getSearchList
Aug 24 00:51:48 wxnuc weewx[10832]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)
Aug 24 00:51:48 wxnuc weewx[10832]:         ****    File "/usr/share/weewx/user/belchertown_highchartsSearchX.py", line 88, in get_extension_list
Aug 24 00:51:48 wxnuc weewx[10832]:         ****      (time_start_vt, time_stop_vt, outTemp_vt) = db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), 'outTemp')
Aug 24 00:51:48 wxnuc weewx[10832]:         ****    File "/usr/share/weewx/weeutil/weeutil.py", line 315, in __new__
Aug 24 00:51:48 wxnuc weewx[10832]:         ****      raise ValueError("start time (%d) is greater than stop time (%d)" % (args[0], args[1]))
Aug 24 00:51:48 wxnuc weewx[10832]:         ****  ValueError: start time (1535065200) is greater than stop time (1483228800)
Aug 24 00:51:48 wxnuc weewx[10832]:         ****  Generator terminated
Aug 24 00:51:48 wxnuc weewx[10832]: copygenerator: copied 22 files to /var/www/html/weewx/belchertown

I have tried it both on an existing WeeWX installation and a clean install. Rig is a RaspberyPi 3+ with Debian Stretch installed.

[Colin Larsen]

I'm about to do a complete rebuild due to a slight (read disastrous) command line slip :( ........ I'll see if I get the same problem in case it's been introduced somehow. Mine is on a Pi3 with Raspian Stretch with desktop.

Luckily I saved the weewx.conf and skin.conf - the rest is toast but gives me a chance to tidy some stuff up

[Pat]

Strange. Can't pinpoint it. You're both on a Pi, let me get a Pi setup and see if I can figure it out. 

@gjr80, have you seen this before with your highcharts extension?

[Colin Larsen]

Just did a brand new install - smooth as butter with no errors that I can see.

Pat, can you explain the temp records below for largest and smallest daily temp range? Using simulator but I would have thought those temps would be in the highest temp above if they were real data?

Highest Temperature	0.4 °C	August 24, 2018 at 4:09 PM	August 24, 2018 at 4:09 PM	0.4 °C
Lowest Temperature	0.2 °C	August 24, 2018 at 4:24 PM	August 24, 2018 at 4:24 PM	0.2 °C
Highest Heat Index	0.4 °C	August 24, 2018 at 4:09 PM	August 24, 2018 at 4:09 PM	0.4 °C
Lowest Wind Chill	0.2 °C	August 24, 2018 at 4:24 PM	August 24, 2018 at 4:24 PM	0.2 °C
Largest Daily Temperature Range	0.3 °C	August 24, 2018 (Min: 32.4 °C - Max: 32.7 °C )	August 24, 2018 (Min: 32.4 °C - Max: 32.7 °C )	0.3 °C
Smallest Daily Temperature Range	0.3 °C	August 24, 2018 (Min: 32.4 °C - Max: 32.7 °C )	August 24, 2018 (Min: 32.4 °C - Max: 32.7 °C )	0.3 °C

[Juan Antonio Mosquera]

Ok, thanks.

I upgrade to 0.4 version and same problem.

Greetings.

[Andrew Milner]

looks to me like a farenheit to Celsius conversion somewhere is doing the -32 without also doing the *5/9 ……..

[gjr80]

Yes I have seen similar errors on numerous occasions, but I have never seen a cause like this one before. I believe the cause is that the SLEs are being run against the [[SummaryByMonth]] and [[SummaryByYear]] reports.

The timespan parameter that is used in the SLE signatures is a TimeSpan object covering the period of the report. For [[ToDate]] reports this is typically the period covered by the current archive record, for example the last 5 minutes. However, the [[SummaryByMonth]] and [[SummaryByYear]] reports are generated by iterating over all the months and years of data in the archive, so the period covered varies as the cheetahgenerator iterates over the months and/or years. For example, the January 2014 NOAA report will use a timespan of midnight 1 January 2014 until midnight 1 February 2014.

The way the SLEs work is that all listed SLEs are evaluated for each report in a skin, irrespective of whether they are [[SummaryByMonth]], [[SummaryByYear]] or [[ToDate]] reports. So when the January 2014 NOAA report is generated the highchartsDay SLE is evaluated using a timespan of midnight 1 January 2014 until midnight 1 February 2014. Midnight today (24 August 2018) is used as the start time and midnight 1 February 2014 is used as the stop time. Consequently the error hits.

So why does this affect some people some of the time? The answer is in how the [[SummaryByMonth]] and [[SummaryByYear]] reports are generated. Normally only the current year and current month reports are generated, these will not produce the error. However, if a user deletes all of the generated [[SummaryByMonth]] and [[SummaryByYear]] reports then all of the reports are regenerated and the error will bite when a historical report is generated. Same for any one with a clean install with data from more than 1 month in their archive.

Solutions, well a couple come to mind:

1. Create a second skin to generate the [[SummaryByMonth]] and [[SummaryByYear]] reports so you can limit the SLE used.
2. It seems your [[SummaryByMonth]] and [[SummaryByYear]] reports are just the NOAA format reports, perhaps the Standard skin versions could be used, but this may be more complex.
3. Still thinking of number 3.

Gary

[Ian Millard]

With my clean install there was no data in the archive, but the error still occurred. After several hours the situation still persisted.

[gjr80]

Can't comment further without seeing a good couple of archive periods of a log extract that includes the full error trace.

Gsry

[Pat O'Brien]

Woke up wondering if this is the classic case of reusing variable names. going to try to replicate it today on a Raspberry Pi. 

On Fri, Aug 24, 2018, 7:22 AM gjr80 <gjrod...@gmail.com> wrote:

Can't comment further without seeing a good couple of archive periods of a log extract that includes the full error trace.

Gsry

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Pat]

Thanks for the insight Gary. I haven't run into this problem yet - even on brand new VM's with only 5 minutes worth of Simulator data.

[gjr80]

No, and you won't run into the situation I described unless you have archive data running across multiple months and you force WeeWX to regenerate the [[SummaryByMonth]] reports in your skin (could happen on [[SummaryByYear]] too if you have data spanning multiple years).

Gary

[Pat]

Interesting, but my lab system has over 3 years of data without trouble with the generator. 

[gjr80]

I'm convinced. Home now so set up a VM running 3.8.2 with no data in the db. Runs fine. Download user.belchertown_highchartsSearchX.highchartsDay and add a search_list_extensions option to Standard skin.conf using user.belchertown_highchartsSearchX.highchartsDay as the only SLE. Runs without issue (expected). Stop WeeWX, swap in a db with a few months of simulated data and restart WeeWX and the error occurs:

Aug 25 00:10:16 stretch21 weewx[26121]: highchartsSearchX: timespan=[2016-01-01 00:00:00 AEST (1451570400) -> 2017-01-01 00:00:00 AEST (1483192800)] _start_ts=1535119200 _end_ts=1535205599
Aug 25 00:10:16 stretch21 weewx[26121]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  start time (1535119200) is greater than stop time (1483192800)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  Traceback (most recent call last):
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 239, in run
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      obj.start()
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 273, in start
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      self.run()
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 158, in run
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 320, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      default_binding)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 372, in _getSearchList
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/user/belchertown_highchartsSearchX.py", line 90, in get_extension_list
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      (time_start_vt, time_stop_vt, outTemp_vt) = db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), 'outTemp')
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weeutil/weeutil.py", line 315, in __new__
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      raise ValueError("start time (%d) is greater than stop time (%d)" % (args[0], args[1]))
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  ValueError: start time (1535119200) is greater than stop time (1483192800)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  Generator terminated

The first line of the log extract is some telemetry I put in highchartsDay to log timespan, _start_ts and _end_ts. You can see how timespan is 1 January 2016 to 1 January 2017 which is the first year span of data used by [[SummaryByYear]], the first report in the skin.

Remove the SLE from skin.conf and all reports are generated without error. Add the SLE back in and the reports continue to be generated without error (expected since no historical [[SumaryByxxxxx]] reports are being generated, only the 'current' ones.

Delete all the NOAA format reports and next report cycle the error is back.

Like I said, I am convinced.

Gary

[Pat]

I admit I was feeling a bit daft reading through this because I couldn't replicate it. Thanks for providing your test information, it allowed me to to replicate it so I can see the problem first hand. I took my 3 year old database (which was working), copied to a new lab system, truncated it to 2018 only (so 8 months of data) and I'm getting the same error. 

As you mentioned, if I remove the NOAA reports from skin.conf, it works. If I add them back, it fails. This is an interesting one! 

I'm a bit lost at this point.... but now that I can see it fail, I can start to try and wrap my head around it... even though you've already pinpointed it to the [[SummaryByMonth]] and [[SummaryByYear]], there's gotta be a way to make them all co-exist without managing 2 skins?

On Friday, August 24, 2018 at 10:19:19 AM UTC-4, gjr80 wrote:

I'm convinced. Home now so set up a VM running 3.8.2 with no data in the db. Runs fine. Download user.belchertown_highchartsSearchX.highchartsDay and add a search_list_extensions option to Standard skin.conf using user.belchertown_highchartsSearchX.highchartsDay as the only SLE. Runs without issue (expected). Stop WeeWX, swap in a db with a few months of simulated data and restart WeeWX and the error occurs:

Aug 25 00:10:16 stretch21 weewx[26121]: highchartsSearchX: timespan=[2016-01-01 00:00:00 AEST (1451570400) -> 2017-01-01 00:00:00 AEST (1483192800)] _start_ts=1535119200 _end_ts=1535205599
Aug 25 00:10:16 stretch21 weewx[26121]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  start time (1535119200) is greater than stop time (1483192800)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****  Traceback (most recent call last):
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 239, in run
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      obj.start()
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 273, in start
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      self.run()
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 158, in run
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 320, in generate
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      default_binding)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 372, in _getSearchList
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/user/belchertown_highchartsSearchX.py", line 90, in get_extension_list
Aug 25 00:10:16 stretch21 weewx[26121]:         ****      (time_start_vt, time_stop_vt, outTemp_vt) = db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), 'outTemp')
Aug 25 00:10:16 stretch21 weewx[26121]:         ****    File "/home/weewx/bin/weeutil/weeutil.py", line 315, in __new__

Aug 25 00:10:16 stretch21 weewx[26121]:<span style="color: #000;" cl

[gjr80]

Just a point of clarification, well wrongness actually. I said that for the [[ToDate]] reports the timespan parameter is the current archive period. It is not, it is in fact the period from the timestamp of first good record in the archive through until the timestamp of the last record in the archive. I had incorrectly stated the start time of the period. This has no effect on my analysis as the timespan start is not used in the portion of the SLE concerned that generates the error.

Gary

[Pat]

Still trying to do some testing, step by step through the process. It looks like if you remove only the SummaryByMonth NOAA report, the extension completes successfully.

So there's some sort of clash between highchartsDay and the NOAA-YYYY-MM.txt.tmpl template?

[gjr80]

If you are only using 2018 data then no, the year NOAA reports have no effect. If you include 2017 data then the year NOAA reports will cause the same issue. Have a look at the very first line of the error trace I posted, that is due to a yearly NOAA report for 2016.

There is no conflict/issue with the template, the issue is how SLEs work when used with reports that iterate over a timespan.

Gary

[Pat]

Fair enough. This is where my knowledge of the SLEs is weakest. 

I'm assuming if this is only affecting the highchartsDay class - and not the others - that there's a way to make the day charts play nice with the other SLEs?

[gjr80]

On Saturday, 25 August 2018 08:14:40 UTC+10, Pat wrote:

I'm assuming if this is only affecting the highchartsDay class - and not the others - that there's a way to make the day charts play nice with the other SLEs?

No, it will affect the others, week will be the same, month and year will likely be the same though there could be some subtleties going on as the period increases to match the SummaryBy period.

Thinking some more you could check timespan.stop against the last good record in the db and only proceed with the SLE if they match. You could also probably do something to turn off 'highcharts plots' just for the SummaryBy reports, so rather than from [Extras] do it as an option under the [[]] stanzas. I think both are pretty kludgey though.

Don't discount a second skin so quickly, it is simple to implement and has the advantage of eliminating a whole pile of db queries. When we put together WeeWX-WD we had 8 reports that used a wide variety of data. Some reports were current point in time and others were full of month, year and alltime aggregates. We split that into 3 skins, it provided a significant improvement in generation time because whole lot of unnecessary db queries no longer occurred. Lazy evaluation of SLE results can improve execution time but I am not sure they will get around this problem though.

Gary

[Pat]

It's seeming like that's my best option - Move the NOAA reports to a 2nd skin, then add that skin to the install.py. 

[John Clark]

My problem is, as follows.

Aug 25 01:10:24 OptiPlex weewx[23765]: acurite: Found station at bus=002 device=006
Aug 25 01:10:24 OptiPlex weewx[23765]: acurite: next read in 12 seconds
Aug 25 01:10:24 OptiPlex weewx[23765]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.belchertown.getAllStats', 'user.belchertown.getForecast', 'user.belchertown.getEarthquake', 'user.belchertown.getNOAAdata', 'user.belchertown.getSocialShare', 'user.belchertown_highchartsSearchX.highchartsDay', 'user.belchertown_highchartsSearchX.highchartsWeek', 'user.belchertown_highchartsSearchX.highchartsMonth', 'user.belchertown_highchartsSearchX.highchartsYear']
Aug 25 01:10:24 OptiPlex weewx[23765]: manager: Daily summary version is 2.0
Aug 25 01:10:24 OptiPlex weewx[23765]: restx: Wunderground-PWS: Published record 2018-08-25 01:10:00 CDT (1535177400)
Aug 25 01:10:24 OptiPlex weewx[23765]: restx: PWSWeather: Published record 2018-08-25 01:10:00 CDT (1535177400)
Aug 25 01:10:24 OptiPlex weewx[23765]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****  Error writing forecast info to public_html/json/darksky_forecast.json. Reason: [Errno 2] No such file or directory: 'public_html/json/darksky_forecast.json'
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****  Traceback (most recent call last):
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 239, in run
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      obj.start()
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 273, in start
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      self.run()
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 158, in run
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 232, in generate
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      ngen += self.generate(section[subsection], gen_ts)
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 320, in generate
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      default_binding)
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 372, in _getSearchList
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****    File "/home/weewx/bin/user/belchertown.py", line 311, in get_extension_list
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****      raise ValueError( "Error writing forecast info to %s. Reason: %s" % ( forecast_file, e) )
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****  ValueError: Error writing forecast info to public_html/json/darksky_forecast.json. Reason: [Errno 2] No such file or directory: 'public_html/json/darksky_forecast.json'
Aug 25 01:10:24 OptiPlex weewx[23765]:         ****  Generator terminated
Aug 25 01:10:24 OptiPlex weewx[23765]: copygenerator: copied 1 files to /home/weewx/public_html
Aug 25 01:10:24 OptiPlex weewx[23765]: reportengine: Running report FTP
Aug 25 01:10:24 OptiPlex weewx[23765]: reportengine: Found configuration file /home/weewx/skins/Ftp/skin.conf for report FTP
Aug 25 01:10:24 OptiPlex weewx[23765]: ftpupload: Attempting connection to ftp.w0avq.org
Aug 25 01:10:25 OptiPlex weewx[23765]: ftpupload: Connected to ftp.w0avq.org
Aug 25 01:10:26 OptiPlex weewx[23765]: ftpupload: Uploaded file /weather/style.css
Aug 25 01:10:27 OptiPlex weewx[23765]: ftpupload: Made directory /weather/json/darksky_forecast.json
Aug 25 01:10:28 OptiPlex weewx[23765]: ftpgenerator: ftp'd 1 files in 3.64 seconds
Aug 25 01:10:28 OptiPlex weewx[23765]: reportengine: Running report RSYNC
Aug 25 01:10:28 OptiPlex weewx[23765]: reportengine: Found configuration file /home/weewx/skins/Rsync/skin.conf for report RSYNC
Aug 25 01:10:28 OptiPlex weewx[23765]: rsyncgenerator: rsync upload not requested. Skipped.
Aug 25 01:10:36 OptiPlex weewx[23765]: acurite: Found station at bus=002 device=006

I could be wrong in my assessment, but my weewx locks up and doesn't update. I am assuming that what is meant by public_html/json/darksky_forecast.json is referring to
/home /weewx/public_html/json/darksky_forecast.json which DOES exist, but isn't getting written to. Is my assumption in error and I should look elsewhere for the answer?

--
John Clark

[Andrew Milner]

I suspect it may be a permissions issue - check the permissions for the directory and file to ensure they are correct.

FYI weewx has not locked up - it still appears to be running doing whatever tasks it can even though the cheetah generator has exited - which will result in a lack of updated reports of course.

[Pat]

What do you have listed for the skin options within weewx.conf? Specifically the HTML_ROOT option?

Andrew's also right. Looks like everything else is updating and FTP'ing fine, just the skin isn't being processed. 

[David Hathaway]

I'm having a different error, I believe.  Any ideas on how to sort it out?

My weewx is quite old indeed.  Ubuntu 18.04LTS and MySQL.

I downloaded the version the day you posted it first, so it is v0.1?  Would the later revisions solve the issue?  I think not, since this appears to be a SQL issue.

13:56:15 localhost weewx[23412]: restx: Wunderground-PWS: Published record 2018-08-26 13:56:00 CDT (1535309760)

Aug 26 13:56:16 localhost weewx[23412]: cheetahgenerator: Generated 14 files for report StandardReport in 0.78 seconds

Aug 26 13:56:16 localhost weewx[23412]: imagegenerator: Generated 16 images for StandardReport in 0.76 seconds

Aug 26 13:56:16 localhost weewx[23412]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator

Aug 26 13:56:16 localhost weewx[23412]:         ****  (1055, "Expression #2 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'weewx.archive_day_rain.dateTime' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by")

Aug 26 13:56:16 localhost weewx[23412]:         ****  Traceback (most recent call last):

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run

Aug 26 13:56:16 localhost weewx[23412]:         ****      obj.start()

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start

Aug 26 13:56:16 localhost weewx[23412]:         ****      self.run()

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 158, in run

Aug 26 13:56:16 localhost weewx[23412]:         ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 26 13:56:16 localhost weewx[23412]:         ****      ngen += self.generate(section[subsection], gen_ts)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 26 13:56:16 localhost weewx[23412]:         ****      ngen += self.generate(section[subsection], gen_ts)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 320, in generate

Aug 26 13:56:16 localhost weewx[23412]:         ****      default_binding)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 372, in _getSearchList

Aug 26 13:56:16 localhost weewx[23412]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/user/belchertown.py", line 82, in get_extension_list

Aug 26 13:56:16 localhost weewx[23412]:         ****      at_rainiest_month = wx_manager.getSql( at_rainiest_month_sql )

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weewx/manager.py", line 394, in getSql

Aug 26 13:56:16 localhost weewx[23412]:         ****      _cursor.execute(sql, sqlargs)

Aug 26 13:56:16 localhost weewx[23412]:         ****    File "/usr/share/weewx/weedb/mysql.py", line 48, in guarded_fn

Aug 26 13:56:16 localhost weewx[23412]:         ****      raise klass(e)

Aug 26 13:56:16 localhost weewx[23412]:         ****  DatabaseError: (1055, "Expression #2 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'weewx.archive_day_rain.dateTime' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by")

Aug 26 13:56:16 localhost weewx[23412]:         ****  Generator terminated

Aug 26 13:56:16 localhost weewx[23412]: copygenerator: copied 1 files to /var/www/html/weewx/belchertown

[Pat]

I haven't come across this error before. How old is your weewx? 

[Andrew Milner]

It looks like this may be an issue with MySQL versions with updates from 5.5 to later:

https://www.sitepoint.com/quick-tip-how-to-permanently-change-sql-mode-in-mysql/

[David Hathaway]

Well, I saw I was running 3.8.0 but it the latest is 3.8.2, so I am upgrading now.  For some reason, it replaced my weewx.conf even though I asked to keep the existing one, so that is slowing things a bit.

[David Hathaway]

Thanks for this Andrew.  It shows great promise.

Unfortunately, I think there are trickle down issues.  I don't know.  Now I am getting exceptions in weewx.cheetahgenerator.CheetahGenerator in sqlite.py... but I am running MySQL?  This not a Belchertown error, so I'll move further conversation about it outside this conversation.

[Andrew Milner]

Well David, I do not know.  I googled the error and only found the MySQL conversation.  I do not know if SQLite has a similar setting for sqlmode.  As you say though, it looks unlikely to be a skin issue.  As to whether you are running SQLite or MySQL only you know.  Weewx defaults to SQLite.

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

[David Hathaway]

FWIW, the CheetahGenerator issue was caused by the borked upgrade to 3.8.2, which set DataBindings.wx_binding.database to archive_sqlite when I have been using mysql.  I really ought to get a better text compare program for times like this.  UltraCompare is awesome, but Windows.

I'm ready to get back to trying Belchertown.  Yay!

Pat, are there known issues moving from v 0.1 to whatever is on github now?  (0.4?)  If I am going to try moving forward, it ought to be on the latest codebase, eh?

[Pat]

I'd suggest using the latest release. No issues with upgrading if all your Belchertown options are within weewx.conf. See the GitHub readme for those options. 

[David Hathaway]

Thanks Pat.  I moved the options into weewx.conf after reading that suggestion.

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[David Hathaway]

I updated MySQL to stop throwing that error, updated to weewx v 3.8.2, and installed Belchertown 0.4. Now I get this error:

Aug 27 09:17:16 localhost weewx[4131]: imagegenerator: Generated 16 images for StandardReport in 0.75 seconds

Aug 27 09:17:47 localhost weewx[4131]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator

Aug 27 09:17:47 localhost weewx[4131]:         ****  start time (1535346000) is greater than stop time (1420092000)

Aug 27 09:17:47 localhost weewx[4131]:         ****  Traceback (most recent call last):

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run

Aug 27 09:17:47 localhost weewx[4131]:         ****      obj.start()

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start

Aug 27 09:17:47 localhost weewx[4131]:         ****      self.run()

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 158, in run

Aug 27 09:17:47 localhost weewx[4131]:         ****      ngen = self.generate(gen_dict[section_name], self.gen_ts)

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 27 09:17:47 localhost weewx[4131]:         ****      ngen += self.generate(section[subsection], gen_ts)

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 232, in generate

Aug 27 09:17:47 localhost weewx[4131]:         ****      ngen += self.generate(section[subsection], gen_ts)

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 320, in generate

Aug 27 09:17:47 localhost weewx[4131]:         ****      default_binding)

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 372, in _getSearchList

Aug 27 09:17:47 localhost weewx[4131]:         ****      searchList += obj.get_extension_list(timespan, db_lookup)

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/user/belchertown_highchartsSearchX.py", line 88, in get_extension_list

Aug 27 09:17:47 localhost weewx[4131]:         ****      (time_start_vt, time_stop_vt, outTemp_vt) = db_lookup().getSqlVectors(TimeSpan(_start_ts, timespan.stop), 'outTemp')

Aug 27 09:17:47 localhost weewx[4131]:         ****    File "/usr/share/weewx/weeutil/weeutil.py", line 315, in __new__

Aug 27 09:17:47 localhost weewx[4131]:         ****      raise ValueError("start time (%d) is greater than stop time (%d)" % (args[0], args[1]))

Aug 27 09:17:47 localhost weewx[4131]:         ****  ValueError: start time (1535346000) is greater than stop time (1420092000)

Aug 27 09:17:47 localhost weewx[4131]:         ****  Generator terminated

Aug 27 09:17:47 localhost weewx[4131]: copygenerator: copied 1 files to /var/www/html/weewx/belchertown

On Monday, August 27, 2018 at 8:35:00 AM UTC-5, David Hathaway wrote:

Thanks Pat.  I moved the options into weewx.conf after reading that suggestion.

[Pat]

Thanks. I'm aware of this error but haven't resolved it yet. It's a problem with NOAA reports being generated. 

My thought is to move the NOAA reports to another skin, but I'm afraid that'll be too much extra work for everyone who installs the skin. 

If you want to stop it from happening, open skin.conf and remove the NOAA_month and NOAA_year sections - which will get the skin working but will not produce the NOAA reports. 

[David Hathaway]

OK, I will try that and see if that gets the rest going.

[David Hathaway]

I feel like a problem child at this point.  Maybe I should email you directly rather than via this group?  No worries if you want to continue this way though.

Commenting out the NOAA reports got it to run...ish.

• I got a "NoColumnError" in the wind section.  See log below.  I commented out the Wind Records section of the records\index.html.tmpl to continue on.
  
• The skin is taking 2 minutes to be generated.  Is this expected?  In contrast, StandardReport takes 0.75 seconds.
• The resulting page is broken as if it is missing all content.  See it here:  http://dave77459.mynetgear.com/weewx/belchertown/

Aug 27 09:53:16 localhost weewx[22819]: cheetahgenerator: Generated 14 files for report StandardReport in 1.18 seconds

Aug 27 09:53:17 localhost weewx[22819]: imagegenerator: Generated 16 images for StandardReport in 1.54 seconds

Aug 27 09:54:14 localhost weewx[22819]: manager: Added record 2018-08-27 09:54:00 CDT (1535381640) to database 'weewx'

Aug 27 09:54:14 localhost weewx[22819]: manager: Added record 2018-08-27 09:54:00 CDT (1535381640) to daily summary in 'weewx'

Aug 27 09:54:14 localhost weewx[22819]: engine: Launch of report thread aborted: existing report thread still running

Aug 27 09:54:15 localhost weewx[22819]: restx: Wunderground-PWS: Published record 2018-08-27 09:54:00 CDT (1535381640)

Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: Generate failed with exception '<class 'weedb.NoColumnError'>'

Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Belchertown/records/index.html.tmpl

Aug 27 09:54:35 localhost weewx[22819]: cheetahgenerator: **** Reason: (1054, "Unknown column 'wind' in 'where clause'")

Aug 27 09:54:35 localhost weewx[22819]: ****  Traceback (most recent call last):

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate

Aug 27 09:54:35 localhost weewx[22819]: ****      print >> _file, compiled_template

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__

Aug 27 09:54:35 localhost weewx[22819]: ****      rc = getattr(self, mainMethName)()

Aug 27 09:54:35 localhost weewx[22819]: ****    File "_etc_weewx_skins_Belchertown_records_index_html_tmpl.py", line 1109, in respond

Aug 27 09:54:35 localhost weewx[22819]: ****    File "_etc_weewx_skins_Belchertown_records_index_html_tmpl.py", line 385, in __errorCatcher46

Aug 27 09:54:35 localhost weewx[22819]: ****    File "<string>", line 1, in <module>

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__

Aug 27 09:54:35 localhost weewx[22819]: ****      return self._do_query(aggregate_type)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query

Aug 27 09:54:35 localhost weewx[22819]: ****      val=val, **self.option_dict)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate

Aug 27 09:54:35 localhost weewx[22819]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate

Aug 27 09:54:35 localhost weewx[22819]: ****      **option_dict)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/manager.py", line 458, in getAggregate

Aug 27 09:54:35 localhost weewx[22819]: ****      _row = self.getSql(select_stmt % interpolate_dict)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weewx/manager.py", line 394, in getSql

Aug 27 09:54:35 localhost weewx[22819]: ****      _cursor.execute(sql, sqlargs)

Aug 27 09:54:35 localhost weewx[22819]: ****    File "/usr/share/weewx/weedb/mysql.py", line 48, in guarded_fn

Aug 27 09:54:35 localhost weewx[22819]: ****      raise klass(e)

Aug 27 09:54:35 localhost weewx[22819]: ****  NoColumnError: (1054, "Unknown column 'wind' in 'where clause'")

Aug 27 09:54:54 localhost weewx[22819]: cheetahgenerator: Generated 13 files for report Belchertown in 96.59 seconds

Aug 27 09:54:54 localhost weewx[22819]: copygenerator: copied 22 files to /var/www/html/weewx/belchertown

On Monday, August 27, 2018 at 9:44:55 AM UTC-5, David Hathaway wrote:

OK, I will try that and see if that gets the rest going.

[Pat]

This way is fine. Problems and resolutions can help everyone if it's in the public forum. 

I think I just resolved the error. In my highcharts Day function I'm defining an end point, yet still using the timespan (which causes the problem). Moving to my defined end point may have fixed it. Doing some testing with it now across a few lab systems to make sure. Then I'll issue 0.5 and we can give that a test. 

To answer some of your questions:

The link to styles.css file is wrong. To fix this, change your weewx.conf's URL to "http://dave77459.mynetgear.com/weewx/belchertown" and re-run the report. 

The 2 minute generation is for the highcharts section and it does seem high. Are you on a Pi? I have 3 years of data in my database and it takes 15 seconds to generate on a PC. A pi would be a bit slower. 

No column error with wind? Hmm. Not too sure on this. Could be database related. You can try to run sudo wee_database --rebuild-daily to rebuild the daily values and see if that helps.  Make sure you have a working backup of your database just in case you need to revert back to it. 

[David Hathaway]

Do you mean this section of weewx.conf?

    # If you have a website, you may specify an URL

    station_url = http://dave77459.mynetgear.com/weewx/

Will that mess up StandardReport?  Heck, I'll try it and see.  Easy to change back.

I am on a quad core Xeon CPU at 3.30Ghz, 16GB RAM, and an SSD main drive.  Hardly anything takes time on this system.

I'll give --rebuild-daily a try.  I have a backup from yesterday, but I'll make another first.

[Pat]

> Do you mean this section of weewx.conf?

Yep

> Will that mess up StandardReport?

Don't believe so. It will change your URL that is displayed on weewx.com if you have register_this_station = true

>  I am on a quad core Xeon CPU at 3.30Ghz, 16GB RAM, and an SSD main drive.

So basically a server. Hmm.... the highcharts do a lot of queries so an extended generation time is expected but not sure why it's 2 minutes. Maybe your tables needs to be optimized?

There are ways to tune MySQL which may be something to check out too?

[David Hathaway]

The .tar.gz backup of the database is 71MB.  That is to say, it is big.  It is possible that it needs tuning, but running a query to create a cumulative rain table for the past year takes less than a second.  When I get home, I can investigate further.  Maybe there is a deadlock or something.

Meanwhile, yay!  It runs and looks nice.  http://dave77459.mynetgear.com/weewx/belchertown/

TO-DO:

• Figure out the wind records
• Figure out the server load issue
• NOAA data when 0.5 comes out
• Personalize the headings, etc
• Figure out how to get my smokeping graph to show
• Figure out how to get pwrstat, which indirectly tracks server loads
• Update my URL

Thanks for the help!  Many hours of fun ahead. :-)

[David Hathaway]

Pat,

If you look at my site and graphs, the year rain is very odd.

http://dave77459.mynetgear.com/weewx/belchertown/graphs/

On the yearly graph, rain is far less than 0.02.  Switch to month, and you can see it exceeds 1/2" a few days.  I also miss the total rainfall line on the year graph.

Dave

[Pat]

The year graph don't total like the others, just shows you which days had rain. That being said your year.json is HUGE at 8mb (by contrast my year.json is 231kb). I wonder why yours is so big? This is why your report generation is so slow. Month and Year look rather large. 

How often are you doing archive? weewx.conf StdArchive.archive_interval

[David Hathaway]

[StdArchive]

    

    # If the station hardware supports data logging then the archive interval

    # will be downloaded from the station. Otherwise, specify it (in seconds).

    archive_interval = 60

[Pat]

Ah, that'll do it. Explains why your database is so large and why it takes a while to generate the graph json files. That's a lot of records to process when generating the graphs. 

You may want to consider going to 300 seconds for archive which is the aggregation of the loops during those 300 seconds. Up to you though, it's your data. 

Changing it now won't cure the size and slowness issues though. Suppose you could run a database query to drop any dateTime from archive that isn't on a 5 minute interval (0, 5, 10, 15, etc). 

Don't know if Tom or Matthew have another way.

[David Hathaway]

I guess your month and year data doesn't use the daily records instead of the interval ones?

At least I know why it is so slow then.

[Pat]

Most of the highcharts code is from @gjr80 so he may be able to speak better to it, but looking at how it gets its data, it's using getSqlVectors which looks like it gets its data from the archive table and not a day summary table. 

[Paul Bartholdi]

The problem you see :

****  DatabaseError: (1055, "Expression #2 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'weewx.archive_day_rain.dateTime' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by")

is related to MySQL : Earlier versions were not to strict and accepted expressions that are correct for usual logic but incorrect according to SQL standard. They are now rejected by default, this is the reason of your error.

You can change this behavior by adding the (MySQL) command " SET @@sql_mode=""; "  before the offending " SELECT " or simply at the beginning of you SQL code. In any case you have to repeat it whenever you reuse MySQL.

Note that this is NOT a MySQL error, the rejection of your command is correct according to the SQL standard.

The better solution is to correct the statement, that is adding " DateTime " in the " GROUP BY " list.  But to do it I need to dig further into weewx code.

An other solution is to change the defaults of MySQL from "sql_mode=only_full_group_by" to "sql_mode="" ", making MySQL non standard for all as it was in earlier version. I would not do it!

Paul

[Douglas Krug]

This looks awesome Pat. Well I may have distracted you with the Ambient Weather to Weewx thing, but you sure did a great service to a lot of people with this. Thank you!

On Tuesday, August 21, 2018 at 2:45:13 PM UTC-4, Pat wrote:

The Belchertown skin for weewx is now available. This skin is modeled after https://belchertownweather.com 

The skin features

• Real time streaming updates on the front page without needing to reload the site (weewx-mqtt extension required)
  
• Forecast data updated every hour without needing to reload (a free DarkSky API key required)
  
• Information on your closest Earthquake updated automatically every 3 hours
• Weather station observation graphs which update without needing to reload
  
• Weather records for the current year, and for all time
  
• A mobile and iPad ready display. You could use an iPad as a "2nd console". 
  

There are dozens of options to configure within the skin. Please view the readme on GitHub for a detailed explanation. 

The GitHub page: https://github.com/poblabs/weewx-belchertown

Direct download of v0.1: https://github.com/poblabs/weewx-belchertown/archive/weewx-belchertown-0.1.tar.gz

Quick install:

wget https://github.com/poblabs/weewx-belchertown/archive/weewx-belchertown-0.1.tar.gz

sudo wee_extension --install weewx-belchertown-0.1.tar.gz

Configure weewx.conf with your options

Restart weewx

Out of the box, it is a very basic theme. Make sure to check out the readme for all options available!

This is my first skin, and I've done all I could to test it out in various environments to iron out any initial bugs. If you do find something that's a little off, or room for improvement, please let me know!

[Philip Kutzenco]

David and Pat,

When I go to David's site, his Windy.Com display is centered on my location, not his. See the png file I attached. I first thought David was my close neighbor, but I think Sugar Land, TX is pretty far away from Western North Carolina :-).

I think I saw an earlier post by someone else who noted a similar issue.Windy.com is updating in realtime using info from the site running the browser. Is there a way that can be resolved?

I hope to work on getting Pat's awesome skin set up for me in the near future.

[Colin Larsen]

Go to Windy.com and use their widget generator, that will give you a URL to copy, replace the one in skin.conf entirely with the new one. That fixed mine @ 41south.net.nz

Colin

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[David Hathaway]

Phillip,

That would explain why I see Dallas on the map... I use a private VPN at work that connects to the net from Dallas.  I thought that was maybe where the regional radar emanated.  Thanks for pointing this out.

I'm attaching the "Day sensors" plot which I repurposed to show my electrical power load; you can see the influence of having this skin active.  My fault for the short archive window, but it does show.  

Dave

[Pat]

If the lat/lon is omitted in the windy embed, it uses your location. Colin's right, using the embed generator is super helpful.

I'm close to releasing 0.5 with a hopeful fix for that timespan error. Make sure you have your custom configs in weewx.conf and not skin.conf. Upgrading the skin will overwrite skin.conf and you'll lose your custom overrides. 

[Pat]

Belchertown skin 0.5 is released. 

This should fix the NOAA reports and day charts. Tested it against sqlite and mysql, and in a couple different timezones. Please let me know if it persists. 

Few other fixes and updates listed in the release notes. 

[Colin Larsen]

Nice :)

Was going to mention the full moon icon, but I see it's been picked up and fixed. Will install it here in a few hours.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.