Weewx V4 : Imagegenerator problems with plot_type = vector and aggregate_type = max
80 views
Skip to first unread message
Jacques Terrettaz
May 7, 2020, 6:04:30 AM5/7/20
to weewx-user
Hi,
I have upgraded to Weewx V4, and since the upgrade I have observed that some charts take quite a long time to render.
For exemple, in skin.conf of Seasons skin :
[[[monthwindvec]]]
[[[[windgustvec]]]]
plot_type = vector
aggregate_type = max
It take 17 minutes to generate this image ! With weewx 3.9.2 (and python 2.7.3), the image was generated instantaneously.
If I remove "aggregate_type = max", the image is generated without delay.
My configuration : Raspberry PI 3B+, Rasbian (Debian 10.3), python 3.7.3, Weewx 4.0.0, mysql database, MariaDB 10.3.22
gjr80
May 7, 2020, 6:08:14 AM5/7/20
to weewx-user
Hi,
Not doubting what you say but can you post a log extract from WeeWX startup showing the issue. Helps to get an overall picture.
Gary
Jacques Terrettaz
May 7, 2020, 6:20:08 AM5/7/20
to weewx-user
Here is the log.
After the capture of the archive record of 09:00, the cheetahgenerator of the SeasonsReport has been completed, but no imagegenerator completion, and no other reports tasks.
At 09:05 : INFO weewx.engine: Launch of report thread aborted: existing report thread still running
At 09:10 : WARNING weewx.engine: Previous report thread has been running 600.0432357788086 seconds. Launching report thread anyway.
At 09:17 : the end of the report thread initiated at 09:00 is terminated.
At 09/20 : everything is OK again, until the next generation of the monthly charts
May 7 09:00:18 meteopi /weewxd: Calculated sunshine_time = 0.000000, based on radiation = 206.000000, and threshold = 274.442784
May 7 09:00:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:00:00 CEST (1588834800) to database 'weewx'
May 7 09:00:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:00:00 CEST (1588834800) to daily summary in 'weewx'
May 7 09:00:18 meteopi weewx[7904] INFO weewx.restx: Emetsys: Published record 2020-05-07 09:00:00 CEST (1588834800)
May 7 09:00:18 meteopi weewx[7904] INFO weewx.restx: Romma: Published record 2020-05-07 09:00:00 CEST (1588834800)
May 7 09:00:18 meteopi weewx[7904] INFO weewx.restx: Windy: Published record 2020-05-07 09:00:00 CEST (1588834800)
May 7 09:00:22 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 4.00 seconds
May 7 09:00:25 meteopi weewx[7904] INFO weewx.restx: Newdata: Published record 2020-05-07 09:00:00 CEST (1588834800)
May 7 09:00:49 meteopi CRON[17963]: (CRON) info (No MTA installed, discarding output)
May 7 09:05:01 meteopi CRON[18131]: (pi) CMD (sleep 45 && php /home/pi/static/static.php --debug=false)
May 7 09:05:18 meteopi /weewxd: Calculated sunshine_time = 5.000000, based on radiation = 349.000000, and threshold = 285.072206
May 7 09:05:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:05:00 CEST (1588835100) to database 'weewx'
May 7 09:05:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:05:00 CEST (1588835100) to daily summary in 'weewx'
May 7 09:05:18 meteopi weewx[7904] INFO weewx.engine: Launch of report thread aborted: existing report thread still running
May 7 09:05:18 meteopi weewx[7904] INFO weewx.restx: Emetsys: Published record 2020-05-07 09:05:00 CEST (1588835100)
May 7 09:05:18 meteopi weewx[7904] INFO weewx.restx: Newdata: Published record 2020-05-07 09:05:00 CEST (1588835100)
May 7 09:05:18 meteopi weewx[7904] INFO weewx.restx: Windy: Published record 2020-05-07 09:05:00 CEST (1588835100)
May 7 09:05:18 meteopi weewx[7904] INFO weewx.restx: Romma: Published record 2020-05-07 09:05:00 CEST (1588835100)
May 7 09:05:49 meteopi CRON[18127]: (CRON) info (No MTA installed, discarding output)
May 7 09:09:01 meteopi systemd[1]: Starting Clean php session files...
May 7 09:09:01 meteopi CRON[18306]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May 7 09:09:01 meteopi systemd[1]: phpsessionclean.service: Succeeded.
May 7 09:09:01 meteopi systemd[1]: Started Clean php session files.
May 7 09:10:01 meteopi CRON[18350]: (pi) CMD (sleep 45 && php /home/pi/static/static.php --debug=false)
May 7 09:10:18 meteopi /weewxd: Calculated sunshine_time = 5.000000, based on radiation = 371.000000, and threshold = 295.767815
May 7 09:10:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:10:00 CEST (1588835400) to database 'weewx'
May 7 09:10:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:10:00 CEST (1588835400) to daily summary in 'weewx'
May 7 09:10:18 meteopi weewx[7904] WARNING weewx.engine: Previous report thread has been running 600.0432357788086 seconds. Launching report thread anyway.
May 7 09:10:18 meteopi weewx[7904] INFO weewx.restx: Newdata: Published record 2020-05-07 09:10:00 CEST (1588835400)
May 7 09:10:18 meteopi weewx[7904] INFO weewx.restx: Emetsys: Published record 2020-05-07 09:10:00 CEST (1588835400)
May 7 09:10:18 meteopi weewx[7904] INFO weewx.restx: Windy: Published record 2020-05-07 09:10:00 CEST (1588835400)
May 7 09:10:18 meteopi weewx[7904] INFO weewx.restx: Romma: Published record 2020-05-07 09:10:00 CEST (1588835400)
May 7 09:10:23 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 4.68 seconds
May 7 09:10:26 meteopi weewx[7904] INFO weewx.imagegenerator: Generated 18 images for report SeasonsReport in 2.87 seconds
May 7 09:10:26 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
May 7 09:10:27 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 1 files for report tagssciez in 1.00 seconds
May 7 09:10:27 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/tagssciez
May 7 09:10:27 meteopi weewx[7904] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_7.4)
May 7 09:10:27 meteopi weewx[7904] INFO paramiko.transport: Authentication (password) successful!
May 7 09:10:27 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
May 7 09:10:41 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
May 7 09:10:41 meteopi weewx[7904] INFO user.sftp: sftpgenerator: transferred 58 files in 14.52 seconds
May 7 09:10:50 meteopi CRON[18346]: (CRON) info (No MTA installed, discarding output)
May 7 09:15:01 meteopi CRON[18518]: (pi) CMD (sleep 45 && php /home/pi/static/static.php --debug=false)
May 7 09:15:18 meteopi /weewxd: Calculated sunshine_time = 5.000000, based on radiation = 396.000000, and threshold = 306.524471
May 7 09:15:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:15:00 CEST (1588835700) to database 'weewx'
May 7 09:15:19 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:15:00 CEST (1588835700) to daily summary in 'weewx'
May 7 09:15:19 meteopi weewx[7904] INFO weewx.restx: Emetsys: Published record 2020-05-07 09:15:00 CEST (1588835700)
May 7 09:15:19 meteopi weewx[7904] INFO weewx.restx: Newdata: Published record 2020-05-07 09:15:00 CEST (1588835700)
May 7 09:15:19 meteopi weewx[7904] INFO weewx.restx: Windy: Published record 2020-05-07 09:15:00 CEST (1588835700)
May 7 09:15:19 meteopi weewx[7904] INFO weewx.restx: Romma: Published record 2020-05-07 09:15:00 CEST (1588835700)
May 7 09:15:25 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 6.04 seconds
May 7 09:15:28 meteopi weewx[7904] INFO weewx.imagegenerator: Generated 18 images for report SeasonsReport in 2.84 seconds
May 7 09:15:28 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
May 7 09:15:29 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 1 files for report tagssciez in 0.98 seconds
May 7 09:15:29 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/tagssciez
May 7 09:15:29 meteopi weewx[7904] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_7.4)
May 7 09:15:29 meteopi weewx[7904] INFO paramiko.transport: Authentication (password) successful!
May 7 09:15:29 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
May 7 09:15:35 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
May 7 09:15:35 meteopi weewx[7904] INFO user.sftp: sftpgenerator: transferred 27 files in 6.55 seconds
May 7 09:15:49 meteopi CRON[18514]: (CRON) info (No MTA installed, discarding output)
May 7 09:17:01 meteopi CRON[18601]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
May 7 09:17:24 meteopi weewx[7904] INFO weewx.imagegenerator: Generated 50 images for report SeasonsReport in 1021.87 seconds
May 7 09:17:24 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
May 7 09:17:24 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 1 files for report tagssciez in 0.29 seconds
May 7 09:17:24 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/tagssciez
May 7 09:17:34 meteopi weewx[7904] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_7.4)
May 7 09:17:35 meteopi weewx[7904] INFO paramiko.transport: Authentication (password) successful!
May 7 09:17:35 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
May 7 09:17:36 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
May 7 09:17:36 meteopi weewx[7904] INFO user.sftp: sftpgenerator: transferred 2 files in 11.47 seconds
May 7 09:20:01 meteopi CRON[18711]: (pi) CMD (sleep 45 && php /home/pi/static/static.php --debug=false)
May 7 09:20:18 meteopi /weewxd: Calculated sunshine_time = 0.000000, based on radiation = 169.000000, and threshold = 317.336947
May 7 09:20:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:20:00 CEST (1588836000) to database 'weewx'
May 7 09:20:18 meteopi weewx[7904] INFO weewx.manager: Added record 2020-05-07 09:20:00 CEST (1588836000) to daily summary in 'weewx'
May 7 09:20:18 meteopi weewx[7904] INFO weewx.restx: Newdata: Published record 2020-05-07 09:20:00 CEST (1588836000)
May 7 09:20:18 meteopi weewx[7904] INFO weewx.restx: Emetsys: Published record 2020-05-07 09:20:00 CEST (1588836000)
May 7 09:20:18 meteopi weewx[7904] INFO weewx.restx: Windy: Published record 2020-05-07 09:20:00 CEST (1588836000)
May 7 09:20:18 meteopi weewx[7904] INFO weewx.restx: Romma: Published record 2020-05-07 09:20:00 CEST (1588836000)
May 7 09:20:22 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 4.10 seconds
May 7 09:20:25 meteopi weewx[7904] INFO weewx.imagegenerator: Generated 18 images for report SeasonsReport in 2.60 seconds
May 7 09:20:25 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
May 7 09:20:26 meteopi weewx[7904] INFO weewx.cheetahgenerator: Generated 1 files for report tagssciez in 0.94 seconds
May 7 09:20:26 meteopi weewx[7904] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/tagssciez
May 7 09:20:26 meteopi weewx[7904] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_7.4)
May 7 09:20:26 meteopi weewx[7904] INFO paramiko.transport: Authentication (password) successful!
May 7 09:20:27 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
May 7 09:20:32 meteopi weewx[7904] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
May 7 09:20:32 meteopi weewx[7904] INFO user.sftp: sftpgenerator: transferred 27 files in 6.62 seconds
Jacques Terrettaz
May 7, 2020, 6:32:18 AM5/7/20
to weewx-user
here is the log after Weewx startup :
May 7 10:44:49 meteopi systemd[1]: Starting LSB: weewx weather system...
May 7 10:44:49 meteopi weewx[22175] INFO __main__: Initializing weewx version 4.0.0
May 7 10:44:49 meteopi weewx[22175] INFO __main__: Using Python 3.7.3 (default, Dec 20 2019, 18:57:59) #012[GCC 8.3.0]
May 7 10:44:49 meteopi weewx[22175] INFO __main__: Platform Linux-4.19.97-v7+-armv7l-with-debian-10.3
May 7 10:44:49 meteopi weewx[22175] INFO __main__: Locale is 'fr_CH.UTF-8'
May 7 10:44:49 meteopi weewx[22175] INFO __main__: PID file is /var/run/weewx.pid
May 7 10:44:49 meteopi weewx[22179] INFO __main__: Using configuration file /etc/weewx/weewx.conf
May 7 10:44:49 meteopi weewx[22163]: Starting weewx weather system: weewx.
May 7 10:44:49 meteopi systemd[1]: Started LSB: weewx weather system.
May 7 10:44:49 meteopi weewx[22179] INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
May 7 10:44:49 meteopi weewx[22179] INFO weewx.engine: StdConvert target unit is 0x1
May 7 10:44:49 meteopi weewx[22179] INFO weewx.wxservices: The following values will be calculated: pressure=prefer_hardware, altimeter=prefer_hardware, appTemp=prefer_hardware, barometer=prefer_hardware, beaufort=prefer_hardware, cloudbase=software, dewpoint=prefer_hardware, ET=prefer_hardware, heatindex=prefer_hardware, humidex=software, inDewpoint=prefer_hardware, maxSolarRad=prefer_hardware, rainRate=prefer_hardware, windchill=prefer_hardware, windrun=prefer_hardware
May 7 10:44:49 meteopi weewx[22179] INFO weewx.wxservices: The following algorithms will be used for calculations: altimeter=aaASOS, maxSolarRad=Bras
May 7 10:44:49 meteopi weewx[22179] INFO weewx.engine: Archive will use data binding wx_binding
May 7 10:44:49 meteopi weewx[22179] INFO weewx.engine: Record generation will be attempted in 'hardware'
May 7 10:44:49 meteopi weewx[22179] INFO weewx.engine: Using archive interval of 300 seconds (specified by hardware)
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: StationRegistry: Station will be registered.
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: Wunderground: Posting not enabled.
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: PWSweather: Posting not enabled.
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: CWOP: Posting not enabled.
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: WOW: Posting not enabled.
May 7 10:44:50 meteopi weewx[22179] INFO weewx.restx: AWEKAS: Posting not enabled.
May 7 10:44:50 meteopi weewx[22179] INFO user.mqtt: service version is 0.22
May 7 10:44:50 meteopi weewx[22179] INFO user.mqtt: binding to ['loop', 'archive']
May 7 10:44:50 meteopi weewx[22179] INFO user.mqtt: topic is weather
May 7 10:44:50 meteopi weewx[22179] INFO user.mqtt: desired unit system is METRIC
May 7 10:44:50 meteopi weewx[22179] INFO user.mqtt: data will be uploaded to mqtt://XXXXXXXXXXX
May 7 10:44:50 meteopi /weewxd: restx: Newdata: service version is 2.0
May 7 10:44:50 meteopi /weewxd: restx: Newdata: Data will be uploaded to http://XXXXXXXXX
May 7 10:44:50 meteopi /weewxd: restx: Romma: service version is 2.0
May 7 10:44:50 meteopi /weewxd: restx: Romma: Data will be uploaded to http://XXXXXXXXXXX
May 7 10:44:50 meteopi /weewxd: restx: Emetsys: service version is 2.0
May 7 10:44:50 meteopi /weewxd: restx: Emetsys: Data will be uploaded to http:/XXXXXXXXXXXXX
May 7 10:44:50 meteopi /weewxd: windy: version is 0.5
May 7 10:44:50 meteopi /weewxd: windy: Data will be uploaded to https://stations.windy.com/pws/update
May 7 10:44:50 meteopi weewx[22179] INFO __main__: Starting up weewx version 4.0.0
May 7 10:44:50 meteopi weewx[22179] INFO weewx.engine: Clock error is -0.23 seconds (positive is fast)
May 7 10:44:50 meteopi weewx[22179] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx'
May 7 10:44:50 meteopi weewx[22179] INFO weewx.manager: Starting backfill of daily summaries
May 7 10:44:50 meteopi weewx[22179] INFO weewx.engine: Starting main packet loop.
gjr80
May 7, 2020, 6:43:37 AM5/7/20
to weewx-user
Ok, it looks like the 0900 report cycle caused all of the Seasons skin images/plots to be generated (50 files), perhaps because they did not exist or were too old. This took some 17 minutes to complete. While a given report thread is running no further report cycles will start a report thread for that report, hence the 0905 warning. Generating all images for a skin such as Seasons can be time consuming, the time taken depending on a number of things such a the processor speed, WeeWX load and system load. You should find that things settle down now, day images will be generated each report cycle, week images every hour, month every three hours and year every 24 hours. I would keep monitoring the log and if this occurs regularly then it’s worth further investigation.
Gary
Gary
Glenn McKechnie
May 7, 2020, 6:48:07 AM5/7/20
to weewx...@googlegroups.com
This sounds like the problem that Luc reported, and I was also experiencing.
Tom came up with a change to xtypes.py and for whatever reason it
doesn't appear to have made it into weewx 4.
This fix cured our problem of hanging reports. Mysql was through the
roof and delaying the report generation.
That file is attached at the following, along with the discussion
https://groups.google.com/d/msg/weewx-development/fmprpi4yJrI/iQ7Rw8srAQAJ
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/a7d49717-c19e-4a15-bf4b-b7883ab9f189%40googlegroups.com.
>
--
Cheers
Glenn
rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie
Tom came up with a change to xtypes.py and for whatever reason it
doesn't appear to have made it into weewx 4.
This fix cured our problem of hanging reports. Mysql was through the
roof and delaying the report generation.
That file is attached at the following, along with the discussion
https://groups.google.com/d/msg/weewx-development/fmprpi4yJrI/iQ7Rw8srAQAJ
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/a7d49717-c19e-4a15-bf4b-b7883ab9f189%40googlegroups.com.
>
--
Cheers
Glenn
rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie
Jacques Terrettaz
May 7, 2020, 6:58:26 AM5/7/20
to weewx...@googlegroups.com
Gary,
Thanks for your reply.
I can confirm that I occurs each time the monthly charts are refreshed. In the log, same problem at 00:00, 03:00, 06:00 , 12:00 ... etc
If I remove the "aggregate_type = max" from this image definition in the Season skin.conf :
[[[monthwindvec]]]
[[[[windgustvec]]]]
plot_type = vector
aggregate_type = max
while keeping all the other daily, weekly, monthly and yearly charts, there is no more delay, and all the charts are produced within seconds.
Le 7 mai 2020 à 12:43, gjr80 <gjrod...@gmail.com> a écrit :
Ok, it looks like the 0900 report cycle caused all of the Seasons skin images/plots to be generated (50 files), perhaps because they did not exist or were too old. This took some 17 minutes to complete. While a given report thread is running no further report cycles will start a report thread for that report, hence the 0905 warning. Generating all images for a skin such as Seasons can be time consuming, the time taken depending on a number of things such a the processor speed, WeeWX load and system load. You should find that things settle down now, day images will be generated each report cycle, week images every hour, month every three hours and year every 24 hours. I would keep monitoring the log and if this occurs regularly then it’s worth further investigation.
Gary
--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/0c5AIxeDJaY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7c1d9d10-f6f3-4232-901c-ed5c75f7b912%40googlegroups.com.
gjr80
May 7, 2020, 7:03:48 AM5/7/20
to weewx-user
Ok, looks indeed like it is the problem highlighted by Glenn. You could try the xtypes.py in the post he linked, it should be safe to use under 4.0.0 as there have been no further commits of xtypes.py. I expect Tom will come along with a committed fix shortly.
Gary
Jacques Terrettaz
May 7, 2020, 7:16:32 AM5/7/20
to weewx-user
Thanks Glenn and Gary,
It is OK with the new xtypes.py.
I deleted all month and yearly images, and it took about 40 seconds to generate them all. Much better that 17 minutes!
Tom Keffer
May 7, 2020, 10:17:08 AM5/7/20
to weewx-user
Somehow that xtypes change didn't make it into the final release. It will appear in v4.0.1.
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c8f1c09c-bf7d-4ed0-bf95-89484588140f%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages