Reportgenerator 'hangs' with vector plot

[Lucas Heijst]

Currently running weewx 4.0.0b18.

The reportgenerator hangs (it never finishes) during calculating of the section below.

        [[[hourwindvec]]]

            [[[[windvec]]]]

                plot_type          = vector

            [[[[windgustvec]]]]

                plot_type          = vector

                aggregate_type     = max

                aggregate_interval = 900    # == 15 min

Luc

[Thomas Keffer]

Worked fine for me. 

Luc: you know better. We need a log! Perhaps the ReportGenerator is not finishing before the next report is due? Perhaps the program segfaulted? Who knows without a log?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com.

[Lucas Heijst]

Tom,

There was not much info in the syslog, thats why I didnt send it.

I included the syslog this time.

I was wrong: the reportgenerator didn't hang, only it is VERY slow

First I generated all other plots: 71 images in 90 seconds

Generated 71 images for report vproReport in 89.58 seconds

Then I added a 6-hour vector plot. Extra time for that 6h plot: 70 seconds

Generated 72 images for report vproReport in 159.32 seconds

Then added a 24-hour vector plot. Extra time for that 24h vector plot 266 seconds

Generated 73 images for report vproReport in 425.79 seconds

Estimated time for the other vector plots:

week vector plot 31 minutes

month vector plot: 137 minutes

year vector plot: 27 hours

Luc

On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:

Worked fine for me. 

Luc: you know better. We need a log! Perhaps the ReportGenerator is not finishing before the next report is due? Perhaps the program segfaulted? Who knows without a log?

-tk

On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst <ljm....@gmail.com> wrote:

Currently running weewx 4.0.0b18.

The reportgenerator hangs (it never finishes) during calculating of the section below.

        [[[hourwindvec]]]

            [[[[windvec]]]]

                plot_type          = vector

            [[[[windgustvec]]]]

                plot_type          = vector

                aggregate_type     = max

                aggregate_interval = 900    # == 15 min

Luc

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-de...@googlegroups.com.

[Glenn McKechnie]

(Sigh, and to the list)

Luc,

I notice a "Launch of report thread aborted: existing report thread
still running"

Do you use mysql (mariadb)?
If you do, does its CPU usage (from top) increase?

If so try dropping the aggregate interval = 900 and see if there is a difference

>>> email to weewx-de...@googlegroups.com <javascript:>.

>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com
>>>

>>> <https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .

>>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-developm...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/weewx-development/afee474f-61cb-4c8d-8bfd-5f1c9a18773f%40googlegroups.com.
>


--


Cheers
Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

[Vince Skahan]

On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:

I included the syslog this time.

I was wrong: the reportgenerator didn't hang, only it is VERY slow

I still think you are doing too much (and too frequently)  for your model pi, the SD card, and your network.

Generated 73 images for report vproReport in 425.79 seconds

ftpgenerator: Ftp'd 79 files in 89.01 seconds 

What's your interval period ?

Just the two items above are over 8.5 minutes for one cycle.

Also switching from ftp to rsync will dramatically improve how long it takes to upload things to your remote site.

I continue to suggest turning everything off but the base Seasons report to baseline your system, then turn other things one-at-a-time.

Or get a much better computer to run weewx on, and a much better computer to ftp to.

[Lucas Heijst]

Vince,

The following set of weewx programs runs for years now on my raspberry pi 2B.

1. Vantage driver

2. Meteostick driver

3. Klimalogg drive4

4. Computer monitor.

The programs run from a memory stick.

The data is written to 4 external mariadb5 databases on my Synology NAS.

The uploaded data is send to my webserver in the Netherlands. 

The FTP speed is about 1 file per second.

My ISP doesn’t offer RSYNC (I know how fast this is on my NASes).

Starting with version 4.0.0b17 I have these slow calculations of vector plots.

Luc

[Lucas Heijst]

Tom, Vince, Glenn,

Glenn,

Yes, I use external mariadb5 databases.

And no, during the creation of the vector plots the memory use of weewx is not much (2.7 %).

Changing the aggregate_interval has effect, but is not the main cause.

Tom, Vince, Glenn,

The main cause of the slow vector calculation is the aggregate_type = max on windgustvec.

Without the max aggregation all 5 vector plots (6h, day, week, month, year) took together 24 seconds.

Generated 76 images for report vproReport in 113.71 seconds

Luc

=====

        [[[hourwindvec]]]

            [[[[windvec]]]]

                plot_type          = vector

            [[[[windgustvec]]]]

                plot_type          = vector

###                aggregate_type     = max

###                aggregate_interval = 900    # == 15 min

        [[[daywindvec]]]

            [[[[windvec]]]]

                plot_type            = vector

            [[[[windgustvec]]]]

                plot_type            = vector

###                aggregate_type       = max

###                aggregate_interval   = 3600    # == 1 hour

        [[[weekwindvec]]]

            [[[[windvec]]]]

                plot_type            = vector

            [[[[windgustvec]]]]

                plot_type            = vector

###                aggregate_type       = max

        [[[monthwindvec]]]

            [[[[windvec]]]]

                plot_type            = vector

            [[[[windgustvec]]]]

                plot_type            = vector

###                aggregate_type       = max

###                aggregate_interval   = 3600    # == 1 hour

        [[[yearwindvec]]]

            [[[[windvec]]]]

                plot_type            = vector

            [[[[windgustvec]]]]

                plot_type            = vector

###                aggregate_type       = max

=====

On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:

(Sigh, and to the list)

Luc,

I notice a "Launch of report thread aborted: existing report thread
still running"

Do you use mysql (mariadb)?
If you do, does its CPU usage (from top) increase?

If so try dropping the aggregate interval = 900 and see if there is a difference

> email to weewx-de...@googlegroups.com.

[Thomas Keffer]

Luc, could you try the attached version of xtypes.py and see if it makes any difference?

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com.

[Glenn McKechnie]

Luc,

It's not weewx increasing in CPU usage, it's the CPU usage for
mariadb. Check on your Synology NAS.

Maybe it's not the same for yours, but I found this occurring in mine.

Worked in 3.9.2, failed in weewx 4.0.06b where it sent the CPU through
the roof, with the reports failing in the same manner as yours.
I assumed mariadb had become a choke point, commenting it out (as
you've done) brought things back to 'normal'.

In fact, looking through my skin.conf shows the note I've left myself.
I see this.

[[[daywindvec]]]
#-# see note below
#-# aggregate_interval = 300
[[[[windvec]]]]
label = Wind Vector (5 min avg.)
plot_type = vector
aggregate_type = avg
aggregate_interval = 300
#[[[[windgustvec]]]]
# label = Gust Wind
# plot_type = vector
###aggregate_type = max
#-# mysqld goes through the roof if this is enabled
#-# aggregate_interval = 300

Obviously I haven't got back to it. Life (and fires) have a knack of
getting in the way sometimes.

Okay, Quick test here:
CPU for mysqld with no report thread running 0.2%
CPU for mysqld with the (seasons) skin.conf as above 16.2%
Now...
Uncommenting those entries (windgustvec) sends CPU to 163% (114%, 133%
155% etc) mysqld is seriously struggling with it. I have to revert
those changes and restart mysqld to get it back down again.

Taking it a little further it's the, not the 'aggregate_interval =
300' that sets it off it's the 'aggregate_type = max '

Changing that to 'aggregate_type = avg' and it's okay, commenting it
out completely is also okay.

Leaving it uncommented (activated) is poison.
Wait for an archive cycle in this high usage state and the CPU is
still maxed out and my wee_reports fail to cycle. That means that with
my quirky setup it's still running and hasn't finished or been reaped.
Interestingly, when I restart weewx that separate wee_report instance
completes.

On 04/04/2020, Lucas Heijst <ljm.h...@gmail.com> wrote:

>> https://groups.google.com/d/msgid/weewx-development/afee474f-61cb-4c8d-8bfd-5f1c9a18773f%40googlegroups.com.
>>
>>
>> >
>>
>>
>> --
>>
>>
>> Cheers
>> Glenn
>>
>> rorpi - read only raspberry pi & various weewx addons
>> https://github.com/glennmckechnie
>>
>

> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-developm...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com.

[Glenn McKechnie]

Well, that fixed it on this machine!

Back to the 16% mysqld usage with both 'aggregate_type = max' and
'aggregate_interval = 300' reinstated in skin.conf

Wouldn't have a fix as smooth as that for covid-19 would you?

>> <https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com?utm_medium=email&utm_source=footer>

>> .
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-developm...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/weewx-development/CAPq0zECKNtZNoPb1hTckb3ttdxzhNNwzXc872du%3DmKDDuGOAtQ%40mail.gmail.com.

[Thomas Keffer]

Before you get too excited, make sure the fix still gives the right answer. It uses a different query strategy.

[Greg Troxel]

Glenn McKechnie <glenn.m...@gmail.com> writes:

> It's not weewx increasing in CPU usage, it's the CPU usage for
> mariadb. Check on your Synology NAS.

I wonder if people have tried postgresql. (I'm still on sqlite.)

[Lucas Heijst]

Tom,

YES, it made a huge difference! 

With the new xtypes.py all reports now runned in only 102 seconds!

Generated 76 images for report vproReport in 101.51 seconds

The report was generated with the original skin.conf file with aggregate_type set to max and the aggregate_intervals set.

Thanks!

Luc

On Saturday, 4 April 2020 09:41:59 UTC-3, Tom Keffer wrote:

Luc, could you try the attached version of xtypes.py and see if it makes any difference?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com.

[Glenn McKechnie]

On 05/04/2020, Thomas Keffer <tke...@gmail.com> wrote:
> Before you get too excited, make sure the fix still gives the right answer.
> It uses a different query strategy.

My first thought was how can I tell without firing up a 3.9.2 version
(Although I think I might have access to one of those :-)

But then if I change the values on this one...
Attached are 3 charts - all at max, None aggregate interval, 300
aggregate interval and 900 aggregate interval. Seem okay - ish

[Lucas Heijst]

Glenn,

The change in Toms xtypes.py is the way how the database query is done for max and min aggregation
So I assume the old query kept the DB manager very busy (but I did not check it because in the new version the query is fast).

Luc

> email to weewx-de...@googlegroups.com.

[Lucas Heijst]

Tom,

The plots look good to me, see attachments.

Luc

>> https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-de...@googlegroups.com.
> To view this discussion on the web visit

[mwall]

greg, 

we started a postgres adapter years ago.  it never made it into weewx core.

the weewx column names are mixed case, like outTemp and windSpeed.  as i recall, postgres lower-cases everything that is not quoted.  so we would need a general way of dealing with case sensitivity that works with mysql, sqlite, and postgres.

i'm pretty sure it would not be too difficult for weewx core, but making it general enough to work for search list extensions and cheetah templates could be a problem (i think that is why it never made it into weewx core).

if you want to hack on it, make a postgresql adapter in bin/weedb/pgsql.py - follow the pattern in sqlite.py or mysql.py in that directory.

m

[mwall]

On Saturday, April 4, 2020 at 9:24:41 AM UTC-4, Greg Troxel wrote:

I wonder if people have tried postgresql.  (I'm still on sqlite.)

here is a summary of how identifiers are treated in various databases:

https://github.com/ontop/ontop/wiki/Case-sensitivity-for-SQL-identifiers 

the original discussion in weewx user group:

https://groups.google.com/forum/#!msg/weewx-development/5fzYoRio2aM/P9qgDqHBEjgJ

a related issue that still exists in mysql (when running on windows or macos):

https://github.com/weewx/weewx/issues/27

[Greg Troxel]

mwall <mw...@users.sourceforge.net> writes:

> we started a postgres adapter years ago. it never made it into weewx core.

Sorry -- I use a lot of programs and my memory was that weewx supported
postgres. I have used it for various things over the years including
one project that had frequent crashes due to laptop batteries dying and
it did amazingly well in terms of robustness.

> the weewx column names are mixed case, like outTemp and windSpeed. as i
> recall, postgres lower-cases everything that is not quoted. so we would
> need a general way of dealing with case sensitivity that works with mysql,
> sqlite, and postgres.

Interesting - I am guessing that postgres is compliant with the SQL
specs, but making a different within-spec choice.

> i'm pretty sure it would not be too difficult for weewx core, but making it
> general enough to work for search list extensions and cheetah templates
> could be a problem (i think that is why it never made it into weewx core).


> if you want to hack on it, make a postgresql adapter in bin/weedb/pgsql.py
> - follow the pattern in sqlite.py or mysql.py in that directory.

I have put your text into my todo list. But it's large :-(

Thanks for taking the time to straighten me out.

[Vince Skahan]

On Friday, April 3, 2020 at 6:56:58 PM UTC-7, Vince Skahan wrote:

On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:

ftpgenerator: Ftp'd 79 files in 89.01 seconds 

Also switching from ftp to rsync will dramatically improve how long it takes to upload things to your remote site.

Luc - just a data point - this is from my Seagate Dockstar which is a 128MB RAM pogoplug, so far slower than even a model-B pi.  Remote host is my Amazon AWS Lightsail instance on Internet.

weeutil.rsyncupload: rsync'd 67 files (962749 bytes) in 1.60 seconds

Suggest trying to switch to rsync to perhaps spin your processing up a bit..... 

[Lucas Heijst]

Vince,

My ISP in the Netherlands doesn't allow the use of rsync.

Luc

[Thomas Keffer]

I had the same issue with my webhost. After 5 years of arguing with them, I gave up and went to AWS Lightsail. Cheaper, faster, and I can do anything I want with my instance. Much happier now.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b76e7a49-5817-490e-b2a3-f6f5e8a345e9%40googlegroups.com.