Question about timing

[Paul Dunphy]

I am testing v5.0.0b15 in a venv (installed on a Debian-12 based 4 GB Pi 4B using Vince's script.)  I'm using the Simulator to add records to a copy of my database with about 860,000 records.  It's working well, but I note the new record is added at XX:05, XX:10, etc, and then the CPU usage jumps up for ~3 minutes. Then it uploads the HTML code to my VPS, and the CPU drops to idle until the next interval ~2 minutes later.

Am I correct that the system is sweeping through the database, calculating the derived parameters' min, max, etc.?  With versions before v5, the upload occurred seconds after the new record was added.

- Paul VE1DX

[Tom Keffer]

Impossible to say without seeing the log. 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com.

[Paul Dunphy]

Nov 29 11:13:33 jade python3[362677]: weewx[362677] INFO user.sftp: sftpgenerator: transferred 27 files in 5.47 seconds
Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:15:16 AST (1701270916) to database 'mem.sdb'
Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:15:16 AST (1701270916) to daily summary in 'mem.sdb'
Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:15:00 AST (1701270900) to database 'weewx.sdb'
Nov 29 11:15:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:15:00 AST (1701270900) to daily summary in 'weewx.sdb'
Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 29 11:17:01 jade CRON[539037]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Nov 29 11:17:01 jade CRON[539036]: pam_unix(cron:session): session closed for user root
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 13 files for report StandardReport in 191.18 seconds
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 12 images for report StandardReport in 0.41 seconds
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 1 images for report mem in 0.04 seconds
Nov 29 11:18:27 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html/mem
Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1)
Nov 29 11:18:28 jade python3[362677]: weewx[362677] INFO paramiko.transport: Authentication (publickey) successful!
Nov 29 11:18:30 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
Nov 29 11:18:34 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
Nov 29 11:18:34 jade python3[362677]: weewx[362677] INFO user.sftp: sftpgenerator: transferred 27 files in 6.95 seconds
Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:20:16 AST (1701271216) to database 'mem.sdb'
Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:20:16 AST (1701271216) to daily summary in 'mem.sdb'
Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:20:00 AST (1701271200) to database 'weewx.sdb'
Nov 29 11:20:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:20:00 AST (1701271200) to daily summary in 'weewx.sdb'
Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 13 files for report StandardReport in 189.78 seconds
Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 12 images for report StandardReport in 0.43 seconds
Nov 29 11:23:25 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html
Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds
Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 1 images for report mem in 0.03 seconds
Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html/mem
Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1)
Nov 29 11:23:26 jade python3[362677]: weewx[362677] INFO paramiko.transport: Authentication (publickey) successful!
Nov 29 11:23:28 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
Nov 29 11:23:31 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
Nov 29 11:23:31 jade python3[362677]: weewx[362677] INFO user.sftp: sftpgenerator: transferred 27 files in 5.72 seconds
Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:25:16 AST (1701271516) to database 'mem.sdb'
Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:25:16 AST (1701271516) to daily summary in 'mem.sdb'
Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:25:00 AST (1701271500) to database 'weewx.sdb'
Nov 29 11:25:15 jade python3[362677]: weewx[362677] INFO weewx.manager: Added record 2023-11-29 11:25:00 AST (1701271500) to daily summary in 'weewx.sdb'
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 13 files for report StandardReport in 189.62 seconds
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 12 images for report StandardReport in 0.41 seconds
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.cheetahgenerator: Generated 1 files for report mem in 0.02 seconds
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.imagegenerator: Generated 1 images for report mem in 0.04 seconds
Nov 29 11:28:25 jade python3[362677]: weewx[362677] INFO weewx.reportengine: Copied 0 files to /home/pdunphy/weewx-data/public_html/mem
Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO paramiko.transport: Connected (version 2.0, client OpenSSH_8.9p1)
Nov 29 11:28:26 jade python3[362677]: weewx[362677] INFO paramiko.transport: Authentication (publickey) successful!
Nov 29 11:28:27 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO paramiko.transport.sftp: [chan 0] sftp session closed.
Nov 29 11:28:31 jade python3[362677]: weewx[362677] INFO user.sftp: sftpgenerator: transferred 27 files in 5.43 seconds

[Tom Keffer]

The templates are taking 3 minutes or more to run. That could be because they are very complex, they use tags that require whole database searches, your disk is on the verge of failure, or the CPUs are busy doing something else.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com.

[Vince Skahan]

Why not try running the benchmark (link) and see what your numbers look like ?

Since you're running the simulator, stop weewx to quiet things down.  The benchmark basically is cpu bound though, but any data would help.

You might also look at the output of my mem extension that you have running too, just to see if you're out of memory.  If you have a memory leak and you are out of RAM, you would be swapping to SD card which is going to be really slow and the pi can slow down a lot as a result.

(I'm running 2x your number of records just fine on a pi4 - takes just a few seconds for my customized variant of the old Standard skin)

[Paul Dunphy]

I asked about the time in the context of providing feedback for
development. If it is something on my end, then it is irrelevant in
terms of an issue with v5 . . . it's a new SD card, and the CPU is doing
nothing else, so I suspect it is a full database search.

One table I generate uses $alltime.humidex.max
($alltime.humidex.max.degree_F)
$alltime.humidex.maxtime.format($ALLTIMEFMT) . . . I assume that would
have to go through the entire database and calculate the value for each
record every time.

Table "Lake Echo extremes since 2015" (lower right of the page) uses
this.  Maybe I need to just calculate it once a day instead of each
interval.  Either way, it's me and not related to the development of
v5.  :-)

https://ve1dx.net/v5/

- Paul VE1DX

> <https://groups.google.com/d/msgid/weewx-development/706963a2-71bd-4577-8bda-5e7a1457fb75n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to weewx-developm...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-development/d3c02ca2-de0d-4191-8060-c59e7c95793an%40googlegroups.com?utm_medium=email&utm_source=footer>.
>

[Paul Dunphy]

If I run " $alltime.humidex.max ($alltime.humidex.max.degree_F) $alltime.humidex.maxtime.format($ALLTIMEFMT) once a day it takes ~3 minutes.  This is not a problem yet.  However, if the database size increases to the point where the calculation time exceeds 5 minutes, will it skip the next interval, do it a minute or two "late," or cause things to blow up?  This is more of a curiosity than anything else, but it seems to me someone will run $alltime.<some derived parameter> that will take more than the loop interval to complete.

- Paul VE1DX

[Tom Keffer]

If a report thread is still running, a new one is not launched. However, if enough time has elapsed (set by option max_wait, default 10 minutes), a new thread is launched anyway on the assumption that the old thread is a zombie thread.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/bd7c4109-c52c-4ccd-bd86-a65ca9741544n%40googlegroups.com.