wee_database --calc-missing / weectl database calc-missing seem to not call shutDown() at the end

Karen K

unread,

Jul 27, 2024, 11:00:03 AM7/27/24

to weewx-user

This refers to https://groups.google.com/g/weewx-user/c/ZLn1scFWuMk

If you call wee_database --calc-missing or weectl database calc-missing all the services defined in weewx.conf are loaded. If some of those services create threads, the command does not finish. Instead it waits indefinitely for the threads finishing.

It seems to me, that wee-database --calc-missing and weectl database calc-missing do not call the shutDown() method of the services after finishing the calculation. So the services do not know it's time to shut down the threads they created.

Could that be?

Excerpt from the log:

Jul 27 16:16:09 LokalWiki wee_database[1320746] INFO __main__: Preparing to calculate missing derived observations...

Jul 27 16:16:09 LokalWiki wee_database[1320746] INFO __main__: Missing derived observations will be calculated from 2021-02-02 00:00:01 CET (1612220401) through to 2021-02-03 00:00:00 CET (1612306800) inclusive.

Jul 27 16:16:34 LokalWiki wee_database[1320746] DEBUG weewx.manager: Daily summary version is 4.0

Jul 27 16:16:34 LokalWiki wee_database[1320746] INFO __main__: Calculating missing derived observations...

Jul 27 16:16:34 LokalWiki wee_database[1320746] DEBUG weewx.engine: Loading service weewx.engine.StdTimeSynch

Jul 27 16:16:34 LokalWiki wee_database[1320746] DEBUG weewx.engine: Finished loading service weewx.engine.StdTimeSynch

Jul 27 16:16:34 LokalWiki wee_database[1320746] DEBUG weewx.engine: Loading service ...

Jul 27 16:16:39 LokalWiki wee_database[1320746] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding

Jul 27 16:16:39 LokalWiki wee_database[1320746] DEBUG weewx.manager: Daily summary version is 4.0

Jul 27 16:16:39 LokalWiki wee_database[1320746] message repeated 3 times: [ DEBUG weewx.manager: Daily summary version is 4.0]

Jul 27 16:16:39 LokalWiki wee_database[1320746] INFO weewx.wxxtypes: Type beaufort has been deprecated. Use unit beaufort instead.

Jul 27 16:16:39 LokalWiki wee_database[1320746] INFO weewx.manager: Starting backfill of daily summaries

Jul 27 16:16:40 LokalWiki wee_database[1320746] INFO weewx.manager: Processed 288 records to backfill 1 day summaries in 0.71 seconds

Jul 27 16:16:40 LokalWiki wee_database[1320746] INFO weecfg.database: Processed 1 day consisting of 288 records. 1 day consisting of 288 records were updated in 6.63 seconds.

Jul 27 16:16:40 LokalWiki wee_database[1320746] INFO __main__: Missing derived observations calculated in 6.64 seconds

...

Jul 27 16:18:53 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): opendata.dwd.de:443

Jul 27 16:18:54 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/radar/composite/wn/WN_LATEST.tar.bz2 HTTP/1.1" 200 5501583

Jul 27 16:18:55 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/radar/composite/wn/WN_LATEST.tar.bz2 in 1.66 seconds

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): opendata.dwd.de:443

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/text_forecasts/html/VHDL50_DWLG_LATEST_html HTTP/1.1" 200 609

Jul 27 16:18:59 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/text_forecasts/html/VHDL50_DWLG_LATEST_html in 0.11 seconds

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/text_forecasts/html/VHDL51_DWLG_LATEST_html HTTP/1.1" 200 478

Jul 27 16:18:59 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/text_forecasts/html/VHDL51_DWLG_LATEST_html in 0.03 seconds

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/text_forecasts/html/VHDL52_DWLG_LATEST_html HTTP/1.1" 200 340

Jul 27 16:18:59 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/text_forecasts/html/VHDL52_DWLG_LATEST_html in 0.04 seconds

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/text_forecasts/html/VHDL53_DWLG_LATEST_html HTTP/1.1" 200 270

Jul 27 16:18:59 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/text_forecasts/html/VHDL53_DWLG_LATEST_html in 0.06 seconds

Jul 27 16:18:59 LokalWiki wee_database[1320746] DEBUG urllib3.connectionpool: https://opendata.dwd.de:443 "GET /weather/text_forecasts/html/VHDL54_DWLG_LATEST_html HTTP/1.1" 200 516

Jul 27 16:18:59 LokalWiki wee_database[1320746] INFO user.DWD.base: successfully downloaded https://opendata.dwd.de/weather/text_forecasts/html/VHDL54_DWLG_LATEST_html in 0.11 seconds

The last lines about downloading are repeated every archive interval. They come from the weewx-DWD data service, which has the purpose of adding data to the archive record. So it is correct that they download files while WeeWX is running. But it is not necessary while calculating missing derived values. That would be no problem, if wee_database / weectl would call the shutDown() method at the end, as the service cares for shutting down the threads appropriately when this method is called, which weewxd does when shutting down.

gjr80

unread,

Jul 27, 2024, 4:07:51 PM7/27/24

to weewx-user

Yes, class CalcMissing should call DummyEngine.shutDown() on closing so that any invoked services have the chance to properly shut down. Fixed at commit bf80fb8 to appear in the next release.

The OP of the referenced thread can apply the fix to an his existing v5 install by downloading the fixed database.py using https://raw.githubusercontent.com/weewx/weewx/development/src/weecfg/database.py and replacing /usr/share/weewx/weecfg/database.py with the downloaded database.py (alternatively he can run WeeWX with calc_missing = False in his import config file to see if this is the source of his problem - as long as he changes back to calc_missing = True before the end of the day there will be no effect on his imported data)

Gary

Karen K

unread,

Jul 28, 2024, 1:52:52 AM7/28/24

to weewx-user

Thank you.

Karen K

unread,

Jul 29, 2024, 5:30:07 AM7/29/24

to weewx-user

gjr80 schrieb am Samstag, 27. Juli 2024 um 22:07:51 UTC+2:

The OP of the referenced thread can apply the fix to an his existing v5 install by downloading the fixed database.py

I was informed that that solved the problem for him.

Reply all

Reply to author

Forward