Catastrophic apt upgrade to V5.02 on Raspberry Pi

312 views
Skip to first unread message

Peter Fletcher

unread,
Feb 20, 2024, 7:14:01 PMFeb 20
to weewx-user
I have had weewx 4.x (apt installed) running happily on my Pi 3b for years. A month ago, since 5.0 seemed to be stable, I allowed the upgrade to the then current release of 5.0. Since there had been a few concerns raised about it, I did a complete image backup of the Pi before allowing the upgrade. This turned out not to be necessary, since the upgrade essentially 'just worked', though I did have to deal with the missing private key issue that has been discussed elsewhere. This evening, as part of a routine apt full-upgrade, , I was updated to 5.02, and the weewx installation appears to have been comprehensively clobbered. Sadly, it didn't occur to me to do an image backup before a minor update, So I don't have an easy way of going back. The initial error appeared to be an access violation to /dev/serial0, such as was described in another recent thread with this version of weewx, but changing the ownership of the serial port to weewx:weewx only changed the problem. systemctl status now reports:
* weewx.service - WeeWX
     Loaded: loaded (/lib/systemd/system/weewx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-02-20 18:48:39 EST; 31s ago
       Docs: https://weewx.com/docs
    Process: 3348 ExecStart=weewxd /etc/weewx/weewx.conf (code=exited, status=1/FAILURE)
   Main PID: 3348 (code=exited, status=1/FAILURE)
        CPU: 1.256s

Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****      callback(event)
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****    File "/etc/weewx/bin/user/crt.py", line 540, in handle_new_loop
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****      self.handle_data(event.packet)
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****    File "/etc/weewx/bin/user/crt.py", line 563, in handle_data
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****      weeutil.weeutil.log_traceback('crt: **** ')
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****  AttributeError: module 'weeutil.weeutil' has no attribute 'log_traceback'
Feb 20 18:48:39 bullseyepi weewxd[3348]: CRITICAL __main__:     ****  Exiting.
Feb 20 18:48:39 bullseyepi systemd[1]: weewx.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 18:48:39 bullseyepi systemd[1]: weewx.service: Failed with result 'exit-code'.
Feb 20 18:48:39 bullseyepi systemd[1]: weewx.service: Consumed 1.256s CPU time.

What is going on?

p q

unread,
Feb 20, 2024, 7:30:42 PMFeb 20
to weewx...@googlegroups.com
https://weewx.com/docs/5.0/usersguide/troubleshooting/what-to-do/

  1. Look at the log file. We are always happy to take questions, but the first thing someone will ask is, "What did you find in the log file?"


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9e86020c-1a88-4b0a-b47f-6133c5432b88n%40googlegroups.com.


--
Peter Quinn
(415)794-2264

vince

unread,
Feb 20, 2024, 7:38:18 PMFeb 20
to weewx-user
Well FWIW the error appears to be for the optional crt extension so I’d perhaps try disabling that and seeing if core weewx is ok. As always set debug=1 and provide logs from weewx startup through two archive periods.

You could also try reinstalling the crt extension.  Back your /etc/weewx tree up just in case,

Tom Keffer

unread,
Feb 20, 2024, 7:48:58 PMFeb 20
to weewx...@googlegroups.com
The "AttributeError" problem is a known problem with crt. See this thread

But, it is not the actual problem --- it's just a reporting problem. For the actual problem, we will need to see more of the log.

On Tue, Feb 20, 2024 at 4:14 PM 'Peter Fletcher' via weewx-user <weewx...@googlegroups.com> wrote:
--

Peter Fletcher

unread,
Feb 20, 2024, 8:41:43 PMFeb 20
to weewx-user
Here is the log from before I applied the apt update, through now. This includes one reboot. The last  start was after setting debug=1 in the config file. I hope that it helps you more than it does me!:

Feb 20 18:20:19 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 2.52 seconds
Feb 20 18:20:21 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report SeasonsReport in 1.47 seconds
Feb 20 18:20:21 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
Feb 20 18:20:23 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 8 files for report PublicReport in 2.19 seconds
Feb 20 18:20:24 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report PublicReport in 1.39 seconds
Feb 20 18:20:24 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx2
Feb 20 18:20:25 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 1 files in 0.75 seconds
Feb 20 18:20:35 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 22 files in 10.12 seconds
Feb 20 18:25:16 bullseyepi weewxd[447]: INFO weewx.manager: Added record 2024-02-20 18:25:00 EST (1708471500) to database 'weewx.sdb'
Feb 20 18:25:16 bullseyepi weewxd[447]: INFO weewx.manager: Added record 2024-02-20 18:25:00 EST (1708471500) to daily summary in 'weewx.sdb'
Feb 20 18:25:19 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 2.53 seconds
Feb 20 18:25:20 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report SeasonsReport in 1.41 seconds
Feb 20 18:25:20 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
Feb 20 18:25:23 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 8 files for report PublicReport in 2.19 seconds
Feb 20 18:25:24 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report PublicReport in 1.40 seconds
Feb 20 18:25:24 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx2
Feb 20 18:25:25 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 1 files in 0.76 seconds
Feb 20 18:25:36 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 22 files in 11.08 seconds
Feb 20 18:30:21 bullseyepi weewxd[447]: INFO weewx.manager: Added record 2024-02-20 18:30:00 EST (1708471800) to database 'weewx.sdb'
Feb 20 18:30:21 bullseyepi weewxd[447]: INFO weewx.manager: Added record 2024-02-20 18:30:00 EST (1708471800) to daily summary in 'weewx.sdb'
Feb 20 18:30:24 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 3.31 seconds
Feb 20 18:30:26 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report SeasonsReport in 1.62 seconds
Feb 20 18:30:26 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
Feb 20 18:30:28 bullseyepi weewxd[447]: INFO weewx.cheetahgenerator: Generated 8 files for report PublicReport in 2.54 seconds
Feb 20 18:30:30 bullseyepi weewxd[447]: INFO weewx.imagegenerator: Generated 16 images for report PublicReport in 1.47 seconds
Feb 20 18:30:30 bullseyepi weewxd[447]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx2
Feb 20 18:30:31 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 1 files in 0.76 seconds
Feb 20 18:30:41 bullseyepi weewxd[447]: INFO weewx.reportengine: ftpgenerator: Ftp'd 22 files in 10.28 seconds
Feb 20 18:32:29 bullseyepi weewxd[447]: INFO __main__: Received signal TERM (15).
Feb 20 18:32:29 bullseyepi weewxd[447]: INFO weewx.engine: Main loop exiting. Shutting engine down.
Feb 20 18:32:29 bullseyepi weewxd[447]: INFO weewx.engine: Shutting down StdReport thread
Feb 20 18:32:29 bullseyepi weewxd[447]: INFO __main__: Terminating weewx version 5.0.0
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Initializing weewxd version 5.0.2
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Command line: /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Using Python 3.9.2 (default, Mar 12 2021, 04:06:34) #012[GCC 10.2.1 20210110]
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Located at /usr/bin/python3
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Platform Linux-6.1.21-v7+-armv7l-with-glibc2.31
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Locale: 'en_US'
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Entry path: /usr/share/weewx/weewxd.py
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: WEEWX_ROOT: /etc/weewx
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Configuration file: /etc/weewx/weewx.conf
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: User module: /etc/weewx/bin/user
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO __main__: Debug: 0
Feb 20 18:33:15 bullseyepi weewxd[18134]: INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Initializing weewxd version 5.0.2
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Command line: /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Using Python 3.9.2 (default, Mar 12 2021, 04:06:34) #012[GCC 10.2.1 20210110]
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Located at /usr/bin/python3
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Platform Linux-6.1.21-v7+-armv7l-with-glibc2.31
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Locale: 'en_US'
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Entry path: /usr/share/weewx/weewxd.py
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: WEEWX_ROOT: /etc/weewx
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Configuration file: /etc/weewx/weewx.conf
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: User module: /etc/weewx/bin/user
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO __main__: Debug: 0
Feb 20 18:37:29 bullseyepi weewxd[589]: INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Initializing weewxd version 5.0.2
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Command line: /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Using Python 3.9.2 (default, Mar 12 2021, 04:06:34) #012[GCC 10.2.1 20210110]
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Located at /usr/bin/python3
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Platform Linux-6.1.21-v7+-armv7l-with-glibc2.31
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Locale: 'en_US'
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Entry path: /usr/share/weewx/weewxd.py
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: WEEWX_ROOT: /etc/weewx
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Configuration file: /etc/weewx/weewx.conf
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: User module: /etc/weewx/bin/user
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO __main__: Debug: 0
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Feb 20 18:48:35 bullseyepi weewxd.py: ArchTH: using /var/tmp/THNow.txt
Feb 20 18:48:35 bullseyepi weewxd.py: LoopTH: using /var/tmp/THNow.txt
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.engine: StdConvert target unit is 0x1
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.engine: Archive will use data binding wx_binding
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.engine: Record generation will be attempted in 'hardware'
Feb 20 18:48:35 bullseyepi weewxd[3348]: INFO weewx.engine: Using archive interval of 300 seconds (specified by hardware)
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO user.crt: service version is 0.21
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO user.crt: realtime txt output goes to /var/www/html/weewx3/realtime.txt
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO user.crt: 'None' values will be displayed as NULL
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO user.crt: zambretti forecast: False
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO user.crt: binding is loop
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: StationRegistry: Station will be registered.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: Wunderground: Posting not enabled.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: PWSweather: Posting not enabled.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: CWOP: Posting not enabled.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: WOW: Posting not enabled.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.restx: AWEKAS: Posting not enabled.
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.engine: 'pyephem' detected, extended almanac data is available
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO __main__: Starting up weewx version 5.0.2
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.engine: Clock error is 0.23 seconds (positive is fast)
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.manager: Starting backfill of daily summaries
Feb 20 18:48:36 bullseyepi weewxd[3348]: INFO weewx.manager: Daily summaries up to date
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:35:00 EST (1708472100) to database 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:35:00 EST (1708472100) to daily summary in 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:40:00 EST (1708472400) to database 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:40:00 EST (1708472400) to daily summary in 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:45:00 EST (1708472700) to database 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.manager: Added record 2024-02-20 18:45:00 EST (1708472700) to daily summary in 'weewx.sdb'
Feb 20 18:48:37 bullseyepi weewxd[3348]: INFO weewx.engine: Starting main packet loop.
Feb 20 18:48:39 bullseyepi weewxd[3348]: INFO weewx.engine: Main loop exiting. Shutting engine down.
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Initializing weewxd version 5.0.2
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Command line: /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Using Python 3.9.2 (default, Mar 12 2021, 04:06:34) #012[GCC 10.2.1 20210110]
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Located at /usr/bin/python3
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Platform Linux-6.1.21-v7+-armv7l-with-glibc2.31
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Locale: 'en_US'
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Entry path: /usr/share/weewx/weewxd.py
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: WEEWX_ROOT: /etc/weewx
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Configuration file: /etc/weewx/weewx.conf
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: User module: /etc/weewx/bin/user
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO __main__: Debug: 1
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Feb 20 20:32:15 bullseyepi weewxd.py: ArchTH: using /var/tmp/THNow.txt
Feb 20 20:32:15 bullseyepi weewxd.py: LoopTH: using /var/tmp/THNow.txt
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.engine: StdConvert target unit is 0x1
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.engine: Archive will use data binding wx_binding
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.engine: Record generation will be attempted in 'hardware'
Feb 20 20:32:15 bullseyepi weewxd[17786]: INFO weewx.engine: Using archive interval of 300 seconds (specified by hardware)
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO user.crt: service version is 0.21
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO user.crt: realtime txt output goes to /var/www/html/weewx3/realtime.txt
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO user.crt: 'None' values will be displayed as NULL
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO user.crt: zambretti forecast: False
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO user.crt: binding is loop
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: StationRegistry: Station will be registered.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: Wunderground: Posting not enabled.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: PWSweather: Posting not enabled.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: CWOP: Posting not enabled.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: WOW: Posting not enabled.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.restx: AWEKAS: Posting not enabled.
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.engine: 'pyephem' detected, extended almanac data is available
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO __main__: Starting up weewx version 5.0.2
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.engine: Clock error is 0.29 seconds (positive is fast)
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.manager: Starting backfill of daily summaries
Feb 20 20:32:16 bullseyepi weewxd[17786]: INFO weewx.manager: Daily summaries up to date
Feb 20 20:32:17 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 18:50:00 EST (1708473000) to database 'weewx.sdb'
Feb 20 20:32:17 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 18:50:00 EST (1708473000) to daily summary in 'weewx.sdb'
Feb 20 20:32:17 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 18:55:00 EST (1708473300) to database 'weewx.sdb'
Feb 20 20:32:17 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 18:55:00 EST (1708473300) to daily summary in 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:00:00 EST (1708473600) to database 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:00:00 EST (1708473600) to daily summary in 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:05:00 EST (1708473900) to database 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:05:00 EST (1708473900) to daily summary in 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:10:00 EST (1708474200) to database 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:10:00 EST (1708474200) to daily summary in 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:15:00 EST (1708474500) to database 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:15:00 EST (1708474500) to daily summary in 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:20:00 EST (1708474800) to database 'weewx.sdb'
Feb 20 20:32:18 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:20:00 EST (1708474800) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:25:00 EST (1708475100) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:25:00 EST (1708475100) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:30:00 EST (1708475400) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:30:00 EST (1708475400) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:35:00 EST (1708475700) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:35:00 EST (1708475700) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:40:00 EST (1708476000) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:40:00 EST (1708476000) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:45:00 EST (1708476300) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:45:00 EST (1708476300) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:50:00 EST (1708476600) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:50:00 EST (1708476600) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:55:00 EST (1708476900) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 19:55:00 EST (1708476900) to daily summary in 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:00:00 EST (1708477200) to database 'weewx.sdb'
Feb 20 20:32:19 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:00:00 EST (1708477200) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:05:00 EST (1708477500) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:05:00 EST (1708477500) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:10:00 EST (1708477800) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:10:00 EST (1708477800) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:15:00 EST (1708478100) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:15:00 EST (1708478100) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:20:00 EST (1708478400) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:20:00 EST (1708478400) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:25:00 EST (1708478700) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:25:00 EST (1708478700) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:30:00 EST (1708479000) to database 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.manager: Added record 2024-02-20 20:30:00 EST (1708479000) to daily summary in 'weewx.sdb'
Feb 20 20:32:20 bullseyepi weewxd[17786]: INFO weewx.engine: Starting main packet loop.
Feb 20 20:32:22 bullseyepi weewxd[17786]: INFO weewx.engine: Main loop exiting. Shutting engine down.

Tom Keffer

unread,
Feb 20, 2024, 8:50:50 PMFeb 20
to weewx...@googlegroups.com
That's the end of the log? Where is the crt error?

The record timestamped 2024-02-20 20:30:00 was downloaded from the logger at 20:32:22. I would expect the next record to be processed at 20:35:16 or so. Did you terminate the program? Is there something later in the log?

Peter Fletcher

unread,
Feb 20, 2024, 10:21:04 PMFeb 20
to weewx-user
That is the entire log from shortly before the update. I am guessing that the known bug in crt.py results in the crash not being properly reported.

The good news is that I was able to figure out what was going on. I am using crt to produce a file that allows the display of current weather conditions on my HOA's website. It is written to a subfolder of its own in /var/www/html and uploaded to the HOA site from there by weewx. Again, it was the change from weewx running as root to it running as weewx that was causing an access privilege error. Simply deleting the old version of the file did not solve the problem, but assigning ownership of the relevant subfolder to weewx:weewx restored normal functionality.

Tom Keffer

unread,
Feb 21, 2024, 10:33:11 AMFeb 21
to weewx...@googlegroups.com
Glad it's working for you, but why were there log entries of the crt exception the first time, but not the last? 

If a thread is exiting without telling us why, that's a bug. Looking through the crt code, the write to the disk is protected by an exception clause, so it should have caught the permissions error and logged it.



Peter Fletcher

unread,
Feb 21, 2024, 1:47:30 PMFeb 21
to weewx-user
I'm blessed if I know why the log doesn't show anything about the crashes. The crash information I showed earlier was actually from systemctl status, after the second crash occurred (the first one reported the serial port access failure). I cannot find a way to get the same detail in the history from journalctl, and the log I am giving you is from running:
sudo grep weewxd /var/log/messages >weewx0.log
This may omit important detail lines, if they don't contain the string: 'weewxd'

Peter Fletcher

unread,
Feb 21, 2024, 3:54:31 PMFeb 21
to weewx-user
It turns out that there were some other privilege problems, resulting from the 5.02 update, but I managed to get journalctl to show me the detailed error logs, which it apparently keeps, and /var/log/messages apparently doesn't. The previous crt exceptions were, indeed, in journalctl's logs and they allowed me to identify and fix the rest. I think that everything is now really sorted, but a nominally minor utility update, liable to be included in a routine apt full-upgrade, should not create this big a mess!

matthew wall

unread,
Feb 21, 2024, 4:02:39 PMFeb 21
to weewx-user
On Wednesday, February 21, 2024 at 3:54:31 PM UTC-5 Peter Fletcher wrote:
It turns out that there were some other privilege problems, resulting from the 5.02 update, but I managed to get journalctl to show me the detailed error logs, which it apparently keeps, and /var/log/messages apparently doesn't. The previous crt exceptions were, indeed, in journalctl's logs and they allowed me to identify and fix the rest. I think that everything is now really sorted, but a nominally minor utility update, liable to be included in a routine apt full-upgrade, should not create this big a mess!

peter,

based on the logs you found, is this what happened:

- crt extension could not write to the file it wanted to write to
- when it tried to report the problem, the crt extension died, because the 'traceback' function it tried to call no longer exists

where does your crt extension put its output?

have you updated to the latest crt extension?  (v0.23)

m

Peter Fletcher

unread,
Feb 21, 2024, 5:18:49 PMFeb 21
to weewx-user
That was one of the many permissions-related problems caused by the update! Changing the registered user of a complex program that reads and writes many files in many places, without making sure that the user knows the consequences of and necessary fixes for this makes changing horses in midstream look like a reasonable and appropriate thing to do! The first problem was that the new registered user didn't have access to the serial port. The crt problem was as you said and was the second one to surface after the first had been fixed. Subsequent to fixing that, I had to correct the access rights to the folders corresponding to a number of 'sites' that my copy of weewx creates.

Life was made substantially more difficult because my ISP has just changed how it manages FTP connections, and there are problems with the new implementation, so when things went wrong with the FTP-uploaded sites I originally blamed them on the ISP's FTP issues.

The crt extension puts its output in a subfolder of /var/www/html, whence weewx FTPs it to a different site (not one hosted by my regular ISP).

I was using V 0.21 of crt, but have now updated to V 0.23.

vince

unread,
Feb 21, 2024, 5:52:14 PMFeb 21
to weewx-user
FWIW - you 'could' have remained on your old version basically forever if you were so inclined.....

But to reply....

On Wednesday, February 21, 2024 at 2:18:49 PM UTC-8 Peter Fletcher wrote:
That was one of the many permissions-related problems caused by the update! Changing the registered user of a complex program that reads and writes many files in many places, without making sure that the user knows the consequences

Disagree.  Documented many many times and places from changelog to the upgrade guide to FAQ and dozens of threads here over almost a year of alpha/beta/release-candidate changes and improvements that resulted from that testing.   The docs were totally redone and enhanced and are now very easily searchable.  Same for the wiki.  Same for the FAQ. 

The switch to running unprivileged is also implementing a decade+ design goal to move away from running as root to the longstanding industry best practice of running as a non-privileged user.  That in itself makes it worth the pain to remove that legacy risk.
 
And what does the user need to learn new ?  Two things.  Use of systemd for startup (the os's essentially force this) and the replacement of wee_xyz utilities with an integrated weectl utility (very very well documented for almost a year now).

And where does the user have to possibly salt to taste ?  Two places.   Access to serial/usb ports and privs to write to their webserver document root.  Just like every other unprivileged app that does either.

... the new registered user didn't have access to the serial port.

Hmmm - the user has to properly configure their os to let the non-privileged user access the peripherals.  This is surprisingly difficult given almost infinite variability of the possible os configurations and the variety of devices that connect to those systems.  But the logs clearly say permission denied when this happens and the how to fix it is very very well documented for many months.

Run debug=1.  See permission denied.  Search the forums/docs/wiki/faq for folks who have been there before and what they did.

 
The crt problem was as you said and was the second one to surface after the first had been fixed. Subsequent to fixing that, I had to correct the access rights to the folders corresponding to a number of 'sites' that my copy of weewx creates.

Yes - the onus is on the user to integrate their weather station software with their user-configured webserver.  That has always been their job.

FWIW:
  • a clean new installation of v5 works right out of the box on debian(ish) os and a raspi (which is debian-based) with no edits required
  • if you are on a pi specifically, the default 'pi' user gets access to serial/usb from the default os for free so no work is required there
  • running the packaged variant frequently requires 'one time' work to salt to taste, so to speak.  Weewx is no different than any other app in this respect.
 Again - this is a major upgrade that is 'optional' although recommended.  In itself making the switch to running unprivileged is reason enough to work a little 'once'.

Or save your db, do a clean installation of v5, copy your db into place, and start it up.  It will almost certainly work.

Peter Fletcher

unread,
Feb 21, 2024, 6:50:52 PMFeb 21
to weewx-user
I understand what you are saying. However, all the discussion that I read was about the change from V4.x to V5.x. I was very careful when I did that update, created a total system backup first, and was fully prepared to sort out issues. I have, in any event, been using systemd for many things, including weewx, for a very long time. There were essentially no issues with the 4.x->5.x update. What I wasn't prepared for was an update from 5.01 to 5.02 clobbering everything that was previously working in 5.01. I don't think that it is reasonable to expect the user to carefully (re-)read all the documentation before doing/allowing a 'second decimal place' update.

I have used the wee_xyz utilities so rarely, that I would need to look up how to do anything with them, anyway. I would expect to do the same with weectl.

My existing installation includes a number of custom skins and custom services. There would be much more than copying over the db involved in the ''clean install' approach, though I did think of doing that after everything fell apart.

matthew wall

unread,
Feb 21, 2024, 7:13:08 PMFeb 21
to weewx-user
On Wednesday, February 21, 2024 at 6:50:52 PM UTC-5 Peter Fletcher wrote:
There were essentially no issues with the 4.x->5.x update. What I wasn't prepared for was an update from 5.01 to 5.02 clobbering everything that was previously working in 5.01. I don't think that it is reasonable to expect the user to carefully (re-)read all the documentation before doing/allowing a 'second decimal place' update.

peter, it would be helpful if you could describe a bit more about your configuration.

are you running with hardware that required separate installation (e.g., sdr) or hardware drivers or kernel modules?

are you running multiple instances of weewd?

are your reports all within a single directory tree, or are they scattered?

did you implement the various report locations using symlinks, or is it all explicitly declared within your weewx config file?

did you have any issues with weewx integration to a local web server?  if so, how did you configure that?

do you have ssh or other configurations for remote access that were affected by the permissions changes?  where are those located, and how are they used?

i'm working on a handful of adjustments to the deb/rpm packaging that will hopefully avoid some of the issues you ran into.  so description of systems that are more extensive than just 'apt install' are very helpful.  in hindsight, we probably should have done the root-to-weewx change in the 5.0.0 release, not a dot-dot.  it was not until 5.0 went out that we realized how many *more* issues there would be if some people end up with weewx running as root while others run unprivileged.

apologies for the disruption it caused.

m

bell...@gmail.com

unread,
Feb 21, 2024, 7:18:58 PMFeb 21
to weewx-user
Mistakes happen. I bet if we asked nicely, the WeeWX team would say they should have implemented the change to running as weewx at 5.0.0. Its a small team doing this in their spare time for free. I’ve paid for worse software and support.
Mini rant over. Huge thanks to TK, Mathew, Gary, and everyone else who has contributed!

vince

unread,
Feb 21, 2024, 7:21:04 PMFeb 21
to weewx-user
On Wednesday, February 21, 2024 at 3:50:52 PM UTC-8 Peter Fletcher wrote:
What I wasn't prepared for was an update from 5.01 to 5.02 clobbering everything that was previously working in 5.01. I don't think that it is reasonable to expect the user to carefully (re-)read all the documentation before doing/allowing a 'second decimal place' update.

Fair enough, but you also need to provide a lot of before+after detail of what you think broke in the process in order to have somebody try to recreate your issue so the next user doesn't run into it too.
  • what was your exact configuration when you started - which extensions and versions (hint - weectl extension list for v5, or wee_extension --list for v4)
  • what do you think broke when you were done upgrading and what debug=1 logs support that ?
  • are you 'sure' it was all working 'before' you upgraded ?
It's a bit hard finding the details in your somewhat over the top wording of your post and followups. What I saw in the thread was:
  • (possibly old) crt extension didn't have permissions to write to the web docroot you wanted it to write to
  • your weewx user likely needed to be added to group dialout to be able to access the serial or usb port 
  • any folders etc. in your web docroot needed permissions tweaked so weewx's user could write there
It would be helpful to know if any of that happened 'during' the 5.0.1 to 5.0.2 upgrade.

My existing installation includes a number of custom skins and custom services. There would be much more than copying over the db involved in the ''clean install' approach, though I did think of doing that after everything fell apart.

For a clean v5 install, yes you would need to install any additions you had added and edit in any pertinent edits you had made to your v4 setup.   FWIW, it's a good opportunity to (re)take control of your system a little and document what you have there.  Over time and a lot of hand tweaks it's easy to lose track of the configuration in good detail.  But yes having 'before' backups always helps.

Lastly - if journalctl is too painful, I'd suggest you add rsyslog to your system and configure that and logrotate with the files in the util/ subdirectory in your weewx installation.  The result after you restart rsyslog and weewx (in that order) is that your logs will go to old-school files under /var/log/weewx which (to me) are much easier to find and read.  Highly recommended.

gary....@gmail.com

unread,
Feb 22, 2024, 9:39:47 AMFeb 22
to weewx-user
Amen to that. Funny how things creep in and when you try to change one thing, others break.
I just did that with mosquitto. Didn't want it running as root anymore. I found it is quite picky about the SSL certs location and permissions.

Take the time to install properly with a fresh OS and weewx. Edit in your changes to weewx.conf and skins.
You'll thank yourself later.

Peter Fletcher

unread,
Feb 22, 2024, 1:42:05 PMFeb 22
to weewx-user
I'm not ignoring you, but today has been a bit busy! I will try to write something up and post it later today or tomorrow.
Reply all
Reply to author
Forward
Message has been deleted
0 new messages