weewx 5.02 crashes (and restarts) on trying to access a docker file?

[valken...@gmail.com]

Since upgrading to weewx 5.02 on ubuntue 22.04, deb instaal. I had some issues becuase weewx does not run as root user anymore. Most of the were easy to solve. However on this ubuntu also a docker instance is running for another application. 

My syslog now tells me this:

INFO weewx.engine: Main loop exiting. Shutting engine down.
Feb 14 16:55:15 kwsweerstation weewxd[9005]: INFO weewx.engine: Shutting down StdReport thread
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__: Caught OSError: [Errno 13] Toegang geweigerd: '/var/lib/docker/overlay2/f40006ded3b1ca4e49c40cb408c89374b8b4a067f4de13711cf4be188137d355/merged'
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  Traceback (most recent call last):
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 210, in run
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet))
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      callback(event)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 660, in check_loop
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      raise BreakLoop
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  weewx.engine.BreakLoop
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  During handling of the above exception, another exception occurred:
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  Traceback (most recent call last):
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 676, in post_loop
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self._catchup(self.engine.console.genArchiveRecords)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 723, in _catchup
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      for record in generator(lastgood_ts):
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/drivers/__init__.py", line 31, in genArchiveRecords
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      raise NotImplementedError("Method 'genArchiveRecords' not implemented")
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  NotImplementedError: Method 'genArchiveRecords' not implemented
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  During handling of the above exception, another exception occurred:
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  Traceback (most recent call last):
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd.py", line 166, in main
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      engine.run()
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 217, in run
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      callback(event)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 678, in post_loop
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self._software_catchup()
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 741, in _software_catchup
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      callback(event)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/etc/weewx/bin/user/cmon.py", line 723, in new_archive_record
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      self.save_data(self.get_data(now, self.last_ts))
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/etc/weewx/bin/user/cmon.py", line 743, in get_data
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      record = self.collector.get_data(now_ts)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****    File "/etc/weewx/bin/user/cmon.py", line 551, in get_data
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****      st = os.statvfs(disk)
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  PermissionError: [Errno 13] Toegang geweigerd: '/var/lib/docker/overlay2/f40006ded3b1ca4e49c40cb408c89374b8b4a067f4de13711cf4be188137d355/merged'
Feb 14 16:55:15 kwsweerstation weewxd[9005]: CRITICAL __main__:     ****  Waiting 10 seconds then retrying...
Feb 14 16:55:25 kwsweerstation weewxd[9005]: INFO __main__: retrying...

As far as I understand it appears weewx is trying to access a docker overlay  file, and doesn´t have the appropriate rights. Now I don't understand what weewx has to do with my docker installation? Or am I totaly missing the point of the issue here? I'm not sure how to fix this one. Stopping docker seems to resolve the issue. 

[gjr80]

Correct. Cmon will try to scan all mounted file systems except for those in an ignore list. Problem is the ignore list is not user configurable via a config file, rather it is hard coded in cmon.py. You have a few options; (1) raise an issue against the weewx-cmon repo and ask for Matthew to make the ignore list user configurable, (2) drop cmon from your system or (3) manually change the ignore list in cmon.py to exclude your docker overlays. I'd suggest a combination of (3) and (1).

Have a look at line 266 in cmon.py, just add your docker overlay to the list. Not sure you can add one wildcard entry for all docker overlays but a bit of experimentation will get you there. 

Gary

And do raise an issue for when Matthew does get some time for cmon.  

[valken...@gmail.com]

Thanx Gary!

It didn't cross my mind that cmon scans for mounted file systems. I added /var/lib/docker/overlay2 to the ignore list, but the error persisted. To save me from further troubles I just removed the cmon extension for the time being. I will inform Matthew.

Op donderdag 15 februari 2024 om 02:03:10 UTC+1 schreef gjr80:

[gjr80]

Appreciate you have removed cmon and solved the problem; however, in having a more leisurely look at the cmon source I see there already s the ability for the user to add mounts to the ignore list via the ignored_mounts config option. If the ignore list is not specified by the user the default ignore list (hard coded in cmon.py) is used, ignored_mounts is specified then that list is used in lieu of the default. Furthermore, when checking a given mount point against the ignore list, the check used is whether the mount point in question starts with any of the entries in the ignore list. So using '/var/lib/docker' in the ignore list should work and cause all docker overlays to be ignored (the source used by cmon as the mount points universe on a system is the contents of /proc/mounts with the mount points being the second entry on each line). For info, the entries in the default ignore list are '/lib/init/rw', '/proc', '/sys', '/dev', '/afs', '/mit', '/run', '/var/lib/nfs'. So something like:

[ComputerMonitor]

    ....

    ignored_mounts = '/lib/init/rw', '/proc', '/sys', '/dev', '/afs', '/mit', '/run', '/var/lib/nfs', '/var/lib/docker'

should work. Note, the apostrophes should not be required but leaving them in won't hurt.

My apologies for the earlier mis-information.

Gary

[valken...@gmail.com]

Dear Gary,

No apologies needed. You just put me on the right track to solve the issue. Actually, there is no explicit need for the cmon plugin on this instance. I used the extension to monitor the hardwae when I first installed. However it is running whithout major issues for years now, and I didn't realy use the cmon extension very often. Anyway, thanks for your further explanation, good to know the ignnored_mounts are configurable after all. Maybe others can use this information as well.

Op vrijdag 16 februari 2024 om 00:21:30 UTC+1 schreef gjr80: