Lose connection to data until reboot

[Johnnie Walker]

Hi,

I'd had a recurring issue for some months that the weewx service loses connection to my Davis Vantage Pro2 / WeatherLink. Basic solution has been to reboot which brings things back into action, although I lose the data for the period during which the problem occurred. Shows up as breaks in the charts (for example):

I have some breaks that are 1-2 days or more if I was on travel and didn't realize until I returned.

I just took time to search /var/log/syslog for the instance that happened today - log is copied in below. And it looks like an out of memory error, so the process is killed. (Log is grep'd for "weewx" , I copied in the log info grep'd "oom" further below). The log entries around 16:12 and after are when I rebooted the server.

I installed weewx well over 2 years ago so can't recollect how I installed. I can post the weewx.conf file if needed.

Could anyone give me a pointer on how to fix, and is there a way to patch up the missing data sections?

Thanks very much - JW

"weewx"

Sep 19 15:38:31 sparta weewx[1081] INFO user.weatherlink_live: Emitting poll packet
Sep 19 15:38:31 sparta weewx[1081] INFO user.weatherlink_live: Emitting push (broadcast) packet
Sep 19 15:38:38 sparta weewx[1081] message repeated 4 times: [ INFO user.weatherlink_live: Emitting push (broadcast) packet]
Sep 19 15:38:41 sparta weewx[1081] INFO user.weatherlink_live: Emitting poll packet
Sep 19 15:38:41 sparta weewx[1081] INFO user.weatherlink_live: Emitting push (broadcast) packet
Sep 19 15:38:51 sparta weewx[1081] message repeated 4 times: [ INFO user.weatherlink_live: Emitting push (broadcast) packet]
Sep 19 15:38:51 sparta weewx[1081] INFO user.weatherlink_live: Emitting poll packet
Sep 19 15:38:53 sparta weewx[1081] INFO user.weatherlink_live: Emitting push (broadcast) packet
Sep 19 15:39:01 sparta weewx[1081] message repeated 3 times: [ INFO user.weatherlink_live: Emitting push (broadcast) packet]
Sep 19 15:39:01 sparta weewx[1081] INFO user.weatherlink_live: Emitting poll packet
Sep 19 15:39:02 sparta kernel: [1192066.955832] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=23c2c8bc241e51af783d9abaf3b8008403f4099db600e63a6284359bda22502e,mems_allowed=0,global_oom,task_memcg=/system.slice/weewx.service,task=python3,pid=1081,uid=0
Sep 19 16:12:45 sparta systemd[1]: Starting LSB: weewx weather system...
Sep 19 16:12:45 sparta weewx[907]:  * Starting weewx weather system weewx
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Initializing weewx version 4.8.0
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Using Python 3.8.10 (default, Jun 22 2022, 20:18:18) #012[GCC 9.4.0]
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Platform Linux-5.4.0-125-generic-x86_64-with-glibc2.29
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Locale is 'C.UTF-8'
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: Debug is 0
Sep 19 16:12:46 sparta weewx[1003] INFO __main__: PID file is /var/run/weewx.pid
Sep 19 16:12:46 sparta weewx[1090] INFO weewx.engine: Loading station type WeatherLinkLive (user.weatherlink_live)
Sep 19 16:12:46 sparta weewx[907]:    ...done.
Sep 19 16:12:46 sparta systemd[1]: Started LSB: weewx weather system.

"oom"

Sep 19 15:39:02 sparta kernel: [1192066.955659] python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 19 15:39:02 sparta kernel: [1192066.955672]  oom_kill_process.cold+0xb/0x10
Sep 19 15:39:02 sparta kernel: [1192066.955745] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 19 15:39:02 sparta kernel: [1192066.955832] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=23c2c8bc241e51af783d9abaf3b8008403f4099db600e63a6284359bda22502e,mems_allowed=0,global_oom,task_memcg=/system.slice/weewx.service,task=python3,pid=1081,uid=0
Sep 19 15:39:02 sparta kernel: [1192066.955996] Out of memory: Killed process 1081 (python3) total-vm:21086716kB, anon-rss:14282388kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:36056kB oom_score_adj:0

[John Kline]

It doesn’t look like you are losing the connection to your Vantage Pro2.  Rather, the kernel is killing WeeWX because you’re running of of memory.

You could write -1000 to the /proc/<pid-of-weewx-process>/oom-kill-adj file to keep it from killing WeeWX, but then something else will be killed to free up memory.

What are you running WeeWX on?  How much memory?  What else is running?

On Sep 19, 2022, at 6:10 PM, Johnnie Walker <joh...@kd2gyo.com> wrote:

Hi,

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6300dba8-2595-45db-b386-465345ef89e2n%40googlegroups.com.

[John Kline]

Correction, the file is /proc/<pid-of-weewx-process>/oom-score-adj

On Sep 19, 2022, at 6:23 PM, John Kline <jo...@johnkline.com> wrote:



[Tom Keffer]

I agree with John that this is an out-of-memory problem, but I wonder why you are getting the offset in humidity during the down time. Is this an artifact of using WeatherLinkLive? Indeed, why are you using WLL? Why not just go to the VP2 directly?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7F064749-72E2-4343-B079-3D2B980E075A%40johnkline.com.

[Johnnie Walker]

Hi John/Tom - many thanks to you both.

On the server instance: running on normal PC hardware, 16GB RAM, Ubuntu Server (think 20.04). I wrote "-1000" to /proc/PROC_ID/oom_score_adj - and we'll see what happens from here. Server mainly used for internal websites, also runs Ubiquiti Unifi Controller.

Tom - I had thought I needed the WLL to serve data to weewx over the LAN, so figured [WeatherLinkLive] in weewx.conf. Have just now checked the file and I have both [Vantage] and [WeatherLinkLive] sections configured. It sounds as if I could remove the WeatherLink one? Am remembering I used a plugin initially to link to the WLL, maybe that's obsolete now.

Both sections copied in below from conf file.

Thanks - JW

[Vantage]
    # This section is for the Davis Vantage series of weather stations.

    # Connection type: serial or ethernet
    #  serial (the classic VantagePro)
    #  ethernet (the WeatherLinkIP or Serial-Ethernet bridge)
    type = ethernet

    # If the connection type is serial, a port must be specified:
    #   Debian, Ubuntu, Redhat, Fedora, and SuSE:
    #     /dev/ttyUSB0 is a common USB port name
    #     /dev/ttyS0   is a common serial port name
    #   BSD:
    #     /dev/cuaU0   is a common serial port name
    port = /dev/ttyUSB0

    # If the connection type is ethernet, an IP Address/hostname is required:
    host = 192.168.1.210

    ######################################################
    # The rest of this section rarely needs any attention.
    # You can safely leave it "as is."
    ######################################################

    # Serial baud rate (usually 19200)
    baudrate = 19200

    # TCP port (when using the WeatherLinkIP)
    tcp_port = 22222

    # TCP send delay (when using the WeatherLinkIP):
    tcp_send_delay = 0.5

    # The type of LOOP packet to request: 1 = LOOP1; 2 = LOOP2; 3 = both
    loop_request = 1

    # The id of your ISS station (usually 1). If you use a wind meter connected
    # to a anemometer transmitter kit, use its id
    iss_id = 1

    # How long to wait for a response from the station before giving up (in
    # seconds; must be greater than 2)
    timeout = 4

    # How long to wait before trying again (in seconds)
    wait_before_retry = 1.2

    # How many times to try before giving up:
    max_tries = 4

    # Vantage model Type: 1 = Vantage Pro; 2 = Vantage Pro2
    model_type = 2

    # The driver to use:
    driver = weewx.drivers.vantage

......

[WeatherLinkLive]
    driver = user.weatherlink_live
    host = 192.168.1.210
    polling_interval = 10
    mapping = th:1, th_indoor, baro, rain:1, wind:1, thw:1, thsw:1, windchill:1

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBDRB7h1QBVOvVFe9Qj_k7WyZDPrqn4Oweix-pqBQ%3DUkA%40mail.gmail.com.

[John Kline]

You still have the problem that something is eating all of your memory.

Assuming you got the process ID correct, something(s) else will be killed. 

On Sep 20, 2022, at 3:40 AM, Johnnie Walker <joh...@kd2gyo.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAM4CCC%2BQ5H0_SeHC%3DvTMgjbkSTjctmweYu06Gh6vdiFqtytXAQ%40mail.gmail.com.

[Cameron D]

looking at the last line of the log, weewx had been allocated VM totalling 21GB, RSS 14GB.  Killing everything else is not going to make much difference.

[Johnnie Walker]

Thanks Cameron - I see that now in the log, any pointers on how I could try and find that mem leak?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/01e1792d-a63e-4ea7-af2c-7706f45d0782n%40googlegroups.com.

[vince]

On Tuesday, September 20, 2022 at 8:27:00 AM UTC-7 joh...@kd2gyo.com wrote:

Thanks Cameron - I see that now in the log, any pointers on how I could try and find that mem leak?

On Tue, Sep 20, 2022 at 9:33 AM 'Cameron D' via weewx-user <weewx...@googlegroups.com> wrote:

looking at the last line of the log, weewx had been allocated VM totalling 21GB, RSS 14GB.  Killing everything else is not going to make much difference.

The only way is to step all the way back to square one and disable 'everything' you have enabled and then reenable them one-by-one until you can figure out the offending piece of the puzzle.    Are you running a large number of skins and extensions and other add-ons perhaps and one is running amok ?  You haven't mentioned what you actually have installed and enabled.

People run weewx on systems with as low as 128 MB of RAM, so this one is highly unusual to say the least.

If it helps any, I have a small extension that will save the weewx memory usage to a separate db and create a minimal skin+graphs of just that - see github (link) if you want to automate getting some data easily.

[Johnnie Walker]

Tom - just replying on the topic of connection: I can't see how to connect to the Vantage Pro 2 directly, my head unit is remote (300ft) and the console only has an RJ-11 - I had purchased the WLL to give me direct connectivity over Ethernet, which has been working with weewx (although acknowledging the issues discussed on this thread). Am I missing something very obvious? (ugh)

On Mon, Sep 19, 2022 at 9:38 PM Tom Keffer <tke...@gmail.com> wrote:

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBDRB7h1QBVOvVFe9Qj_k7WyZDPrqn4Oweix-pqBQ%3DUkA%40mail.gmail.com.

[Johnnie Walker]

Vince - thank you. I've installed the memory util and it's working great. Screenshot below.

I'm running the absolute basic config I think (or at least I thought) - basic skin (one of the default options) and only extension I think (or additional driver) is the WeatherLinkLive.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0f0d9f38-151f-4e91-bd2f-12eac8869611n%40googlegroups.com.

[Warren Gill]

If you have the Vantage Pro2 Console, you can use the add-on data logger expansion module to connect directly by USB. https://www.davisinstruments.com/products/weatherlink-windows-usb

I use the Wireless Weather Envoy, which bypasses the console and connects by radio to the station, and USB to weewx https://www.davisinstruments.com/products/wireless-weather-envoy

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAM4CCC%2B95-ZVCOL3ZyWfEPhQznWWEp2e7_xUJO0LjBfWDyJH4g%40mail.gmail.com.

[Tom Keffer]

On Tue, Sep 20, 2022 at 3:07 PM Johnnie Walker <joh...@kd2gyo.com> wrote:

Tom - just replying on the topic of connection: I can't see how to connect to the Vantage Pro 2 directly, my head unit is remote (300ft) and the console only has an RJ-11 - I had purchased the WLL to give me direct connectivity over Ethernet, which has been working with weewx (although acknowledging the issues discussed on this thread). Am I missing something very obvious? (ugh)

Let's make sure we're talking about the same thing. The WeatherLink Live is a standalone unit that contains its own barometer and temperature sensor, while communicating with the ISS wirelessly to get wind and rain data. It then uploads to the Davis cloud. It looks like this. There are several drivers for this unit, but I'm not familiar with them. I think they listen to the UDP broadcasts from the unit to create LOOP packets.  

Davis used to offer a logger for their console called "WeatherlinkIP", which is essentially the same thing as the USB and serial loggers, except it communicates over ethernet. This is the hardware that is referred to under the [Vantage] section. I don't think Davis makes this anymore.

So, given the remote location of your station and the discontinuation of the WeatherLinkIP, you have probably found the best solution. The only other approach I can think of is to use a serial logger and a serial-to-ethernet converter to get the data onto your network.

Or, perhaps I'm misunderstanding your configuration.

[Cameron D]

I have been tracking my memory usage in Weewx for some years.  I was seeing slow memory leak until 10 months ago.  Since then VM total has bounced between  290 or 350 MB - it would start at 290 then after some time jump to 350 and stay there until reboot. For the last 5 months has been a flat 290MB.

I'd start looking at those step increments and see if they coincide with any particular events.

[vince]

Your graphs make no sense unless your os is somehow misconfigured.  The mem_size is 3x or more what I'd expect.  The mem_rss and mem_share look reasonable.   Did you do something to your os setup that alters how it handles memory ?

Here's mine from a pi4 with a lot of skins including Belchertown, MQTT subscribe, MQTT publish, and some custom stuff I wrote for here.  I show slight memory usage growth (bottom-left graph) in the almost 4 weeks since I built a clean pi and moved my old setup over to it, but given I have a 4GB pi4 I'm expecting/hoping that I'm good for a long long time.  Units are in MB.

Another way of working this one would be to (temporarily) switch back to the simulator driver, restart weewx, and let it run for a day and see if it has the expected under 100 MB of RAM usage.  That might hone in on your driver being the culprit.

[Cameron D]

I think the numbers vary a lot with the system architecture.  My numbers are for Intel x64 - RSS is half yours while VM_total is double.  Then again, I run only Seasons skin.

However, when I had the memory leak it was all due to data increment (showing up as RSS).  The jump in total while RSS is stable could be loading more code, so I expect  it should only happen once or twice then stabilise.  We need to see longer term trends.

[Johnnie Walker]

Hi - well, hope this doesn't jinx it but seem to have fixed the memory leak. Should have occurred to me already, but I had not been updating the weewx weatherlinklive plugin and turned out the version I had was over 2 years old. Installed the latest and restarted weewx, now have some pretty stable memory readings, updating around 6am this morning  - been stable all day:

[Johnnie Walker]

Tom - just in response: Yes, I have the WeatherLink unit and am getting the barometer and temp info on the console and on weewx etc. 

This is the weewx driver I found for it : https://github.com/michael-slx/weewx-weatherlink-live

As just mentioned in an email 10mins ago, I had not been updating this driver and so on checking this morning found it was 2 years out of date (I installed weewx about 2 years ago). A reinstall seems to have fixed (for now) the memory problem.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/yJNYSpx-ihI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECZiftXHkvj_ewd3yvf2PLr4pRUJFUPB%2BRcfQC9it7JYg%40mail.gmail.com.