RAM is running full. Apparently it's due to the weectl import

[Sebastian E]

Hello everyone, I currently have the problem that my RAM is regularly full and I have to restart the server.

I have now seen that the weectl import service is not stopped. I'm attaching the output. Does anyone have an idea what this Area 31 30 means? That's where it seems to hang. Thank you

root@weewx5:~# weectl import --import-config=/etc/weewx/import/wu-example.conf --no-prompt --verbose
Using configuration file /etc/weewx/weewx.conf
Starting weectl import...
Observation history for Weather Underground station 'I*****' will be imported.
The following options will be used:
config=/etc/weewx/weewx.conf, import-config=/etc/weewx/import/wu-example.conf
station=I******, from=None, to=None
apiKey=******************************
dry-run=False, calc_missing=True, ignore_invalid_data=True
tranche=250, interval=5, wind_direction=[0.0, 360.0]
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x10' (METRIC).
The following imported field-to-WeeWX field map will be used:
source field 'epoch' in units 'unix_epoch' --> WeeWX field 'dateTime'
source field 'tempAvg' in units 'degree_F' --> WeeWX field 'outTemp'
source field 'humidityAvg' in units 'percent' --> WeeWX field 'outHumidity'
source field 'dewptAvg' in units 'degree_F' --> WeeWX field 'dewpoint'
source field 'heatindexAvg' in units 'degree_F' --> WeeWX field 'heatindex'
source field 'windchillAvg' in units 'degree_F' --> WeeWX field 'windchill'
source field 'pressureAvg' in units 'inHg' --> WeeWX field 'barometer'
source field 'precipTotal' in units 'inch' --> WeeWX field 'rain'
(source field 'precipTotal' will be treated as a cumulative value)
source field 'precipRate' in units 'inch_per_hour' --> WeeWX field 'rainRate'
source field 'windspeedAvg' in units 'mile_per_hour' --> WeeWX field 'windSpeed'
source field 'winddirAvg' in units 'degree_compass' --> WeeWX field 'windDir'
source field 'windgustHigh' in units 'mile_per_hour' --> WeeWX field 'windGust'
source field 'solarRadiationHigh' in units 'watt_per_meter_squared' --> WeeWX field 'radiation'
source field 'uvHigh' in units 'uv_index' --> WeeWX field 'UV'
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Obtaining raw import data for period 1 ...
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 139 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
139 records identified for import.
Unique records processed: 139; Last timestamp: 2024-07-22 11:34:52 CEST (1721640892)
Mapped data saved to archive successfully for period 1.
Calculating missing derived observations ...
Processing record: 140; Last record: 2024-07-23 00:00:00 CEST (1721685600)
Recalculating daily summaries...

Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
139 records were processed and 139 unique records imported in 0.28 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the weectl log file.
area 30 31

[Sebastian E]

One more note

The message appears after cancellation

Finished calculating missing derived observations
Finished import

248 records were processed and 248 unique records imported in 0.42 seconds.

Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the weectl log file.

^CException ignored in: <module 'threading' from '/usr/lib/python3.11/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1590, in _shutdown
    lock.acquire()
KeyboardInterrupt:
root@weewx5:~#

[gjr80]

A few questions for you:
- why do you believe weectl import is the cause of your memory leak problem?
- what does the WeeWX log say?
- what do you mean by "I have now seen that the weectl import service is not stopped". weectl import is a plain ordinary python utility that is run from the command line, it is not a WeeWX (or system) service.
- regards your second post, I'm not seeing anything unusual there: weectl import was run and appears to have completed execution normal, then ctl-C seems to have been pressed. Exactly what did you do (ie commands entered, keys pressed)?

Gary

[Karen K]

Gary,

may be, it is a little bit of a language problem. I heard of this case before, so I may add some explanation, but I cannot explain all.

The user seems to call weectl import from the command line or from crontab to retrieve data from Wunderground. The command first runs as expected. Data is imported and saved to the database. But then, the command does not end. It hangs. So I suggested to the user to press CTRL-C several times. This resulted in an error message saying that the command hang in the threading module while waiting for a lock.

First question: Where does weectl import use threading? Or does this point to the problem that threading is involved at all?

I had a short glance at the sources, but I did not see it.

Then, there are some hints that third party extensions are involved in the problem. Does weectl import load WeeWX extensions? I do not see a reason to do so. But the output "area 30 31" at the end of the first post is from an extension, not from the command itself.

Karen

[Graham Eddy]

is the transaction buffer over-running avail memory? can the tranche of updates be reduced per transaction?

⊣GE⊢

[gjr80]

So if ctl-C is being used to interrupt a script, the script must be being run from the command line; ctl-C will have no effect on scripts executed by cron. weectl import does not directly use threading, however, weectl import uses a number of WeeWX 'API' calls to do things (save data to database and calculate missing/derived obs are the two main ones). I would suggest this use of threading is possibly due to an xtype call when calculating missing/derived obs. weectl import does not 'load' any extensions per se; however, that is not to say that some other parts of WeeWX called by weectl import do not. Again an xtype being used to calculate missing/derived obs is an obvious candidate. I don't see the 'area 30 31' output being direct weectl import output, quite possibly it is output from some other extension.

To start tracking this down we need to go back to basics with log and config. What does the WeeWX log shown when weectl import is run? And what is in weewx.conf (and to a lesser extent) the import config file /etc/weewx/import/wu-example.conf.

Gary

[Sebastian E]

Hello, first of all thank you very much for your help.

Here is the content of htop. There I see that the import has started several times. According to crontab it starts every 15 minutes. Because it doesn't seem to finish, there are so many views. Just my guess.

[gjr80]

OK. At the moment my best guess is that when weectl import calculates missing derived obs one of the GTS derived obs is not being calculated cleanly/causing a process to remain in memory. Over time the overall memory use increases until the system hangs/crashes due to lack of free memory. A restart relieves the symptoms by clearing the old weectl import processes and the cycle repeats. weectl import is completing execution (the printing of the two lines starting 'Those records with a timestamp already in the archive will not have been' are the last thing weectl import does), but I suspect the thread launched by the GTS type is continues once weectl import has otherwise finished.

You still haven't provided any log extracts, my suggestion is to edit weewx.conf, set debug = 1 then save weewx.conf. The next time your cron job runs weectl import it should use debug = 1 and there may be some clue in the log as to what is happening. If your system is using systemd-journald for logging you may need to poke around using journalctl to find the weectl import log output, this section of the view logs wiki page may help. You might want to start with sudo journalctl -t weectl.

Gary

[Sebastian E]

Hey, attached is an excerpt from journalctl.

Greetings Sebastian

[Karen K]

If - as Gary suggests - GTS is the reason, you could try and comment out user.GTS.GTSService in weewx.conf and try again. If that makes a difference, this would be a path to look further.

[gjr80]

Was there anything after the last two lines? The 'calculate missing' part of the log is where there may be a clue. The 'UNIQUE constraint failed' error messages are expected and can be ignored; once WeeWX has written a given record to archive any attempt to write the same record again will result in such an error.

Gary

[Sebastian E]

Unfortunately without success

[Sebastian E]

Here is a longer excerpt from journalctl

[Karen K]

Sebastian E schrieb am Dienstag, 23. Juli 2024 um 13:41:03 UTC+2:

Here is a longer excerpt from journalctl

What I get from that log is, that weectl starts report generation after importing data. And I see, the Belchertown skin is involved. So there is a possibility that this problem is involved: v5 performance troubleshooting.

 

[gjr80]

It's possible the so called 'Belchertown v5 issue' is involved but a few things suggest it is not, or if it is it's effect is insignificant:

- weewx.conf indicates the user is using the wview_extended schema

- the Belchertown issue raises it's head when a report is run and missing derived obs are calculated on the fly, in this case weectl import is calculating those missing derived obs before reports are run

- the Belchertown report generation is completed within 10 seconds

- the 'Ctl-C response' shows weectl import was caught executing code that was not Belchertown code (Belchertown does not use locks or threading)

Gary

[vince]

One thing to check is that while weewx.conf might say it's using the wview_extended schema, that is only positively true if the db was created at weewx v4 or later.  Suggest the original poster check their actual db files to see what the database is actually using.   You should probably check all the databases you have.

[gjr80]

True, but I suspect Belchertown is not playing a significant role here, the log extract shows Belchertown completes in less than 10 seconds. My understanding tis that the v5 Belchertown issue is characterised by excessively long Belchertown report generation times.

Gary

[vince]

Agree, but it eliminates some possibilities.   They also have a secondary db which might be where the issue is.   Quick thing to try to pin down the cause...

[bell...@gmail.com]

Probably enough chefs in the kitchen already…. But I can’t help myself.

Did this work at one time and now doesn’t? If so, can you remember what you changed?

If it never ‘worked’ or you can’t remember what you changed, I’d step back and ‘disable’ all non core   WeeWX extensions. A quick scan of the weewx.conf, this is Belchertown skin, weatherservices.DWDservice, and GTS.GTSService.  If it works, add them back one at a time. Or perhaps you don’t need them for the import… I don’t understand exactly how the scheduled import interacts or replaces a standard WeeWX install. But if you are ‘just’ importing data, running the reports after import only to have them run at the archive interval might be redundant…

-rich

[Sebastian E]

Hello everyone,

I'm now further, if I remove the part from weewx.conf, it works

data_services = user.weatherservices.DWDservice

[Karen K]

Sebastian E schrieb am Mittwoch, 24. Juli 2024 um 11:35:09 UTC+2:

data_services = user.weatherservices.DWDservice

Which services are still activated now?

[Sebastian E]

Hello,

this services are still activated now 

prep_services = weewx.engine.StdTimeSynch
        data_services = 
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
        xtype_services = weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta, user.GTS.GTSService
        archive_services = weewx.engine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport

[Karen K]

Digging further into this issue I found out, that weectl loads all the services including restful services and report services to calculate missing derived values. This makes no sense to me, because those services are not used during calculation of missing derived values. The same applies to data_services. Or do I miss something?

[Tom Keffer]

See issue #95.

It's true that the reporting service probably does not need to be loaded, but it's difficult for weectl to make a decision on what's needed and what is not. Hence the decision to load everything.

On Thu, Jul 25, 2024 at 11:45 AM Karen K <kk44...@gmail.com> wrote:

Digging further into this issue I found out, that weectl loads all the services including restful services and report services to calculate missing derived values. This makes no sense to me, because those services are not used during calculation of missing derived values. The same applies to data_services. Or do I miss something?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/964bde45-4c97-4edb-b363-425d295877d3n%40googlegroups.com.

[Karen K]

Thank you for clarification.

[Karen K]

Sebastian E schrieb am Montag, 22. Juli 2024 um 13:56:45 UTC+2:

Hello everyone, I currently have the problem that my RAM is regularly full and I have to restart the server.

I have now seen that the weectl import service is not stopped. I'm attaching the output. Does anyone have an idea what this Area 31 30 means? That's where it seems to hang. Thank you

See https://groups.google.com/g/weewx-user/c/x0YJIz-aMYg/m/wx3ZgWGpAAAJ for explanation and further instructions.