Cascading errors for the Seasons base skin.

[Sylvain M]

Hello,

I have just installed weewx under opensuse with the repository according to the user manual method.
My HTML_ROOT = /srv/www/htdocs/weewx/

My base is mysql.

I managed to put my translation into French but my web interface does not have an image in the base Seasons skin to get back to basics.

I set debug to 1 and my system log gives me lots of errors. Here are the last 3000 lines: https://pastebin.com/dJKQ53Kw

Can you help me resolve these errors?
What's not working?

Thank you for your help.

[Tom Keffer]

As you say, there are several errors here.

The first is caused by changes in Pillow, the library WeeWX uses for imaging. A number of features were deprecated in Pillow 10.0. WeeWX Version 5 will have workarounds for these features, but, for now, the solution is to go back to Pillow version 9.5. I don't know how to do that on the SuSE platform, but I'm sure it's possible.

It seems that you are using the Bootstrap skin? It is causing the 2nd error (KeyError: 'minvalues'). I do not know anything about this skin, but make sure your copy is up to date --- perhaps the author has fixed it.

One of the templates in one of your skin is using the unit 'hpa'. The name is actually "hPa". Note the capital "P".

Finally, you are getting a number of "Duplicate entry" errors caused by trying to insert data into the database for time values that already exist in the database. In this case, for records back in July 2022, over a year ago. You didn't say what kind of weather station you are using, but its implementation of genArchiveRecords() is not quite right. Or, perhaps the log is from an archive dump? It's impossible to tell without seeing earlier parts of the log from when weewxd starts up.

Hope that gets you started.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2c404e14-b1ec-485d-bdaf-99bfa4c76e6fn%40googlegroups.com.

[Sylvain M]

It is not possible to downgrade the Pillow version to 9.5 under OpenSuse Tumbleweed (OpenSuse forum).
The repository I used to install is dedicated to OpenSuse Leap.
However, I had asked the question of compatibility with Tumbleweed on this weewxx-user forum but without an answer which I interpreted as obvious.

I'm going to have to stop weewx until there is an update that is compatible with Pillow 10.0.1 at best and above all that is compatible with OpenSuse Tumbleweed.

My weather station is a Bresser Wifi Comfort Weather Center 7-in-1 which sends data to Wunderground.
I imported the data from Wunderground with the import tool to populate my mysql database with the wee_import tool (perhaps not adapted to my version?). I updated it every day.

The goal for me is to have a history of past months and years to compare different parameters such as temperatures, rain or wind. It’s tracking weather history that fascinates me.
I have done very little manipulation since I recently installed weewx (October 12, 2023).

As I do not have a computer permanently connected, importing data seemed ideal to me, especially since my station is not supported by current drivers.

[Tom Keffer]

Tumbleweed doesn't show earlier versions of packages? What do you see if you do (note the -v flag):

zypper search -v python3-Pillow

If I do it on Leaf, I see that V8.3.1 is available. Then

sudo zypper install python3-Pillow-8.3.1-lp152.5.3.1 

would install it. Any version of Pillow later than v5.2 will work.

Alternatively, the beta for V5 has a patch for Pillow 10.0.  Installation using pip

While it's in beta, it's pretty stable.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b9ef6ce2-8874-4d0b-83bd-ac94a2269f90n%40googlegroups.com.

[Sylvain M]

To order

zypper search -v python3-Pillow
 
I have in return

No matching items found.

I couldn't find version 5 beta for download in .tar.gz.
I just found it in .deb

Is there an address to download this version 5 to install with pip?
I will be happy to test it.

[Tom Keffer]

See my previous email for instructions for installing with pip.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3d319fb6-bc1f-4db5-b2c2-4b3f129d2defn%40googlegroups.com.

[Sylvain M]

THANKS. The installation is done with pip without any hassle.

Everything works fine with my installation with pip.
Weewx does not want to work on my Apache server although I have set the WEEWX_ROOT = /srv/www/htdocs/weewx/
I put it back to default.
So I made some links and everything works. I will see the cause later since you indicate in the documentation that it is possible.

The most annoying thing is that I have errors with wee_import to feed my mysql database:

sylvain@gandalf:~/weewx-venv/lib/python3.11/site-packages> python3 wee_import.py --config=/home/sylvain/weewx-data/weewx.conf --import-config=/home/sylvain /weewx-data/util/import/wu-example.conf --dry-run --from=2022-7-12 --to=2023-10-24 --verbose
Using WeeWX configuration file /home/sylvain/weewx-data/weewx.conf
Starting wee_import...
Observation history for Weather Underground station 'I********' will be imported.
The following options will be used:
      config=/home/sylvain/weewx-data/weewx.conf, import-config=/home/sylvain/weewx-data/util/import/wu-example.conf
      station=I*******, from=2022-7-12, to=2023-10-24
      apiKey=****************************0b80
      dry-run=True, calc_missing=True, ignore_invalid_data=True
      slice=250, interval=x, wind_direction=[0.0, 360.0]
Using database binding 'wx_binding', which is bound to database 'weewx'
Destination table 'archive' unit system is '0x11' (METRICWX).
Missing derived observations will be calculated.
Observations timestamped after 2022-07-12 00:00:00 CEST (1657576800) and up to and
including 2023-10-25 00:00:00 CEST (1698184800) will be imported.
This is a dry run, imported data will not be saved to archive.
Starting dry run import...
Records covering multiple periods have been identified for import.
Period 1...
Obtaining raw import data for period 1 ...
Traceback (most recent call last):
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/wee_import.py", line 899, in <module>
     hand()
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/wee_import.py", line 829, in main
     source_obj.run()
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/weeimport/weeimport.py", line 375, in run
     _raw_data = self.getRawData(period)
                 ^^^^^^^^^^^^^^^^^^^^^
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/weeimport/wuimport.py", line 249, in getRawData
     _raw_data = f.read().decode(char_set)
                 ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: decode() argument 'encoding' must be str, not None

What did I configure incorrectly?

[gjr80]

It's is not your config, it looks like WU has changed the header content in one (or more) of their API responses. I should be able to have a look tonite and provide a fix.

Gary

[gjr80]

I've patched wee_import to handle the changed WU API response. The patched code will appear in the v5.0.0 (when released). In the meantime if you wish to use it now with any other the v5.0.0 beta releases you can by downloading the patched wuimport.py and replacing your existing version. The location of the v5 WeeWX files is significantly different to that for v4. You indicated you installed v5 via pip, so the following instructions should work in this case (or until I have a better approach for installing patched file):

1. download the patched wuimport.py:

$ wget -P /var/tmp https://raw.githubusercontent.com/weewx/weewx/V5/src/weeimport/wuimport.py

2. locate your existing wuimport.py. If you followed the v5 install instructions and installed via pip you may find wuimport.py in something like ~/weewx-venv/lib/python3.9/site-packages/weeimport . If you used a different virtual environment name to weewx-venv or are running a different python version change the path accordingly, or depending on your system the path may be different again. Either way you need to find wuimport.py. 

3. If you cannot find wuimport.py on your system or are not sure do not proceed or guess - stop and seek clarification. Once you have found wuimport.py copy the downloaded version in /var/tmp over the existing version. 

4. you can now use wee_import to import from WU.

Gary

[Sylvain M]

Thank you Gary for your work and your explanations.

Your explanations were very clear and I had already spotted all these files.

I had some errors which did not seem to me to be related to the import, I tried an update with
python3 -m pip install weewx --upgrade

who actually updated my weewx and everything was back to normal.

I was able to do my wee_import without difficulty. I even found that the process was faster than the first time.

However, I have a question, when I do an update with the --date=2023-10-27 option, why does the import want to do an import of 2023-10-28 too?

This gives me the errors:

___________________________________________________________________________________________________________________________________________________________________________________
$ python3 wee_import.py --config=/home/sylvain/weewx-data/weewx.conf --import-config=/home/sylvain/weewx-data/util/import/wu-example.conf --date=2023- 10-27 --verbose

Using WeeWX configuration file /home/sylvain/weewx-data/weewx.conf
Starting wee_import...

Observation history for Weather Underground station 'IG******' will be imported.

The following options will be used:
      config=/home/sylvain/weewx-data/weewx.conf, import-config=/home/sylvain/weewx-data/util/import/wu-example.conf

      station=IG******, date=2023-10-27
      apiKey=****************************0b80
      dry-run=None, calc_missing=True, ignore_invalid_data=True
      slice=250, interval=1, wind_direction=[0.0, 360.0]

Using database binding 'wx_binding', which is bound to database 'weewx'
Destination table 'archive' unit system is '0x11' (METRICWX).
Missing derived observations will be calculated.

Observations timestamped after 2023-10-27 00:00:00 CEST (1698357600) and up to and
including 2023-10-28 00:00:00 CEST (1698444000) will be imported.
Starting import...

Records covering multiple periods have been identified for import.
Period 1...
Obtaining raw import data for period 1 ...

Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 155 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 155; Last timestamp: 2023-10-27 12:54:52 CEST (1698404092)
Mapped data saved to archive successfully ""for period 1.
Period 2...
Obtaining raw import data for period 2 ...
**** Unable to load source data for period 2.
**** Possibly a bad station ID, an invalid date or data does not exist for this period.
****Period 2 will be skipped. Proceeding to next period.
Calculating missing derived observations ...
Processing record: 155; Last record: 2023-10-28 00:00:00 CEST (1698444000)
Recalculating daily summaries...

Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import

Traceback (most recent call last):
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/wee_import.py", line 899, in <module>
     hand()
   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/wee_import.py", line 829, in main
     source_obj.run()

   File "/home/sylvain/weewx-venv/lib/python3.11/site-packages/weeimport/weeimport.py", line 509, in run
     _msg = "%d records were processed and %d unique records " \
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^
TypeError: must be real number, not NoneType

______________________________________________________________________________________________________________________________________________________________________________________________

I had the same errors with the --from=xxxx --to=xxxxx, it adds an extra day to the import.

Do you have a tip for not exceeding the date limit?

I wanted to edit a cron to automatically update every 15 minutes or according to the last date recorded with a bash script but this could be complicated for me especially since these errors are annoying and I have to deal with a datetime in the format of the Epoch.

[vince]

For the question about dates, you specified --date=2023-10-27 and it looked for all records up to and the end of that date, which is 00:00:00 2023-10-28, so it seems to have worked as expected according to the description in  https://www.weewx.com/docs/5.0/devnotes/#archive-records

Without seeing your .conf files (please remove any identifiers and keys) it's difficult to guess about your TypeError in the transcript you provided

[Sylvain M]

Yes indeed, I understand the overflow on 2023-20-28. It’s a notion of milestone that I didn’t understand.
The language barrier and translation approximations do not help in understanding the subtleties of language.

I just understood that the --from determines the inclusive close date values while the --to determines the exclusive date of the imported records.
So why these errors?

For the .conf files that you tell me about must be
weewx.conf: https://pastebin.com/91m9eRPK
and my import file
wu-example.conf: https://pastebin.com/NzwNsgcM

You may discover a configuration error.

While manipulating the dates and their formatting to be compatible with wu__import's formatting, I discovered that specifying dates with times --from=2023-10-27T12:54 --to=2023 -10-28T00:45 does not generate any errors and exits with the success message.

[gjr80]

Let me sit down and work through the WU import. WU has a history for changing things and not for the better. They have changed the encoding of their API response so goodness knows what else may have changed as well.

Gary

[gjr80]

I have been able to reproduce your symptoms and I now believe I understand what is happening.

In terms of why wee_import wants to import from 27 and 28 October Vince has covered the main point. So as far as WeeWX is concerned (and assuming a five minute archive period) the first archive record of the 27 October covers the five minute period from midnight at the start of 27 October until 00:05, so the first archive period of the day is timestamped at five minutes after midnight. The last archive period covers the five minute period from 23:55 until (and including) midnight at the end of 27 October, so the last archive period of 27 October is timestamped at midnight on 28 October. 

The other factor at play is how WU provides data via its API. The WU API call used by wee_import provides one full day of data at a time, the problem being that WU defines a day as starting at midnight at the start of the day though until (but not including) midnight at the end of the day. So when the WU API responds with data for 27 October the first record provided is the archive record timestamped at midnight at the start of 27 October, this archive record actually contains data from 26 October so it is discarded. The last record provided by WU is timestamped at 23:55 27 October, which is only the second last archive record for the day. WU does not provide the last archive record of the day, to get that archive record we need to ask WU for the data for 28 October. The first record from the WU 28 October data is used and the rest are discarded.

That is why when you attempt to import a given day of data from WU you will actually see wee_import obtain two days data from WU.

Now for your error message. To cut a long story short there was a flaw in how wee_import calculated the total elapsed time for an import. I believe I have fixed it. You can use the new code by downloading the updated weeimport.py and replacing weeimport.py on your system with the downloaded version. In an earlier post I gave you instructions for replacing wuimport.py, this time you need to replace weeimport.py, it is in the same directory as wuimport.py and the process is the same just a different file. You can download the updated weeimport.py via the following command:

$ wget -P /var/tmp https://raw.githubusercontent.com/weewx/weewx/V5/src/weeimport/weeimport.py

Do not get weeimport.py confused with wee_import.py, they are in different directories but if you replace the wrong file you will break wee_import.

Once you have replaced weeimport.py wee_import should work without error.

Gary

[Sylvain M]

Hello Gary.

Thank you very much for your work !
I updated the weeimport.py file at 6:50 p.m.

I created a small bash script (I'm not a programmer) to download my WU data every hour (I didn't dare to be more greedy). The script is triggered by the crontab.
My script retrieves the last timestamp from the mysql database and uses it as dateFrom=%Y-%m-%dT%H:%M. The recovery date is dateTo=%Y-%m-%dT%H:%M. In principle everything works well.

If I understood correctly, the calculation error that you detected can explain my mysql errors of this type 1062, "Duplicate entry '1698663292' for key 'PRIMARY":

# journalctl -u weewx.service -p err -b -f --since today
...
Oct 30 17:28:01 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 03:55:00 CEST (1657590900) to database 'weewx': (1062, " Duplicate entry '1657590900' for key 'PRIMARY'")
Oct 30 17:33:02 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:00:00 CEST (1657591200) to database 'weewx': (1062, " Duplicate entry '1657591200' for key 'PRIMARY'")
Oct 30 17:38:03 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:05:00 CEST (1657591500) to database 'weewx': (1062, " Duplicate entry '1657591500' for key 'PRIMARY'")
Oct 30 17:43:04 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:10:00 CEST (1657591800) to database 'weewx': (1062, " Duplicate entry '1657591800' for key 'PRIMARY'")
Oct 30 17:48:04 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:15:00 CEST (1657592100) to database 'weewx': (1062, " Duplicate entry '1657592100' for key 'PRIMARY'")
Oct 30 17:53:04 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:20:00 CEST (1657592400) to database 'weewx': (1062, " Duplicate entry '1657592400' for key 'PRIMARY'")
Oct 30 17:58:05 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:25:00 CEST (1657592700) to database 'weewx': (1062, " Duplicate entry '1657592700' for key 'PRIMARY'")
Oct 30 18:03:05 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:30:00 CEST (1657593000) to database 'weewx': (1062, " Duplicate entry '1657593000' for key 'PRIMARY'")
Oct 30 18:08:05 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:35:00 CEST (1657593300) to database 'weewx': (1062, " Duplicate entry '1657593300' for key 'PRIMARY'")
Oct 30 18:13:05 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:40:00 CEST (1657593600) to database 'weewx': (1062, " Duplicate entry '1657593600' for key 'PRIMARY'")
Oct 30 18:18:07 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:45:00 CEST (1657593900) to database 'weewx': (1062, " Duplicate entry '1657593900' for key 'PRIMARY'")
Oct 30 18:23:07 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:50:00 CEST (1657594200) to database 'weewx': (1062, " Duplicate entry '1657594200' for key 'PRIMARY'")
Oct 30 18:28:07 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 04:55:00 CEST (1657594500) to database 'weewx': (1062, " Duplicate entry '1657594500' for key 'PRIMARY'")
Oct 30 18:33:07 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:00:00 CEST (1657594800) to database 'weewx': (1062, " Duplicate entry '1657594800' for key 'PRIMARY'")
Oct 30 18:38:08 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:05:00 CEST (1657595100) to database 'weewx': (1062, " Duplicate entry '1657595100' for key 'PRIMARY'")
Oct 30 18:43:08 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:10:00 CEST (1657595400) to database 'weewx': (1062, " Duplicate entry '1657595400' for key 'PRIMARY'")
Oct 30 18:48:08 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:15:00 CEST (1657595700) to database 'weewx': (1062, " Duplicate entry '1657595700' for key 'PRIMARY'")
Oct 30 18:53:08 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:20:00 CEST (1657596000) to database 'weewx': (1062, " Duplicate entry '1657596000' for key 'PRIMARY'")
Oct 30 18:58:09 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:25:00 CEST (1657596300) to database 'weewx': (1062, " Duplicate entry '1657596300' for key 'PRIMARY'")
Oct 30 19:03:09 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:30:00 CEST (1657596600) to database 'weewx': (1062, " Duplicate entry '1657596600' for key 'PRIMARY'")
Oct 30 19:08:09 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:35:00 CEST (1657596900) to database 'weewx': (1062, "Duplicate entry '1657596900' for key 'PRIMARY'")
Oct 30 19:13:09 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:40:00 CEST (1657597200) to database 'weewx': (1062, " Duplicate entry '1657597200' for key 'PRIMARY'")
Oct 30 19:18:10 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:45:00 CEST (1657597500) to database 'weewx': (1062, " Duplicate entry '1657597500' for key 'PRIMARY'")
Oct 30 19:23:10 gandalf python3[1261]: weewx[1261] ERROR weewx.manager: Unable to add record 2022-07-12 05:50:00 CEST (1657597800) to database 'weewx': (1062, " Duplicate entry '1657597800' for key 'PRIMARY'")


I don't understand why there are Duplicate entry errors on different keys?

Do you have any idea what's going on?

[gjr80]

When importing records with wee_import primary key duplicate entry log entries indicate a record with the timestamp of the current imported record already exists in the database. This most commonly occurs if the import period overlaps with existing database data. This is normal and not considered an error. In such cases the imported record is discarded and no database changes are made. If the imported record is required rather than the corresponding record in the database, the database record must be deleted and the import performed again. If the database record is to be retained then nothing need be done.

Gary

[Sylvain M]

It's not joy.
I no longer have access to my Wunderground data with a 401 error message like this:

# ./maj_meteo.sh

Using WeeWX configuration file /home/sylvain/weewx-data/weewx.conf
Starting wee_import...

Observation history for Weather Underground station 'I*****' will be imported.

Using database binding 'wx_binding', which is bound to database 'weewx'
Destination table 'archive' unit system is '0x11' (METRICWX).
Missing derived observations will be calculated.

Observations timestamped after 2023-10-31 00:05:00 CET (1698707100) and up to and
including 2023-10-31 12:15:00 CET (1698750900) will be imported.

Starting import...
Records covering multiple periods have been identified for import.

Unable to open Weather Underground station I***** or HTTP Error 401: Unauthorized
**** Unable to load config file.
****HTTP Error 401: Unauthorized
****Nothing done, exiting.

I sent an email to the Feeback service to ask the cause of this blockage since there is no quota imposed when the API key is provided to us.
I also requested the restoration of access to my data, explaining that I was not a flow vacuum cleaner. Just an amateur tweaking his weather software.
I am waiting for a response and action.

Have there ever been flow blockages before?

Furthermore, I still have this duplicate primary key error but in 2022, which is the first day of sending data from my weather station. So every 5 minutes I get these errors that I have absolutely no idea how to resolve. I'll only paste the last 10 lines:

# journalctl -u weewx.service -p err -b -f --since today

.....
Oct 31 16:49:35 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 05:45:00 CEST (1657597500) to database 'weewx': (1062, " Duplicate entry '1657597500' for key 'PRIMARY'")
Oct 31 16:54:35 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 05:50:00 CEST (1657597800) to database 'weewx': (1062, " Duplicate entry '1657597800' for key 'PRIMARY'")
Oct 31 16:59:36 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 05:55:00 CEST (1657598100) to database 'weewx': (1062, " Duplicate entry '1657598100' for key 'PRIMARY'")
Oct 31 17:04:36 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:00:00 CEST (1657598400) to database 'weewx': (1062, " Duplicate entry '1657598400' for key 'PRIMARY'")
Oct 31 17:09:36 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:05:00 CEST (1657598700) to database 'weewx': (1062, " Duplicate entry '1657598700' for key 'PRIMARY'")
Oct 31 17:14:36 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:10:00 CEST (1657599000) to database 'weewx': (1062, " Duplicate entry '1657599000' for key 'PRIMARY'")
Oct 31 17:19:37 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:15:00 CEST (1657599300) to database 'weewx': (1062, " Duplicate entry '1657599300' for key 'PRIMARY'")
Oct 31 17:24:37 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:20:00 CEST (1657599600) to database 'weewx': (1062, " Duplicate entry '1657599600' for key 'PRIMARY'")
Oct 31 17:29:37 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:25:00 CEST (1657599900) to database 'weewx': (1062, " Duplicate entry '1657599900' for key 'PRIMARY'")
Oct 31 17:34:37 gandalf python3[1297]: weewx[1297] ERROR weewx.manager: Unable to add record 2022-07-12 06:30:00 CEST (1657600200) to database 'weewx': (1062, " Duplicate entry '1657600200' for key 'PRIMARY'")

If you have an explanation and a solution to resolve this error, it would be appreciated?

Thank you for your help.

[Sylvain M]

Linux Opensuse made a big update that forced all python programs to use the python3.11 version.
This way of doing things caused a big mess in programs like tuned, firewalld or sendmail.
Weewx had become completely inoperable.

I found a solution by making a link from python3.11 to python3 in the /usr/bin/ folder.
It is effective and solves the problem.
If this solution can help.

The solution to my blockage and some practical information about Wunderground.

My blockage was due to an update with pip which allowed me to receive the weewx version 5.0.0b15.
At the same time, I updated the documentation with the command
weectl station upgrade --what docs examples util
which replaced my wu-example.conf file. He was a virgin.

I made a copy/paste error filling this file with my API key information. This explains the 401 error.

The exchange with Wunderground Feedback taught me that we had a quota of transactions which is not indicated anywhere on the page dedicated to the API key.
It’s limited to 1500 calls per day and 30 calls per minute.
Always good to know.

Since November 6 I have been receiving erratic data with negative temperatures even though they are around 11°C. There is also the example of "current conditions" which indicate 1.4°C and the "statistics" indicate 21.1°C while my station indicates 11.2°C.

I think strongly of Gary who previously explained that there were unexpected changes in data flows. Is that the case ?

There are also holes in my diagrams.
I don't know how to check the continuity of temporal data in my database?

What mistake did I make to have hatched diagrams instead of standard curves?

I still can't convert my wind data from m/s to km/h as well as mbar to hPa. However, I have read the documentation and forum threads which for some no longer apply to version 5 of weewx.
I used the settings recommended in weewx.conf:
[[Defaults]]
[[[Units]]]
[[[Groups]]]]
                  group_altitude = meter # Options are 'foot' or 'meter'
                  group_pressure = hPa # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'
                  group_rain = mm # Options are 'inch', 'cm', or 'mm'
                  group_rainrate = mm_per_hour # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'
                  group_temperature = degree_C # Options are 'degree_C', 'degree_F', or 'degree_K'
                  group_speed = km_per_hour # Wind and gust speed
                  group_speed2 = km_per_hour2

Thank you for your help.

[Sylvain M]

Hello Gary,

Do you think there has been a new change in the data flow protocol coming from Wunderground?
I always receive erratic data with no relation to reality.

On my diagrams, I no longer have hatched areas but still discontinuous curved lines.

[gjr80]

My apologies for not replying to your previous post.

I have no knowledge of any changes to the data being provided by WU. In working on some wee_import. change over theist week I have done a number of imports for my station and they have worked without error. I have not checked the veracity of the actual PWS data though. In the past WU has been somewhat flaky/unreliable/hit and miss, call it what you may. However, in recent times they have been more stable. If you are having issues with wildly inaccurate data you might like to do a few checks on your end before taking the issue up with WU. Can you monitor the data that is being uploaded by your station to WU and compare it against what is actually recorded by WU (as opposed to what wee_import obtains via the WU API). If you log onto WU and look at your station towards the bottom of the page you should be able to display your stations historical data either graphically or in tabular form. Does that data  look correct or does it show the non-sense values. If the graphical/tabulated data is correct and wee_import is receiving non-sense values then perhaps it is the WU API or wee_import.

On my diagrams, I no longer have hatched areas but still discontinuous curved lines.

 

I am not sure what you mean by this, do you mean gaps in plot lines, do you mean something like the second plot in the Line gaps section of the Customisation Guide? Gaps are typically caused by a lack of data over a period of time. Given you are importing data from WU this could be a lack of data on WU or a problem with your import. It could also be that you have archive records in the WeeWX database that cover that period but those records are missing some data fields. I would suggest identifying the time/period of the gap(s), looking at WU to see if the missing data exists and re-importing if the WU data does exist. If this does not fix the problem I would check your database to see if you do indeed have archive records covering that period, if you do delete those records and then re-import the same period. 

I still can't convert my wind data from m/s to km/h as well as mbar to hPa. However, I have read the documentation and forum threads which for some no longer apply to version 5 of weewx.
I used the settings recommended in weewx.conf:
[[Defaults]]
[[[Units]]]
[[[Groups]]]]
                  group_altitude = meter # Options are 'foot' or 'meter'
                  group_pressure = hPa # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'
                  group_rain = mm # Options are 'inch', 'cm', or 'mm'
                  group_rainrate = mm_per_hour # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'
                  group_temperature = degree_C # Options are 'degree_C', 'degree_F', or 'degree_K'
                  group_speed = km_per_hour # Wind and gust speed
                  group_speed2 = km_per_hour2

Units for plots (among other things) can set in a number of locations within WeeWX. The base default settings are in the weewx.defaults, units can be set by the user/skin developer in the skin config file (skin.conf) [ImageGenerator] stanza, the [StdReport] [[Defaults]] [[[Units]]] [[[[Groups]]]] stanza in weewx.conf or the skin report stanza under [StdReport] in weewx.conf (eg [StdReport] [[SeasonsReport]] for the Seasons skin). If you are using a skin that supports internationalisation the internationalisation settings may also change the plot units. The Processing order section of the Customization Guide explains the processing order of these locations/options.

Looking at your plots, and given you have an internationalised skin, I suspect that you might find your internationalisation settings are forcing the use of MetricWx units. Under [StdReport] [[SeasonsReport]] I expect you currently have something like:

[StdReport]

    ....

    [[SeasonsReport]]

        ....

        lang = fr

Try adding an override for the units you wish to display:

[StdReport]

    ....

    [[SeasonsReport]]

        ....

        lang = fr

        [[[Units]]]

            [[[[Groups]]]]

                group_pressure = hPa

                group_speed = km_per_hour

I've not used internationalisation/localisation so I might have the exact overrides wrong, but see how that goes.

Gary

[Sylvain M]

As my English is no longer correct as before, I use an online translator with the approximations that this entails. I am sorry for that.

Concerning erratic values.
I actually took the same approach as you recommend, that is to say monitoring the graphics of the WU pages which are correct and consistent with the values displayed by the console of my PWS. I deduce that the values sent by my PWS station are correct and correctly recorded by WU since the graphs correspond to reality. Note that I also send my station data to weathercloud.net whose graphics are identical to WU.

To import my data, I of course use wee_import in a simple bash script which retrieves the last recorded timestamp which is converted into dateTime which will serve as a variable for the dateFrom. The date of moment T is used for the dateTo variable.
I submit my script on pastbin for your study and the community in case I have made a programming error:
https://pastebin.com/biYjHe1E

The strange thing is that the values recorded in mysql for today 11/16/2023 are correct except from 11:59 p.m. On the other hand, the graph of the outTemp column (outside temperature?) of weewx is completely erratic since it indicates temperatures of -5°C while the database indicates 8.27°C. Same after erasing and importing data.
Is there a problem interpreting the weewx script?
The second strange part is the recording of the outTemp temperature at 0.32°C and the outHumidity which suddenly dropped from 95% to 79.99% at 11:59 p.m.
From a WU quota point of view, my 286 recordings for the day are far from 1500. I don't understand.

Concerning the units.
Indeed, I use the season skin since I use weewx version 5.0.0b15.
My understanding is that weewx.conf settings take precedence over other settings in other files.
I think I understood your advice.
So that you have an overall view you will find my weewx.conf file
https://pastebin.com/BtUtTZzB

I hope it will be more meaningful this way.

Sylvain