import wu daily summariies are not complete

[Artvd]

 I am trying  to import wunderground data with weectl import using the Weewx 5.1 guidance and standard settings in wu.config

During dry run there were no problems. But in the " wet" run i get the message:

"Daily summaries are not complete. Try again without from/to dates" 

I would like to import 4 years and hesitate to work without dates. Is there another possibility for resolve this problem? 

Below the relevant part from the terminal. 

$ weectl import --import-config=/var/tmp/wu.conf  --from=2021-01-01T00:00 --to=2021-01-10T00:00
Using configuration file /home/avd/weewx-data/weewx.conf
Starting weectl import...
Observation history for Weather Underground station 'IVAUXS11' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Imported records will not overwrite existing database records.
Missing derived observations will be calculated.
Observations timestamped after 2021-01-01 00:00:00 CET (1609455600) and up to and
including 2021-01-10 00:00:00 CET (1610233200) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 74; Last timestamp: 2021-01-01 23:45:54 CET (1609541154)
Period 2 ...
Unique records processed: 74; Last timestamp: 2021-01-02 23:45:54 CET (1609627554)
Period 3 ...
Unique records processed: 94; Last timestamp: 2021-01-03 23:45:53 CET (1609713953)
Period 4 ...
Unique records processed: 96; Last timestamp: 2021-01-04 23:45:54 CET (1609800354)
Period 5 ...
Unique records processed: 96; Last timestamp: 2021-01-05 23:45:53 CET (1609886753)
Period 6 ...
Unique records processed: 84; Last timestamp: 2021-01-06 23:45:54 CET (1609973154)
Period 7 ...
Unique records processed: 70; Last timestamp: 2021-01-07 23:45:53 CET (1610059553)
Period 8 ...
Unique records processed: 71; Last timestamp: 2021-01-08 23:45:53 CET (1610145953)
Period 9 ...
Unique records processed: 71; Last timestamp: 2021-01-09 23:45:54 CET (1610232354)
Period 10 ...
Period 10 - no records identified for import.
Calculating missing derived observations ...
Processing record: 730; Last record: 2021-01-10 00:00:00 CET (1610233200)
Recalculating daily summaries...
**** Daily summaries are not complete. Try again without from/to dates.
**** Nothing done, exiting.

[gjr80]

It appears that your daily summaries have somehow ended up in a state where they are out of sync with the archive records in the database. When the daily summaries are in this state the partial update of the daily summaries that occurs after missing derived records are calculated is prevented from being run. This is what you are seeing. The solution is to rebuild your daily summaries separately before re-importing the data concerned. To do this try running:

weectl database rebuild-daily

This will rebuild the daily summaries and should see the daily summaries back in sync with the archive. You should then be able to re-import your WU data. Note the original non-dry-run import should have resulted in the imported WU records being saved to archive (it was just the re-calculated derived obs and daily summary rebuild that failed), so when you re-import the same data you will likely see many 'duplicate record' warnings in the log. These warnings can be ignored, the important thing is the import completes and exits normally.

Gary

[Artvd]

Thank for your reaction. I run the rebuild  but it  went in a slit second. Indeed the data was in the archive and i therefor tried an second import with only changing the dates. The first time it was confiirmed to be OK but there was no data in the archive. Again with only changing the dates i get the following message from the terminal. 

eectl import --import-config=/var/tmp/wu.conf  --from=2023-01-01T00:00 --to=2023-01-06T00:00

Using configuration file /home/avd/weewx-data/weewx.conf
Starting weectl import...
Observation history for Weather Underground station 'IVAUXS11' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Imported records will not overwrite existing database records.
Missing derived observations will be calculated.

Observations timestamped after 2023-01-01 00:00:00 CET (1672527600) and up to and
including 2023-01-06 00:00:00 CET (1672959600) will be imported.

Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...

Unable to open Weather Underground station IVAUXS11  or  HTTP Error 401: Unauthorized
**** Unable to load config file.
**** HTTP Error 401: Unauthorized
Traceback (most recent call last):
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_actions.py", line 58, in obs_import
    source_obj.run()
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weeimport/weeimport.py", line 367, in run
    _raw_data = self.get_raw_data(period)
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weeimport/wuimport.py", line 322, in get_raw_data
    response = urllib.request.urlopen(request_obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/avd/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectl.py", line 67, in main
    namespace.func(namespace)
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py", line 90, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_cmd.py", line 85, in import_func
    weectllib.import_actions.obs_import(config_dict,
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_actions.py", line 110, in obs_import
    log.info(f"**** " % e)
             ~~~~~~~~~^~~
TypeError: not all arguments converted during string formatting
(weewx-venv) avd@raspberrypi:~ $ weectl import --import-config=/var/tmp/wu.conf  --from=2021-01-01T00:00 --to=2022-01-06T00:00

Using configuration file /home/avd/weewx-data/weewx.conf
Starting weectl import...
Observation history for Weather Underground station 'IVAUXS11' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Imported records will not overwrite existing database records.
Missing derived observations will be calculated.
Observations timestamped after 2021-01-01 00:00:00 CET (1609455600) and up to and

including 2022-01-06 00:00:00 CET (1641423600) will be imported.

Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...

Unable to open Weather Underground station IVAUXS11  or  HTTP Error 401: Unauthorized
**** Unable to load config file.
**** HTTP Error 401: Unauthorized
Traceback (most recent call last):
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_actions.py", line 58, in obs_import
    source_obj.run()
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weeimport/weeimport.py", line 367, in run
    _raw_data = self.get_raw_data(period)
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weeimport/wuimport.py", line 322, in get_raw_data
    response = urllib.request.urlopen(request_obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/avd/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectl.py", line 67, in main
    namespace.func(namespace)
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py", line 90, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_cmd.py", line 85, in import_func
    weectllib.import_actions.obs_import(config_dict,
  File "/home/avd/weewx-venv/lib/python3.11/site-packages/weectllib/import_actions.py", line 110, in obs_import
    log.info(f"**** " % e)
             ~~~~~~~~~^~~
TypeError: not all arguments converted during string formatting

Op maandag 17 februari 2025 om 21:56:43 UTC+1 schreef gjr80:

[gjr80]

The speed of the rebuild is not the issue, the speed will vary depending on the number of archive records that need to be read to do the rebuild. Better to post the console output than try to describe it, the console output will give details on the number of days rebuilt, the number of records used and the time taken. 

As for the 401 error, that is indicative of an incorrect API key, though if you completed a previous WU import using the same import config file perhaps not. In any case you might like to check your API key in your import config file against what is recorded on your WU account. The other possibility is WU misbehaviour; WU has had a long history of random dropouts, failures and generally not working at times. In that case all you can do is wait and try again later. 

Gary

[Artvd]

I have renewed the API. Also changed the skin to standard. And retried many times to import without succes (both with conf and with derive). I can login normally to WU 

My website and WU are updated with my current data.  It is so strange i could import 2021, but not the subsequent years of data from WU.

I am posting the terminal output, but it does not give much info as you can see.  And also the end of my many efforts to import the WU data.

Unique records processed: 96; Last timestamp: 2022-02-25 23:45:53 CET (1645829153)

Period 57 ...
Unique records processed: 96; Last timestamp: 2022-02-26 23:45:54 CET (1645915554)
Period 58 ...
Unique records processed: 96; Last timestamp: 2022-02-27 23:45:53 CET (1646001953)
Period 59 ...
Unique records processed: 96; Last timestamp: 2022-02-28 23:45:53 CET (1646088353)
Period 60 ...
Period 60 - no records identified for import.

Calculating missing derived observations ...

Processing record: 5648; Last record: 2022-03-01 00:00:00 CET (1646089200)

Recalculating daily summaries...
**** Daily summaries are not complete. Try again without from/to dates.
**** Nothing done, exiting.

(weewx-venv) avd@raspberrypi:~ $ source ~/weewx-venv/bin/activate
(weewx-venv) avd@raspberrypi:~ $ weectl database rebuild-daily
Using configuration file /home/avd/weewx-data/weewx.conf
All daily summaries will be rebuilt.
Rebuild the daily summaries in the database 'weewx.sdb' (y/n)? y
Rebuilding daily summaries in database 'weewx.sdb' ...
Daily summaries up to date in 'weewx.sdb'.

Op dinsdag 18 februari 2025 om 03:37:24 UTC+1 schreef gjr80:

[Artvd]

I have  again unsuccesfully retried to import WU data  to  my archives. Please see the final lines of the terinal output below. 

At the  end of the import Weewx refers  to  the weectl log file. Can somebody please tell me where this log file should be located and what is the correct name of that log file. 

Secondly question i  have is about NOAA files on my server. Is it correct that i find these files not only in the archives  map but also in a map called NOAA? 

" Recalculating daily summaries...
Records processed: 5000; time: 2022-02-20 05:45:53 CET (1645332353)
Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
5840 records were processed and 5840 unique records imported in 78.19 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the weectl log file." 

Op dinsdag 18 februari 2025 om 14:21:17 UTC+1 schreef Artvd: