import from WU returns only 1 row from actual date

[Sebastian Eckelmann]

Hello everyone,

I would like to retrieve the data from WU via wee_import with API. At the moment I only have the data set from the night of 00:04:52. With the command: wee_import --import-config=/etc/weewx/import/wu-example.conf --from=2022-09-20T22:30 --to=2022-09-27T06:00 I get the entire records. But the last one is from the night 00:04. Is it even possible to call up current data at intervals of about 10 minutes or only from the previous day? Thank you

    

[Rainer Lang]

Maybe WU only allows downloads from the last or earlier completed day(s) (of the user's time zone) via the API with wee_import (or any other download 'tool') ...
That would explain this behaviour.

Anyone has any information of some such restriction ?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f902011a-35f8-4ba1-8042-263630b5a87fn%40googlegroups.com.

[gjr80]

Before I look into this I would like to understand exactly what it is you are doing and the results you are seeing.

As I understand it:

1. you are running wee_import --import-config=/etc/weewx/import/wu-example.conf --no-prompt via cron on the hour and at 10,20,30,40 and 50 minutes past the hour.

2. each time wee_import is run via cron the only record that is imported is 00:04:52 (note the emphasis on each and only)

3. when you run wee_import --import-config=/etc/weewx/import/wu-example.conf --from=2022-09-20T22:30 --to=2022-09-27T06:00 from the command line all expected records are imported except for the last day, on the last day only the 00:04:52 record is imported

Is this understanding correct?

Also, in future please avoid posting terminal screen captures as images; they are difficult to read on mobile devices and are not searchable. Posting terminal captures as text makes our job easier.

Gary

[Sebastian Eckelmann]

First, if i query the WU API manually, i get all results from the actual date. There is not such a restriction in the API itself.

@Gary:

thanks for your interest, i will try to explain the situation as good as possible.

1. Yes, that's right. 

2. Yes, each time only this record is imported. wee_import says each time:

Starting import ...
Records covering multiple periods have been identified for import.
1 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 1; Last timestamp: 2022-09-28 00:04:52 CEST (1664316292)
Calculating missing derived observations ...
Processing record: 1; Last record: 2022-09-29 00:00:00 CEST (1664402400)
Recalculating daily summaries...

Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
1 records were processed and 1 unique records imported in 2.56 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.

In the syslog, it says that importing failed

Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Raw import data read successfully for period 1.
Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Mapping raw import data for period 1 ...
Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Mapped 1 records.
Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Raw import data mapped successfully for period 1.
Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Saving mapped data to archive for period 1 ...
Sep 28 13:50:02 weewx wee_import[3536] ERROR weewx.manager: Unable to add record 2022-09-28 00:04:52 CEST (1664316292) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Sep 28 13:50:02 weewx wee_import[3536] INFO weeimport.weeimport: Mapped data saved to archive successfully for period 1.

3. Not at all, i observed a more complex behavior.

i tried it on two days (today and yesterday) with the exact same result:

From the actual day, only one record (0:04:52) is found, from one day ago, there are all records, but only before 21:44:52, from two days ago all records are found.

This behavior was identical on both days, today and yesterday with the same results. (today 1 record, yesterday all until 21:44, two days ago all records)

1. i run
wee_import --import-config=/etc/weewx/import/wu-example.conf --date=2022-09-28 [present day]
[...]
Observations timestamped after 2022-09-28 00:00:00 CEST (1664316000) and up to and
including 2022-09-29 00:00:00 CEST (1664402400) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 1; Last timestamp: 2022-09-28 00:04:52 CEST (1664316292)
Period 2 ...
**** Unable to load source data for period 2.
**** Possibly a bad station ID, an invalid date or data does not exist for this period.
**** Period 2 will be skipped. Proceeding to next period.
Calculating missing derived observations ...
Processing record: 1; Last record: 2022-09-29 00:00:00 CEST (1664402400)
Recalculating daily summaries...

2. i run
wee_import --import-config=/etc/weewx/import/wu-example.conf --date=2022-09-27 [yesterday]
[...]
Observations timestamped after 2022-09-27 00:00:00 CEST (1664229600) and up to and
including 2022-09-28 00:00:00 CEST (1664316000) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 261; Last timestamp: 2022-09-27 21:44:52 CEST (1664307892)
Period 2 ...
Period 2 - no records identified for import.
Calculating missing derived observations ...
Processing record: 261; Last record: 2022-09-28 00:00:00 CEST (1664316000)
Recalculating daily summaries...
[...]

3. i run 
wee_import --import-config=/etc/weewx/import/wu-example.conf --date=2022-09-26 [two days before]

[..]
Observations timestamped after 2022-09-26 00:00:00 CEST (1664143200) and up to and
including 2022-09-27 00:00:00 CEST (1664229600) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 288; Last timestamp: 2022-09-26 23:59:52 CEST (1664229592)
Period 2 ...
Period 2 - no records identified for import.
Calculating missing derived observations ...
Processing record: 288; Last record: 2022-09-27 00:00:00 CEST (1664229600)
Recalculating daily summaries...

Over all, if i query over the last three days, i get the records from yesterday until 22:24:52 an the other days as before

Observations timestamped after 2022-09-25 18:00:00 CEST (1664121600) and up to and
including 2022-09-28 06:00:00 CEST (1664337600) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 72; Last timestamp: 2022-09-25 23:59:52 CEST (1664143192)
Period 2 ...
Unique records processed: 288; Last timestamp: 2022-09-26 23:59:52 CEST (1664229592)
Period 3 ...
Unique records processed: 269; Last timestamp: 2022-09-27 22:24:52 CEST (1664310292)
Period 4 ...
Unique records processed: 1; Last timestamp: 2022-09-28 00:04:52 CEST (1664316292)
Calculating missing derived observations ...
Processing record: 630; Last record: 2022-09-29 00:00:00 CEST (1664402400)
Recalculating daily summaries...

so, i'm confused, where to locate this behavior.

best regards

sebastian