Missing archive data / wee_device --dump

[Richard G]

I have had a Davis Vantage Pro2 and weewx running on a Pi for several years now. I had some issues last year and have a few weeks of data which I can browse on the Davis console but are not in the weewx database. I had assumed that running

wee_device --dump

Would go through all the archive records stored in the Davis console and then dump them into weewx but that doesn't appear to work. I was expecting loads of duplicate key entries but I get none and it only seems to go back 1 month and says added 2560 records. If I run the command again then it does exactly the same thing and says added 2560 records.

Firstly I wasn't expecting any recent records to be added as they are unto date.

Secondly I would have though the first run would have imported the data and the 2nd run would have resulted in duplicates but it doesn't.

Is there any other way of importing the missing data?

Any help appreciated.

[gjr80]

What does the log say? It will tell you what was/was not written to database during the wee_device —dump runs. I can’t recall the exact console output from wee_device —dump, but wee_device just causes the archive records to be downloaded from the logger, it doesn’t actually know whether they are saved to database or not. I would take the reported figure from wee_device as the number of records downloaded. The log will tell you exactly how many were actually saved to database.

The only ways I am aware of to obtain records from the logger are:

1.  normal WeeWX operation (using hardware record generation)

2. automatic download of new records during WeeWX startup (only records newer than the most recent database record are downloaded)

3. wee_device —dump

Gary

[Richard G]

Thanks for the prompt response. Nothing was in the log which confused me but then I realised my syslog filter didn't redirect the output from wee_device....

Yes the log file says no data added due to unique constraint failed as expected. However, it only goes back until the 2021-01-28 whereas the console has 10 years of data stored. I'm wondering whether the older archive data stored in the console is only summary or aggregated data so can't be used by weewx?

Richard

[gjr80]

If you have a genuine Davis logger it can only store 2560 records, how long that goes back in time depends on the archive interval used by the console. If using a 2 hour interval (the maximum) you get 213 days in the logger, if using the common five minute archive interval you get about 8 days which sort of ties in with the 28 January date you saw.

Gary

[Richard G]

Gary,

Thank you that explains it. 

I was hoping to be able to get the Davis console monthly summaries out so at least these were correct on weewx and therefore the yearly summaries as well. I’m guessing the only way I could do that is manually work out the difference between the weewx monthly summary and the console summary and then insert a few dummy records into the database to make up the missing data?

Richard

[gjr80]

Richard,

Not saying that can’t be done but the WeeWX database structure may make that challenging.

In broad terms, WeeWX has the archive table which stores archive records and then there are the daily summary tables, one daily summary table per observation stored in the archive table, that stores summary/limited aggregate data for each day for that observation. Aggregates based on multiples of one day (week, month, year etc) are obtained from the daily summary tables (aggregates that are not based on multiples of a whole day and plots are derived from the archive table).

Gary