Inserting missing data using wee_import

[Mike Revitt]

I am having some interesting challenges around updating my database with some missing data.

Now that I have integrated my Caravan temperatures into WeeWX I am have extracted the data from the period that I was recording it but not updating WeeWX.

I have created a css file with the data and wee_import runs perfectly with no errors, but no new data appears in the database.

I suspect that this is because I am trying to do an update rather than an insert. So I have created an update script for. the archive table and this runs perfectly, I used python to generate the epoc dates, and I ran this on a copy of the database so I did not screw anything up.

The challenge is updating the archive tables archive_day_extraHumid1 etc. I can relatively easily construct the min and max values for a time period but constructing the min time is a little more challenging. So before I do that is there some tool I can invoke that will populate these archive tables from the new data in the main archive table

[Mike Revitt]

I have also discovered some columns  sum REAL, count INTEGER, wsum REAL, sumtime INTEGER

and have no idea what to put in them

[gjr80]

That’s one of the reasons we discourage writing to the database directly, far better to let the WeeWX machinery take care of everything. In answer to your question, if you have inserted the missing data in the archive then just use the wee_database utility to rebuild the daily summaries for the period concerned, that will take care of everything in the daily summary tables.

Gary

[Mike Revitt]

Thanks Gary, will give that a go.

And am I correct in my suspicion that wee_import didn't work because it only does inserts not updates

[gjr80]

Yes, wee_import inserts and does not update. If you have a look back at the WeeWX log from when you did the import you will probably find a lot of error messages about ‘unique constraint failures’ on dateTime, this is because wee_import tried to add a record with a timestamp that already existed in the database. wee_import uses code used by StdArchive for adding records to the database and unfortunately wee_import is unable to tell if the insert was successful or not, that is why you if you look closely there is some oddly worded summary text after a wee_import run. It advises you of how many records were processed and not how many were saved, and you are directed to the log to check success or otherwise.

Gary 

[Mike Revitt]

All appear to have worked perfectly thanks Gary,

I can see all the history now and just need to wait for the week, month and year images to refresh

[Mike Revitt]

For those that are interested in this, I have now completed the update of the missing data and documented how I did that here, turned out to be a lot simpler than I thought it would be

https://www.cougar.eu.com/useful-guides/weewx-guides/add-missing-data.html