Recurrence of unique constraint issues - Weewx 4.3.0, HP1000

[icoj...@gmail.com]

Hi All,

I'm seeing a recurrence of catastrophic failures with my setup that I last saw around this time last year.

There are numerous unique constraint errors that appear to be due to the HP1000 trying to insert historical data when there is already data there.  This results in reports and data never being updated and showing as offline.

Jan 18 08:40:15 weatherpi weewx[1780] ERROR weewx.manager: Unable to add record 2021-02-09 10:15:36 GMT (1612865736) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

The solution I used last year was to do a full clean install.  I'd rather not go down that path this year as I've reconfigured weewx to the simulator and all is well.  So that suggests the issue is with the HP1000 driver.  I reinstalled the driver but still see the same issue.

My next guess would be to uninstall the driver but I don't know how to do that.

My original post is here

Any guidance would be very much appreciated.

Thank you,

Matt

[Susan Mackay]

The folks that know the WeeWx code better than I may have other ideas but what I do is to use your favourite SQLLite database tool to delete from where the duplications start (1612865736 in your case) to the end and then delete and rebuild the daily summary records.

What the HP1000 driver does is to ask WeeWx for the date of the last record and then checks with the console for any records that are *after* that date and treats those as 'historic' records.

Can you look at the database and see if there are records (possibly with dateTime values that don't seem to follow the general trend - for example nearly all of mine end in '0') that might be causing the issue.

Also can you please provide the traceback for the error, not just the error itself. I'd like to see where the error is originating in the driver source file.

Susan

[icoj...@gmail.com]

Thanks Susan, sent a PM

[Susan Mackay]

I have received the PM but I think the discussion is better served by getting everyone's understanding of the situation.

The PM contains a rather long log file and the important excerpt is:

Jan 19 09:28:13 weatherpi weewx[608] INFO root: HP100: Established contact at 19/01/22 09:28:13
Jan 19 09:28:13 weatherpi weewx[608] INFO root: HP100: Retrieving startup records

Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:39:15 GMT (1642523955) to database 'weewx.sdb'
Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:39:15 GMT (1642523955) to daily summary in 'weewx.sdb'
Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:44:15 GMT (1642524255) to database 'weewx.sdb'
Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:44:15 GMT (1642524255) to daily summary in 'weewx.sdb'
Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:49:15 GMT (1642524555) to database 'weewx.sdb'
Jan 19 09:28:30 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-18 16:49:15 GMT (1642524555) to daily summary in 'weewx.sdb'

.......

Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 07:54:15 GMT (1642578855)
Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-19 09:14:15 GMT (1642583655) to database 'weewx.sdb'
Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-19 09:14:15 GMT (1642583655) to daily summary in 'weewx.sdb'
Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 07:59:15 GMT (1642579155)
Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:04:15 GMT (1642579455)
Jan 19 09:29:32 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:09:15 GMT (1642579755)
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:14:15 GMT (1642580055)
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:19:15 GMT (1642580355)
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-19 09:19:15 GMT (1642583955) to database 'weewx.sdb'
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:24:15 GMT (1642580655)
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.manager: Added record 2022-01-19 09:19:15 GMT (1642583955) to daily summary in 'weewx.sdb'
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:29:15 GMT (1642580955)
Jan 19 09:29:33 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:34:15 GMT (1642581255)
Jan 19 09:29:34 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:39:15 GMT (1642581555)
Jan 19 09:29:34 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:44:15 GMT (1642581855)
Jan 19 09:29:34 weatherpi weewx[608] INFO weewx.restx: PWSWeather: Published record 2022-01-19 08:49:15 GMT (1642582155)
Jan 19 09:29:34 weatherpi weewx[608] ERROR weewx.manager: Unable to add record 2021-01-24 08:53:12 GMT (1611478392) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Jan 19 09:29:34 weatherpi weewx[608] ERROR weewx.manager: Unable to add record 2021-01-24 08:58:12 GMT (1611478692) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Jan 19 09:29:34 weatherpi weewx[608] ERROR weewx.manager: Unable to add record 2021-01-24 09:03:12 GMT (1611478992) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Jan 19 09:29:34 weatherpi weewx[608] ERROR weewx.manager: Unable to add record 2021-01-24 09:08:12 GMT (1611479292) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Jan 19 09:29:34 weatherpi weewx[608] ERROR weewx.manager: Unable to add record 2021-01-24 09:13:12 GMT (1611479592) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

The driver gets access to the console at local time 2022-01-19 09:28:13 (to use the date/time format of the other records)

What I see is that the driver is correctly retrieving historical records from the console up to and including the one with the timestamp "2022-01-19 09:19:15 GMT".  There is then a jump back in time to 2021-01-24 08:53:12 or nearly a year before! This is where the records become duplicated.

I must admit I can't understand what is happening other than the data being held in your console has become corrupted in some way. 

Normally the driver simply asks the console for the 'HISTORY_DATA' records starting from a specific date. The log shows that it is somehow jumping back a year and asking for those records, which are already in the database.

I know the area of the code where it retrieves these records and I'll see if there is a logic error but I'll be surprised if there is and it has not shown itself earlier.

HOWEVER, (this needs to be verified by others) it seems that it is logging the error and it may well be that you get a number of these error messages but my guess is that it will eventually stop and you will start to retrieve the current LOOP records.

Susan