Problem in forecast extension using darsky as source
Anders Fosgerau
For the record: In using the modified forecast extension posted
earlier on the list as I'm running under python3, mysql and a metric
database. But I guess this is a more generic problem than just me
deviating from standard configuration.
The problem appears "periodically" when using the forecast extension with Darsky as source for the forecast module. The short version is that a metric/US conversion without float rounding seems to cause a database schema "violation".
This is the log:
Jan 16 18:10:17 hjernen weewx[8513] INFO user.forecast: DSThread: DS:
downloading forecast from
'https://api.darksky.net/forecast/XXXXXXXXXXXXXXXXXXXXXXXXXXXX6bb8/55.71275,12.395433?exclude=currently,minutely,hourly,alerts,flags&lang=en&units=us'
Jan 16 18:10:18 hjernen weewx[8513] INFO user.forecast: DSThread: DS: got 8 forecast records
Jan 16 18:10:18 hjernen weewx[8513] INFO weewx.manager: Added record
2021-01-16 18:10:18 CET (1610817018) to database 'weewx_forecast'
Jan 16 18:10:18 hjernen weewx[8513] ERROR user.forecast: DSThread: DS:
save failed (attempt 1 of 3): (1406, "Data too long for column 'qsf' at
row 1")
Jan 16 18:10:28 hjernen weewx[8513] INFO weewx.manager: Added record
2021-01-16 18:10:18 CET (1610817018) to database 'weewx_forecast'
Jan 16 18:10:28 hjernen weewx[8513] ERROR user.forecast: DSThread: DS:
save failed (attempt 2 of 3): (1406, "Data too long for column 'qsf' at
row 1")
Jan 16 18:10:38 hjernen weewx[8513] INFO weewx.manager: Added record
2021-01-16 18:10:18 CET (1610817018) to database 'weewx_forecast'
Jan 16 18:10:38 hjernen weewx[8513] ERROR user.forecast: DSThread: DS:
save failed (attempt 3 of 3): (1406, "Data too long for column 'qsf' at
row 1")
Jan 16 18:10:48 hjernen weewx[8513] ERROR user.forecast: DSThread: DS:
forecast failure: save failed after 3 attempts, dbm_dict: {'manager':
'weewx.manager.Manager', 'schema': [('method', 'VARCHAR(10) NOT NULL'),
('usUnits', 'INTEGER NOT NULL'), ('dateTime', 'INTEGER NOT NULL'),
('issued_ts', 'INTEGER NOT NULL'), ('event_ts', 'INTEGER NOT NULL'),
('duration', 'INTEGER'), ('location', 'VARCHAR(64)'), ('desc',
'VARCHAR(256)'), ('zcode', 'CHAR(1)'), ('hour', 'INTEGER'), ('tempMin',
'REAL'), ('tempMax', 'REAL'), ('temp', 'REAL'), ('dewpoint', 'REAL'),
('humidity', 'REAL'), ('windDir', 'VARCHAR(3)'), ('windSpeed', 'REAL'),
('windGust', 'REAL'), ('windChar', 'VARCHAR(2)'), ('clouds',
'VARCHAR(2)'), ('pop', 'REAL'), ('qpf', 'VARCHAR(8)'), ('qsf',
'VARCHAR(5)'), ('rain', 'VARCHAR(2)'), ('rainshwrs', 'VARCHAR(2)'),
('tstms', 'VARCHAR(2)'), ('drizzle', 'VARCHAR(2)'), ('snow',
'VARCHAR(2)'), ('snowshwrs', 'VARCHAR(2)'), ('flurries', 'VARCHAR(2)'),
('sleet', 'VARCHAR(2)'), ('frzngrain', 'VARCHAR(2)'), ('frzngdrzl',
'VARCHAR(2)'), ('hail', 'VARCHAR(2)'), ('obvis', 'VARCHAR(3)'),
('windChill', 'REAL'), ('heatIndex', 'REAL'), ('uvIndex', 'INTEGER'),
('airQuality', 'INTEGER'), ('hilo', 'CHAR(1)'), ('offset', 'REAL'),
('waveheight', 'REAL'), ('waveperiod', 'REAL')], 'table_name':
'archive', 'database_dict': {'database_name': 'weewx_forecast',
'driver': 'weedb.mysql', 'host': 'localhost', 'user': 'weewx',
'password': 'weewx'}}
The DS forecast retrieved was:
{"latitude":55.71275,"longitude":12.395433,"timezone":"Europe/Copenhagen","daily":{"summary":"Light rain on Monday through Friday.","icon":"rain","data":[{"time":1610751600,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1610782200,"sunsetTime":1610809980,"moonPhase":0.12,"precipIntensity":0,"precipIntensityMax":0.0002,"precipIntensityMaxTime":1610820540,"precipProbability":0.02,"temperatureHigh":29.65,"temperatureHighTime":1610795280,"temperatureLow":24.04,"temperatureLowTime":1610858700,"apparentTemperatureHigh":23.98,"apparentTemperatureHighTime":1610795220,"apparentTemperatureLow":19.81,"apparentTemperatureLowTime":1610863200,"dewPoint":19.52,"humidity":0.79,"pressure":1023.3,"windSpeed":4.61,"windGust":8.64,"windGustTime":1610820900,"windBearing":337,"cloudCover":0.22,"uvIndex":0,"uvIndexTime":1610796000,"visibility":10,"ozone":334.4,"temperatureMin":20.57,"temperatureMinTime":1610759820,"temperatureMax":29.65,"temperatureMaxTime":1610795280,"apparentTemperatureMin":14.84,"apparentTemperatureMinTime":1610759640,"apparentTemperatureMax":23.98,"apparentTemperatureMaxTime":1610795220},{"time":1610838000,"summary":"Mostly cloudy throughout the day.","icon":"partly-cloudy-day","sunriseTime":1610868480,"sunsetTime":1610896500,"moonPhase":0.16,"precipIntensity":0.0004,"precipIntensityMax":0.0033,"precipIntensityMaxTime":1610907180,"precipProbability":0.05,"precipType":"snow","precipAccumulation":0.11,"temperatureHigh":31.36,"temperatureHighTime":1610883000,"temperatureLow":26.98,"temperatureLowTime":1610906400,"apparentTemperatureHigh":26.04,"apparentTemperatureHighTime":1610884860,"apparentTemperatureLow":20.44,"apparentTemperatureLowTime":1610906400,"dewPoint":23.04,"humidity":0.83,"pressure":1019.2,"windSpeed":4.98,"windGust":19.66,"windGustTime":1610924400,"windBearing":187,"cloudCover":0.7,"uvIndex":0,"uvIndexTime":1610882700,"visibility":9.994,"ozone":339.1,"temperatureMin":24.04,"temperatureMinTime":1610858700,"temperatureMax":31.36,"temperatureMaxTime":1610883000,"apparentTemperatureMin":19.52,"apparentTemperatureMinTime":1610864220,"apparentTemperatureMax":26.04,"apparentTemperatureMaxTime":1610884860},{"time":1610924400,"summary":"Possible light rain in the evening.","icon":"rain","sunriseTime":1610954820,"sunsetTime":1610983020,"moonPhase":0.19,"precipIntensity":0.0041,"precipIntensityMax":0.0163,"precipIntensityMaxTime":1610982120,"precipProbability":0.47,"precipType":"rain","temperatureHigh":35.94,"temperatureHighTime":1610967360,"temperatureLow":33.61,"temperatureLowTime":1610992800,"apparentTemperatureHigh":27.53,"apparentTemperatureHighTime":1610967000,"apparentTemperatureLow":26.04,"apparentTemperatureLowTime":1610992800,"dewPoint":29.55,"humidity":0.87,"pressure":1014.3,"windSpeed":9.65,"windGust":24.29,"windGustTime":1610967360,"windBearing":210,"cloudCover":0.88,"uvIndex":0,"uvIndexTime":1610968980,"visibility":9.379,"ozone":322.1,"temperatureMin":28.65,"temperatureMinTime":1610927580,"temperatureMax":35.94,"temperatureMaxTime":1610967360,"apparentTemperatureMin":20.98,"apparentTemperatureMinTime":1610927400,"apparentTemperatureMax":28.17,"apparentTemperatureMaxTime":1611003300},{"time":1611010800,"summary":"Light rain throughout the day.","icon":"rain","sunriseTime":1611041160,"sunsetTime":1611069540,"moonPhase":0.22,"precipIntensity":0.0076,"precipIntensityMax":0.0167,"precipIntensityMaxTime":1611050280,"precipProbability":0.91,"precipType":"rain","temperatureHigh":37.04,"temperatureHighTime":1611057240,"temperatureLow":35.07,"temperatureLowTime":1611079200,"apparentTemperatureHigh":28.03,"apparentTemperatureHighTime":1611051300,"apparentTemperatureLow":26.52,"apparentTemperatureLowTime":1611079200,"dewPoint":33.53,"humidity":0.9,"pressure":1004,"windSpeed":12.69,"windGust":27.58,"windGustTime":1611067980,"windBearing":193,"cloudCover":0.97,"uvIndex":0,"uvIndexTime":1611055380,"visibility":7.712,"ozone":303.6,"temperatureMin":33.91,"temperatureMinTime":1611010800,"temperatureMax":39.7,"temperatureMaxTime":1611097200,"apparentTemperatureMin":26.23,"apparentTemperatureMinTime":1611074760,"apparentTemperatureMax":31.15,"apparentTemperatureMaxTime":1611097200},{"time":1611097200,"summary":"Light rain throughout the day.","icon":"rain","sunriseTime":1611127440,"sunsetTime":1611156060,"moonPhase":0.25,"precipIntensity":0.0103,"precipIntensityMax":0.0315,"precipIntensityMaxTime":1611119160,"precipProbability":0.97,"precipType":"rain","temperatureHigh":44.19,"temperatureHighTime":1611153180,"temperatureLow":41.44,"temperatureLowTime":1611208800,"apparentTemperatureHigh":36.17,"apparentTemperatureHighTime":1611162240,"apparentTemperatureLow":33.87,"apparentTemperatureLowTime":1611208800,"dewPoint":39.37,"humidity":0.89,"pressure":994.3,"windSpeed":17.13,"windGust":37.51,"windGustTime":1611117780,"windBearing":212,"cloudCover":1,"uvIndex":0,"uvIndexTime":1611141660,"visibility":9.657,"ozone":297.2,"temperatureMin":38.71,"temperatureMinTime":1611097200,"temperatureMax":44.34,"temperatureMaxTime":1611170820,"apparentTemperatureMin":31.15,"apparentTemperatureMinTime":1611097200,"apparentTemperatureMax":36.52,"apparentTemperatureMaxTime":1611170100},{"time":1611183600,"summary":"Light rain until afternoon, starting again overnight.","icon":"rain","sunriseTime":1611213780,"sunsetTime":1611242580,"moonPhase":0.28,"precipIntensity":0.0053,"precipIntensityMax":0.0228,"precipIntensityMaxTime":1611230220,"precipProbability":0.85,"precipType":"rain","temperatureHigh":43.81,"temperatureHighTime":1611223020,"temperatureLow":39.86,"temperatureLowTime":1611273420,"apparentTemperatureHigh":34.78,"apparentTemperatureHighTime":1611221820,"apparentTemperatureLow":30.23,"apparentTemperatureLowTime":1611252240,"dewPoint":37.87,"humidity":0.85,"pressure":987.9,"windSpeed":20.07,"windGust":40.15,"windGustTime":1611253800,"windBearing":199,"cloudCover":0.87,"uvIndex":0,"uvIndexTime":1611227760,"visibility":10,"ozone":323.2,"temperatureMin":39.99,"temperatureMinTime":1611270000,"temperatureMax":43.81,"temperatureMaxTime":1611223020,"apparentTemperatureMin":30.23,"apparentTemperatureMinTime":1611252240,"apparentTemperatureMax":35.25,"apparentTemperatureMaxTime":1611200640},{"time":1611270000,"summary":"Possible drizzle in the morning.","icon":"rain","sunriseTime":1611300120,"sunsetTime":1611329100,"moonPhase":0.31,"precipIntensity":0.0025,"precipIntensityMax":0.0061,"precipIntensityMaxTime":1611275880,"precipProbability":0.65,"precipType":"rain","temperatureHigh":42.68,"temperatureHighTime":1611316620,"temperatureLow":35.21,"temperatureLowTime":1611381600,"apparentTemperatureHigh":34.07,"apparentTemperatureHighTime":1611317220,"apparentTemperatureLow":28.18,"apparentTemperatureLowTime":1611381600,"dewPoint":36.44,"humidity":0.85,"pressure":986.7,"windSpeed":17.26,"windGust":37.77,"windGustTime":1611270000,"windBearing":209,"cloudCover":0.94,"uvIndex":0,"uvIndexTime":1611314400,"visibility":10,"ozone":334.5,"temperatureMin":38.1,"temperatureMinTime":1611356400,"temperatureMax":42.68,"temperatureMaxTime":1611316620,"apparentTemperatureMin":30.26,"apparentTemperatureMinTime":1611272820,"apparentTemperatureMax":34.07,"apparentTemperatureMaxTime":1611317220},{"time":1611356400,"summary":"Overcast throughout the day.","icon":"cloudy","sunriseTime":1611386400,"sunsetTime":1611415620,"moonPhase":0.34,"precipIntensity":0.0001,"precipIntensityMax":0.0006,"precipIntensityMaxTime":1611414000,"precipProbability":0.16,"precipType":"rain","temperatureHigh":41.47,"temperatureHighTime":1611403980,"temperatureLow":31.98,"temperatureLowTime":1611453480,"apparentTemperatureHigh":34.65,"apparentTemperatureHighTime":1611404340,"apparentTemperatureLow":26.26,"apparentTemperatureLowTime":1611457080,"dewPoint":32.16,"humidity":0.82,"pressure":995.9,"windSpeed":9.36,"windGust":19.71,"windGustTime":1611432240,"windBearing":250,"cloudCover":0.97,"uvIndex":0,"uvIndexTime":1611400740,"visibility":10,"ozone":348.3,"temperatureMin":33.02,"temperatureMinTime":1611442800,"temperatureMax":41.47,"temperatureMaxTime":1611403980,"apparentTemperatureMin":28.04,"apparentTemperatureMinTime":1611384180,"apparentTemperatureMax":34.65,"apparentTemperatureMaxTime":1611404340}]},"offset":1}The last line in the code below taken from forecast.py is probably the cause:
# Dark Sky provides snowfall in cm in optional precipAccumulation field
if 'precipAccumulation' in period:
_qsf = Forecast.str2float('precipAccumulation',
period['precipAccumulation'],
DS_KEY)
r['qsf'] = _qsf / 2.54 if _qsf is not None else None
precipAccumulation was 0.11 in the retrieved forecast. 0.11 divided by 2.54 gives result 0,0433070866141732 which doesn't fit in a VARCHAR(5).
Implemented a poor man's solution by extending qsf to a varchar(20)
in the database (mysql: alter table archive modify qsf varchar(20);) and
problem seems to be gone.