Enabling historical data collection on new install

[icoj...@gmail.com]

Hi,

When I first installed weewx a year ago I noticed that it went through a process of adding all the historical data over the course of the first few days.  This then gave me a good historical picture for trend analysis.

I've done a fresh install of 4.1.1 on a separate RPi this week and after 2 days I cannot see any historical data.  Is there something I need to do?

Many thanks

[Graham Eddy]

if you were using vantage pro, it would have (pretty promptly) backfilled whatever it had in datalogger memory.

i think anything else would require manual intervention, and, of that, the only thing that comes to mind is backfilling from wunderground (using wunderfixer tool)

you’ll need to give some pertinent details like which weather station and which driver to get any answer...

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/dbc86e88-e951-482e-92fc-bf086a21c783n%40googlegroups.com.

[steeple ian]

Matt,

From previous experience using an HP1000 it takes a long time to download the data stored on your device. How many years worth?

-im

[Matt C]

Hi,

 

I think I’ve got about 4-5 years worth but the issue is I’m not seeing any of that yet, I don’t see any data prior to yesterday, so I don’t think the process has even started. 

 

Regards

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/RovNaHpvlJs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CADASSaRQ_Y4f1NQJEkDk5zg9PFcYm%3DWmJJwW2jxg%2BksxzjrZAA%40mail.gmail.com.

 

[steeple ian]

I assume yours is a Maplin badged device? From what I remember when I had the same model it took nearly a day to download a couple of years’ worth of data. Have you looked into the syslog to see what is happening in real time? From the command line run to watch the progress: -

sudo tail -f /var/log/syslog 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2310EA5B-2C2F-4904-A8CE-AAEF21E23E70%40hxcore.ol.

[icoj...@gmail.com]

Hi,

It's a aercus badged device.  I have looked at the syslog periodically since it's been up and running and I'm only seeing writes to the database on each current loop, shown below, there are no log records of any other database activity.

Sep  4 12:20:30 rasptest weewx[17241] INFO weewx.manager: Added record 2020-09-04 12:20:00 BST (1599218400) to database 'weewx.sdb'

Sep  4 12:20:30 rasptest weewx[17241] INFO weewx.manager: Added record 2020-09-04 12:20:00 BST (1599218400) to daily summary in 'weewx.sdb'

Thanks for your help

[steeple ian]

Patience is a virtue:-)

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5b96f3f1-09a9-4e7d-8e02-5f81fa6d1e13n%40googlegroups.com.

[icoj...@gmail.com]

Still nothing...

[steeple ian]

what is your syslog showing?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e80cba1f-3faa-4573-9cb5-43a751d8fdcan%40googlegroups.com.

[Matt C]

Exactly the same as my previous reply.  It appears to be adding a record to the DB on each loop but there’s no log entry for historical data.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CADASSaQ7Kup6Ki9R4Kh%3DYaMABGxzpN634w3962a-KuwPj8fApw%40mail.gmail.com.

 

[steeple ian]

Change you debug level to debug = 2 and see if that reveals any more information.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9423CBF9-AA82-426D-8A5D-11FEC3342E78%40hxcore.ol.

[icoj...@gmail.com]

Thank you.  I've done that and pasted below is what I am seeing...

Sep 7 12:55:35 rasptest weewx[340] INFO weewx.cheetahgenerator: Generated 7 files for report w34Highcharts in 1.30 seconds
Sep 7 12:55:35 rasptest weewx[340] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/weather34/w34highcharts
Sep 7 12:56:39 rasptest weewx[340] DEBUG user.weather34: CloudCover url1 exit code 0
Sep 7 12:56:39 rasptest weewx[340] DEBUG user.weather34: CloudCover url2 exit code 0
Sep 7 12:56:39 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'IHDR' 16 13
Sep 7 12:56:39 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'gAMA' 41 4
Sep 7 12:56:39 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'pHYs' 57 9
Sep 7 12:56:39 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'IDAT' 78 65458
Sep 7 12:56:39 rasptest weewx[340] DEBUG PIL.Image: Error closing: 'Image' object has no attribute 'fp'
Sep 7 12:57:37 rasptest weewx[340] ERROR user.weather34: Failed getting web service data. URL: http://000.000.000.000/weewx/weather34/eualert.php?country=UK&region=013 Header: ["'User-Agent'", "'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3'"], Error: HTTP Error 400: Bad Request
Sep 7 13:00:29 rasptest weewx[340] DEBUG user.weather34: CloudCover: Percent value is 55.0
Sep 7 13:00:29 rasptest weewx[340] INFO weewx.manager: Added record 2020-09-07 13:00:00 BST (1599480000) to database 'weewx.sdb'
Sep 7 13:00:29 rasptest weewx[340] INFO weewx.manager: Added record 2020-09-07 13:00:00 BST (1599480000) to daily summary in 'weewx.sdb'
Sep 7 13:00:30 rasptest weewx[340] DEBUG weewx.reportengine: Running reports for latest time in the database.
Sep 7 13:00:30 rasptest weewx[340] DEBUG weewx.reportengine: Running report 'SeasonsReport'
Sep 7 13:00:30 rasptest weewx[340] DEBUG weewx.reportengine: Found configuration file /home/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Sep 7 13:00:31 rasptest weewx[340] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Sep 7 13:00:31 rasptest weewx[340] DEBUG weewx.manager: Daily summary version is 2.0
Sep 7 13:00:32 rasptest weewx[340] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79 seconds
Sep 7 13:00:32 rasptest weewx[340] DEBUG weewx.manager: Daily summary version is 2.0
Sep 7 13:00:38 rasptest weewx[340] INFO weewx.imagegenerator: Generated 30 images for report SeasonsReport in 5.35 seconds
Sep 7 13:00:38 rasptest weewx[340] INFO weewx.reportengine: Copied 0 files to /var/www/html/
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Report 'MobileReport' not enabled. Skipping.
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Report 'StandardReport' not enabled. Skipping.
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Report 'FTP' not enabled. Skipping.
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Report 'RSYNC' not enabled. Skipping.
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Running report 'Weather34Report'
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Found configuration file /home/weewx/skins/Weather34/skin.conf for report 'Weather34Report'
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', u'user.stats.MyStats', u'user.lastrain.lastRainTags']
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.manager: Daily summary version is 2.0
Sep 7 13:00:38 rasptest weewx[340] INFO weewx.cheetahgenerator: Generated 2 files for report Weather34Report in 0.34 seconds
Sep 7 13:00:38 rasptest weewx[340] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/weather34/
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Running report 'w34Highcharts'
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.reportengine: Found configuration file /home/weewx/skins/w34Highcharts/skin.conf for report 'w34Highcharts'
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', u'user.w34highchartsSearchX.w34highcharts_temp_week', u'user.w34highchartsSearchX.w34highcharts_bar_rain_week', u'user.w34highchartsSearchX.w34highcharts_wind_week', u'user.w34highchartsSearchX.w34highcharts_solar_week', u'user.w34highchartsSearchX.w34highcharts_indoor_derived_week', u'user.w34highchartsSearchX.w34highcharts_wind_rose_week', u'user.w34highchartsSearchX.w34highchartsYear']
Sep 7 13:00:38 rasptest weewx[340] DEBUG weewx.manager: Daily summary version is 2.0
Sep 7 13:00:39 rasptest weewx[340] DEBUG weewx.cheetahgenerator: Skip 'year.json': last_mod=1599479735.01 age=304.582433701 stale=600
Sep 7 13:00:39 rasptest weewx[340] INFO weewx.cheetahgenerator: Generated 6 files for report w34Highcharts in 0.79 seconds
Sep 7 13:00:39 rasptest weewx[340] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/weather34/w34highcharts
Sep 7 13:01:40 rasptest weewx[340] DEBUG user.weather34: CloudCover url1 exit code 0
Sep 7 13:01:40 rasptest weewx[340] DEBUG user.weather34: CloudCover url2 exit code 0
Sep 7 13:01:40 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'IHDR' 16 13
Sep 7 13:01:40 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'gAMA' 41 4
Sep 7 13:01:40 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'pHYs' 57 9
Sep 7 13:01:40 rasptest weewx[340] DEBUG PIL.PngImagePlugin: STREAM 'IDAT' 78 65458
Sep 7 13:01:40 rasptest weewx[340] DEBUG PIL.Image: Error closing: 'Image' object has no attribute 'fp'
Sep 7 13:02:37 rasptest weewx[340] ERROR user.weather34: Failed getting web service data. URL: http://000.000.000.000/weewx/weather34/eualert.php?country=UK&region=013 Header: ["'User-Agent'", "'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3'"], Error: HTTP Error 400: Bad Request

[steeple ian]

That all seems to be as I would expect except you have not setup the Meteoalarm web service hence that error. It certainly looks like it has finished downloading the historic data.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/cb3b5d66-c153-4458-aa28-88597da58002n%40googlegroups.com.

[icoj...@gmail.com]

Thank you.  I've set up meteoalarm, I think, and have included a screenshot to show what I'm seeing.  There is no historical data and at no point during my random checks of the log over the almost 5 days it's been running show any records to suggest it's recorded any.  I'm stumped, the config is the same as the previous installation which updated historical data within the first 24 hours.

[steeple ian]

Very odd. Is there any chance you may have accidentally flushed your weather station memory.

Did you save a copy of your old database? If you did you can update that to the new schema as per the WeeWX documentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8d8f9083-8720-4892-be77-c67f051dddb4n%40googlegroups.com.

[icoj...@gmail.com]

I've checked the weather station and the historic data is present.  I've still got the old installation, it's still running at least the back end is.  I'll see what's involved in migrating that to the new installation.

Thanks for your help.

[icoj...@gmail.com]

I'm looking through the various weewx utilities and using wee_debug I've come across this...

Archive info
Database name: weewx.sdb
Table name: archive
Version 2.0
Unit system: 1 (US)
First good timestamp: 2020-09-03 12:10:00 BST (1599131400)
Last good timestamp: 2020-09-07 21:15:00 BST (1599509700)
Number of records: 1284
weewx (weewx.conf) is set to use an archive interval of 300 seconds.
The station hardware was not interrogated in determining archive interval.

Databases configured in weewx.conf
Database name: weewx.sdb
Database driver: weedb.sqlite

Database name: weewx
Database driver: weedb.mysql
Database host: localhost

Comparing this to the old installation below...

Archive info
Database name: weewx.sdb
Table name: archive
Version 2.0
Unit system: 1 (US)
First good timestamp: 2018-04-14 17:06:58 BST (1523722018)
Last good timestamp: 2020-09-06 17:16:01 BST (1599408961)
Number of records: 319647
weewx (weewx.conf) is set to use an archive interval of 300 seconds.
The station hardware was not interrogated in determining archive interval.


Databases configured in weewx.conf
Database name: weewx.sdb
Database driver: weedb.sqlite


Database name: weewx
Database driver: weedb.mysql
Database host: localhost


Database name: forecast.sdb
Database driver: weedb.sqlite

This clearly shows that there are no records prior to the time the installation completed.  I'm also wondering why the database host entries are missing in the new installation.  Is there anyone who understands what this may be telling me please?!

[icoj...@gmail.com]

I've obtained the historical data directly from the weather station and am trying to import using wee_import.  I have configured the required info but am facing the following problem and don't know what it means and how to fix it.  Can anyone help please?

pi@rasptest:/home/weewx/bin $ sudo ./wee_import --import-config=/home/weewx/import200908.conf --dry-run
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/2018comma.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
**** 'ascii' codec can't encode character u'\xb0' in position 29: ordinal not in range(128)

[icoj...@gmail.com]

wow, this is an exercise in troubleshooting!  It seems that copy and paste is not friendly when doing this sort of thing.  Typing out all the values, units, etc appears to have worked.  The other issue I encountered is that it doesn't handle the change from BST to GMT as there are duplicate records and it simply throws an error.  To get around this I've removed the dupes from the source file.