Manually adding historical data to weewx database

[dhin...@djhindley.com]

Hi

I have just started using weewx on a Raspberry Pi with a Davis Vantage using Weatherlink IP. The weather station has been running for 2 years, but I only got weewx working from today. All is working fine, except that since I updated my archive record to 5mins immediately prior to the installation, weewx only shows the recent records. Does anyone know if there is a way I can manually add the historical data to the weewx database, so that the full history is shown on my weewx webpage? 

Thanks

[Andrew Milner]

where is the historical data kept??

[David Hindley]

The data has been downloaded to a text file from the WeatherLink software, stores in a txt file on my Windows PC. 

David. 

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/VCwuWaT2kwY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Milner]

if you can massage the data into a suitable format you can always use wee_import

http://weewx.com/docs/utilities.htm#wee_import_utility

[dhin...@djhindley.com]

Great.  Thanks.  I will try that.

[dhin...@djhindley.com]

The import facility looks like the right way to go - but unfortunately my source data has variable archive intervals. Is there any way of dealing with this in the weewx import utlility, or is it best to convert all the source data to have the same archive interval (painful!)?

Thanks.

[Andrew Milner]

As far as I know the latest version of weewx casn handle variable intervals- so just let wee_import derive the interval from the time stamps should work (I think!!)

someone more knowledgeable maywish to comment ....

[gjr80]

David,

Variable archive intervals have been supported since weeWX v3.0.0, though I believe there was a display/report calculation bug that was addressed in 3.7.0 (of course we are all running 3.8.0 so neither of these should be an issue :))

One thing to be mindful of when using wee_import to import data is that when you are deriving the archive interval from the source data (ie finding the difference between successive timestamps) missing records can skew this significantly. A single missing record in a data set with an interval of 5 minutes will likely have negligible effect but if you have several hours or days missing it may be a different matter. For this reason when importing from WeatherUnderground if your WeatherUndergroud data was posted every 10 minutes it is quite often handy to force wee_import to use a 10 minute archive interval as WeatherUnderground often does not return all available data, plus it is not unusual for users to have large gaps in their WeatherUnderground data.

If you have a changing archive interval in your data and are concerned about data gaps, it might be useful to break the import down into smaller chunks that use a single archive interval and import that data with a set rather derived interval.

Gary

PS. If there is significant reformatting required to import the ex-WeatherLink data I am happy to put together a WeatherLink module for wee_import, I will just need to know the data format. Only thing is it will take time for me to put the module together, though I guess your old data is not going anywhere :)

[dhin...@djhindley.com]

Gary

Many Thanks for your very helpful reply.  So which setting do I use for the archive interval in the config file? "derive" seems the most logical, but the instructions say that "derive" is suitable when the data was recorded at fixed intervals.

I do have some gaps in my data - up to two weeks in some cases.  I plan to try to fill as many of these gaps as possible using a combination of downloaded data from weathercloud (for my weather station) and also using the Met office's 5km grid data as a suitable proxy (for the grid containing my PWS).  I will attempt to process this all into a single archive database using R software and then use weewx import. If I need to take up your kind offer of a Weatherlink import module, I will let you know.

David.

[dhin...@djhindley.com]

I have successfully imported my 2018 data and the statistics all reconcile with my own analysis of the same data, so looks a good start.  However, the weewx charts don't show the historical data at all. How do I get them to do this, please?

[Andrew Milner]

a quick way is to delete all the .png files and let weewx regenerate them

[gjr80]

David,

Perhaps a bit late but the answer is it depends. 'derive' will attempt to derive the interval from the incoming data, it's not a very sophisticated algorithm, it is simply the difference in timestamps of successive records. So derive can suffer if you have big gaps. 'conf' will use the archive interval in your weewx.conf (or whatever weeWX config file wee_import is told to use). This will be fine if all of your source data say came from weeWX and uses the same interval as the archive interval specified in weewx.conf. The advantage here is that large data gaps don't have the disruptive impact they did with 'derive'. Of course the downside is that your source data and weewx.conf may have different intervals. 'x' (where x is the number of minutes) allows you to force all imported records to use x as the interval. Again this can be handy if your source data has a fixed interval but there are large data gaps. Also useful for importing tranches of source data that may have different intervals.

In summary - beware of 'derive', you really need a good solid (regular) data source for it to be effective.

You will find the interval config option described in the wee_import section of the Utilities Guide.

Gary

[David Hindley]

Many Thanks. I had an interval field in my history, so I used that, which seemed to work OK.  I have successfully loaded the 2018 history; now the rest. 

David. 

--