Computer offline for 3 days --Davis weather logger did not upload data stored during this time.

[lecoqacr...@gmail.com]

Have a raspberry pi 3b running weewx 4.1 (thank you for a great program and so much hard work and the online support), but am curious as to why the data which should be up to 8 days for the data logger was not uploaded automatically when restarting the raspberry pi. There is a gap of 3 days that I don't understand if the logger keeps 8 days in memory. Apparently there is a setting that needs to be changed but am unsure of this setting. 

thank you for your time.

[gjr80]

Hi,

Whilst it's true the vantage driver will backfill records generated during a WeeWX outage there are some conditions/limitations. Firstly, in weewx.conf under [StdArchive] option no_catchup must either not exist or must be set to False otherwise the hardware record catchup will not be attempted (the no_catchup option is not included in weewx.conf for a default install so likely it will not exist on your system). Secondly, and what probably prevents most catchups occurring, is that the catchup only downloads those records that are newer/younger/more recent than the timestamp of the last good archive record in your database. Whilst your missing data may well be in the data logger it could be the you have a record in your archive that is timestamped after the logger data of interest. In this case the easiest solution is to stop WeeWX and use the wee_device utility with --dump to dump all of the archive records stored in the logger to WeeWX. You can check the WeeWX log to see what records were downloaded during the dump. --dump will download all stored archive records but only those not already saved in the WeeWX archive will be saved, those already in the archive will generate a duplicate key error which can be ignored.

Just for completeness there is one other situation where backfill of logged data did not occur. WeeWX v3.9.0 introduced a change whereby backfill was only attempted on startup if hardware record generation was in use. This behaviour was changed as of v3.9.1 and backfill is now attempted irrespective of whether hardware or software archive record generation is used.

Gary

[Greg Sinclair]

So to do the backfill,

[StdArchive] 

  no_catchup = True

Thanks

Greg

[gjr80]

No. no_catchup was implemented to give users the ability to turn off the automatic catch-up/backfill that occurs on WeeWX startup. To disable automatic catch-up/backfill on WeeWX startup you need to set no_catchup = True; if no_catchup is omitted or no_catchup = False (the default) automatic catch-up/backfill on WeeWX startup will be enabled. A fresh WeeWX install will not include the no_catchup setting in weewx.conf but automatic catch-up/backfill on WeeWX startup will still occur due to the default being False.

I would expect not many folks would use no_catch to disable automatic catch-up/backfill on WeeWX startup.

Gary

[lecoqacr...@gmail.com]

Thank you for the replies. Yes, as stated the"no_catchup" setting does not exist in my /home/weewx/weewx.conf file. Used the setup.py method of installation using latest weewx as of 10/5/2020. Therefore based on the replies no_catchup is already set to false by default somewhere in the weewx program and therefore should attempt to catch up when offline for several days. That would only leave the second reason stated in the first reply--that there is a time stamp already newer in my database that has prevented the catchup from reaching further back than this this newer time stamp. How could that be possible since the machine was started after 3 days without an ability to receive data that would include the newer time stamped data?

I did stop weewx and invoke the wee_device --dump command but received the response of  bash : wee_device command not found. I then tried the following command that did work: /home/weewx/util/scripts/wee_device /home/weewx/weewx.conf --dump

I was then prompted for for a yes or no which proceeded with a yes it and it dumped data through 10/07/2020  which should include data through the offline time. This data is to be placed automatically into the database file according to the documentation when using the dump command. How long does it take to show on the plot graphs (/home/weewx/public_html/index.html) which would remove the blank data line of temp, humidity etc.? Does dumping the data clear the memory of the data logger or is it still there as well as now on my dump file?  Incidentally, should the full paths have been needed to  invoke the wee_device command. Thank you and my apologies for so many questions.

[gjr80]

On Thursday, 15 October 2020 at 13:58:25 UTC+10 lecoqacr...@gmail.com wrote:

Thank you for the replies. Yes, as stated the"no_catchup" setting does not exist in my /home/weewx/weewx.conf file. Used the setup.py method of installation using latest weewx as of 10/5/2020. Therefore based on the replies no_catchup is already set to false by default somewhere in the weewx program and therefore should attempt to catch up when offline for several days.

Correct. 

That would only leave the second reason stated in the first reply--that there is a time stamp already newer in my database that has prevented the catchup from reaching further back than this this newer time stamp. How could that be possible since the machine was started after 3 days without an ability to receive data that would include the newer time stamped data?

Hard to say. RPis are prone to time issues, particularly on startup and after power outages, and it’s not unknown for a future dated record to be created. Or it could some other cause/issue.

I did stop weewx and invoke the wee_device --dump command but received the response of  bash : wee_device command not found. I then tried the following command that did work: /home/weewx/util/scripts/wee_device /home/weewx/weewx.conf --dump. I was then prompted for for a yes or no which proceeded with a yes it and it dumped data through 10/07/2020  which should include data through the offline time. This data is to be placed automatically into the database file according to the documentation when using the dump command. How long does it take to show on the plot graphs (/home/weewx/public_html/index.html) which would remove the blank data line of temp, humidity etc.?

Short answer is it depends. Data in reports should be updated on the next report cycle. Data in plots depends on the plot aggregate interval used in the plot, plots that include aggregates are only generated every ‘aggregate interval’; current plots re-generate every report cycle, week plots every hour, month every three hours and year every 24 hours. NOAA format reports are a little different, the current month and year reports are updated every report cycle. Historical NOAA format month and year reports are never re-generated. The easiest way to force and update of the NOAA format reports is to delete all of the generated reports, this will cause all NOAA format reports to be re-generated on the next report cycle. A similar approach can be taken with the plots by deleting the generated plot files to force all plots to be re-generated on the next report cycle.

 

Does dumping the data clear the memory of the data logger or is it still there as well as now on my dump file? 

No, —dump just reads all of the stored archive records and attempts to save them to archive. —clear-memory will clear the logger memory.

 

Incidentally, should the full paths have been needed to  invoke the wee_device command.

setup.py installs typically require you to be in the directory containing the executable (in this case wee_device) or to use an appropriate path to the executable. Package installs usually don’t require the path as the appropriate directory is added to the PATH environment variable on installation. And we probably don’t help too much as we tend not to mention adding the path when listing commands, probably a bit of assumed knowledge there as well.

 

Thank you and my apologies for so many questions.

No problems, we’ve all been in your shoes at some time or other.

Gary 

[lecoqacr...@gmail.com]

Thank you again for your reply. Found the below error in the /var/log/syslog file  for the dump command output that covered parts of 10/06/2020 to 10/14/2020

raspverrypi wee_device [1263] ERROR weewx.manager: Unable to add record 2020-10-07 14:45:00 CDT (1602099900) to database 'weewx.sdb': attempt to write a readonly database

Used the "--dump" command today to recheck after clearing the memory "--clear memory" and received the same above error as yesterday. Weewx was stopped prior to the dump command. This error occurs  every 5 minutes for the periods stated above and for today as well. 

Could this be corrupted memory and the reason for the catchup problem? Should the vantage console be powered down and reset or is there an easier way to solve this error? Thank you

[gjr80]

It looks like the user you were logged in as when you ran wee_device does not have sufficient privileges to write to the database. Try prefixing wee_device with sudo, ie:

$ sudo /home/weewx/bin/wee_device —dump

Gary

[Timothy L]

Thank you again..

Ran the command suggested, sudo /home/weewx/bin/wee_device —dump, and output contained a slightly different message as seen below.

raspberrypi wee_device[1516] ERROR weewx.manager: Unable to add record 2020-10-16 19:10:00 CDT (1602893400) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

The vantage pro2 has a couple of sensors disconnected since trying to finalize setup--unsure if that could cause above error.   thank you

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8529a7da-5b5c-4433-8681-2c50e4eaaad5n%40googlegroups.com.

[gjr80]

That is completely normal. —dump causes all records in the logger to be downloaded and saved to archive. Since the logger just keeps a round robin of the last 2560 records chances are some of those already exist in the archive, those are rejected with the error message you just posted. The records from your outage are hopefully are still in the logger and they will be saved without error. Look through your log file, there will be about 2560 entries, the ones from your three day period where you were offline should be saved without error.

Gary