Weewx seems to have stopped working when time changed for DST

[Robert Westrich]

Does anyone else have this problem or a fix? The last successful update was at 3:00 am when time changed here from 2:00 am to 3:00 am for DST...

Stopped/started weewx, didn't help. Rebooted Ubuntu, that didn't help.

Attached is syslog:

Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Using configuration file /etc/weewx/weewx.conf.
Mar 10 17:54:22 ubuntu-01 weewx[5026]: VantagePro: Opened up serial port /dev/ttyUSB0, baudrate 19200
Mar 10 17:54:22 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdConvert
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: StdConvert target unit is 0x1
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdConvert
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdCalibrate
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdCalibrate
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdQC
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdQC
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdArchive
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Using station hardware archive interval of 60
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Record generation will be attempted in 'hardware'
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Using archive database: archive_sqlite
Mar 10 17:54:22 ubuntu-01 weewx[5026]: stats: Schema exists with 16 elements
Mar 10 17:54:22 ubuntu-01 weewx[5026]: stats: Backfilling stats database.
Mar 10 17:54:22 ubuntu-01 weewx[5026]: stats: stats database up to date.
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Using stats database: stats_sqlite
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdArchive
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdTimeSynch
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdTimeSynch
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdPrint
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdPrint
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdRESTful
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Data will be posted to Wunderground
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Data will not be posted to PWSweather
Mar 10 17:54:22 ubuntu-01 weewx[5026]:     ****  'station'
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Data will be posted to CWOP
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Started thread for RESTful upload sites.
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdRESTful
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Loading service weewx.wxengine.StdReport
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Finished loading service weewx.wxengine.StdReport
Mar 10 17:54:22 ubuntu-01 weewx[5026]: wxengine: Starting up weewx version 2.2.1.
Mar 10 17:54:22 ubuntu-01 weewx[5026]: VantagePro: Getting archive packets since 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:54:23 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:54:25 ubuntu-01 weewx[5026]: VantagePro: Retrieving 513 page(s); starting index= 4
Mar 10 17:54:25 ubuntu-01 weewx[5026]: VantagePro: DMPAFT complete: page timestamp 2013-03-08 22:15:00 EST (1362798900) less than final timestamp 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:54:25 ubuntu-01 weewx[5026]: VantagePro: Catch up complete.
Mar 10 17:54:25 ubuntu-01 weewx[5026]: wxengine: Starting main packet loop.
Mar 10 17:54:26 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:54:26 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:54:26 ubuntu-01 weewx[5026]: wxengine: Clock error is -0.11 seconds (positive is fast)
Mar 10 17:54:26 ubuntu-01 weewx[5026]: VantagePro: Requesting 200 LOOP packets.
Mar 10 17:54:27 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:55:11 ubuntu-01 weewx[5026]: VantagePro: Getting archive packets since 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:55:12 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:55:13 ubuntu-01 weewx[5026]: VantagePro: Retrieving 513 page(s); starting index= 0
Mar 10 17:55:13 ubuntu-01 weewx[5026]: VantagePro: DMPAFT complete: page timestamp 2013-03-08 22:16:00 EST (1362798960) less than final timestamp 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:55:13 ubuntu-01 weewx[5026]: VantagePro: Catch up complete.
Mar 10 17:55:13 ubuntu-01 weewx[5026]: reportengine: Running reports for latest time in the database.
Mar 10 17:55:13 ubuntu-01 weewx[5026]: reportengine: Running report StandardReport
Mar 10 17:55:14 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Standard/skin.conf for report StandardReport
Mar 10 17:55:14 ubuntu-01 weewx[5026]: stats: Schema exists with 16 elements
Mar 10 17:55:14 ubuntu-01 weewx[5026]: filegenerator: generated 1 'SummaryByMonth' files in 0.04 seconds
Mar 10 17:55:14 ubuntu-01 weewx[5026]: filegenerator: generated 1 'SummaryByYear' files in 0.09 seconds
Mar 10 17:55:14 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:55:14 ubuntu-01 weewx[5026]: VantagePro: Requesting 200 LOOP packets.
Mar 10 17:55:14 ubuntu-01 weewx[5026]: filegenerator: generated 14 'toDate' files in 0.82 seconds
Mar 10 17:55:15 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:55:15 ubuntu-01 weewx[5026]: genimages: Generated 30 images in 0.58 seconds
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: copied 9 files to /var/www/weewx
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: Running report FTP
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report FTP
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: FTP upload not requested. Skipped.
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: Running report RSYNC
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report RSYNC
Mar 10 17:55:15 ubuntu-01 weewx[5026]: reportengine: rsync upload not requested. Skipped.
Mar 10 17:56:11 ubuntu-01 weewx[5026]: VantagePro: Getting archive packets since 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:56:12 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:56:13 ubuntu-01 weewx[5026]: VantagePro: Retrieving 513 page(s); starting index= 1
Mar 10 17:56:13 ubuntu-01 weewx[5026]: VantagePro: DMPAFT complete: page timestamp 2013-03-08 22:17:00 EST (1362799020) less than final timestamp 2013-03-10 03:00:00 EDT (1362898800)
Mar 10 17:56:13 ubuntu-01 weewx[5026]: VantagePro: Catch up complete.
Mar 10 17:56:13 ubuntu-01 weewx[5026]: reportengine: Running reports for latest time in the database.
Mar 10 17:56:13 ubuntu-01 weewx[5026]: reportengine: Running report StandardReport
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Standard/skin.conf for report StandardReport
Mar 10 17:56:14 ubuntu-01 weewx[5026]: stats: Schema exists with 16 elements
Mar 10 17:56:14 ubuntu-01 weewx[5026]: filegenerator: generated 1 'SummaryByMonth' files in 0.03 seconds
Mar 10 17:56:14 ubuntu-01 weewx[5026]: filegenerator: generated 1 'SummaryByYear' files in 0.06 seconds
Mar 10 17:56:14 ubuntu-01 weewx[5026]: filegenerator: generated 14 'toDate' files in 0.15 seconds
Mar 10 17:56:14 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console
Mar 10 17:56:14 ubuntu-01 weewx[5026]: VantagePro: Requesting 200 LOOP packets.
Mar 10 17:56:14 ubuntu-01 weewx[5026]: genimages: Generated 30 images in 0.58 seconds
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: copied 0 files to /var/www/weewx
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: Running report FTP
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report FTP
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: FTP upload not requested. Skipped.
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: Running report RSYNC
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report RSYNC
Mar 10 17:56:14 ubuntu-01 weewx[5026]: reportengine: rsync upload not requested. Skipped.
Mar 10 17:56:15 ubuntu-01 weewx[5026]: VantagePro: successfully woke up console

[Ted Garrison]

No issue here.  Never glitched as it went from 1:55 to 3:00...

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Thomas Keffer]

(NB: my station had no problems either.)

I'm not sure exactly what's going on here. The record for 0300 was correctly downloaded from the console and put in the database, but nothing has appeared in the console memory since then. 

I suspect the memory got corrupted somehow. Try the following in order:

1. Stop weewx.

2. Run the configuration utility with the '--info' flag and send the results to me before doing anything else:

wee_config_vantage /home/weewx/weewx.conf --info

3. You may be able to recover by running the same utility with the '--dump' flag, then restarting weewx. This will result in lots of warnings of duplicate records.

wee_config_vantage /home/weewx/weewx.conf --dump

4. If that doesn't work, clear the memory. 

wee_config_vantage /home/weewx/weewx.conf --clear

then unplug, take the batteries out, put back together, and restart everything.

Let me know how it goes.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Robert Westrich]

Tom,

Thanks for the reply. I had already done the --info and saw that my station was not configured for DST the way the example is in the user doc, so I fixed that first. Then I did the --dump, and then restarted without clearing, and that seemed to solve the issue. The DST changes I made were prior to the --dump and the --dump was required to solve the issue. Not sure why a --clear wasn't needed, as I think the --dump was only there to take what is in the station and force it into the db...

In any case, it appears to be working now. I used wunderfixer to update Weather Underground for the missing data, and noticed something else...

I have my archive interval set to 60 seconds in the station. I do see in the syslog that weewx processes the wu update, but wu was only showing 5 minute interval in the tabular data. After wunderfixer, at least for today, I do see the data for every archive interval. Not sure if that is a wu issue or not. Any idea why wunderfixer can send data for every minute but wu only takes data every 5 minutes from weewx? There are no errors in the syslog to indicate wu update failed. I do see the proper messages about CWOP waiting to update until the 5 minute window is reached....

Bob

[Thomas Keffer]

The --dump worked because of the way the memory in the Vantage works. You ask for all data records 'since' some time, and then it starts returning pages starting with that time. They continually march forward in time and then you stop when you see something in the past, signalling that the memory has wrapped around and you're done. Unfortunately, for reasons I don't understand, it occasionally returns a starting page with a date before the given starting date, so it looks like everything has been read.

By dumping all the data you get around this problem. Now you have all the data, so the 'since' date becomes the current time. It seems to work fine then.

As for the rest of your question, I have never tries such a short archive interval. If syslog is showing that all the WU updates went out, but WU is only showing one every 5 minutes, well, I guess that's all they will accept. 

I don't know why wunderfixer works any differently. Maybe because it is sending data from the past? Dunno.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Westrich]

Thanks again Tom

[Peter Finley]

Hi Tom,

I ran into this on a station that I manage as well.

Here's the result of querying the console:

Davis Vantage EEPROM settings:
    
    CONSOLE TYPE:                   VantageVue
    
    CONSOLE FIRMWARE:
      Date:                         Oct 12 2009
      Version:                      2.12
    
    CONSOLE SETTINGS:
      Archive interval:             1800 (seconds)
      Altitude:                     769 (foot)
      Wind cup type:                large
      Rain bucket type:             0.01 inches
      Rain year start:              1
      Onboard time:                 2013-03-13 21:12:43
      
    CONSOLE DISPLAY UNITS:
      Barometer:                    inHg
      Temperature:                  degree_F
      Rain:                         inch
      Wind:                         mile_per_hour
      
    CONSOLE STATION INFO:
      Latitude (onboard):           41.8
      Longitude (onboard):          -88.1
      Time zone code:               6
      Use manual or auto DST?       MANUAL
      DST setting:                  OFF
      GMT offset:                   -6.0 hours
      Use GMT offset or time zone?  GMT_OFFSET
        
    RECEPTION STATS:
      Total packets received:       29742
      Total packets missed:         98
      Number of resynchronizations: 0
      Longest good stretch:         1268
      Number of CRC errors:         21
      
    BAROMETER CALIBRATION DATA:
      Current barometer reading:    30.370 inHg
      Altitude:                     769 feet
      Dew point:                    15 F
      Virtual temperature:          27 F
      Humidity correction factor:   5
      Correction ratio:             1.030
      Correction constant:          +0.000 inHg
      Gain:                         0.000
      Offset:                       -57.000

The DST is set to MANUAL. Must this be AUTO? Should this be set to AUTO? I don't see anything in the docs that say what it should be set to.

Here are the logs from around 3 AM on the 10th:

Mar 10 01:30:16 weather weewx[5345]: Archive: added archive record 2013-03-10 01:30:00 CST (1362900600)
Mar 10 01:30:16 weather weewx[5345]: restful: Published record 2013-03-10 01:30:00 CST (1362900600) to Wunderground station 
Mar 10 01:30:16 weather weewx[5345]: filegenerator: generated 1 'SummaryByMonth' files in 0.34 seconds
Mar 10 01:30:17 weather weewx[5345]: filegenerator: generated 1 'SummaryByYear' files in 1.04 seconds
Mar 10 01:30:18 weather weewx[5345]: filegenerator: generated 12 'toDate' files in 0.61 seconds
Mar 10 01:30:18 weather weewx[5345]: genimages: Generated 11 images in 0.42 seconds
Mar 10 03:00:16 weather weewx[5345]: Archive: added archive record 2013-03-10 03:00:00 CDT (1362902400)
Mar 10 03:00:16 weather weewx[5345]: restful: Published record 2013-03-10 03:00:00 CDT (1362902400) to Wunderground station 
Mar 10 03:00:16 weather weewx[5345]: filegenerator: generated 1 'SummaryByMonth' files in 0.34 seconds
Mar 10 03:00:17 weather weewx[5345]: wxengine: Clock error is -3599.76 seconds (positive is fast)
Mar 10 03:00:17 weather weewx[5345]: filegenerator: generated 1 'SummaryByYear' files in 1.03 seconds
Mar 10 03:00:17 weather weewx[5345]: VantagePro: Clock set to 2013-03-10 03:00:17 CDT (1362902417)
Mar 10 03:00:18 weather weewx[5345]: filegenerator: generated 12 'toDate' files in 0.61 seconds
Mar 10 03:00:20 weather weewx[5345]: genimages: Generated 33 images in 2.21 seconds
Mar 10 03:00:21 weather weewx[5345]: filegenerator: generated 1 'SummaryByMonth' files in 0.34 seconds
Mar 10 03:00:22 weather weewx[5345]: filegenerator: generated 1 'SummaryByYear' files in 1.04 seconds
Mar 10 03:00:23 weather weewx[5345]: filegenerator: generated 12 'toDate' files in 0.61 seconds
Mar 10 03:00:25 weather weewx[5345]: genimages: Generated 33 images in 2.18 seconds
Mar 10 03:30:18 weather weewx[5345]: filegenerator: generated 1 'SummaryByMonth' files in 0.34 seconds
Mar 10 03:30:19 weather weewx[5345]: filegenerator: generated 12 'toDate' files in 0.67 seconds
Mar 10 03:30:21 weather weewx[5345]: genimages: Generated 33 images in 2.18 seconds

It looks like the clock was adjusted right at that same moment.

Performing the dump appears to have fixed it. Thanks for the tip!

Peter

On Sun, Mar 10, 2013 at 9:09 PM, Robert Westrich <rswes...@gmail.com> wrote:

Thanks again Tom

[Clive Johnston]

I'm based in the UK and we have moved into British Summer Time (BST) which is GMT + 1.  I have experienced exactly the same symptoms with a Davis Vantage Pro2 running WeeWX.  In our case the time moves forward at 1am and becomes 2am.  The record for 2am was retrieved from the console and uploaded to Wunderground, but WeeWX got stuck.  I couldnt get it to work at all and ended up clearing the memory with the config tool, resulting in losing the data between 2am and 7:30pm. 

I am also looking for a fix for this issue, any ideas?   I looked at the console settings and can also see that my DST settings are manual. 

Use manual or auto DST?       MANUAL
DST setting:                  OFF

Is this the problem?

[Thomas Keffer]

Yes, it is.

It's too bad you cleared the memory of the console, because it was recoverable using the --dump option in wee_config_vantage.

But, users shouldn't get punished like this for having manual DST on their consoles. I'll see if I can change the driver to be more forgiving.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Pete]

Just to spoil the manual DST theory:

Use manual or auto DST?       AUTO

Weewx stopped updating on the change last night. using wee_config_vantage with --dump did sort it out though in my case.

[Thomas Keffer]

Rats! You had to spoil it!

.... still scratching my head on this ...

Pete, do you have the system log from last night? Particularly around the transition time? I'm interested in how the Vantage DMPAFT command works during that period.

-tk

[Pete]

Until today the station was running quite happily with a fairly old version (1.10.2!) and the issue triggered at around the switch over. Given it was no longer doing its thing uninterrupted though I thought it was a good time to upgrade! Now running version 2.2.1 (only the archive.sdb copied over) it still persisted until doing wee_config_vantage --dump.

Mar 31 00:30:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 00:30:00 GMT (1364689800)
Mar 31 00:30:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.07 seconds
Mar 31 00:30:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.08 seconds
Mar 31 00:30:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.08 seconds
Mar 31 00:30:23 wx-host weewx[594]: genimages: Generated 11 images in 0.10 seconds
Mar 31 00:45:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 00:45:00 GMT (1364690700)
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.10 seconds
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.11 seconds
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.11 seconds
Mar 31 00:45:23 wx-host weewx[594]: genimages: Generated 11 images in 0.14 seconds
Mar 31 02:00:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 02:00:00 GMT (1364691600)
Mar 31 02:00:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.07 seconds
Mar 31 02:00:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.09 seconds
Mar 31 02:00:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.08 seconds
Mar 31 02:00:23 wx-host weewx[594]: genimages: Generated 22 images in 0.29 seconds
Mar 31 02:15:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.07 seconds
Mar 31 02:15:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.09 seconds
Mar 31 02:15:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.08 seconds
Mar 31 02:15:24 wx-host weewx[594]: genimages: Generated 22 images in 0.30 seconds
Mar 31 02:30:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.07 seconds
Mar 31 02:30:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.09 seconds
Mar 31 02:30:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.08 seconds
Mar 31 02:30:24 wx-host weewx[594]: genimages: Generated 22 images in 0.29 seconds
Mar 31 02:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.11 seconds
Mar 31 02:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.08 seconds
Mar 31 02:45:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.08 seconds
Mar 31 02:45:23 wx-host weewx[594]: genimages: Generated 22 images in 0.30 seconds
Mar 31 02:45:27 wx-host weewx[594]: VantagePro: Clock error is -5.35 seconds (positive is fast)

[Pete]

Here is it reproduced with debug = 1

Apr  4 18:31:34 wx-host weewx[9858]: VantagePro: Getting archive packets since 2013-03-31 02:00:00 BST (1364691600)
Apr  4 18:31:36 wx-host weewx[9858]: VantagePro: successfully woke up console
Apr  4 18:31:40 wx-host weewx[9858]: VantagePro: Retrieving 513 page(s); starting index= 4
Apr  4 18:31:41 wx-host weewx[9858]: VantagePro: DMPAFT complete: page timestamp 2013-03-09 01:45:00 GMT (1362793500) less than final timestamp 2013-03-31 02:00:00 BST (1364691600)
Apr  4 18:31:41 wx-host weewx[9858]: VantagePro: Catch up complete.
Apr  4 18:31:41 wx-host weewx[9858]: wxengine: Starting main packet loop.
Apr  4 18:31:43 wx-host weewx[9858]: VantagePro: successfully woke up console

[Pete]

So after a bit more digging the Davis tech ref says

In order to use the "DMPAFT" command you need to determine the time and date-stamp of the last
record that you already have, AND this record should match one of the records already archived
in the WeatherLink data logger. (if the data is not found, then the entire contents of the data
archive will be downloaded.)

In my previous post you can see the station is dumping all its memory (1 partial + 511 records + 1 partial = 513) possibly indicating that the requested data record doesn't exist. Tweaking the timestamp passed to genArchiveRecords (since_ts) by one archive interval to 00:45 the station returns the correct number of pages, gets a single primary key duplicate error and then inserts all of the missing records.

So my guess is that at 01:00 an archive packet is generated and logged by weewx. The DST change kicks in and the console clock jumps forward an hour to 02:00. Now either the data record is stored as 02:00 or the 01:00 entry is overwritten, either way the record from 01:00 is missing.

Any thoughts for handling this elegantly?

[Thomas Keffer]

I think you're getting close. Here's my timeline:

1364690700 or 00:45 standard time. Weewx logs the 00:45 record as unix time 1364690700.

1364691600, or 01:00 standard time, 02:00 DST. Weewx asks for all records since 1364690700 . However, since the console has no notion of UTC (damn them!), this requires conversion to local time. Python does this, and comes up with 01:00 standard time. The console recognizes this and returns the next archive record, that is the one in the logger memory under date/time 01:00. All is well.

1364692500, or 02:15 DST. Weewx asks for all records since 1364691600. Again, this has to be converted to local time. Python comes up with 02:15. Now when the console looks in the logger memory what it sees depends on whether or not the console is honoring the daylight savings time transition. If it is, then all should be well: the next archive record is in 02:15 and should be found. However, if it is not, the record is in 01:15 and will not be found. 

This is why I think whether or nor the console's honoring DST makes a difference.

What am I missing?

-tk

[Thomas Keffer]

Pete, one other thing. In your log I see:

Mar 31 00:45:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 00:45:00 GMT (1364690700)
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByMonth' files in 0.10 seconds
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 1 'SummaryByYear' files in 0.11 seconds
Mar 31 00:45:23 wx-host weewx[594]: filegenerator: generated 7 'toDate' files in 0.11 seconds
Mar 31 00:45:23 wx-host weewx[594]: genimages: Generated 11 images in 0.14 seconds
Mar 31 02:00:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 02:00:00 GMT (1364691600)

But, I thought GMT does not honor DST. How come there is the one hour jump?

The string comes from the utility function weeutil.timestamp_to_string, which uses the "%Z" formatter for time zone. It should be your local time zone, not GMT. Anything funny about your locale?

-tk

On Thu, Apr 4, 2013 at 12:34 PM, Pete <pww...@gmail.com> wrote:

[Pete]

You are correct, it should be "02:00:00 BST" not "02:00:00 GMT". There is something strange going on with the locale:

>>> import locale, time
>>> locale.setlocale(locale.LC_ALL, "")
'en_GB.UTF-8'
>>> time.strftime("%X %x %Z", time.localtime())
'15:10:02 05/04/13 GMT'
>>> time.strftime("%X %x %Z")
'15:10:06 05/04/13 BST'
>>> time.tzname
('GMT', 'BST')

Times are correct local times in both cases just not the zone from %Z. The version of Python I'm using for Weewx is 2.6.6, essentially I think this is wrong:

>>> time.strftime("%Z") == time.strftime("%Z", time.localtime())
False

vs Python 2.7.3 on another box:

>>> time.strftime("%Z") == time.strftime("%Z", time.localtime())
True

when it really should be equal for both, given that time.strftime without the second parameter uses localtime(). That must not be quite true for python 2.6.6 though.

[Thomas Keffer]

So, I am wondering if this is related to your experience with weewx failing at the DST crossover despite you having 'AUTO' DST set on your console?

Your observation is making me suspicious that your version of Python may have botched the unix epoch time to local time conversion. Or, is there something odd about your locale?

-tk

[Thomas Keffer]

Pete,

I just uploaded V2.3.0b1, a beta of V2.3. It includes the ability to create a summary of the logger memory, page by page. If your console still has the DST transition in memory, and if you're willing to try a beta, I'd appreciate you giving it a go. 

It's in http://sourceforge.net/projects/weewx/files/development_versions

To use:

cd /home/weewx

./bin/wee_config_vantage weewx.conf --logger-summary=FILE'

where FILE is a path to a filename where you have write privileges.

Send me the file (or post here).

Thanks in advance!

-tk

On Fri, Apr 5, 2013 at 7:29 AM, Pete <pww...@gmail.com> wrote:

[Pete]

I'm not sure the locale/strftime issue extends beyond incorrectly printing the zone for our needs:

>>> import time
>>> last_db_datetime = 1364691600        #01:00 UTC/02:00 DST (localtime)
>>> time.localtime(last_db_datetime - 1)
time.struct_time(tm_year=2013, tm_mon=3, tm_mday=31, tm_hour=0, tm_min=59, tm_sec=59, tm_wday=6, tm_yday=90, tm_isdst=0)
>>> time.localtime(last_db_datetime)
time.struct_time(tm_year=2013, tm_mon=3, tm_mday=31, tm_hour=2, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=90, tm_isdst=1)

The first call to localtime has tm_isdst=0 while the second (the first second of BST) shows tm_isdst=1 and is an hour ahead of GMT correctly.

If the DMPAFT timestamp sent is 02:00 BST localtime (1364691600) the Davis logger returns all (512/513) of the pages indicating the record doesn't exist.

If the DMPAFT timestamp is for the non existant localtime 01:00 (really 02:00 DST/01:00 UTC, but the logger expects localtime) the logger returns pages from 02:15 onwards as we expect with no duplicates or missing data points (the archive interval is 15 minutes).

I'll send you a copy of the logger-summary when it is completed, although it looks as if that may be a while!

[Thomas Keffer]

On Fri, Apr 5, 2013 at 10:29 AM, Pete <pww...@gmail.com> wrote:

I'll send you a copy of the logger-summary when it is completed, although it looks as if that may be a while!

? It should only take 1 or 2 minutes to do the summary. Remember to stop weewx before running. 

Or, are you referring to the time to install the beta version?

-tk

[Pete]

Davis Vantage EEPROM settings:

    CONSOLE TYPE:                   VantagePro2

    CONSOLE FIRMWARE:
      Date:                         Sep 29 2009
      Version:                      1.90

    CONSOLE SETTINGS:
      ...
      Onboard time:                 2013-04-05 18:31:11

    CONSOLE STATION INFO:
      ...
      Time zone code:               19

      Use manual or auto DST?       AUTO

      DST setting:                  N/A
      GMT offset:                   +1.0 hours

      Use GMT offset or time zone?  GMT_OFFSET

I notice the time zone is incorrectly set to 19 (GMT Monrovia, Casablanca) instead of 18 ((GMT) Greenwich Mean Time, Dublin, Edinburgh, Lisbon, London).

Maybe this is at fault? It looks as if time zone 19 is possibly GMT without DST, however the console is still using GMT offset (indicating it is following DST)?

It seems that the console advances at 02:00 instead of 01:00. I understand that in the US the daylight switch happens at 02:00 local time while in Europe it is 01:00 UTC. So GMT switches 01:00 -> 02:00, CET switches 02:00 -> 03:00 and EET switches 03:00 -> 04:00 (when moving into summer time).

Here are the relevant sections for a few hours either side of the DST switch

Python 2.6.6:

2200  440    0 | 2013-03-30 23:00 | 2013-03-30 23:00:00 GMT (1364684400)
2201  440    1 | 2013-03-30 23:15 | 2013-03-30 23:15:00 GMT (1364685300)
2202  440    2 | 2013-03-30 23:30 | 2013-03-30 23:30:00 GMT (1364686200)
2203  440    3 | 2013-03-30 23:45 | 2013-03-30 23:45:00 GMT (1364687100)
2204  440    4 | 2013-03-31 00:00 | 2013-03-31 00:00:00 GMT (1364688000)
2205  441    0 | 2013-03-31 00:15 | 2013-03-31 00:15:00 GMT (1364688900)
2206  441    1 | 2013-03-31 00:30 | 2013-03-31 00:30:00 GMT (1364689800)
2207  441    2 | 2013-03-31 00:45 | 2013-03-31 00:45:00 GMT (1364690700)
2208  441    3 | 2013-03-31 01:00 | 2013-03-31 02:00:00 GMT (1364691600)    #Out of sync
2209  441    4 | 2013-03-31 01:15 | 2013-03-31 02:15:00 GMT (1364692500)
2210  442    0 | 2013-03-31 01:30 | 2013-03-31 02:30:00 GMT (1364693400)
2211  442    1 | 2013-03-31 01:45 | 2013-03-31 02:45:00 GMT (1364694300)
2212  442    2 | 2013-03-31 03:00 | 2013-03-31 03:00:00 GMT (1364695200)    #Back in sync
2213  442    3 | 2013-03-31 03:15 | 2013-03-31 03:15:00 GMT (1364696100)
2214  442    4 | 2013-03-31 03:30 | 2013-03-31 03:30:00 GMT (1364697000)
2215  443    0 | 2013-03-31 03:45 | 2013-03-31 03:45:00 GMT (1364697900)
2216  443    1 | 2013-03-31 04:00 | 2013-03-31 04:00:00 GMT (1364698800)
2217  443    2 | 2013-03-31 04:15 | 2013-03-31 04:15:00 GMT (1364699700)
2218  443    3 | 2013-03-31 04:30 | 2013-03-31 04:30:00 GMT (1364700600)
2219  443    4 | 2013-03-31 04:45 | 2013-03-31 04:45:00 GMT (1364701500)
2220  444    0 | 2013-03-31 05:00 | 2013-03-31 05:00:00 GMT (1364702400)
2221  444    1 | 2013-03-31 05:15 | 2013-03-31 05:15:00 GMT (1364703300)
2222  444    2 | 2013-03-31 05:30 | 2013-03-31 05:30:00 GMT (1364704200)
2223  444    3 | 2013-03-31 05:45 | 2013-03-31 05:45:00 GMT (1364705100)
2224  444    4 | 2013-03-31 06:00 | 2013-03-31 06:00:00 GMT (1364706000)

Python 2.7.3:

2200  440    0 | 2013-03-30 23:00 | 2013-03-30 23:00:00 GMT (1364684400)       
2201  440    1 | 2013-03-30 23:15 | 2013-03-30 23:15:00 GMT (1364685300)       
2202  440    2 | 2013-03-30 23:30 | 2013-03-30 23:30:00 GMT (1364686200)       
2203  440    3 | 2013-03-30 23:45 | 2013-03-30 23:45:00 GMT (1364687100)       
2204  440    4 | 2013-03-31 00:00 | 2013-03-31 00:00:00 GMT (1364688000)       
2205  441    0 | 2013-03-31 00:15 | 2013-03-31 00:15:00 GMT (1364688900)       
2206  441    1 | 2013-03-31 00:30 | 2013-03-31 00:30:00 GMT (1364689800)       
2207  441    2 | 2013-03-31 00:45 | 2013-03-31 00:45:00 GMT (1364690700)       
2208  441    3 | 2013-03-31 01:00 | 2013-03-31 02:00:00 BST (1364691600)        #out of sync
2209  441    4 | 2013-03-31 01:15 | 2013-03-31 02:15:00 BST (1364692500)       
2210  442    0 | 2013-03-31 01:30 | 2013-03-31 02:30:00 BST (1364693400)       
2211  442    1 | 2013-03-31 01:45 | 2013-03-31 02:45:00 BST (1364694300)       
2212  442    2 | 2013-03-31 03:00 | 2013-03-31 03:00:00 BST (1364695200)        #back in sync
2213  442    3 | 2013-03-31 03:15 | 2013-03-31 03:15:00 BST (1364696100)       
2214  442    4 | 2013-03-31 03:30 | 2013-03-31 03:30:00 BST (1364697000)       
2215  443    0 | 2013-03-31 03:45 | 2013-03-31 03:45:00 BST (1364697900)       
2216  443    1 | 2013-03-31 04:00 | 2013-03-31 04:00:00 BST (1364698800)       
2217  443    2 | 2013-03-31 04:15 | 2013-03-31 04:15:00 BST (1364699700)       
2218  443    3 | 2013-03-31 04:30 | 2013-03-31 04:30:00 BST (1364700600)       
2219  443    4 | 2013-03-31 04:45 | 2013-03-31 04:45:00 BST (1364701500)       
2220  444    0 | 2013-03-31 05:00 | 2013-03-31 05:00:00 BST (1364702400)       
2221  444    1 | 2013-03-31 05:15 | 2013-03-31 05:15:00 BST (1364703300)       
2222  444    2 | 2013-03-31 05:30 | 2013-03-31 05:30:00 BST (1364704200)       
2223  444    3 | 2013-03-31 05:45 | 2013-03-31 05:45:00 BST (1364705100)       
2224  444    4 | 2013-03-31 06:00 | 2013-03-31 06:00:00 BST (1364706000)

[Thomas Keffer]

I would imagine the internal algorithm in the console would know the rules for switching DST according to the local time zone code. At least, the VP2 manual would leave one to believe that.

By default, weewx corrects the clock on the console to match your computer's clock. With a time zone difference between them, the console would have a different idea of what that time represents than the computer. I can easily see how a record could not be found under these circumstances.

-tk

[Pete]

The computer's clock must have updated correctly going by the consecutive timestamps in the syslog:

Mar 31 00:45:23 wx-host weewx[594]: genimages: Generated 11 images in 0.14 seconds
Mar 31 02:00:22 wx-host weewx[594]: Archive: added archive record 2013-03-31 02:00:00 GMT (1364691600)

The syslog doesn't show the VP2 clock being updated until 02:45 and even then it is only by -5.35 seconds (this is common, the WeatherlinkIP can be slow to respond):

Mar 31 02:45:27 wx-host weewx[594]: VantagePro: Clock error is -5.35 seconds (positive is fast)

So in the spirit of testing the console a bit more I've just tried a few things with the data logger disconnected. Setting the console time to 00:59 on 2013-03-31, the clock ticked to 01:00. Trying 01:59 it moved over to 03:00 (DST started an hour late for the timezone). The same also happened when I restarted the console and set timezone=18 (GMT UK where 00:59 should be followed by 02:00).

Apologies if I'm completely missing something here!

[Pete]

Just to add to this Eastern European Time also switched 01:59 -> 03:00 on the console (it should switch 02:59 -> 04:00).

So for the timezones I've tested it seems the console (with firmware 1.90) always assumes the DST change always occurs at 02:00, when it can happen at least at 01:00 and 03:00 in some zones.

Does anyone else still have their Weewx syslog output from when the DST change happened?

[Andy Lodge]

I've got exactly the same problem since the clocks went forward...  how do i download wee_config_vantage to my raspberry to run the dump command to try to fix it ?

Cheers, Andy

[Thomas Keffer]

See this more recent thread: https://groups.google.com/d/msg/weewx-user/rM2IDREAWiI/oloGcj7aAAAJ

-tk

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.