Re: [weewx-user] 4 beta test report

[Greg Troxel]

Greg Troxel <g...@lexort.com> writes:

(Earlier, on weewx-user, I posted that I was seeing a backtrace from
v4beta (recent master). I'm following up here now that I've read the
code and have something perhaps useful to say.)

> Traceback (most recent call last):
> File "/usr/weewx/bin/weewx/reportengine.py", line 197, in run
> obj.start()
> File "/usr/weewx/bin/weewx/reportengine.py", line 280, in start
> self.run()
> File "/usr/weewx/bin/weewx/imagegenerator.py", line 41, in run
> self.genImages(self.gen_ts)
> File "/usr/weewx/bin/weewx/imagegenerator.py", line 180, in genImages
> aggregate_interval=aggregate_interval)
> File "/usr/weewx/bin/weewx/xtypes.py", line 86, in get_series
> aggregate_interval)
> File "/usr/weewx/bin/weewx/xtypes.py", line 138, in get_series
> for stamp in weeutil.weeutil.intervalgen(startstamp, stopstamp, aggregate_interval):
> File "/usr/weewx/bin/weeutil/weeutil.py", line 339, in intervalgen
> stamp2 = int(time.mktime(dt2.timetuple()))
> OverflowError: mktime argument out of range

I am using python 2.7, NetBSD 8 on earmv7hf-el. weewx 3.x has run for
over 2 years with essentially no problems.

Ihave added printfs to weeutil.py, in intervalgen. The crash happens
during the loop in else block. The last dt2 printout is

dt2 2020-03-08 02:00:00

which is a time that does not exist (here in EST5EDT). So apparently
the python time doing the addition in the datetime object is not going
back to timeval each time.

I am using the older skin that was standard before the Seasons skin
became normal.

Is anyone else seeing this? You can tell you are having this problem
two ways:
traceback in log during html report generation
graphs on week/month/year page are stale



I looked at python datetime docs:
https://docs.python.org/3/library/datetime.html
https://docs.python.org/2/library/datetime.html

There is a notion of "aware" vs "naive" datetime objects, and as I read
it the code is creating naive objects, which don't have an associated
timezone. So adding 3600s to 2020-03-08T01:00 leads to 02:00, even
though I think that time doesn't exist and the right answer is 03:00.

time.mktime() says it will raise an error if the time cannot be
represented:
https://docs.python.org/2/library/time.html


I think to fix, we need a try: block that just does mktime on dt2 and
discards the result, with an catch block that does continue, so we
increment dt2 again. Or, to pass the local tz in the constructor,
perhaps.

I suppose it is possible that the underlying mktime library function on
NetBSD is stricter than on Other Systems. The man page says:

The function returns the specified calendar time; if the calendar
time cannot be represented, it returns (time_t)-1. This can happen
either because the resulting conversion would not fit in a time_t
variable, or because the time specified happens to be in the
daylight savings gap and tm_isdst was set to -1. Other mktime()
implementations do not return an error in the second case and
return the appropriate time offset after the daylight savings gap.
There is code to mimick this behavior, but it is not enabled by
default.

and also makes a claim of conforming to C89.

I have previously observed test failures in bup (a backup program using
git data storage, written in python), and reproduced problems in git. I
didn't ever chase that down, but I am now suspecting the same underlying
cause, namely invocations of mktime which are incorrect according to
standards, but happen to work anyway on Linux. (That is of course
guess, not a substantiated conclusion.)

[Thomas Keffer]

Try running this little program:

import os
import time
import datetime

os.environ['TZ'] = 'America/New_York'
time.tzset()

dt = datetime.datetime(2020,3,8,1)
tt = dt.timetuple()
print("Starting datetime is %s" % dt)
print("Represented as a time tuple, this is %s" % tt)

delta = datetime.timedelta(seconds=3600)
dt1 = dt + delta
print("After adding an hour, the resultant datetime is %s" % dt1)

tt1 = dt1.timetuple()
print("Represented as a timetuple this is %s" % tt1)

ts = time.mktime(tt1)
print("In unixtime: %f" % ts)
print("As time.ctime(), this is %s" % time.ctime(ts))

I get:

Starting datetime is 2020-03-08 01:00:00
Represented as a time tuple, this is time.struct_time(tm_year=2020, tm_mon=3, tm_mday=8, tm_hour=1, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68, tm_isdst=-1)
After adding an hour, the resultant datetime is 2020-03-08 02:00:00
Represented as a timetuple this is time.struct_time(tm_year=2020, tm_mon=3, tm_mday=8, tm_hour=2, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68, tm_isdst=-1)
In unixtime: 1583650800.000000
As time.ctime(), this is Sun Mar  8 03:00:00 2020

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmipndlv950.fsf%40s1.lexort.com.

[Greg Troxel]

I get

Starting datetime is 2020-03-08 01:00:00
Represented as a time tuple, this is time.struct_time(tm_year=2020, tm_mon=3, tm_mday=8, tm_hour=1, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68, tm_isdst=-1)
After adding an hour, the resultant datetime is 2020-03-08 02:00:00
Represented as a timetuple this is time.struct_time(tm_year=2020, tm_mon=3, tm_mday=8, tm_hour=2, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68, tm_isdst=-1)

Traceback (most recent call last):

File "./test.py", line 22, in <module>
ts = time.mktime(tt1)

OverflowError: mktime argument out of range

I wrote a test program for mktime. With the isdst/gmtoff values from
localtime, it works. With this, I can provoke EINVAL, which I think is
per the spec.

----------------------------------------
#include <errno.h>
#include <stdio.h>
#include <time.h>

int
main()
{
struct tm *t;
time_t start = 1583647200; /* 20200308T0100 EST */
time_t plus1;

t = localtime(&start);

printf("isdst %d tm_gmtoff %ld\n", t->tm_isdst, t->tm_gmtoff);
t->tm_isdst = -1;
t->tm_gmtoff = 0;
printf("isdst %d tm_gmtoff %ld\n", t->tm_isdst, t->tm_gmtoff);

t->tm_hour += 1;
printf("tm_hour %d\n", t->tm_hour);

plus1 = mktime(t);

printf("plus1 %jd errno %d\n", (intmax_t) plus1, errno);

printf("result: %s\n", plus1 == 1583650800 ? "ok" : "bad");


return 0;
}
----------------------------------------
isdst 0 tm_gmtoff -18000
isdst -1 tm_gmtoff 0
tm_hour 2
plus1 -1 errno 22
result: bad

[Thomas Keffer]

OK, so we've isolated the issue to a problem with how NetBSD implements time.mktime().

I suppose we could trap the exception in intervalgen() and, instead, return the simple arithmetic result of adding increment. Something like:

        delta = datetime.timedelta(seconds=interval)
        last_stamp1 = 0
        while dt1 < stop_dt:
            dt2 = min(dt1 + delta, stop_dt)
            stamp1 = int(time.mktime(dt1.timetuple()))

            try:
                stamp2 = int(time.mktime(dt2.timetuple()))

            except OverflowError:

                stamp2 = stamp1 + interval
            if stamp2 > stamp1 > last_stamp1:
                yield TimeSpan(stamp1, stamp2)
                last_stamp1 = stamp1
            dt1 = dt2

[Greg Troxel]

Thomas Keffer <tke...@gmail.com> writes:

> OK, so we've isolated the issue to a problem with how NetBSD implements
> time.mktime().

I don't think it's a problem in the implementation, just not matching
the weewx code's beyond-standards expectation. As I understand it, an
error return for times that do not exist in the local timezone is the
standard approach for mktime, and guessing what it might mean, while
obviously useful, is an extension. (If it is a NetBSD bug, I'll file a
ticket and see about fixing it; we are as a group pretty hard-core about
standards compliance.)

I just looked at the sources, and there is a default-off option
NO_ERROR_IN_DST_GAP. Reading the logs, it seems people think the error
behavior in the gap is correct.

But even if is a bug, it would be nice to avoid it.

> I suppose we could trap the exception in intervalgen() and, instead, return
> the simple arithmetic result of adding increment. Something like:
>
> delta = datetime.timedelta(seconds=interval)
> last_stamp1 = 0
> while dt1 < stop_dt:
> dt2 = min(dt1 + delta, stop_dt)
> stamp1 = int(time.mktime(dt1.timetuple()))
> try:
> stamp2 = int(time.mktime(dt2.timetuple()))
> except OverflowError:
> stamp2 = stamp1 + interval
> if stamp2 > stamp1 > last_stamp1:
> yield TimeSpan(stamp1, stamp2)
> last_stamp1 = stamp1
> dt1 = dt2

We could, and I think that would work fine.

But, taking a step back, I think the code is ending up running in a
situation where it isn't helpful, and in the case mktime doens't raise
an exception, returning not what's expected.

First, I get it that with dt1 being some date and 0000, and interval
24h, we want the posix time_t values corresponding to local midnight,
even if a day is 23h or 25h. And simlarly when interval is 3h, 6h, or
12h. (I have seen the graphs that result in plotting programs that
aren't aware of this, and they are difficult to read and ugly!)

But when interval is 1h, I think we want to use and display the local tz
naming of times that are actually 1h apart. So for the day in question,
that's

0000
0100
0300
0400

and we want those to appear equi-spaced on the graph.

With the try/except above, that I think will happen on systems where
mktime rejects 0200.

With systems where mktime of 0200 returns the time_t for 0300 (which is
3600 after the time_t for 0100), I think it will result in

0000
0100
0300
0300
0400

So I wonder about another if block that checks for interval == 3600 (or
maybe <= 3600?) and just does straightforward time_t operations. For
these intervals, a one hour change will not cause misrounding.

I suppose there is the issue of DST shifts that are less than 1h, but my
impression is that DST is always 1h or 2h, even if TZ offsets are :30 or
:15.