Monthly reports for current and previous months are generated in different display units

[Mickael Hoareau]

Hello,

I have a strange problem.

I'm using weewx 3.8.2 with a Davis Vantage Pro 2.

On a station with about 6 years of data (http://stations.meteor-oi.re/ESH555/), I display the reports in metric units.

[Units]

    #
    # This section is for managing the selection and formatting of units.
    #
   
    [[Groups]]
        #
        # For each group of measurements, this section sets what units to use for it.
        # NB: The unit is always in the singular. I.e., 'mile_per_hour', NOT 'miles_per_hour'
        #
        group_altitude     = meter                 # Options are 'foot' or 'meter'
        group_degree_day   = degree_C_day         # Options are 'degree_F_day' or 'degree_C_day'
        group_direction    = degree_compass
        group_moisture     = centibar
        group_percent      = percent
        group_pressure     = hPa                  # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'
        group_radiation    = watt_per_meter_squared
        group_rain         = mm                   # Options are 'inch', 'cm', or 'mm'
        group_rainrate     = mm_per_hour          # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'
        group_speed        = km_per_hour          # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'
        group_speed2       = km_per_hour2         # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'
        group_temperature  = degree_C             # Options are 'degree_F' or 'degree_C'
        group_uv           = uv_index
        group_volt         = volt

        # The following unit groups are used internally and should not be changed:
        group_count        = count
        group_interval     = minute
        group_time         = unix_epoch

In my database, data are stored in METRIC units too

[StdConvert]
    target_unit = METRIC    # Options are 'US', 'METRICWX', or 'METRIC'

The pages from my skin (a cutomized skin from fuzzy-archer bootstrap skin) are correctly displayed.

There is a problem with the monthly reports (NOAA reports) : the current month is displayed in the unit specified in the skin.conf, but all the previous months are in US units.

Current month : http://stations.meteor-oi.re/ESH555/NOAA/NOAA-2019-06.txt

                  TEMPERATURE (C), RAIN (mm), WIND SPEED (km/h)

                                         HEAT   COOL         AVG
      MEAN                               DEG    DEG          WIND                   DOM
DAY   TEMP   HIGH   TIME    LOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH   TIME    DIR
---------------------------------------------------------------------------------------
 01   20,3   22,9  10:41   17,6  03:30    0,0    3,5    0,0    1,5   22,5  05:31    354
 02   19,7   21,8  10:30   18,3  21:15    0,0    2,5    3,6    9,3   43,5  22:15    102
 03   18,1   19,2  08:02   16,2  20:51    0,4    0,0    3,0   19,7   51,5  21:00    103

Previous month : http://stations.meteor-oi.re/ESH555/NOAA/NOAA-2019-05.txt

                   TEMPERATURE (F), RAIN (in), WIND SPEED (mph)

                                         HEAT   COOL         AVG
      MEAN                               DEG    DEG          WIND                   DOM
DAY   TEMP   HIGH   TIME    LOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH   TIME    DIR
---------------------------------------------------------------------------------------
 01   67,4   70,6  16:10   65,2  06:18    0,0    4,4   0,00    5,4   23,0  11:23    108
 02   70,1   75,8  13:49   65,7  05:16    0,0    9,1   0,00    2,2   19,0  11:30     51
 03   71,2   77,0  12:34   67,7  23:58    0,0   11,2   0,00    1,6   12,0  13:20      7

In the database, in case of there was a mixup somehow in the stored data, I verified the data for the current (june) month and the previous (may) month :

MariaDB [weewx_ESH555]> SELECT usUnits, FROM_UNIXTIME(dateTime), outTemp FROM archive WHERE dateTime >= 1556654400 ORDER BY dateTime DESC LIMIT 5;
+---------+-------------------------+------------------+
| usUnits | FROM_UNIXTIME(dateTime) | outTemp          |
+---------+-------------------------+------------------+
|      16 | 2019-06-20 20:00:00     | 18.1666666666667 |
|      16 | 2019-06-20 19:55:00     | 18.0555555555556 |
|      16 | 2019-06-20 19:50:00     | 17.8333333333333 |
|      16 | 2019-06-20 19:45:00     | 17.7222222222222 |
|      16 | 2019-06-20 19:40:00     | 17.7777777777778 |
+---------+-------------------------+------------------+
5 rows in set (0.00 sec)

MariaDB [weewx_ESH555]> SELECT usUnits, FROM_UNIXTIME(dateTime), outTemp FROM archive WHERE dateTime >= 1556654400 ORDER BY dateTime ASC LIMIT 5;
+---------+-------------------------+------------------+
| usUnits | FROM_UNIXTIME(dateTime) | outTemp          |
+---------+-------------------------+------------------+
|      16 | 2019-05-01 00:00:00     | 19.1111111111111 |
|      16 | 2019-05-01 00:05:00     | 19.1111111111111 |
|      16 | 2019-05-01 00:10:00     | 19.1111111111111 |
|      16 | 2019-05-01 00:15:00     | 19.0555555555556 |
|      16 | 2019-05-01 00:20:00     |               19 |
+---------+-------------------------+------------------+
5 rows in set (0.00 sec)

It seems OK, the data for the two months are stored in metric units (usUnits=16).

I deleted the generated files, but nothing changed

There is no specific messages in the logs.

I really don't understand what is the problem.

How two files, generated with the same skin configuration, the same weewx, the same data format in the database can have different display format at the end ?

I must be missing something ...

If someone can help me :)

[gjr80]

Hi,

Some questions below.

Gary

On Friday, 21 June 2019 03:43:49 UTC+10, Mickael Hoareau wrote:

The pages from my skin (a cutomized skin from fuzzy-archer bootstrap skin) are correctly displayed.

There is a problem with the monthly reports (NOAA reports) : the current month is displayed in the unit specified in the skin.conf, but all the previous months are in US units.

What skin is generating the NOAA format reports? Standard? customised fuzzy-archer ?

I deleted the generated files, but nothing changed

Did you delete all NOAA format files, not just a few or the last one but every single one on the WeeWX machine in the directory where the skin generates them (not somewhere where they may be copied).

[Mickael Hoareau]

Hi,

Le vendredi 21 juin 2019 03:41:09 UTC+4, gjr80 a écrit :

What skin is generating the NOAA format reports? Standard? customised fuzzy-archer ?

It is the Bootstrap (fuzzy-archer) skin.
In skins/Bootstrap/skin.conf, I have this :

[CheetahGenerator]

    # This section is used by the generator CheetahGenerator, and specifies
    # which files are to be generated from which template.

    encoding = html_entities    # Possible encodings are 'html_entities', 'utf8', or 'strict_ascii'

    search_list_extensions = user.historygenerator.MyXSearch, user.timeperiods.TimePeriod

    [[SummaryByMonth]]
        #
        # Reports that summarize "by month"
        #
        [[[NOAA_month]]]
            encoding = strict_ascii
            template = NOAA/NOAA-YYYY-MM.txt.tmpl

    [[SummaryByYear]]
        #
        # Reports that summarize "by year"
        #
        [[[NOAA_year]]]
            encoding = strict_ascii
            template = NOAA/NOAA-YYYY.txt.tmpl

 

Did you delete all NOAA format files, not just a few or the last one but every single one on the WeeWX machine in the directory where the skin generates them (not somewhere where they may be copied).

 

Yes, and I even tried to delete ALL the files from public_html repository of weewx (on the raspberry pi running weewx), but the re-generated files from the previous months came back in US units

At first, I thought it was a bad data import : I recently had a SD card crash on this Rasperry Pi, so I made another SD card and I imported a backup of the database, but I don't see how this could be a problem since the data are in correct unit format.

I have some missing data to import for march and april, but I prefer to resolve the current problem before.

I love weewx (I'm using it for 4 years now and deployed it on more than 30 stations), but this time, I'm a little lost ^^

[gjr80]

OK. Let's have a look at those SLEs, perhaps one of those is having some unintended consequences on the NOAA templates. Looking at fuzzy-archer the NOAA templates appear on first glance to be identical to the WeeWX Standard skin NOAA templates. I see a historygenerator.py that contains a MyXSearch SLE but I don't see timeperiods.py nor a TimePeriod SLE. A few questions:

1. Did you modify the NOAA templates in any way? If so please post your modified templates.

2. Did you modify the historygenerator.MyXSearch SLE in any way? Again if so please post the modified copy.

3. What does the timeperiods.TimePeriod SLE do?

4. Are there any other templates being run in this skin or are the NOAA monthly and yearly templates the only ones? If there are others please post the whole skin.conf.

5. If no other templates what happens when you comment out the line starting search_list_extensions = and then delete all the NOAA format reports and wait for regeneration ?

Gary

[Mickael Hoareau]

Le vendredi 21 juin 2019 08:08:28 UTC+4, gjr80 a écrit :

OK. Let's have a look at those SLEs, perhaps one of those is having some unintended consequences on the NOAA templates. Looking at fuzzy-archer the NOAA templates appear on first glance to be identical to the WeeWX Standard skin NOAA templates. I see a historygenerator.py that contains a MyXSearch SLE but I don't see timeperiods.py nor a TimePeriod SLE. A few questions:

1. Did you modify the NOAA templates in any way? If so please post your modified templates.

I did not modify the monthly reports for NOAA templates.

But I did modify the yearly reports to add more infos on rain accumulation.

The modified file is joined to this message

 

2. Did you modify the historygenerator.MyXSearch SLE in any way? Again if so please post the modified copy.

No, I didn't modify it, but I use an older version you can here : https://github.com/mhoareau/fuzzy-archer/tree/meteor-oi/bin/user (I'v forked the original repository)

3. What does the timeperiods.TimePeriod SLE do?

It creates customized timespan used on this page for rain accumulation : http://stations.meteor-oi.re/ESH555/stats.html

You can find the code here : https://github.com/mhoareau/fuzzy-archer/blob/meteor-oi/bin/user/timeperiods.py (sorry for the wrong author info at the start of the file, bad copy-paste habit, I will modify that)

 

4. Are there any other templates being run in this skin or are the NOAA monthly and yearly templates the only ones? If there are others please post the whole skin.conf.

Yes, there is other templates. I attached the whole skin.conf file in this post.

5. If no other templates what happens when you comment out the line starting search_list_extensions = and then delete all the NOAA format reports and wait for regeneration ?

There is other templates, so I cannot try it "as is".
I will try to desactivate all other templates and only generate NOAA reports.

I did try to delete another time all generated files, just in case I didn't do it right the last time, I regenerated them, but no luck.
New files appeared in public_html/, other templates genetated files in metrics units (including NOAA report for current month and current year), but all the NOAA reports from previous months and previous years are in US units. :-/

Thanks for trying to help me :)

[Mickael Hoareau]

I did attach the monthly reports (not modified) by error.

Here the yearly customized report.

[gjr80]

Thanks. I have not forgotten this. I have looked through what you have provided and I cannot see anything that could be causing the problem. So as the next step I copied your fork and installed it on a VM here to see if I could reproduce problem. Unfortunately it all works fine for me, metric throughout all NOAA format reports. I even converted the test database to METRIC and same result. Can you confirm that the code/templates/config you are running on the problem machine is the same as your that committed in your fork?

One last thing, could you post a wee_debug report. Before posting the wee_debug output check it for any sensitive info such as passwords, user names, api keys etc, it does a good job at hiding these but it is not perfect.

Gary

[Mickael Hoareau]

Hello,

There some really minor adjustements made on the skin that are not commited in the fork.
But these minor adustements are running on other weewx instances (like http://stations.meteor-oi.re/BER590/history.html) and working just fine with previous NOAA reporting in the good units, so they can't be the problem.

You will find a wee_debug report attached to this message.

Mickael

[gjr80]

Aha, I think i see what may be going on. You have the Standard skin and the Bootstrap skin both generating to public_html. Both the Standard skin and the Bootstrap skin have NOAA sub-directories with identical (or near identical) templates. So you have two skins generating the same reports into the same directory. If the Standard skin was generating US customary NOAA reports, the Bootstrap skin would then come along and produce its Metric NOAA reports. But since the previous months/years reports already exist courtesy of the Standard skin, the Bootstrap skin only produces the current month/year NOAA report. Hence you see US customary reports for all but the current reports which are Metric.

When you delete the reports to force regeneration the Standard skin does the heavy lifting and re-generates all the NOAA reports on the next report cycle. The Bootstrap skin sees all the old reports already exist so it just produces just the current month/year reports. This behaviour could be verified by careful analysis of the number of files generated by each skin in the first report cycle after deleting all the NOAA files, the Standard skin will produce many and the Bootstrap skin will not.

To check if this is indeed the issue:
1. comment out the entire [[StandardReport]] stanza in weewx.conf
2. delete all NOAA format reports
3. restart WeeWX
4. wait for the first report cycle and then check the NOAA format reports to see if they are all Metric

Gary

[Mickael Hoareau]

Wow ... I'm embarassed O_o

I was sure the Standard skin was disabled.

I disabled it and it indeed worked.

As always, it's was something really simple  but it takes someone else to point it out ! Haha.

Thank you a lot for the time you took to help me. :)

Mickael

[gjr80]

You don't know how glad I was that this was the problem - I was out of ideas after this one!

Gary