Parsing some NWS Area Forecast Matrices fails

[Michael Bruski]

Hi all,

First of all, I am running WeeWX 4.1.1 with forecast extension 3.4.0b10.

I had recently managed to use a Area Forecast Matrix for one location and that is working fine.   Recently I tried to setup a second location to use the AFM (different LID and FOID) and it fails to get a forecast even though I see it in the web browser using the URL that forecast.py uses.  After careful application of some trace print statements, I see the forecast is succesfully download from the FOID and that the forecast for the specified LID is also located and extracted but nothing gets parsed.  Below is a copy of my trace output from the log:

Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: forecast downloaded successfully from 'http://forecast.weather.gov/product.php?site=NWS&product=AFM&format=txt&issuedby=LWX'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'MDZ005-072100-'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Carroll-'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Including the cities of Eldersburg and Westminster'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: '137 PM EDT Wed Oct 7 2020'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Date             Wed 10/07/20            Thu 10/08/20            Fri 10/09/20'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'EDT 3hrly     05 08 11 14 17 20 23 02 05 08 11 14 17 20 23 02 05 08 11 14 17 20'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'UTC 3hrly     09 12 15 18 21 00 03 06 09 12 15 18 21 00 03 06 09 12 15 18 21 00'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Max/Min                75 77 80    47 48 52    65 67 70    42 44 48    68 70 73'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Temp                   76 76 68 61 56 51 50 60 65 64 55 50 47 45 45 61 68 67 59'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Dewpt                  52 52 53 50 48 46 45 45 43 42 43 42 41 40 41 45 45 46 47'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'RH                     43 43 59 67 74 83 83 58 45 45 64 74 79 82 86 56 43 47 64'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Wind dir                W  W NW NW NW NW NW NW NW NW NW NW  W  W  W SW SW SW  S'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Wind spd               19 17 10  6  5  5  6 11 12 10  5  3  1  2  1  4  6  8  4'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Wind gust              33 31'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Clouds                 CL CL SC FW FW CL CL FW CL CL CL CL FW FW FW FW FW FW SC'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'PoP 12hr                     20          20           0           0           0'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'QPF 12hr                      0           0           0           0           0'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Rain shwrs                 S  S'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Date          Sat 10/10/20  Sun 10/11/20  Mon 10/12/20  Tue 10/13/20'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'EDT 6hrly     02 08 14 20   02 08 14 20   02 08 14 20   02 08 14 20'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'UTC 6hrly     06 12 18 00   06 12 18 00   06 12 18 00   06 12 18 00'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Min/Max          54    75      56    69      53    66      54    68'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Temp          55 56 73 66   61 58 66 60   56 55 63 59   56 56 65 61'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Dewpt         50 54 59 60   59 56 55 54   54 53 56 56   56 56 58 57'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'PWind dir         S    SW      SW    SE       E     E       E    SE'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Wind char        LT    GN      LT    LT      GN    GN      LT    LT'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Avg clouds    SC B1 B1 B1   B1 B1 B1 B1   B2 B2 B2 B2   B2 B2 B2 B1'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'PoP 12hr          5    20      30    30      40    40      30    30'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: 'Rain shwrs              S    C  C  C  C    C  C  C  C    C  C  C  C'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: ''
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: '$$'
Oct  7 14:36:45 server weewx[3014] INFO user.forecast: NWSThread: NWS: successfully extracted forecast for 'MDZ005'

Examining NWSParseForecast, nothing is parsed until the Date/Time Group (a line with 7 tokens) is located.  In the above AFM another line preceeds the DTG which also happens to have 7 tokens.  I think this line is being passed to date2ts() which in turn fails and causes the parser to stop looking for further input.  I didn't see a way to raise the issue on GitHub so I thought I would bring this up here.

Mike/AJ9X

[John Kline]

Perhaps Area Forecast Matrix responses can’t be treated just like Point Forecast Matrix responses.  AFMs are only supported to the extent that they provide an identically formatted response as the PFM response.

What do you know about AFMs and the differences between them and PFMs?

On Oct 7, 2020, at 12:18 PM, Michael Bruski <michael...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/05fa293c-659c-4619-9b06-9ed97a30efa4o%40googlegroups.com.

[Michael Bruski]

Hi John,

I've looked at several forecast matrices from my local and surrounding NWS forecast offices and haven't seen anything format wise that would lead me to think they would cause a problem for the forecast extension.  I have selected several different AFMs and see they parse fine.  But I ran into this issue with my local AFM and spent several hours trying to track down the cause.   The issue is not with the matrix itself but with text preceeding the DTG.   To test further, I put a cheap hack into the parse loop so that date2ts() isn't called if the first word in 'line' is 'Including'.  This actually works (in this specific instance) and a valid forecast is produced.  What other lines of text might NWS put in the header preceeding the DTG, I just don't know.    The URLs mentioned in forecast.py are dead links so I couldn't find any specifics on decoding the reports.

I'm not terribly comfortable programming in python so I'm not sure the best approach to work this issue but sending bad data to time.mktime() via date2ts() is probably bad unless there is a way to trap the error.  Not meant to criticize...  just my thoughts on how to make it better.

Mike/AJ9X

[John Kline]

Absent a spec on area forecast matrices, and absent any problem reports on point forecast matrices, there isn’t much to go on here.

On Oct 7, 2020, at 2:35 PM, Michael Bruski <michael...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3936dc80-0d29-414f-a5b9-5f286bd659b1n%40googlegroups.com.

[Ernest Jillson]

Here is the spec on AFMs from the NWS:

https://www.weather.gov/media/notification/dir/AFM_Format.pdf

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/85F7E477-9A8C-4AF3-8713-4A121E5AE3A3%40johnkline.com.

[Michael Bruski]

I queried the NWS Baltimore/Washington D.D. forecast office about the differences between AFM and PFM and received this reply:

     Hello,

     Thanks for writing.  The AFMs and PFMs are set to a national standard. So they should be the same across all the forecast offices.

     Our Directive that talks about these products, as well as links to format decoding, are in this NWS Directive between pages 10-12.

     Hope that helps.

     Chris Strong

     NWS Baltimore/Washington

The reference Chris sites contains links to AFM/PFM format PDFs and the element specifications PDF for both reports.  The only difference between the two that I could detect was in lines 2 and 3 of the report (Description and Location in the PFM;  zone and included cities in the AFM).

[John Kline]

This is all very helpful.

Would you please send your NWS configuration for the problem AFM report as well as sample output from the URL you specified that the extension chokes on?

Have you determined if the output is valid as specified by the AFM format doc?

I’ll add proper support for AFM in the coming week or so.

Cheers,

John

On Oct 8, 2020, at 6:08 AM, Michael Bruski <michael...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/05c0fc4d-63d1-4e42-acc5-590605ae924en%40googlegroups.com.

[Michael Bruski]

I attached a text file containing one of the latest results from URL https://forecast.weather.gov/product.php?site=NWS&product=AFM&format=txt&issuedby=LWX.

These are the options I have configured in my weewx.conf file:

# Options for extension 'forecast'
[Forecast]
    data_binding = forecast_binding
    [[XTide]]
        location = "INSERT_LOCATION_HERE (e.g., Boston)"
    [[Zambretti]]
        hemisphere = NORTH
    [[NWS]]
        lid = "MDZ005"
        foid = "LWX"
        url = http://forecast.weather.gov/product.php?site=NWS&product=AFM&format=txt&issuedby=LWX
    [[WU]]
        api_key = INSERT_WU_API_KEY_HERE
    [[OWM]]
        api_key = INSERT_OWM_API_KEY_HERE
    [[UKMO]]
        api_key = INSERT_UKMO_API_KEY_HERE
        location = INSERT_UK_LOCATION_HERE
    [[Aeris]]
        client_id = INSERT_AERIS_CLIENT_ID_HERE
        client_secret = INSERT_AERIS_CLIENT_SECRET_HERE
    [[WWO]]
        api_key = INSERT_WWO_API_KEY_HERE
    [[DS]]
        api_key = INSERT_DS_API_KEY_HERE

I appreciate any effort you put into resolving this.  It will be nice to see the extension handle these instead of silently failing.

Also, a thank you to the other group member who supplied a link to the AFM_format.pdf document.  I mentioned to Chris at NWS Balt/Wash this morning that it is difficult to find any specifications for the AFM/PFM on the NWS web site.  Perhaps that will get addressed in the future as well.

Mike/AJ9X

[Michael Bruski]

I forgot to mention that the AFM format is correct and the extension will handle it (in this particular case for MDZ005 and for MDZ006) if I force the scanner/parser to ignore the 'Including' line.  I'm pretty sure that 'cheat' will fail though on MDZ013 and MDZ014 in that same dataset.  Perhaps need to check the line splits to 7 tokens and the first token is a numeric value or a numeric value less than 1300 before passing it to date2ts() and not worry about first token is 'Including'?

Mike/AJ9X

[John Kline]

Of course, these lines being one of the ways the spec differs between the area and point forecasts.

In the email you include from Chris Strong, he is saying AFM forecasts should not differ and across the country and PFM forecasts should not differ across the country.  He is not saying AFM and PFM forecasts are the same.

On Oct 8, 2020, at 11:04 AM, Michael Bruski <michael...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8e330107-4462-4d0c-ab87-ce12f9760f81n%40googlegroups.com.