Monthly NOAA data stopped populating

[Steve Barge]

My  Monthly Climatological Summary data suddenly stopped populating (See attached). All the data is getting populated in the database, and is all there when I look at the Week, Month and Year data.

I upgraded to Weewx 3.8.0 a while back (don't recall the exact date) but could coincide with the date that this stopped working.

I've run the wee-database command to drop and rebuild the daily but this did not resolve the issue.

Any help would be greatly appreciated.

[gjr80]

Hi,

Could be any one of a number of causes. Best to look at the log and see what is going on. Please edit weewx.conf, set debug=1 then stop/start weeWX.let weeWX run for a couple of archive periods and the post the log from weeWX startup.

Gary

[Steve Barge]

Attached is the log file.  Looks like my pressure and barometer readings are outside the limit values, but I wouldn't think this should affect the monthly reports.

[gjr80]

On Thursday, 8 February 2018 00:44:35 UTC+10, Steve Barge wrote:

 Looks like my pressure and barometer readings are outside the limit values, but I wouldn't think this should affect the monthly reports.

Any fields with values outside the QC limits are set to None, there could be an affect on the NOAA format reports but only in as much as the barometer values may be missing; but you are correct in as much as it will not prevent the NOAA format reportsbeing produced and populated with (available) data.

Feb  7 08:25:32 raspberrypi weewx[26967]: ftpupload: Uploaded file /weewx/NOAA/NOAA-2018-02.txt
Feb  7 08:25:33 raspberrypi weewx[26967]: ftpupload: Uploaded file /weewx/NOAA/NOAA-2018.txt

These lines indicate the 2018 and February 2018 NOAA format reports were produced and uploaded. The report you cite as missing data is the January 2018 report, what does the February 2018 report look like? Is it complete? The way the NOAA format reports are generated is the current month and year reports are generated every report cycle, past months (and years)  reports are never generated (again) unless the report does not exist (ie you deleted it) and there are no reports for the subsequent months and years. What this means is that if there was some issue with your install that meant the January 2018 report is incomplete, given that we are now in February 2018 the January 2018 report will never be updated again (even if all the data is there) unless you delete the January and February 2018 reports. If you are confident the data is in you system why don't you try deleting the January and February 2018 reports to force weeWX to regenerate the Janauary report and see if that fixes it (February report will be automatically generated as it is the current month).

Gary

[Steve Barge]

Hi Gary -

Thanks for the reply.  I write all of my files to a temp file system, so if I reboot, they are all deleted and rebuilt by weewx.  I've rebooted several times and each time the January report looks the same (missing data from the 16th on and there is nothing in the February report).

I did some queries against the archive table and all of the temp data is there - the barometer and pressure data are null, but again I wouldn't think that should affect the monthly reports.

I also checked to see when I upgraded from weewx 3.7 to 3.8 and I did it on December 29, so that isn't the issue.  The only thing that I can see that changed on January 16th is that my barometer data started to be outside the bounds.  My weather station is not local, so I'm not sure what's going on there.

[gjr80]

Aha, the problem is your baromater data (or lack thereof), but the effect is not in the way I was expecting. The answer is in the skins/Standard/NOAA/NOAA-YYY-MM.txt.tmpl template file, the meat of the template is:

 HEAT COOL AVG
 MEAN DEG DEG WIND DOM
DAY TEMP HIGH TIME LOW TIME DAYS DAYS RAIN SPEED HIGH TIME DIR
---------------------------------------------------------------------------------------
#for $day in $month.days
#if $day.barometer.count.raw
$day.dateTime.format($D) $day.outTemp.avg.nolabel($Temp,$NONE) $day.outTemp.max.nolabel($Temp,$NONE) $day.outTemp.maxtime.format($Time) $day.outTemp.min.nolabel($Temp,$NONE) $day.outTemp.mintime.format($Time) $day.heatdeg.sum.nolabel($Temp,$NONE) $day.cooldeg.sum.nolabel($Temp,$NONE) $day.rain.sum.nolabel($Rain,$NONE) $day.wind.avg.nolabel($Wind,$NONE) $day.wind.max.nolabel($Wind,$NONE) $day.wind.maxtime.format($Time) $day.wind.vecdir.nolabel($Dir,$NONE)
#else
$day.dateTime.format($D)
#end if
#end for

Note the if statement, the template checks for any barometer readings for the day, if there are some it displays all data for the day, if there are none it displays nothing for the day. This was done to check if there was any data to display for the day, I expect barometer was an arbirary choice, you could just change it to outTemp if you want, but keep in mind you have an orphan that will be overwritten when you upgrade.  The sma emechanism is also at play in the NOAA-YYYY.txt.tmpl template.

I'll have a closer look at that template, I would have thought there may have been a more robust way to achieve the same ends.

Gary

[Steve Barge]

Fantastic - thanks for finding that!  I'll update that template for now.

[Thomas Keffer]

I'll have a closer look at that template, I would have thought there may have been a more robust way to achieve the same ends.

​Good sleuthing!

Don't know why I chose 'barometer', because there is indeed a better way: do the check on dateTime, which is required for every record. So, it becomes:

#if $day.dateTime.count.raw

...

#end if

Thanks for finding this, Gary

-tk

​

[Andrew Milner]

FYI there are also three similar occurrences in the yearly NOAA template ....

[pravee...@gmail.com]

The way the NOAA format reports are generated is the current month and year reports are generated every report cycle, past months (and years)  reports are never generated (again) unless the report does not exist (ie you deleted it) and there are no reports for the subsequent months and years"

Raking up an old thread. How is this done? I m looking to create some reports and have same requirement that i want only current month report to be generated. Others to be generated only if they don't exist already.

[gjr80]

The behaviour you describe is inherent in reports named [[SummaryByMonth]] and [[SummaryByYear]]  ([[SummaryByDay]] can be used as well). Any templates included under reports using these names (in any skin config file) will be generated in the same manner. 

Gary

[Praveen Chandrasekaran]

The ones under summarybyyear or summaryby month requires both year and month to be supplied I think. I want to supply only month and build historical records for the month across years . I want this to be done only at end of each month for current month or if files don't exist.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/pN6KA5_N2S0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/abe67f2a-ed15-4f32-b501-63fd440853a4n%40googlegroups.com.

[gjr80]

I don't know what you mean by "requires both year and month to be supplied". [[SummaryByMonth]] and [[SummaryByYear]] iterate over all available archive records in spans of one month and one year respectively and the respective template is generated against each span (which is why you end up with numerous generated files from each template). No 'year' or 'month' is supplied.

If you only want a report generated at a specific time perhaps you should look at the Scheduling report generation section of the Customization Guide, in particular the report_timing option which allows for 'cron like' control over report generation. There is no reason I can think of why report_timing could not be used with a [[SummaryByXxxx]] report to give the multi-span reports like the NOAA summary reports but only generated/updated a specific times. Though I have not seen this approach used before.

Gary

[Praveen Chandrasekaran]

Thanks. That clarified things for me quite a bit. I need my report to iterate over all years too. So my report will be lik NOAA-MM.txt.tmpl. No year in file name.

I guess i can't put this under existing summaryByyear but need to create a new ones?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1fcb2f71-63bf-429a-aae5-d10f14d7592an%40googlegroups.com.

[an oldman]

don't know if this will help

using weewx 4.8.0 python 3
i have in Seasons skin.conf
[CheetahGenerator]
    [[SummaryByMonth]]
        # Reports that summarize "by month"
        [[[NOAA_month]]]
            encoding = normalized_ascii
            template = NOAA/NOAA-%Y-%m.txt.tmpl

        # html to show noaa year month side by side
        [[[HtmlNOAA_month]]]
            encoding = html_entities
            template = NOAA/NOAA-EXTRA-YYYY-MM.html.tmpl

        # 5n1 noaa year month text file
        [[[ExtraNOAA_month]]]
            encoding = normalized_ascii
            template = NOAA/NOAA-EXTRA-%Y-%m.txt.tmpl

    [[SummaryByYear]]
        # Reports that summarize "by year"
        [[[NOAA_year]]]
            encoding = normalized_ascii
            template = NOAA/NOAA-%Y.txt.tmpl

        # html to show noaa year side by side
        [[[HtmlNOAA_year]]]
            encoding = html_entities
            template = NOAA/NOAA-EXTRA-YYYY.html.tmpl

        # 5n1 noaa year file
        [[[ExtraNOAA_year]]]
            encoding = normalized_ascii
            template = NOAA/NOAA-EXTRA-%Y.txt.tmpl

this makes files like this
normal NOAA year file NOAA-2014.txt

normal NOAA month file NOAA-2014-12.txt

html year file NOAA-EXTRA-2014.html

html month file NOAA-EXTRA-2014-12.txt
extra year file NOAA-EXTRA-2014.txt
extra month file NOAA-EXTRA-2014-12.txt