Export and Append to Text File Each Report Cycle

[Rory Gillies]

Hi, at first I thought this would be quite easy but I'm struggling a bit. The background is I need a text file to import into WXSim using the Cumulus %Y%mlog.txt format. I created a simple skin that generates the %Y%mlog.txt each reporting cycle (5 minutes in my case), but is there a way to append the next cycle to the existing file as a new line rather than overwrite it?

My skin.conf looks like this:

[CheetahGenerator]
    encoding = utf8

    [[ToDate]]
        [[[text-data]]]
            template = %Y%mlog.txt.tmpl

[CopyGenerator]
    copy_always = *.txt
   
[Generators]
    generator_list = weewx.cheetahgenerator.CheetahGenerator, weewx.reportengine.CopyGenerator

And the template:

#encoding UTF-8
#errorCatcher Echo
## +------------------------------------------------------+
## |           Cumulus Monthly Output Template            |
## +------------------------------------------------------+
## | Produces a YYYYmmlog.txt output for import to WXSim  |
## +------------------------------------------------------+


$current.dateTime,$current.outTemp.format(add_label=False),$current.outHumidity.format(add_label=False),$current.dewpoint.format(add_label=False),$current.windSpeed.format(add_label=False),$current.windGust.format(add_label=False),$current.windDir.format(add_label=False),,,$current.barometer.format(add_label=False),$day.rain.sum.format(add_label=False),,,,,,,$current.radiation.format(add_label=False),

Which produces a text file called 202412log.txt:

18/12/24 19:40:00,6.3,82,3.4,10,37,266,,,982.8,15.4,,,,,,,0,

WXSim can read this, however it only contains a single observation period. I would like to append the data to the file until the month is complete, then generate a new file for the next month. Any ideas? It will likely need some Python which is not my strongpoint, although I can get around it a bit...

I'm running WeeWX 5.1 if it makes it easier. TIA

[p q]

Have you thought about doing this with a database query?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/92a4cb34-0516-44f3-8406-2a08b25a4d77n%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[Rory Gillies]

Hi Peter,

Yes that might be easier but it’s a bit beyond my skill set. As long as it can be kept in WeeWX I’m open to suggestions

[gjr80]

So you want to build a file with potentially 8928 rows (31 days per month 288 records per day)? 

If so this should be able to be done solely in a Cheetah template using just a few lines of code by iterating over the archive records in the month. Something like (untested):

#for $record in $month.records

$record.dateTime,$record.outTemp.format(add_label=False),.....

#end for

Like I said untested and I haven't used record iterators for a long time but at worst a little finessing should get you there. One thing to note, report generation time will gradually increase as the month progresses peaking on the last record of the last day of the month.

Gary

[Rory Gillies]

Thanks Gary, that did the trick! Doesn't take long on my RPi 5 for 19 days in December:

2024-12-19T08:55:33.656223+00:00 weewx5 weewxd[1317867]: INFO weewx.cheetahgenerator: Generated 1 files for report cumulus in 3.50 seconds

Cheers,

Rory

[Tom Keffer]

Your variable $rainacc is just a raw number. It's not the ValueHelper that WeeWX normally traffics in, so suffixes such as .round() and .format() won't work.

However, Python syntax will work. Try this: 

$("%.2f" % $rainacc)

On Thu, Dec 19, 2024 at 5:21 AM 'Rory Gillies' via weewx-user <weewx...@googlegroups.com> wrote:

Thanks for the help, I've got this running pretty well now, however I have an issue with formatting a variable. The WXSim developer has told me the expected rain value is an accumulator for the year. I created a $rainacc so the template code looks like this:

#set $rainacc = $year.rain.sum.raw - $month.rain.sum.raw
#for $record in $month.records
#set $rainacc = $rainacc + $record.rain.raw
$record.dateTime.format("%d/%m/%Y %H:%M"),$record.outTemp.format(add_label=False),$record.outHumidity.format(add_label=False),$record.dewpoint.format(add_label=False),$record.windSpeed.format(add_label=False),$record.windGust.format(add_label=False),$record.windDir.format(add_label=False),0,0,$record.barometer.format(add_label=False),$rainacc,0,0,0,0,0,0,$record.radiation.format(add_label=False),0
#end for

This works, however the output for rain is to 13 decimals: 1083.8000001314047. No matter what I do I can't seem to format it to 2 decimal places. I have tried $rainacc.round(2)and $rainacc.format(%.2f)

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/9b766655-b90d-4a85-8d48-f1d4e9fb7e96n%40googlegroups.com.

[Rory Gillies]

Thanks Tom, I figured it out but posted the message by mistake instead of deleting it. I used #set $rainacc = round($rainacc + $record.rain.raw,2)

[Rory Gillies]

I thought I had this working, but due to an intermittently faulty anemometer some of the output has N/A instead of 0:

07/02/25,08:25,7.1,89,5.5,18,25,225,0.0,0.0,1040.0,119.8,18.5,58,25,2.9,7.1,0.0,9,0.0,0.00,0.0,0.0,0.0,225,0.0,0.0,0.0,7.1
07/02/25,08:30,7.1,89,5.5,18,25,225,0.0,0.0,1039.9,119.8,18.5,58,25,2.9,7.1,0.0,9,0.0,0.00,0.0,0.0,0.0,225,0.0,0.0,0.0,7.1
07/02/25,08:35,7.2,89,5.5,18,25,225,0.0,0.0,1039.9,119.8,18.5,58,25,3.0,7.2,0.0,11,0.0,0.00,0.0,0.0,0.0,225,0.0,0.0,0.0,7.2
07/02/25,08:40,7.1,89,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.9,119.8,18.5,58,   N/A,   N/A,7.1,0.0,12,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,08:45,7.2,90,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.5,58,   N/A,   N/A,7.2,0.0,12,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.2
07/02/25,08:50,7.1,90,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.6,58,   N/A,   N/A,7.1,0.0,13,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,08:55,7.1,90,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.6,58,   N/A,   N/A,7.1,0.0,15,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,09:00,7.1,89,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.6,58,   N/A,   N/A,7.1,0.0,16,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,09:05,7.1,89,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.6,58,   N/A,   N/A,7.1,0.0,18,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,09:10,7.1,89,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.6,58,   N/A,   N/A,7.1,0.0,18,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,09:15,7.1,89,5.5,   N/A,   N/A,   N/A,0.0,0.0,1039.8,119.8,18.7,58,   N/A,   N/A,7.1,0.0,21,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1
07/02/25,09:20,7.1,89,5.4,   N/A,   N/A,   N/A,0.0,0.0,1039.7,119.8,18.7,58,   N/A,   N/A,7.1,0.0,20,0.0,0.00,0.0,0.0,0.0,   N/A,0.0,0.0,0.0,7.1

This causes problems with WXSim-Lite which can't read the file properly and crashes. I thought this would be an easy fix by using the 'is not None' to test for data in $windDir. My template looks like this (the new bit in bold):

#encoding UTF-8
#errorCatcher Echo
## +----------------------------------------------------+

## |          Cumulus Monthly Output Template           |

## +----------------------------------------------------+
## | Produces a YYmmlog.txt output for import to WXSim  |
## +----------------------------------------------------+

#set $rainacc = $year.rain.sum.raw - $month.rain.sum.raw

#set $dayrainacc = 0
#set $daycount = 1
#set $wndspd = 0
#set $wnddir = 0
#set $wndgst = 0
#set $wndchl = 0
#for $record in $month.records
#set $rainacc = round($rainacc + $record.rain.raw,1)
#set $dayrainacc = round($dayrainacc + $record.rain.raw,1)
#set $daycount = $daycount + 1
#if $daycount == 289
#set $daycount = 1
#set $dayrainacc = round($record.rain.raw,1)
#end if
#if $record.windDir.raw is not None
#set $wndspd = round($record.windSpeed.raw,0)
#set $wnddir = round($record.windDir.raw,0)
#set $wndgst = round($record.$windGust.raw,0)
#set $wndchl = round($record.$windchill.raw,0)
#end if
$record.dateTime.format("%d/%m/%y"),$record.dateTime.format("%H:%M"),$record.outTemp.format(add_label=False),$record.outHumidity.format(add_label=False),$record.dewpoint.format(add_label=False),$wndspd,$wndgst,$wnddir,$record.rainRate.format(add_label=False),$dayrainacc,$record.barometer.format(add_label=False),$rainacc,$record.inTemp.format(add_label=False),$record.inHumidity.format(add_label=False),$wndgst,$wndchl,$record.heatindex.format(add_label=False),$record.UV.format(add_label=False),$record.radiation.format(add_label=False),$record.ET.format(add_label=False),0.00,0.0,0.0,0.0,$wndgst,0.0,$dayrainacc,0.0,$record.humidex.format(add_label=False)
#end for

When the report runs I get the error: 'ValueHelper' object is not callable. Any ideas? I'm not sure where the ValueHelper object is. I tried #if $record.windDir.raw != 'N/A' but got the same error.

Any ideas how I can test for N/A in the records?

[Tom Keffer]

Why not use something like $record.windSpeed.format(format_string="%.0f", None_string="0")

This will round to zero decimal digits and emit "0" instead of "N/A".

See format() in the Customization Guide.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/b887fd8e-20a7-40b6-a4fa-59ecd42ddd9dn%40googlegroups.com.

[Rory Gillies]

Thanks Tom, the None_string=“0” did the trick

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/kLC0osy-AsQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDCzzC1hvwFxU7vaxmxcwZk8O611nekT7qouHq0HOJo9g%40mail.gmail.com.