Creation of Cumulus realtime.txt file

[PWS]

Has anyone been successful in extending weewx to create a realtime.txt file in Cumulus format? Perhaps based on the WURF service so that a new file could be created each LOOP?

[mwall]

On Thursday, April 24, 2014 4:25:17 PM UTC-4, PWS wrote:

Has anyone been successful in extending weewx to create a realtime.txt file in Cumulus format? Perhaps based on the WURF service so that a new file could be created each LOOP?

here is a service that will emit all of the LOOP values in a single comma-delimited line, appending to file, default to /var/tmp/realtime.txt:

import weewx
from weewx.wxengine import StdService

class CRTService(StdService):
    def __init__(self, engine, config_dict):
        d = config_dict.get('CRTService', {})
        self.filename = d.get('filename', '/var/tmp/realtime.txt')
        self.bind(weewx.NEW_LOOP_PACKET, self.writeData)

    def writeData(self, event):
        with open(self.filename, 'a') as f:
            f.write(','.join(["%s" % x for x in event.packet.values()]))
            f.write("\n")

put it in bin/user/crt.py, then add this to your weewx.conf:

[CRTService]
    filename = /path/to/realtime.txt

[Engines]
    [[WxEngine]]
        process_services = ..., user.crt.CRTService

[PWS]

Sorry to be a nuisance, but as  a Linux newbie, I'm a bit hazy about the correct location for crt.py. The path looks like a reference relative to $WEEWX_ROOT but in a default weewx installation from .deb, then $WEEWX_ROOT doesn't appear to have definitive value? Or is the implication that attempts to extend weewx are better made from a setup.py installation?

[mwall]

On Sunday, April 27, 2014 4:35:54 PM UTC-4, PWS wrote:

Sorry to be a nuisance, but as  a Linux newbie, I'm a bit hazy about the correct location for crt.py. The path looks like a reference relative to $WEEWX_ROOT but in a default weewx installation from .deb, then $WEEWX_ROOT doesn't appear to have definitive value? Or is the implication that attempts to extend weewx are better made from a setup.py installation?

put crt.py in the user directory.

in a default setup.py installation the user directory is /home/weewx/bin/user

in a .deb installation the user directory is /usr/share/weewx/user

m

[wysiwyg]

PWS :
In case you are not sure, you may try:

locate *user/*.py

and should be able to find the directory.
if it founds nothing, try:

sudo updatedb

if you are linux newbie, this command may help for many things: locate is use to find directories or files, updatedb is used to keep updated the database that locate uses.






Regarding the CRT things,I am interested too, so I tried it, and it looks it make weewx stop on my side...

I copy paste the code in /usr/share/weewx/user/crt.py
added this in weewx.conf:
[CRTService]
    filename = /var/www/weewx/realtime2.txt

and:
[Engines]
    # This section configures the internal weewx engines.
    # It is for advanced customization.

    [[WxEngine]]
   
        # The list of services the main weewx engine should run:
        prep_services = weewx.wxengine.StdTimeSynch
        process_services = weewx.wxengine.StdConvert, weewx.wxengine.StdCalibrate, weewx.wxengine.StdQC , user.crt.CRTService
        archive_services = weewx.wxengine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground,  weewx.restx.StdAWEKAS
        report_services = weewx.wxengine.StdPrint, weewx.wxengine.StdReport


here is syslog:

Apr 28 14:33:39 gunther weewx[5276]: wxengine: Initializing weewx version 2.6.3
Apr 28 14:33:39 gunther weewx[5276]: wxengine: Using Python 2.7.6 (default, Mar 22 2014, 22:59:56) #012[GCC 4.8.2]
Apr 28 14:33:39 gunther weewx[5276]: wxengine: pid file is /var/run/weewx.pid
Apr 28 14:33:39 gunther weewx[5278]: wxengine: Using configuration file /etc/weewx/weewx.conf
Apr 28 14:33:39 gunther weewx[5278]: wxengine: Loading station type WMR100 (weewx.drivers.wmr100)
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Loading service weewx.wxengine.StdTimeSynch
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Finished loading service weewx.wxengine.StdTimeSynch
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Loading service weewx.wxengine.StdConvert
Apr 28 14:33:40 gunther weewx[5278]: wxengine: StdConvert target unit is 0x1
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Finished loading service weewx.wxengine.StdConvert
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Loading service weewx.wxengine.StdCalibrate
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Finished loading service weewx.wxengine.StdCalibrate
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Loading service weewx.wxengine.StdQC
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Finished loading service weewx.wxengine.StdQC
Apr 28 14:33:40 gunther weewx[5278]: wxengine: Loading service user.crt.CRTService


Nothing after this, but if I do 'sudo service weewx status', it says it's not running.
did I miss something ?

[mwall]

On Monday, April 28, 2014 8:44:12 AM UTC-4, wysiwyg wrote:

Nothing after this, but if I do 'sudo service weewx status', it says it's not running.
did I miss something ?

when debugging changes to code or configuration, first run weewx directly, e.g.,

sudo weewxd /etc/weewx/weewx.conf

at the same time, monitor the log file in another shell, e.g.,

sudo tail -f /var/log/syslog

once you are sure that everything works you can go back to running weewx as a daemon, e.g.,

sudo /etc/init.d/weewx start

m

[wysiwyg]

Thanks for helpful information!


regarding realtime.txt, in the mean time, I found this (which is fine for my use),
https://groups.google.com/forum/#!searchin/weewx-user/android/weewx-user/GPehU2_HYsM/_xbAy-bZoZ8J

[PWS]

I haven't had a chance to look at this in detail yet so ICBW, but I'd question whether this is likely to be what I'd regard as a genuine realtime.txt generator - 'genuine' in the sense that realtime should mean what it says, ie reflect LOOP values for rapidly changing parameters like wind speed/direction and be capable of updates/uploads every few seconds and not just be another skin reflecting the latest archive data, eg every 5 minutes. In other words, similar to a WURF upload generator.

[Andrew Milner]

PWS - Is such precision really and truly required??  Most weather stations available to consumers probably do not provide an update on a particular sensor any more frequent that say every minute - even if they emit 'something' more often.  In particular, with wind, one needs a timeframe during which one can actually measure/count something.  Temperatures and pressures do not usually change faster than every 2-5 minutes.  Rainfall requires a high precision measuring instrument to make it wirthwhile reporting a rainfall amount .... and so on .....

FO weather consoles themselves only update their screens every 50 seconds for example - so updating or reading off those stations more frequently gets you nowhere.  WMR stations sould well emit sensor readings more freqhently, but only archive data every 60 seonds - so the only time you have a 'complete' picture is every 60 seconds.  Some meteoroligical 'readings' are composed from multiple values - eg temperature, wind .. with the best will in the world the result is computed from the values avaiable at a time ... which may or may not actually have been taken at EXACTLY the same time anyway.

In other words - getting readings and updating screens for the hell of it is quite likely doing loads of work for no benefit!!

[mwall]

On Monday, April 28, 2014 10:44:51 AM UTC-4, PWS wrote:

I haven't had a chance to look at this in detail yet so ICBW, but I'd question whether this is likely to be what I'd regard as a genuine realtime.txt generator - 'genuine' in the sense that realtime should mean what it says, ie reflect LOOP values for rapidly changing parameters like wind speed/direction and be capable of updates/uploads every few seconds and not just be another skin reflecting the latest archive data, eg every 5 minutes. In other words, similar to a WURF upload generator.

the reason for doing a service (rather than a weewx template) is so that realtime.txt is emitted on every LOOP. 

so the example above is, in fact, "genuine" where "genuine" = "emits data on every LOOP value".

the part you will need to tweak is the content of the realtime.txt file.  but maybe i'll do that too so that anyone using weewx will have a drop-in replacement for any cumulus template that requires a cumulus realtime.txt file...

m

[PWS]

Yes, absolutely. Users of weather station software fall into several groups and I do agree that for many users, updates eg every 5 minutes are sufficient. But others eg for wind-dependent sports (including eg small private airfields) do prefer real-time wind information. And there's no doubt also that even for pure hobbyists having real-time wind available on a website has a powerful appeal of its own that provides sufficient justification. 

[Andrew Milner]

Even the airfield works with a time lag, and one is usually landing with one eye on the sock anyway!  The last wind given (if the airfield was manned) would have been probably 30 seconds - 1 minute before touchdown.  having the precise conditions NOW is rarely required, and even more rarely available.  A pilot needs to know an average wind over previous 5 mins to 30 mins, and likely gust strength and gust frequency - but in fact will only obtain a gust strength and will not be aware in any way of how often the wind is gusting ..... so actually he cannot get the info he could ideally do with.

Of course all pilots are also able to land without knowing the exact wind .... as is very often the case at small airfields - the best they can get is visual from the sock. - which also gives them the frequency indication (and direction variability also) .... the station and the computer only give approximations at the best of times .....

More frequently does not necessarily equate to more accuracy or better information!

[PWS]

Ah, but the customer is always right! ;-)

[Andrew Milner]

I'll check with the Captain.

[PWS]

Maybe I should just clarify that slightly flippant comment, to avoid any misunderstanding. What I meant was I've long since given up on trying to second-guess what a given user or customer might want either in the way of weather hardware or software. Whatever logical argument I think I can marshal (which may well be incomplete in fact when exposed to counter-argument) the user will still want what they think they want. And who am I to disagree with what is ultimately often essentially their opinion.

For example, I cannot fathom why anyone would want to use a 1-minute archive interval - it's neither fish nor fowl. Far too long to capture any useful wind gust data, but most other weather parameters don't change significantly on a minute-to-minute basis. If I was writing AWS software from scratch, I'd be tempted to:

** Capture all gust speed/direction data at LOOP resolution (in a Davis sense) and save as an array at archive resolution (it's the only way of ensuring complete wind data)

** Capture rainfall data as date/time of each LOOP that shows one tip ==> gives best available rainfall rate data

** Capture most other data at eg 5min AI

But even this isn't perfect. For instance there can be interesting fine structure in the pressure record when an active front passes over - you almost need LOOP resolution to record this even though 5min values are fine >99% of the time

On the airfield requirement: What we're most often asked for (and what I think corresponds to common professional practice) is the current rolling 2min mean along with highest gust in past 2mins (if over a certain threshold - can't recall the value offhand, is it 15kn?).Davis LOOP2 comes closest to providing these values, but AFAIK isn't implemented in weewx? But hence the interest in accessing LOOP data in real time since that's the only way of achieving finer-grained calculation of the values that some users require. The Cumulus realtime.txt is just one quick way of visualising what's happening.

Sorry - this has strayed a long way from the original weewx code question and answer.

[mwall]

On Monday, April 28, 2014 10:44:51 AM UTC-4, PWS wrote:

I haven't had a chance to look at this in detail yet so ICBW, but I'd question whether this is likely to be what I'd regard as a genuine realtime.txt generator - 'genuine' in the sense that realtime should mean what it says, ie reflect LOOP values for rapidly changing parameters like wind speed/direction and be capable of updates/uploads every few seconds and not just be another skin reflecting the latest archive data, eg every 5 minutes. In other words, similar to a WURF upload generator.

here is a fully functional service that emits a cumulus realtime.txt file:

https://sourceforge.net/p/weewx/wiki/crt/

the cumulus 'specification' for the file is here:

http://wiki.sandaysoft.com/a/Realtime.txt

the many ambiguities in the specification are enumerated in the crt.py comments.

m

[PWS]

OK - I can see you've uploaded a full CRT service, but I was in the process of taking a look at this earlier version, so I'd just like to get that up and running first. I think the file location is sorted out OK, but I'm getting a syntax error in line 1 of crt.py when I try to start weewx with the revised weewx.conf. My version of crt.py is attached. I can't spot a problem with it, but perhaps someone with sharper eyes...

On Thursday, 24 April 2014 21:57:08 UTC+1, mwall wrote:

[PWS]

OK just spotted the typo - will try again!

[mwall]

my original crt.py posting failed to initialize the class.  it should be this:

import weewx
from weewx.wxengine import StdService

class CRTService(StdService):
    def __init__(self, engine, config_dict):

        super(CRTService, self).__init__(engine, config_dict)

        d = config_dict.get('CRTService', {})
        self.filename = d.get('filename', '/var/tmp/realtime.txt')
        self.bind(weewx.NEW_LOOP_PACKET, self.writeData)

    def writeData(self, event):
        with open(self.filename, 'a') as f:
            f.write(','.join(["%s" % x for x in event.packet.values()]))
            f.write("\n")

most of the code in the full implementation is to calculate all of the fields that cumulus requires for the official realtime.txt format.

m

[PWS]

Making progress in that the version with the updated config file and crt.py present will now launch without errors, but syslog reveals that it's not saving data because it thinks that the database is read only. Any idea why this should be? The only change that I've made in addition to those suggested in this thread are to save the updated weewx.conf file with a different name and starting weewx with the new conf file as the argument.

[mwall]

On Tuesday, April 29, 2014 4:57:55 PM UTC-4, PWS wrote:

Making progress in that the version with the updated config file and crt.py present will now launch without errors, but syslog reveals that it's not saving data because it thinks that the database is read only. Any idea why this should be? The only change that I've made in addition to those suggested in this thread are to save the updated weewx.conf file with a different name and starting weewx with the new conf file as the argument.

sounds like you're running weewx as a user that does not have write permission on the database.  if you're running from a .deb installation, you need to sudo since by default everything is owned by root.

m

[Nico wsMaurik]

I agree on all your points but would like to add the sun-radiation sensor to the LOOP resolution as this can be used to check/compare solar panel production. Only on the Davis it's updated every 50-60secs so here's your fish interval :-)

rgds

nico

Op maandag 28 april 2014 22:24:27 UTC+2 schreef PWS:

[PWS]

OK, thanks, got something working now. 

But I'm struggling to map the values in the CSV file to the expected sequence of values in a Davis LOOP packet. There are 81 fields which I guess is something about the right number (the uncertainty being whether the 3 initial LOO bytes are lost and exactly how you count eg the various Alarm bytes, plus presumably the last 4 bytes (LF+CR+CRC) are also truncated). Field 3 (1-based) looks like pressure, but beyond that the sequence is difficult to fathom - certainly it doesn't look like raw LOOP bytes or fields, but values that are presumably processed somehow and perhaps resequenced? 

I don't suppose that there's a key anywhere? (I mean to the file output, not the expected raw LOOP pattern, which I know).

On Tuesday, 29 April 2014 16:11:51 UTC+1, mwall wrote:

[mwall]

On Wednesday, April 30, 2014 5:04:41 PM UTC-4, PWS wrote:

I don't suppose that there's a key anywhere? (I mean to the file output, not the expected raw LOOP pattern, which I know).

try something like this instead.  it will emit two lines to the file, a commented header in the first line and the data in the second line.

import weewx
from weewx.wxengine import StdService

class CRTService(StdService):
    def __init__(self, engine, config_dict):

        super(CRTService, self).__init__(engine, config_dict)

        d = config_dict.get('CRTService', {})
        self.filename = d.get('filename', '/var/tmp/realtime.txt')
        self.bind(weewx.NEW_LOOP_PACKET, self.writeData)

    def writeData(self, event):

        with open(self.filename, 'w') as f:
            keys = []
            values = []
            for key in event.packet:
                keys.append(key)
                values.append(event.packet[key])
            f.write('#')
            f.write(','.join(keys))
            f.write("\n")
            f.write(','.join(values))
            f.write("\n")

 

[PWS]

The line:

f.write(','.join(values))

is throwing an error: 'TypeError sequence item 0: expected string, float found'

Presumably the values just need forcing to strings. Is that just .join(values.tostring) ? (Sorry don't know enough Python to be sure of the type conversion syntax.)

[mwall]

On Thursday, May 1, 2014 11:36:49 AM UTC-4, PWS wrote:

The line:

f.write(','.join(values))

is throwing an error: 'TypeError sequence item 0: expected string, float found'

Presumably the values just need forcing to strings. Is that just .join(values.tostring) ? (Sorry don't know enough Python to be sure of the type conversion syntax.)

python's lack of strict typing bites me every time, and i can never be bothered to remember when automatic conversion will happen. 

force conversion to string with something like this:

f.write(','.join(["%s" % x for x in values]))

[PWS]

@mwall

 Thanks again - that's really helpful - now managed to get a Loop file output working that means something. But looking at the output , I'm scarcely surprised that I couldn't decode the nominal Loop output - the sequence and nature of the fields is hugely different from eg the standard Davis LOOP. But it doesn't really matter - once you have the key then it can all start to make sense. I just need to work with it now to understand exactly which fields are present and how to work with them. I'll be back...

[Thomas Keffer]

Because dictionaries in Python are unordered, you cannot predict the order that the items come out. If you care, you can sort the keys, then ask for each one in order.

The names were inherited from wview. 

-tk

On Thu, May 1, 2014 at 12:01 PM, PWS <gpro...@gmail.com> wrote:

@mwall

 Thanks again - that's really helpful - now managed to get a Loop file output working that means something. But looking at the output , I'm scarcely surprised that I couldn't decode the nominal Loop output - the sequence and nature of the fields is hugely different from eg the standard Davis LOOP. But it doesn't really matter - once you have the key then it can all start to make sense. I just need to work with it now to understand exactly which fields are present and how to work with them. I'll be back...

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PWS]

Haven't had a chance to try the full service yet - hopefully over the weekend. But just to comment on some of the FIXME points:

All time values: I don't think that Cumulus provides any absolute definition - it's user choice AFAIK. But typical usage would be for local time adjusted for DST (though not everyone will apply the DST change - some users preferring to stick to GMT or local equivalent year-round)

Is-sunny: will obviously only be relevant to those with a solar irradiance sensor. But if it is fitted then Cumulus will be able to estimate (from comparing actual with theoretical maximum for time of day and day of year) whether conditions are likely to be bright sunshine or cloudy. It's the same (relatively straightforward) algorithm used to estimate daily hours of bright sunshine from the irradiance record. It's certainly an imperfect calculation but a lot better than nothing for those users interested in continuity with the extensive past records of bright sunshine hours.

Barometer/pressure: Not sure which definition of pressure you're meaning here, but general usage is QNH (unless you live at higher altitude when there's obviously considerable scope for debate about which pressure correction should be used).

Interval for pressure trend is conventionally 3 hours.

Interval for av wind speed: Again I'm not sure that there's a definitive answer here. 10-min means are the most commonly used, but 2-min means are IIRC more widely used in aviation. Personally I'm happier to see 2-min means, but 10-min means seem to be suggested by professionals as the correct basis for Beaufort speed conversions. 

Interval for av wind dir: should presumably be the same as chosen for wind speed.

NB My overall take on this is: The reason for being interested in the Cumulus real-time file is to be able to take advantage of the various real-time webpage presentations that are available using this file. I don't think it's necessary critical to follow the exact internal calculations in generating realtime.txt that Cumulus might use (obviously as long as the values used are of the same type and precision), especially where there may be some debate about which formula may be the 'correct' one.

[mwall]

On Friday, May 2, 2014 4:55:37 PM UTC-4, PWS wrote:

Haven't had a chance to try the full service yet - hopefully over the weekend. But just to comment on some of the FIXME points:

thank you - that is helpful.

the crt extension has been updated to version 0.2.  from the changelog:

* aggregate excluding previous period
* fix trend calculations
* parameterize the wind speed/dir avg intervals
* ensure NULL instead of None in output
* use barometer (slp) instead of pressure (gauge)
* calculate temperature trend
 
max_rad is not calculated - anyone have the math for that?

is_sunny is the same thing as is_daylight.  something based on solar radiation would be more appropriate, but then we need a threshold.  can anyone suggest a measure in W/m^2 that indicates "it is sunny now"?

[PWS]

I'm assuming that is_daylight simply means is it daytime between the hours of civil twilight (or whatever exact definition of daytime might be preferred), whereas is_sunny refers to whether there's bright sunshine or not. So is_daylight can just be calculated from astronomical data whereas is_sunny must use a measured value, typically from from a global solar irradiance sensor and then suitably processed (or less commonly a sunshine hours duration sensor, which would give a Boolean value directly).

The calculation for is_sunny is around on the web, but I don't have a reference to hand - I'll see if I can track one down when I get a moment. I did write up some notes on this several years ago but rather stupidly didn't include the calculation there. I do have the notes (but not the formula!) on a personal website at: http://www.elyweather.co.uk/Sunshine1.aspx but more for the chart I was using in some software at the time which illustrates the principle more easily than I can describe it. (The notes were only very rough and haven't been touched for many years - I'm surprised they're still there!). But the principle is to calculate the envelope of maximum irradiance through the day (for the current day of the year) and then also calculate a similar envelope for some percentage of the maximum (I think I used about 45%, but I'll need to double-check). If the current irradiance value exceeds the threshold then you define that as bright sunshine. If you look at the shape of the envelope you can see why just using a single fixed threshold value for irradiance gives bad results - bearing in mind also how the height of the envelope is likely to vary through the year, especially at higher latitudes.

[wysiwyg]

I am also interested in realtime data.

I tried the setup from here: http://sourceforge.net/p/weewx/wiki/crt/

But it does not work on my side: it make weewx stops and set those error message in syslog:

Jun  3 08:51:35 gunther weewx[3749]: wxengine: Starting main packet loop.
Jun  3 08:51:48 gunther weewx[3749]: wxengine: Caught unrecoverable exception in wxengine:
Jun  3 08:51:48 gunther weewx[3749]:     ****  unsupported operand type(s) for -: 'NoneType' and 'float'
Jun  3 08:51:48 gunther weewx[3749]:     ****  Traceback (most recent call last):
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 954, in main

Jun  3 08:51:48 gunther weewx[3749]:     ****      engine.run()
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 181, in run
Jun  3 08:51:48 gunther weewx[3749]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 210, in dispatchEvent
Jun  3 08:51:48 gunther weewx[3749]:     ****      callback(event)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 282, in handle_new_loop
Jun  3 08:51:48 gunther weewx[3749]:     ****      data = self.calculate(event.packet, archive)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 312, in calculate
Jun  3 08:51:48 gunther weewx[3749]:     ****      archive, ts)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 151, in calcTrend
Jun  3 08:51:48 gunther weewx[3749]:     ****      return new_val - old_val[0]
Jun  3 08:51:48 gunther weewx[3749]:     ****  TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
Jun  3 08:51:48 gunther weewx[3749]:     ****  Exiting.
Jun  3 08:54:36 gunther weewx[3849]: wxengine: Initializing weewx version 2.6.3
Jun  3 08:54:36 gunther weewx[3849]: wxengine: Using Python 2.7.6 (default, Mar 22 2014, 22:59:56) #012[GCC 4.8.2]
Jun  3 08:54:36 gunther weewx[3849]: wxengine: pid file is /var/run/weewx.pid
Jun  3 08:54:36 gunther weewx[3851]: wxengine: Using configuration file /etc/weewx/weewx.conf
Jun  3 08:54:36 gunther weewx[3851]: wxengine: Loading station type WMR100 (weewx.drivers.wmr100)
Jun  3 08:54:37 gunther weewx[3851]: wxengine: StdConvert target unit is 0x1
Jun  3 08:54:37 gunther weewx[3851]: forecast: MainThread: Zambretti: interval=600 max_age=604800 hemisphere=NORTH lower_press

In case that help, but you can see it in logs: I use WMR100 driver.

[mwall]

On Tuesday, June 3, 2014 3:57:02 AM UTC-4, wysiwyg wrote:

I tried the setup from here: http://sourceforge.net/p/weewx/wiki/crt/

But it does not work on my side: it make weewx stops and set those error message in syslog:

Jun  3 08:51:35 gunther weewx[3749]: wxengine: Starting main packet loop.
Jun  3 08:51:48 gunther weewx[3749]: wxengine: Caught unrecoverable exception in wxengine:
Jun  3 08:51:48 gunther weewx[3749]:     ****  unsupported operand type(s) for -: 'NoneType' and 'float'
Jun  3 08:51:48 gunther weewx[3749]:     ****  Traceback (most recent call last):
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 954, in main

Jun  3 08:51:48 gunther weewx[3749]:     ****      engine.run()
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 181, in run
Jun  3 08:51:48 gunther weewx[3749]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 210, in dispatchEvent
Jun  3 08:51:48 gunther weewx[3749]:     ****      callback(event)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 282, in handle_new_loop
Jun  3 08:51:48 gunther weewx[3749]:     ****      data = self.calculate(event.packet, archive)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 312, in calculate
Jun  3 08:51:48 gunther weewx[3749]:     ****      archive, ts)
Jun  3 08:51:48 gunther weewx[3749]:     ****    File "/usr/share/weewx/user/crt.py", line 151, in calcTrend
Jun  3 08:51:48 gunther weewx[3749]:     ****      return new_val - old_val[0]
Jun  3 08:51:48 gunther weewx[3749]:     ****  TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
Jun  3 08:51:48 gunther weewx[3749]:     ****  Exiting.

at line 147, change this:

def calcTrend(label, new_val, archive, ts, n=3):
    lastts = ts - n * 3600
    qstr = "SELECT %s FROM archive WHERE dateTime>? AND dateTime<=?" % label
    old_val = archive.getSql(qstr, (lastts, ts))

    return new_val - old_val[0]

to this:

def calcTrend(label, new_val, archive, ts, n=3):
    if new_val is None:
        return None
    lastts = ts - n * 3600
    qstr = "SELECT %s FROM archive WHERE dateTime>? AND dateTime<=?" % label
    old_val = archive.getSql(qstr, (lastts, ts))

    return new_val - old_val[0]

or just download version 0.3 of crt.

m

[wysiwyg]

One step  further :-)

now error with line 351:
Jun  3 23:32:25 gunther weewx[5249]:     ****    File "/usr/share/weewx/user/crt.py", line 351, in calculate
Jun  3 23:32:25 gunther weewx[5249]:     ****      dp_K = dp_C + 273.15
Jun  3 23:32:25 gunther weewx[5249]:     ****  TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'
Jun  3 23:32:25 gunther weewx[5249]:     ****  Exiting.


So I try samekind of thing (at least to try to see what happend next.
So I have modified line 351 like this:
dp_C = weewx.units.convert(vt, 'degree_C')[0]
        if dp_C is not None:
            dp_K = dp_C + 273.15
        else:
            dp_K = 273.15
        v = calcHumidex(t_C, dp_K);

Then I get again this error on an other item
Jun  3 23:39:25 gunther weewx[5370]:     ****    File "/usr/share/weewx/user/crt.py", line 208, in calcHumidex
Jun  3 23:39:25 gunther weewx[5370]:     ****      return t_C + v
Jun  3 23:39:25 gunther weewx[5370]:     ****  TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'
Jun  3 23:39:25 gunther weewx[5370]:     ****  Exiting.

 

[wysiwyg]

Could it be that WMR100 send a packet per sensor at the time, so that CRT.py would try to output all data but has only one sensor in hand (so many undefined values)?

Just an hypothesis, I may be completly wrong.
 

[mwall]

That is exactly what is happening. Try v0.3 of crt - it has fixes for all (i think) null sensors.

M

[wysiwyg]

just tried: the 1st one I mentionned (now line 355) is still not ok:

Jun  4 01:46:05 gunther weewx[7144]:     ****    File "/usr/share/weewx/user/crt.py", line 355, in calculate
Jun  4 01:46:05 gunther weewx[7144]:     ****      dp_K = dp_C + 273.15
Jun  4 01:46:05 gunther weewx[7144]:     ****  TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'
Jun  4 01:46:05 gunther weewx[7144]:     ****  Exiting.

[mwall]

at line 355 change this:

        dp_C = weewx.units.convert(vt, 'degree_C')[0]

        dp_K = dp_C + 273.15

        v = calcHumidex(t_C, dp_K);

to this:

        dp_C = weewx.units.convert(vt, 'degree_C')[0]

        dp_K = None if dp_C is None else dp_C + 273.15
        v = calcHumidex(t_C, dp_K);
 
m

[wysiwyg]

Hello mwall !

Nice! now it's running !

I checked with sudo weewxd weewx.conf and I can see the LOOP message every few seconds with the different packets and REC message every 5min

But...no file in my /var/www/weewx/realtime directory.

Here is my config (in weewx.conf):
[CumulusRealTime]
    filename = /var/www/weewx/realtime/realtime.txt

and:
[Engines]
    # This section configures the internal weewx engines.
    # It is for advanced customization.

    [[WxEngine]]

        # The list of services the main weewx engine should run:
        prep_services = weewx.wxengine.StdTimeSynch
        process_services = weewx.wxengine.StdConvert, weewx.wxengine.StdCalibrate, weewx.wxengine.StdQC, user.forecast.ZambrettiForecast, user.forecast.WUForecast, user.crt.CumulusRealTime
        archive_services = weewx.wxengine.StdArchive
        #restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, $
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdAWEKAS, user.owm.OpenWeatherMap
        report_services = weewx.wxengine.StdPrint, weewx.wxengine.StdReport

[mwall]

On Wednesday, June 4, 2014 3:34:55 AM UTC-4, wysiwyg wrote:

Hello mwall !

Nice! now it's running !

I checked with sudo weewxd weewx.conf and I can see the LOOP message every few seconds with the different packets and REC message every 5min

But...no file in my /var/www/weewx/realtime directory.

check the log file.  check the output from weewxd.

does /var/www/weewx/realtime exist before you start weewx?  (crt does not do a mkdir - it just tries to create a file)

m

[wysiwyg]

Hello,

Yes the directory /var/www/weewx/realtime exists.
I tried to create an empty realtime.txt (juste in case) but no difference.

Regarding logs: syslog nothing noticable (but CRT.py has only one syslog call.

Regarding weewxd : nothing but LOOP message with data from the differents sensors and time to time a REC message (archiving of data).

Is there some syslog or weewxd ouput I could add to investigate deeper ?

(thanks for supporting!)

[mwall]

On Wednesday, June 4, 2014 10:52:23 AM UTC-4, wysiwyg wrote:

Is there some syslog or weewxd ouput I could add to investigate deeper ?

try adding a syslog call or even a print statement in write_data, then restart weewx
 

[wysiwyg]

Ok, I tried this:

    def write_data(self, data):

        with open(self.filename, 'w') as f:

            f.write(self.create_realtime_string(data))
            f.write("\n")
            syslog.syslog(syslog.LOG_NOTICE,"Hey, I'm wriiiiting")

The bad point is that it made weewxd stops:

Jun  4 17:39:53 gunther weewx[2172]: wxengine: Caught unrecoverable exception in wxengine:
Jun  4 17:39:53 gunther weewx[2172]:     ****  global name 'syslog' is not defined
Jun  4 17:39:53 gunther weewx[2172]:     ****  Traceback (most recent call last):
Jun  4 17:39:53 gunther weewx[2172]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 954, in main
Jun  4 17:39:53 gunther weewx[2172]:     ****      engine.run()
Jun  4 17:39:53 gunther weewx[2172]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 181, in run
Jun  4 17:39:53 gunther weewx[2172]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Jun  4 17:39:53 gunther weewx[2172]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 210, in dispatchEvent
Jun  4 17:39:53 gunther weewx[2172]:     ****      callback(event)
Jun  4 17:39:53 gunther weewx[2172]:     ****    File "/usr/share/weewx/user/crt.py", line 289, in handle_new_loop
Jun  4 17:39:53 gunther weewx[2172]:     ****      self.write_data(data)
Jun  4 17:39:53 gunther weewx[2172]:     ****    File "/usr/share/weewx/user/crt.py", line 295, in write_data
Jun  4 17:39:53 gunther weewx[2172]:     ****      syslog.syslog(syslog.LOG_NOTICE,"Hey, I'm wriiiiting")
Jun  4 17:39:53 gunther weewx[2172]:     ****  NameError: global name 'syslog' is not defined
Jun  4 17:39:53 gunther weewx[2172]:     ****  Exiting.

I think it might be an "import syslog" that is missing at beginning of crt.py ?

The good point is that this crash means I was in write_data :-D


Oh my god, it is in /var/tmp !!! :-) and works !!

did I miss something in weewx.conf ?

[mwall]

On Wednesday, June 4, 2014 11:51:19 AM UTC-4, wysiwyg wrote:

I think it might be an "import syslog" that is missing at beginning of crt.py ?

correct.

 

Oh my god, it is in /var/tmp !!! :-) and works !!

did I miss something in weewx.conf ?

the code was looking for [CumulusRealTimeService] when it should have been looking for [CumulusRealTime]

i have made these fixes for crt 0.4

m

[wysiwyg]

Yesterday it has worked all afternoon but stopped a quarter after midnight with this message:

Jun  5 08:34:05 gunther weewx[3974]: wxengine: Caught unrecoverable exception in wxengine:
Jun  5 08:34:05 gunther weewx[3974]:     ****  'NoneType' object has no attribute '__getitem__'
Jun  5 08:34:05 gunther weewx[3974]:     ****  Traceback (most recent call last):
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 954, in main
Jun  5 08:34:05 gunther weewx[3974]:     ****      engine.run()
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 181, in run
Jun  5 08:34:05 gunther weewx[3974]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/weewx/wxengine.py", line 210, in dispatchEvent
Jun  5 08:34:05 gunther weewx[3974]:     ****      callback(event)
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/user/crt.py", line 288, in handle_new_loop
Jun  5 08:34:05 gunther weewx[3974]:     ****      data = self.calculate(event.packet, archive)
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/user/crt.py", line 325, in calculate
Jun  5 08:34:05 gunther weewx[3974]:     ****      v,t = calcMinMax('outTemp', archive, ts, 'MAX')
Jun  5 08:34:05 gunther weewx[3974]:     ****    File "/usr/share/weewx/user/crt.py", line 197, in calcMinMax
Jun  5 08:34:05 gunther weewx[3974]:     ****      tstr = time.strftime("%H:%M", time.gmtime(t[0]))
Jun  5 08:34:05 gunther weewx[3974]:     ****  TypeError: 'NoneType' object has no attribute '__getitem__'
Jun  5 08:34:05 gunther weewx[3974]:     ****  Exiting.
Jun  5 08:34:34 gunther weewx[4033]: wxengine: Initializing weewx version 2.6.3


I notice this stop this morning and try to restart, but it stops immediatly with same error



By the way, is there a possibility to make such issue dont stops completly weewx?

[mwall]

On Thursday, June 5, 2014 2:38:07 AM UTC-4, wysiwyg wrote:

By the way, is there a possibility to make such issue dont stops completly weewx?

done.  try crt v0.4

m

[wysiwyg]

Works well upto now!
many thanks !

I will confirm tomorrow if no more "night error" happens

[Jarmo Seppänen]

When starting with weeWX 2.6.3 and Mysql db got this errormessage:

Jun  5 18:38:38 raspberrypi weewx[17718]: crt: **** Traceback (most recent call last):
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****   File "/home/weewx/bin/user/crt.py", line 314, in handle_new_loop
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****     data = self.calculate(event.packet, archive)
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****   File "/home/weewx/bin/user/crt.py", line 344, in calculate
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****     data['wind_run'] = calcWindRun(archive, ts)
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****   File "/home/weewx/bin/user/crt.py", line 151, in calcWindRun
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****     (sod_ts, ts)):
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****   File "/home/weewx/bin/weewx/archive.py", line 297, in genSql
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****     for _row in _cursor.execute(sql, sqlargs):
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****   File "/home/weewx/bin/weedb/mysql.py", line 189, in execute
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: ****     raise weedb.OperationalError(e)
Jun  5 18:38:38 raspberrypi weewx[17718]: crt: **** OperationalError: (1064, "You have an error in your SQL syntax; check the manual t
hat corresponds to your MySQL server version for the right syntax to use near 'windSpeed FROM archive WHERE dateTime>1401915600 AND da
teTime<=1401982718' at line 1")


Traced it back to
# calculate wind run since midnight
def calcWindRun(archive, ts):
    run = 0
    sod_ts = weeutil.weeutil.startOfDay(ts)
    for row in archive.genSql("SELECT `interval`,windSpeed FROM archive "
                              "WHERE dateTime>? AND dateTime<=?",
                              (sod_ts, ts)):
        run += row[1] * (row[0] / 60)
    return run

and put the interval inside `` as it seems that in mySQL the interval is "reserved" word.

Although I don't understand anything about coding this seems to be working ;)

[mwall]

On Thursday, June 5, 2014 11:53:14 AM UTC-4, Jarmo Seppänen wrote:

and put the interval inside `` as it seems that in mySQL the interval is "reserved" word.

Although I don't understand anything about coding this seems to be working ;)

thank you jarmo.  your fix is included in crt v0.5

m

[wysiwyg]

Hello

I am trying a small customisation of this service to output xml file instead of txt, just playing with write_data function.

here is my code:
at top of the file:
import xml.etree.cElementTree as ET

and then in write_data

 def write_data(self, data):

        with open(self.filename, 'w') as f:
            f.write(self.create_realtime_string(data))
            f.write("\n")

        root = ET.Element("root")
        doc = ET.SubElement(root, "doc")
        field1 = ET.SubElement(doc, 'windSpeed_avg')
        windSpeed_avg.text = "value1"  # here it does not work with data['windSpeed_avg']instead of "value1"
        field2 = ET.SubElement(doc, "field2")
        field2.text = "some value2"
        tree = ET.ElementTree(root)
        # tree.write(self.filenamexml)
        tree.write('/var/tmp/realtime.xml')



I thought I could access to values in data by using like data['windSpeed_avg'] but it is not working.

Is there a simple way to get each values separatly inside "data" ?

As the second step, is there a way to get corresponding label (like 'windSpeed_avg) so I can set automaticaly my subelement label ?

I'm a total newbie in python :-)


Anyway, my xml file creation is working and realtime.xml looks like this:

fred@gunther:/var/tmp$ cat realtime.xml
<root><doc><windSpeed_avg>value1</windSpeed_avg><field2>some vlaue2</field2></doc></root>

[wysiwyg]

ok, solution found!

 def write_data(self, data):
        with open(self.filename, 'w') as f:
            f.write(self.create_realtime_string(data))
            f.write("\n")
        root = ET.Element("root")
        doc = ET.SubElement(root, "doc")
        field1 = ET.SubElement(doc, 'windSpeed_avg')

        field1.text = "test1"
        # data['windSpeed_avg']
        field2 = ET.SubElement(doc, "outTemp")
        field2.text = self.format(data, 'outTemp')
        #data['outTemp']

        tree = ET.ElementTree(root)
        # tree.write(self.filenamexml)
        tree.write('/var/tmp/realtime.xml')

I will play a little bit more and share final result in case of interesting someone.

[Marco Bakker]

Great job !

is it possible to get values in Celcius, hPa and mm ?

And if so, how ?

Thanks !

Marco 

[Frederic Stuyk]

No clue how (I am total newbie in python), but in case you use those values in js or php or whatever, I guess you may be able to convert there ?

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/-tihrtU-iyQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[galfert]

Would it be possible to run WeeWx and this CRT add-on to then be able to run Cumulus? My station is not directly supported by Cumulus. I thought this might be a way to feed data to Cumulus.

[gjr80]

Don't know, I thought realtime.txt was a product of Cumulus, does Cumulus accept it as a source? Guess that is something for the Cumulus folks.

You might also want to have a look at the assumptions/limitations listed in the up front comments in bin/user/crt.py. On the face of it they do not seem particularly significant but who knows if they are significant if used to feed Cumulus.

Gary