Format of weewx-variables/datafields for arithmetic (service development)?

[Michael Frotscher]

All,

I'm trying to write a simple service that creates a new derived measurement based on values already in the weewx database. Ultimately, I want to get a running total of the rainfall, but I'm not there yet. Also, not a Python developer by trade, but I'm trying.

My issue is: when I try to do arithmetic, the variables have, of course, to be numbers, be it int or float. I'll take the 'pressure' measurement as an example. Looking into the database, it seems to be a float, but it could also be a string.

But trying to use it that way, or trying to convert the string into a float, always gives me errors (could not convert string to float). I've tried stripping invisible whitespaces from the string (if it is one), but the error remains.

Here's the relevant code, one of the many I've tried.

So how do I get a weewx-variable into usable form? I'm probably missing something pretty basic here.

Thanks!

BTW, is there a way to test-run a service file? Like have the python code run in the weewx-environment?

foo = (float('pressure') * 10)

or

pressure_trim = 'pressure'.strip()  
foo = (float(pressure_trim) * 10)

[Tom Keffer]

When a value comes off the database it's always either a number, or the Python value "None", with very few exceptions. You didn't show us the error, but the issue is likely to be that "pressure" has the value "None", which signals bad or missing data.

Be sure to read the document Notes for developers. It includes some information about None.

So, your example becomes,

foo = pressure * 10 if pressure is not None else None

Then, of course, you have to remember that "foo" could be None. This is a very common pattern in WeeWX.

Then again, perhaps the error is completely different and it's not due to the value None. If you're still having trouble, show us the error.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f08bf769-d2b2-4840-b0a3-2feeef33ea67n%40googlegroups.com.

[Michael Frotscher]

Tom,

Thanks for the reply.

The exact error is: CRITICAL __main__:     ****  ValueError: could not convert string to float: 'pressure'
and weewx crashes when it encounters that.

I have considered that the value might be Null (or None, rather), which is why I chose a variable that's fed from my barometer, so should always contain a valid value.

But I have also added a check for that into my code, with the same result:

if 'pressure' != None:

  foo = do arithmetic

else:

  foo = do nothing

FWIW, I'm using the "electricity" service example, and the code in question is in the "def new_archive_record(self, event):" section.

But is it correct to refer to the variable as 'pressure'? With the single quotes?

Because if I don't, like you have in your example, I get this:

NameError: name 'pressure' is not defined

[Tom Keffer]

You're going to have to give us more information about where your variable "pressure" came from. 

Other bits, probably not related to your problem.

1. Never assume a variable always contains a valid value. Every instrument fails.

2. To compare with None use

if pressure is not None:

  foo = do arithmetic

else:

  foo = do nothing

3. Pressure with single quotes ('pressure') is a string. Variables are unadorned.

4. The variable pressure in my example is undefined because in my little example I didn't define it. I'm assuming you defined it somewhere, which brings us back to my first paragraph above.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/09981f6c-0d06-4a4c-b438-91c46cc01c84n%40googlegroups.com.

[Michael Frotscher]

Here's the full code of my service:

For testing purposes I try to retrieve the pressure value, do the arithmetic and write it to a different, unused datafield. The MySQLdb is in there to get the last non-null value of the field im interested in, but that's not implemented so far.

#!/usr/bin/env python
import MySQLdb
import weewx
from weewx.engine import StdService

class AddRaintotal(StdService):

   def __init__(self, engine, config_dict):
     # Initialize my superclass first:
     super(AddRaintotal, self).__init__(engine, config_dict)
     # Bind to any new archive record events:
     self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_packet)
#      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

#(The example uses new_archive_record, but that gives me another, totally different error)
     
   def new_archive_packet(self, event):

      if 'pressure' != None:
        newpressure = ('pressure' * 10)

        event.record['pb'] = newpressure

[Tom Keffer]

On Thu, May 2, 2024 at 6:32 PM Michael Frotscher <frot...@gmail.com> wrote:

Here's the full code of my service:

     

   def new_archive_packet(self, event):

      if 'pressure' != None:
        newpressure = ('pressure' * 10)

        event.record['pb'] = newpressure

Strictly interpreted, you first ask whether the string "'pressure'" is equal to None. It's not, so we proceed to the next statement. Now you're trying to multiply a string ('pressure') by 10. Surprisingly, this will succeed, but likely does not give you the results you expect. The variable "newpressure" will actually be set to the string 'pressurepressurepressurepressurepressurepressurepressurepressurepressurepressure'. That is, the string 'pressure' concatenated 10 times.

You want a variable pressure, not the literal string 'pressure'. Where will it come from?

I would suggest taking an online Python course if this is unfamiliar to you. 

-tk

[František Slimařík]

I guess you want something like this:

if event.record['pressure'] != None:

  newpressure = (event.record['pressure'] * 10)

  event.record['pb'] = newpressure

Dne pátek 3. května 2024 v 3:38:01 UTC+2 uživatel Tom Keffer napsal:

[Michael Frotscher]

Thanks, guys!

I see my mistake now. It's not about strings/floats at all, but I wrongly assumed that all measurements that have values assigned to them (via LOOP or whatever) would already be defined as global variables.

That's obviously not the case, but assigning the last value to the variable via "event.record['pressure']" (as Frantisek pointed out) is necessary. I'm still not sure why that needs to be in single quotes, indicating a string and not a number.

I would have expected something like "event.record(pressure)".

I have to admit that I have not found this in any examples out there, and not in the documentation for sure.

But it's now doing what I want.

[Michael Frotscher]

Hmm,

ok, so it works for "pressure", which contains data in every archive period and is not None.

I've tried to substitute that with "rain", which is what I really want. That is usually "Null/None" in the database, but gets the amount of a bucket tip added to it every time that happens.

It is fed by MQTT, which publishes the bucket amount on every tip, and that works. It even sums them up correctly if several bucket tips occur in an archive period.

However, trying to retrieve that with "if event.record['rain'] != None:"

I get a Key Error. Meaning the variable doesn't exist. Isn't every weewx measurement in the dictionary?

That happens with any measurement that's in the database but normally "Null".

I've tried to define it as Null/None in the init-part of the service 

rain = None

But still get the key error.

Thanks!

[František Slimařík]

I believe "rain" is specific in this case. I was checking raw packets previously on my device and normally is rain "none" if it´s not raining ;) So it depends what your device is sending to weewx.

raw packet: {'dateTime': 1642794129, 'usUnits': 1, 'rain_total': 0.0, 'barometer': 30.22, 'temperature_out': 28.7, 'dewpoint': 20.1, 'humidity_out': 69.0, 'wind_speed': 2.6, 'wind_gust': 2.6, 'wind_dir': 268.0, 'solar_radiation': 0.0, 'uv': 0.0, 'temperature_in': 70.1, 'humidity_in': 51.0, 'rain': None}

pá 3. 5. 2024 v 15:37 odesílatel Michael Frotscher <frot...@gmail.com> napsal:

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/8a2868e8-7f39-4038-acd5-c783572e2226n%40googlegroups.com.

[Michael Frotscher]

Same here, 'rain' is None unless it's raining, then it'll show data. I haven't checked the raw packets, but in the archive table the numbers are all multiples of my tipping bucket size, so that all makes sense.

That's why I added the "check for Null/None" if-clause.

But to be populated with data, 'rain' has to be defined. And weewx crashes on that 

if event.record['rain'] != None:

clause. Do I need to check for a packet? like event.package?

[Michael Frotscher]

Even more unexpected, it seems I can write to those "undefined" variables, but trying to read them gives me a Key Error.

I have created a new field in the database called "raintotal", where I want to store the running total of the rainfall.

Trying to test this for being "None", gives:

CRITICAL __main__:     ****      if event.record['raintotal'] == None:
CRITICAL __main__:     ****         ~~~~~~~~~~~~^^^^^^^^^^^^^
CRITICAL __main__:     ****  KeyError: 'raintotal'
CRITICAL __main__:     ****  Exiting.

But if I just write to it:

event.record['raintotal'] = 12345

That ends up in the database and doesn't throw an error.

[František Slimařík]

Hello,

event.record is dictionary. If you do event.record["rain" ] = something you add new key/value into dictionary (if doesn´t exists or overwrite current value). In case packet doesn´t contain no value for rain, key/value pair is not added. If you do something and you are not sure if key exist in dictionary you should do:

if "rain" in event.record:

  do_something

pá 3. 5. 2024 v 21:31 odesílatel Michael Frotscher <frot...@gmail.com> napsal:

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/cc1d11ae-3adf-46f5-a65a-c6da05880e73n%40googlegroups.com.

[Michael Frotscher]

That did it, thanks, Frantisek!

I now check that if 'rain' is in event.record, then check if it is non-Null, and if those two conditions are true, 'rain' has a valid value and I can use it for arithmetic.

[Tom Keffer]

Or, combining into one step...

if event.record.get('rain') is not None:

    do something

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/44a5a9c6-491f-4c10-a777-78f310a2a04dn%40googlegroups.com.

[Michael Frotscher]

Admittedly more elegant, thanks!

Like I said, I'm still getting into Python, so I wasn't aware one could apply get() here.

I usually have a fair understanding of what code does and can generate pseudocode of what I want, but implementing that in proper Python code is what I struggle with.

[Joel Bion]

I guess this is as good a time as any to ask if None is the way Weewx represents a nonexistent element of data for the result of a query to any of its API functions. For example, if I want the barometric pressure at 9am, and no pressure was recorded around that time, is it None that would be returned?

Is there ever a case where (like in some other Python packages) NaN is used? 

Sent from my iPhone

On May 4, 2024, at 2:05 PM, Michael Frotscher <frot...@gmail.com> wrote:

Admittedly more elegant, thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/31ee4807-ea36-4fb1-b355-b9759d0a1798n%40googlegroups.com.

[Vince Skahan]

It would return None - no value is present for that query's results.

A quick grep of an installed v5 shows NaN is never even referenced other than in a comment or two.

(from https://www.datacamp.com/tutorial/python-nan-missing-values-in-python#)

"NaN is a floating-point representation of "Not a Number," used primarily in numerical computations. None, on the other hand, is Python's object representing the absence of a value akin to null in other languages. While NaN is used in mathematical or scientific computations, None is more general-purpose, indicating the lack of data."