VatnagePro driver: what is the best way to get archiveRecords every minute?

[Saverio Guzzo]

Hey community,

I have a fleet of Davis weather stations connected to as many RaspberryPis, on which I am running the vantage drivers as a standalone program on docker container, deployed using Balena.

I'd like my program to send data every minute to a REST API and I've been looking into some way to get archive data from a in loop, but I'm not sure I'm doing it the right way.

What I did was defining in the driver's main method something like:

while True:

since_ts = datetime.datetime.timestamp(datetime.datetime.now() - datetime.timedelta(minutes=1))

for packet in vantage.genDavisArchiveRecords(since_ts):

try:

new_packet = weewx.units.to_METRICWX(packet)

davispusher.send_message(payload=new_packet)

log.debug(davispusher.payload)

except Exception as e:

log.debug("Found an exception: %s" % e)

time.sleep(1.2)

continue

time.sleep(60)

Where davispusher is an instance of a simple class that is needed to add some values to the archive record and send the message and is defined as:

class DavisPusher:

def __init__(self, host='https://mywebsite.somewhere, port=8080, endpoint='davis'):

self._endpoint = "{host}:{port}/{endpoint}".format(host=host,

endpoint=endpoint,

port=port)

self.payload = {}

def format_message(self,

payload: dict,

sensor_id: str = environ['SENSOR_ID'],

latitude: str = environ['LATITUDE'],

longitude: str = environ['LONGITUDE'],

altitude: str = environ['ALTITUDE'],

) -> dict:

"""Formats message for backend."""

payload_copy = payload.copy()

try:

self.payload = payload_copy

self.payload['sensor_id'] = sensor_id

self.payload['altitude'] = float(altitude)

self.payload['longitude'] = float(longitude)

self.payload['latitude'] = float(latitude)

except Exception:

self.payload = {}

def send_message(self, payload):

'''formats and sends message to backend'''

self.format_message(payload)

resp = requests.post(self._endpoint, data = json.dumps(self.payload), allow_redirects = True)

log.debug("got HTTP statuscode %s", resp.status_code)

However, my logs are full of logging errors ("Bad file descriptor", "File "/usr/local/lib/python3.8/logging/handlers.py" No such file or directory")

I guess those are errors due to logging configuration, but I'm not sure how to address them. I created a GitHub Gist in order not to pollute this post!

Also, what I would like to achieve is the possibility to configure the various consoles (set time, set coordinates) at startup using environment variables. How could I do this using the VantageConfigurator class?

Thanks a lot in advance for your help, I'm kind of alone in my work and I'd really appreciate your help.

Friendly greetings,

Saverio

[Saverio Guzzo]

PS. One detail that I forgot to mention: Balena uses docker images, for running the program I'm using balenalib/raspberrypi4-64-python:3.8-stretch-run

[Doug Jenkins]

Saverio:

Reading your post, have you considered using mqtt for each station node publishing it's archive data to a single broker?

I wonder if you could do the following to achieve the same goal:

1. Set each station node archive interval to 1 minute.

2. Install weewx mqtt extension on each node 

3. Configure each node to publish to a different topic, for example weather-n1, n2,etc.

4. Configure the mqtt driver to just publish archive data.

5. Setup a mosquito mqtt broker on the same network to collect all the data. It can be setup in a container.

This assumes all of your nodes are on the same network, although you could setup the mqtt broker in a cloud instance if the nodes are not on the same network.

That way you will get the full archive record published from each station. You would just need to focus on consuming the published data from the broker. At least all the data will be in a single place.

Just a thought.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/55d40e74-aa67-40c0-bcb6-8b8a03c9a665n%40googlegroups.com.

[Saverio Guzzo]

Hi, thanks for your answer!

Yes, this is a very good insight and suggestion. Before, I was using Apache Pulsar as a broker, but somehow the people working on the backend have decided to move to a REST service.

Anyhow, I think my issue is strictly related to logging.. if you have any suggestions on how to solve it, please let me know! :)

Thank you very much,

Saverio

[vince]

"Bad file descriptor", "File "/usr/local/lib/python3.8/logging/handlers.py" No such file or directory"

Seems pretty clear to me.  Your docker image doesn't appear to have the logging stuff built into it at the location python expects.

Poke around in a shell to see what's in there with something like "docker run --rm -it yourimagename bash" 

[Saverio Guzzo]

Thanks for you answer Vince.

True, the message is clear! However, the file "/usr/local/lib/python3.8/logging/handlers.py" is there as expected! That's why I'm surprised by the logging errors.

It's already a few weeks I'm chasing this error :(

If you guys have additional suggestions, please let me know

[Saverio Guzzo]

Regardless of the errors. Do you think that my code is acceptable to get archive data every minute?

Thanks,

Saverio