Adding a pitft or piscreen to show current weather data

[William Phelps]

I have several Pi's running weewx installed now, and I had a request from a user to add a real time display, both so they can see weather data and so they know the Pi is working properly.  Has anyone else tried this? 

I thought I would start by adding the weewx-wd extension, then have a separate process that parsed the various data files produced by weewx-wd to update graphic displays on the screen.  This keeps it separate from weewx.  Once I get it working then perhaps it could be incorporated into a weewx extension, not sure yet.  As a start I'll probably just display temperature and wind speed...

Another approach I thought of is to try to pull the data back off a WU web page running one of their rapid-fire widgets, but they use flash and I think that's probably more trouble than it's worth...

William

[mwall]

william,

weewx-wd has a lot of overhead for what you're trying to do.

you could use a csv service to emit loop data on every sensor reading, then have another process that monitors that file to display on lcd or web page or whatever.  emit the data as csv or json or whatever format works best for the javascript side of things.

its trivial when weewx is running on the same host as the display.  a bit more work if the display is remote from weewx (then you need to ftp/rsync/whatever on every loop)

i'm working on a wdcr (weather-display clientraw) extension that emits clientraw.txt so that wd live or any other software that expects clientraw would work.  it has built-in ftp and rsync so you can transfer loop records if necessary.  but its not ready yet - even just emitting a clientraw.txt (not the other 3 clientraw* files) has a bunch of extra junk that most people will never need.

here is the csvwriter service:

import os
import os.path
import time

import weewx
import weewx.engine

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):
        super(CSVWriter, self).__init__(engine, config_dict)

        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):
            self.write_line(self.sort_keys(event.record))
        self.write_line(self.sort_data(event.record))

    def write_line(self, fields):
        with open(self.filename, "a") as f:
            f.write('%s\n' % ','.join(fields))

    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

    def sort_data(self, record):
        fields = [str(record['dateTime'])]
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(str(record[k]))
        return fields
 

[William Phelps]

Wow, that's great! Just what I needed.  You just saved me a *bunch* of time, thanks!!!

Graphics on the locally attached display is done with pygame, at least that's what I use.  I've already done a Pi/TFT/GPS module that shows when the ISS is going to make a visible pass and plot it on a star chart, so I plan to reuse as much of that as possible.  One feature I will add is the ability to stop and start weewx, and shutdown or reboot the Pi, from the touch screen.

On Saturday, 2 May 2015 14:58:41 UTC-7, mwall wrote:

weewx-wd has a lot of overhead for what you're trying to do.

you could use a csv service to emit loop data on every sensor reading, then have another process that monitors that file to display on lcd or web page or whatever.  emit the data as csv or json or whatever format works best for the javascript side of things.

[...]

[William Phelps]

Ah, but I realize now in looking at it that this only updates when there is a new archive record, and I want something much more frequent, whenever weewx polls the Davis...

[mwall]

try this:

https://github.com/weewx/weewx/wiki/csv

it is now a packaged extension.  use the 'binding' option to determine whether you get LOOP packets or archive records.

m

[William Phelps]

One change:

        if self.binding == 'loop':
            self.bind(weewx.NEW_LOOP_PACKET, self.handle_new_loop)
        else:
            self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

How about an option to change it from appending to re-writing?  Maybe an "append = true"?

[mwall]

On Saturday, May 2, 2015 at 7:39:26 PM UTC-4, William Phelps wrote:

How about an option to change it from appending to re-writing?  Maybe an "append = true"?

[CSV]
    append = True
    header = True
 

[mwall]

try 0.3. 

the 'mode' option can be 'append' or 'write'.  the default is 'append'.

the 'header' option can be true or false.  default is true.

m

[William Phelps]

In csv.py:

    def write_data(self, data):
#        if self.header and not os.path.exists(self.filename):
#            self.write_line(self.sort_keys(data))
#        self.write_line(self.sort_data(data))
        flag = "a" if self.mode == 'append' else "w"
        with open(self.filename, flag) as f:
            if self.header and not os.path.exists(self.filename) or flag == "w":
                f.write('%s\n' % ','.join(self.sort_keys(data)))
            f.write('%s\n' % ','.join(self.sort_data(data)))

#    def write_line(self, fields):
#        flag = "a" if self.mode == 'append' else "w"
#        with open(self.filename, flag) as f:
#            f.write('%s\n' % ','.join(fields))

[William Phelps]

A little progress - nothing fancy yet, just some text, but just getting the screen to actually display something is a fair bit of work.  Now I can add to it - maybe a wind direction/speed display, barometer trend plot, who knows?  Certainly some buttons, at least, to be able to shutdown the Pi gracefully for example.

I have it updating the display at a 2 second interval.  From my plots it looks like this adds about 6% to the Pi's CPU usage.

[Vince Skahan]

very nice....

I have my Pi driving a cheapo Nokia 5110 which is fading visually after a few months of 24x7 on, but I only refresh once every 3 minutes via cron since the Pi is out in a detached building anyway.  I had a 16x2 LCD wired up previously, but that used a whole lot of pins on the Pi.

For your display, do you know if those are susceptible to burn-in on the display since you have a static text display ?

[William Phelps]

I do not know how susceptible to burn-in these LCD displays are, I guess we'll find out...  They're not terribly expensive, the Adafruit 2.8" PiTFT is $35.

I've added a wind gauge now.

[William Phelps]

I decided to try a different approach. Rather than write the loop data to a file, I'm now using IPC.  I've got an extension running that tries to connect to a local process, then it sends the data packet.  One advantage is that the data does not go thru a string conversion. Also, watching the display updating, you can see it's updating more frequently.  The process that updates the LCD display is running as a Server, with the weewx extension running as a Client.

Another approach would be to add a Server to weewx that responds to requests for data.  I'm not sure how to do this though so that it can access the most recent LOOP packet, so I think I'll just stick with what I have.

[Vince Skahan]

On Tuesday, May 5, 2015 at 6:04:38 PM UTC-7, William Phelps wrote:

I decided to try a different approach. Rather than write the loop data to a file, I'm now using IPC.  I've got an extension running that tries to connect to a local process, then it sends the data packet.  One advantage is that the data does not go thru a string conversion. Also, watching the display updating, you can see it's updating more frequently.  The process that updates the LCD display is running as a Server, with the weewx extension running as a Client.

Kill the local process your extension connects to and see if weewx aborts.   I know I messed my extension up quite a few times before getting the right try/except logic in it to handle the possible cases.

My setup is a little different, Pi with some DS18B20 sensors on it has a cron job writes a JSON-formatted file to tmpfs which the Pi's nginx webserver can make available.    Pi also drives a Nokia 5110 LCD display based on that data.  The weewx computer has an extension to grab the nginx-served data from the Pi.    My wifi here can be bouncy so I needed to refactor how my extension handles the remote webserver not being there periodically.

One of these days I'll find the neighbor who set his stuff to a whacked wifi channel who overlaps with the rest of the neighborhood's channels. 

[William Phelps]

On Wednesday, 6 May 2015 08:19:29 UTC-7, Vince Skahan wrote:

Kill the local process your extension connects to and see if weewx aborts.   I know I messed my extension up quite a few times before getting the right try/except logic in it to handle the possible cases.

I trap errors so it won't crash weewx:

    def send_data(self, data):
      try:
        conn = Client(self.address, authkey=self.key)
        conn.send(data)
        conn.close()

      except Exception, e:
#        no logging - too many log entries!
#        syslog.syslog(syslog.LOG_ERR, "wxdata: send error. %s" % e)
#        conn.close()

 

[Vince Skahan]

I'd be interested in what you wind up doing for a case.   I found this unrelated project on Instructables today, but the case presentation looked interesting for a Pi-based console http://www.instructables.com/id/Touchscreen-Internet-Radio-Raspberry/?ALLSTEPS - I know there are a million different cases out there, some native supporting the PiTFT.

[William Phelps]

I'm using the PiScreen and case from ozzmaker.com - I don't know if he still sells the case, I do not see it on the web site.  It's a simple design made from clear acrylic and doesn't fit very well.

The case in the link you shared does indeed look interesting. I like that it is free-standing and tilted.  It could probably be modified for the OzzMaker PiScreen.

[William Phelps]

It dawned on me today that the coolest thing about using IPC to communicate between weewx and my graphic display is that they no longer have to be on the same machine. I moved weewx back to the Beagle Bone Black that it was running on before I started this effort.  The Pi with the touch screen display can be anywhere I want it to be as long as it has a network connection.

[mwall]

On Thursday, May 7, 2015 at 3:01:30 AM UTC-4, William Phelps wrote:

It dawned on me today that the coolest thing about using IPC to communicate between weewx and my graphic display is that they no longer have to be on the same machine. I moved weewx back to the Beagle Bone Black that it was running on before I started this effort.  The Pi with the touch screen display can be anywhere I want it to be as long as it has a network connection.

william,

a weewx service that publishes all of the LOOP data via socket/http is pretty powerful. that would let you make the weather data from *any* type of hardware available to any number of clients, concurrently.  the trick is to design a schema/protocol (or use an existing one, if such exists?) that has legs.

m

[William Phelps]

On Thursday, 7 May 2015 03:10:28 UTC-7, mwall wrote:

a weewx service that publishes all of the LOOP data via socket/http is pretty powerful. that would let you make the weather data from *any* type of hardware available to any number of clients, concurrently.  the trick is to design a schema/protocol (or use an existing one, if such exists?) that has legs.

m

In my current version, the weewx service is the client and the display program is the server. Each time a LOOP packet is received, the service opens a client connection to a fixed ip address (could also be a hostname), does a send, and closes the connection. The server updates the display with the new data and then waits for another connection. This about as simple as it gets. 

It's really easy with Python's built-in Multiprocessing.  In python-to-python communication, you just send the LOOP packet variable across, no extra protocol or wrapper needed.

What I am thinking of doing is turning this around, with a Server running on the weewx side.  The only way I can think of to do this is to use two IPC's (InterProcess Connection).  The present weewx extension would receive the LOOP packet and send it to another process on the same machine.  That process would open up two server connections - one to receive the data from weewx, and a second to listen for data requests.  When it got a request it would send back the latest LOOP packet.  Anyone with the correct ip address, port number, and authentication key could request data from this server.  I have to do some testing to see if I can work out how to handle running two server processes; I'm thinking they need to be separate threads.

Is there another way to obtain the current LOOP packet or does this make sense?

William

[mwall]

On Thursday, May 7, 2015 at 12:14:17 PM UTC-4, William Phelps wrote:

What I am thinking of doing is turning this around, with a Server running on the weewx side.  The only way I can think of to do this is to use two IPC's (InterProcess Connection).  The present weewx extension would receive the LOOP packet and send it to another process on the same machine.  That process would open up two server connections - one to receive the data from weewx, and a second to listen for data requests.  When it got a request it would send back the latest LOOP packet.  Anyone with the correct ip address, port number, and authentication key could request data from this server.  I have to do some testing to see if I can work out how to handle running two server processes; I'm thinking they need to be separate threads.

Is there another way to obtain the current LOOP packet or does this make sense?

you could run a weewx service that spawns a thread on startup and registers itself for NEW_LOOP_PACKET events.  each time the service gets a new loop packet, it simply sets a 'current data' member of the thread to be a copy of the loop packet.  the thread simply listens for requests (on a socket, via http, whatever).  if you want to handle multiple clients, then have the thread spawn a separate thread to handle each request.

m

[William Phelps]

On Thursday, 7 May 2015 10:29:43 UTC-7, mwall wrote:

you could run a weewx service that spawns a thread on startup and registers itself for NEW_LOOP_PACKET events.  each time the service gets a new loop packet, it simply sets a 'current data' member of the thread to be a copy of the loop packet.  the thread simply listens for requests (on a socket, via http, whatever).  if you want to handle multiple clients, then have the thread spawn a separate thread to handle each request.  

That's pretty much what I'm doing now, using a modified version of your CSV service to capture the LOOP packets and send them on.  I'll modify that to have 2 threads, one to catch the LOOP packets and the other to be a server and answer requests for them.  Should be fun!

The goal of all this is to have an HDMI touch screen with a bunch of lovely weewx graphs on it and a current data display, with buttons to show day, week, month, and year graphs.  This will be set up in the Interp centers of a couple of local parks.

[William Phelps]

I got it working! For now it uses threading to spawn the server thread.  I want to try creating a process instead, but this works:  File wxdata.py attached - put in in /home/weewx/bin/user and add it as a process service.

It reads configuration settings from weewx.conf:

[WXDATA]
    binding = loop
    address = hhhhhhhh.local # default is "localhost"
    port = 6000
    key = "wxdata key"

If you're connecting from the same machine "localhost" works; to connect from another system you have to put the actual address (or hostname if the client is able to resolve it)

[William Phelps]

I tried using python's multiprocessing to create a queue manager process and a server process to handle the requests.  It was a PITA to get working, but I did finally sort it all out.  Not worth the effort.  It's a *lot* slower than my first solution with threading, and it chews up a significant chunk of the processor.  After a few hours the server quit responding, and the memory usage was triple what it had been.  Don't really need a queue for this simple task, so down the bit bucket for that one!

I do like the idea of the cllient sending in a request though, so I changed the threading version to work that way.  The client sends in a request for "wxdata" and the server responds with the most recent LOOP data packet.  I guess I might as well expand this so the server catches both LOOP and Archive data and makes them available, then we'll have a handy general tool for anyone to use.