Graphs, databases and networking

[Bogdan Stochin]

So, would it be possible to add a basic realtime chart? There are many nice chart libraries for example Dislin is one I like: https://www.mps.mpg.de/dislin or Gnuplot

Or the online paid or free ones: Grafana (https://grafana.com/), Chart.js (https://www.chartjs.org/), Plotly (https://plot.ly/), Matplotlib (https://matplotlib.org/) for Python, etc.

Currently I am trying to use Python to graph the output in real time but the charts are blocking, in other words after ploting a set of points, it won't return.

My current goal is to save data and into a MySQL database and plot it live on a chart. I would like to chart the barometric pressure in particular to get a basic barograph indicating weather trends.

Example code writing to MySQL database is the  rtl_433_wrapper.py posted in this forum.

Here are some Python codes I stiched togheter from examples:

This script is passed to rtl_433 command line and transmits data over tcp to another client:

*****************************************************************************************************************

# import the socket library
import socket              
# next create a socket object
s = socket.socket()        
print "Socket successfully created"
# reserve a port on your computer in our
# case it is 12345 but it can be anything
port = 43261              
# Next bind to the port
# we have not typed any ip in the ip field
# instead we have inputted an empty string
# this makes the server listen to requests
# coming from other computers on the network
s.bind(('', port))       
print "socket binded to %s" %(port)
 
# put the socket into listening mode
s.listen(5)    
print "socket is listening"          
 
# a forever loop until we interrupt it or
# an error occurs
while True:
 
   # Establish connection with client.
   c, addr = s.accept()    
   print 'Got connection from', addr
 
   # send a thank you message to the client.
   c.send('Thank you for connecting')
 
   # Close the connection with the client
   c.close()






    #loads a line from rtl_433 console
    tjson = json.loads(line)
    dt = datetime.datetime.strptime(tjson['time'], '%Y-%m-%d %H:%M:%S')
    print (dt)
   
    if 'id' in tjson:   
        sensorID = int(tjson['id'])
        print(sensorID)  

        if sensorID == 231: #my sensors I want to process
            print('Found sensor 231')
            temp = float(tjson['temperature_F'])
            pressure = float(tjson['pressure_inHg'])
           
            x1 = [dt]
            y1 = [temp]           
           
            x2 = [dt]
            y2 = [pressure]           
           
        if sensorID == 243:#my sensors I want to process
            print('Found sensor 243')
            temp = float(tjson['temperature_F'])
            pressure = float(tjson['pressure_inHg'])

            x3 = [dt]
            y3 = [temp]           
           
            x4 = [dt]
            y4 = [pressure]           
           
        if sensorID == 15454:#my sensors I want to process
            print('Found sensor 15454')
            temp = float(tjson['temperature_F'])
            humid = float(tjson['humidity'])
           
            x5 = [dt]
            y5 = [temp]           
           
            x6 = [dt]
            y6 = [humid]           
           
       

    if 'temperature_F' in tjson:
        temp = "%.3f" % tjson['temperature_F']
        print('temp %s' % temp)
        #to do chart data
    else:
        print("no temp data")


    if 'humidity' in tjson:
        humidity = "%.3f" % tjson['humidity']
        print('humidity %s' % humidity)
        #to do chart data
    else:
        print("no humidity data")

*****************************************************************************************************************

This is the listening client script:

*****************************************************************************************************************

import socket # for socket
import sys
import sqlite3
import json


# Create a TCP/IP socket
try:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print ("Socket successfully created")
except socket.error as err:
    print ("socket creation failed with error %s" %(err))
    sys.exit()


# Bind the socket to the port
server_address = ('127.0.0.1', 43261)
try:
    sock.bind(server_address)
    print ('starting up on %s port %s' % server_address)
except:
    print ('Cant bind to %s port %s' % server_address)
    sys.exit()

# Listen for incoming connections
sock.listen(10)
FullMsg = ''
# Wait for a connection
Processing = True
while Processing:
    print ('waiting for a connection')
    (connection, client_address) = sock.accept()
    print ('connection from', client_address)
    Processing = True
    while True:
        data = connection.recv(4096).decode()
        print (data)
        if data == 'Exit':
            Processing = False
            break

        if data:
            print ('sending data back to the client')
            #FullMsg = FullMsg % data #cocantenate
            print (FullMsg)
        else:  
            print(FullMsg, client_address)
            print ('no more data from', client_address)
            break     
    Processing = False

print ("Exitin the listening thread")
# Clean up the connection
connection.close()
*****************************************************************************************************************

[Christian Zuckschwerdt]

I guess those examples are rather old. rtl_433 can output events in JSON format to network with -F syslog. Some example bridges and a short guide how to set things up for time series logging are on http://triq.net/articles/rtl-433-networking

I would recommend Graphite or InfluxDB as true TSDB and rather not use MySQL or Sqlite. Beautiful and versatile display is Grafana.

[Bogdan Stochin]

Thanks for the article.

[Bogdan Stochin]

So I looked into Graphite but it seems it only works on Linux and very complex.

Grafana looks promising.

But they all are an overkill in my oppinion. All I would like to see is a simple plot, time vs pressure. 

[DaveNF2G]

Maybe import into Excel spreadsheet and output graphically?

[Bogdan Stochin]

That solution again is not real time.

I would like a live graph, updating in real time.

[Bogdan Stochin]

With both, Grafana and Graphite, you have to write to a database first then have Graphite or Grafana query the database at preset intervals. That's how I understand these work

and they are ok, but why use 3 different types of software to do a simple task when rtl_433 could do it automatically?

I propose that rtl_433 make and write to a graph as the data comes in. I proposed Dislin chart control because it is free, easy to use, flexible. To add to a project all is needed is one
library and one include file, that's it.

[Greg Troxel]

Bogdan Stochin <bogda...@gmail.com> writes:

> With both, Grafana and Graphite, you have to write to a database first then
> have Graphite or Grafana query the database at preset intervals. That's how
> I understand these work
> and they are ok, but why use 3 different types of software to do a simple
> task when rtl_433 could do it automatically?

Because that's the unix way! But seriously, many people want to do many
different things. As it is, rtl_433 receives packets -- not even
suppressing duplicates -- and outputs them. That's a clean division of
effort; it is structurally easy to write something something that reads
the json and stores and/or graphs it.

> I propose that rtl_433 make and write to a graph as the data comes in. I
> proposed Dislin chart control because it is free, easy to use, flexible. To
> add to a project all is needed is one
> library and one include file, that's it.

It seems that currently rtl_433 does not require python.

But why don't you just have a python program that reads the json and
does what you want and pipe the json output to it? And then publish it
- or even have the script in contrib directory in the rtl_433 sources?
That seems far simpler than encoding how one person likes graphs into
the main progam, and making decisions about direct local graphing vs
MQTT vs other approaches.

[Bogdan Stochin]

It seems that currently rtl_433 does not require python.

Dislin is not a Python library, it's a C library, a component, a DLL, etc. That's why I proposed it, it could easily be added to the project

 

But why don't you just have a python program that reads the json and
does what you want and pipe the json output to it?  And then publish it
- or even have the script in contrib directory in the rtl_433 sources?
That seems far simpler than encoding how one person likes graphs into
the main progam, and making decisions about direct local graphing vs
MQTT vs other approaches.

I do have a script that does that but there is a snag, the graphs don't update in real time, the calls to the graph routine are blocking.

Notice I said "I propose", I did not make any decisions.

And MQTT is just another big mess.

I mean, really, do you need a server, a database and a client to plot 2 numbers on a graph?

 

[Greg Troxel]

Bogdan Stochin <bogda...@gmail.com> writes:

>> It seems that currently rtl_433 does not require python.
>
> Dislin is not a Python library, it's a C library, a component, a DLL, etc.

I see - I misread.

However, it is not Free Software.

> I do have a script that does that but there is a snag, the graphs don't
> update in real time, the calls to the graph routine are blocking.

It would seem that moving it into the rtl_433 process would only make
that worse.

> I mean, really, do you need a server, a database and a client to plot 2
> numbers on a graph?

Of course not - but good system design has clean, narrow, interfaces
when that makes sense. A stream of json messages describing received
frames is exactly that, and I don't see how putting what you want in the
main program is any better than just having another process that reads
that stream and does what you want.

[Bogdan Stochin]

It would seem that moving it into the rtl_433 process would only make
that worse.

Really, an additional thread would impact the main thread how?

> I mean, really, do you need a server, a database and a client to plot 2
> numbers on a graph?

Of course not - but good system design has clean, narrow, interfaces
when that makes sense.  A stream of json messages describing received
frames is exactly that, and I don't see how putting what you want in the
main program is any better than just having another process that reads
that stream and does what you want.

This is not a "system", this is just a program, it doesn't even have an interface.

[Christian Zuckschwerdt]

As said, currently the only option is to consume the JSON data (perhaps with the Syslog output) and then transform it as needed.

An API is already on it's way expect something to land in a few weeks. Then your suggestion is perfectly doable and even recommended: clone the rtl_433.c, change a few things, and add it as a new tool alongside the original rtl_433 executable.

Interfacing with other stuff is something we'll explore -- just not with the rtl_433 executable ;)

re Graphite, you only really need the Whisper database files (think Sqlite for TSDB, or something like RRD).