Homemade weather station / Raspberry Pi

[Christopher McDonald]

I am building my own station from scratch.  The unit will be a Raspberry Pi, with sensors going directly into the GPIO ports, and into an MPC2008 ADC.  Because of this, I will just dump the data directly into a database, so I don't 'need' any of the front end code for reading the sensors (Davis, Oregon Scientific, etc).  I just need to pull the data from the DB and display it.  All of my own code will take care of the data logging.  

Has anyone else done this?  I always like to learn by the way others have done it instead of reinventing the wheel.  Any suggestions or nudges in the right direction would be appreciated.

[vds]

On Wednesday, May 15, 2013 1:02:52 PM UTC-7, Christopher McDonald wrote:

I am building my own station from scratch.  The unit will be a Raspberry Pi, with sensors going directly into the GPIO ports, and into an MPC2008 ADC.  Because of this, I will just dump the data directly into a database, so I don't 'need' any of the front end code for reading the sensors (Davis, Oregon Scientific, etc).  I just need to pull the data from the DB and display it.  All of my own code will take care of the data logging.  

Uncertain which part of it you're asking about, but here's one Pi article that maybe is of interest

http://www.adafruit.com/blog/2013/03/08/raspberry-pi-as-a-weather-underground-personal-weather-station-piday-raspberrypi-raspberry_pi/

 

[Christopher McDonald]

On Wednesday, May 15, 2013 7:59:59 PM UTC-6, vds wrote:

Uncertain which part of it you're asking about, but here's one Pi article that maybe is of interest

http://www.adafruit.com/blog/2013/03/08/raspberry-pi-as-a-weather-underground-personal-weather-station-piday-raspberrypi-raspberry_pi/

 

No, I have the data collection under control, I'm trying to get a visual representation of the data.  Most of the programs I have found want to be the data collection and visual generation of the data.  I'm trying to skip the collection step and just have the program display the data.... Kind of like permanently running in simulation mode, and having my code generate the simulation data, except that it's actual data.  Does that make sense?

[Thomas Keffer]

Some things that come to mind:

• Run the weewx utility wee_reports over and over again.
  
• I believe the weather program wfrog has separate data collection and display functions (but, I could be wrong).
  
• Take a look at Peter Finley's display program. 
  

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Christopher McDonald]

Thanks Tom.  I'll see about dumping the data into a format that wee_reports can just generate the pages on a 1 minute cron job.  If not, wfrog looks interesting.

[vds]

On Wednesday, May 15, 2013 7:13:27 PM UTC-7, Tom Keffer wrote:

• I believe the weather program wfrog has separate data collection and display functions (but, I could be wrong).
  

-tk

Tom - I took a quick look at the wfrog link and I have to say that frog is absurdly cute.

weewx needs a mascot for your logo.  Something wee.  Like a leprechaun perhaps ?

[Andrew Milner]

.... or even mannekin pis perhaps??!!

ouch - did I really suggest that??!!

[Thomas Keffer]

LOL! Any artists out there?

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Christopher McDonald]

I've been reading through the documentation and laying out my pseudo code....  Is there a list of the tables in the archive_mysql and stats_mysql?   I know they are generated once the system is fired up with the weather station running, but my RPi doesn't have a weather station, so weewxd wont start (except in simulation), so the tables aren't created......

I think I'm going to attempt to just put the data in the archive_mysql table and then let weewx just do it's thing, but I really need to know the 'default' tables and values...  Is it in any of the documentation and I'm just missing it?

On Wednesday, May 15, 2013 8:13:27 PM UTC-6, Tom Keffer wrote:

[Andrew Milner]

try /bin/user/schemas.py for the archive schema, and from memory each of the stats types corresponds to a stats table

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TdbIIG_P49g/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Christopher McDonald]

Thanks Andrew.  After poking around I found the schema.py file and then reading through some of the other code, found the wee_config_database and had it create the archive and stats DB's.   One step closer!

Now, for the driver part.....  I'm thinking (and I may be totally off base here) that I need to create a custom driver, lets call it PiWx ( the name of my project to make a Raspberry Pi a Wx station).  My theory is that since weewx wants to run with driver or a simulator, and I want it to run with neither, that I need to create the custom driver to look at the data and not worry about the data collection aspect.  Here is where I get lost and maybe you or Tom can point me in the right direction....

I want weewx to run as a daemon as it is designed to.  I want it to generate reports, (html, ftp, restful, etc) just as it normally would with a Davis Vantage or other station attached.  However, there is no external station to get the data from.  The data will just be residing in a db.  I just need to know if I should:

1.    Have my data in it's own database, let weewx query it and then generate the reports and archive, etc.
   
2.    Put my data directly into the weewx database ( This seems more logical. Clean data in the DB and then the driver would just get it and run the reports.  Most efficient route IMO)
3.    Do something entirely different since I really don't have a firm grasp yet on the complete logic flow of weewx, and there is a different way to do it that is much better.

Thanks for the input,

-Chris

[Christopher McDonald]

What is the prefered format of dateTime in stats.outTemp and other stats.* tables?  TIMESTAMP, seconds since the epoch?  hh:mm:ss DD:MM:YYYY?   Is it time.time(), or datetime.datetime()? Can I just alter all of the tables to use dateTime as AI as well as PK, NN, and UQ and let it timestamp automagically?  In my test database I was logging stats with the column 'datetime. as a datatype of DATETIME and it set as PK, NN, and AI.  

Or should I be asking Tom this?

On Thursday, May 23, 2013 2:34:09 PM UTC-6, Andrew Milner wrote:

[Thomas Keffer]

All times in the databases are in unix epoch time.

There are provisions for using other time standards (provided they can be expressed as an integer), but they have not been tested.

-tk

[Christopher McDonald]

Ok, so I'm inserting data into the STATS DB, but I can't run wee_reports because my config does not have a station type.  How do I build a 'null' station so that I can put data into the STATS tables, but no data collection?

[Thomas Keffer]

My memory is a little hazy on this (I'm away from home), but I think the station type is used solely for the "About this station" section in the templates. You can enter anything there.

Or, I could be wrong about this. :-)

-tk

[Christopher McDonald]

Nope,  if you set it to something other than Simulator in the /etc/weewx/weewx.conf file it then proceeds to blow up the weewxd because it firsts tries to open up a serial or USB port, and they don't exist, so it dies.....

I think maybe wfrog might be the answer.  I really like the layout of weewx better though.

wfrog has the same issue.  Will not run without a station attached.  Back to hacking weewx......

[Andrew Milner]

you could always create a simulator driver that did nothing!!

[Thomas Keffer]

Hi, Chris

Finally got home and had a chance to try this. When I set 

station_type = Foobar

in my weewx.conf I am able to run wee_reports just fine.

What do you mean by "blow up weewxd"? Can you please post a log.

-tk

[Christopher McDonald]

root@PiWx:~# weewxd /etc/weewx/weewx.conf

Traceback (most recent call last):

  File "/usr/bin/weewxd", line 68, in <module>

    weewx.wxengine.main(options, args)

  File "/usr/share/weewx/weewx/wxengine.py", line 864, in main

    engine = EngineClass(config_dict)

  File "/usr/share/weewx/weewx/wxengine.py", line 64, in __init__

    self.setupStation(config_dict)

  File "/usr/share/weewx/weewx/wxengine.py", line 82, in setupStation

    driver = config_dict[stationType]['driver']

  File "/usr/lib/python2.7/dist-packages/configobj.py", line 567, in __getitem__

    val = dict.__getitem__(self, key)

KeyError: 'Foobar'

root@PiWx:~#

I can't find a log file....

[Christopher McDonald]

Found the log.

On Saturday, June 1, 2013 7:52:19 AM UTC-6, Tom Keffer wrote:

[Andrew Milner]

Have you tried running just wee-reports manually rather than the full demon - which I think is what Tom said worked when he tried it. 

 

I think weewx itself will require simulator to be the device - since as you say it otherwise opens a device, whilst wee-reports just uses the same information on the 'about station' page and does not care what the device is..

 

I forget how you run reports manually though .. before you ask!!

[Thomas Keffer]

Hi, Chris,

Let's clarify what we're trying to do here. 

It was my understanding that you wanted to just run the report engine, wee_reports, which will not require a station type. You were going to get the instrument data into the databases by independent means.

However, you are running the full weewx daemon which, of course, will require a station type (and a corresponding driver). With this approach, weewx would put the data in the databases.

These will be two completely different approaches to how you're going to get your data into the databases. Once we settle that, we can proceed.

-tk

[Christopher McDonald]

I don't know WHAT I'm trying to do anymore....  8^).  This is what I originally thought when I first saw weewx:

I have python gathering data from my sensors.  It puts it in its own database called PiWx.

mysql> use PiWx

Database changed

mysql> show tables;

+----------------+

| Tables_in_PiWx |

+----------------+

| temperature    |

| wind           |

+----------------+

2 rows in set (0.00 sec)

mysql> select * from temperature order by datetime DESC limit 10;

+---------------------+------+----------+-----------+

| datetime            | temp | humidity | barometer |

+---------------------+------+----------+-----------+

| 2013-06-06 10:25:06 |   76 |       42 |     30.11 |

| 2013-06-06 10:20:03 |   76 |       43 |     30.11 |

| 2013-06-06 10:15:02 |   76 |       42 |     30.11 |

| 2013-06-06 10:10:02 |   76 |       42 |     30.11 |

| 2013-06-06 10:05:03 |   76 |       42 |     30.11 |

| 2013-06-06 10:00:03 |   76 |       42 |     30.11 |

| 2013-06-06 09:55:03 |   77 |       42 |     30.11 |

| 2013-06-06 09:50:03 |   76 |       43 |     30.11 |

| 2013-06-06 09:45:03 |   76 |       42 |     30.11 |

| 2013-06-06 09:40:10 |   76 |       43 |     30.12 |

+---------------------+------+----------+-----------+

10 rows in set (0.00 sec)

So my assumption was that weewx was doing something similar, and I could just have wee_reports look at my data and generate the pretty pages.  After actually reading the documentation and some of the code, I have come to the conclusion I should not assume I can look at someone elses program and guess the logic flow and that I can just change a few lines and make it do what I want.  Ah, the hubris of naive wannabe programmers!

So I asked about just running wee_reports with faulty assumptions.  I am know thinking I have two paths:

1.  Have my code put the data directly into the weewx.stats table and run wee_reports as a cron job only, not using any of weewx as a daemoned (I may have just made up a word!) program.  This might be the easier road for me and would probably allow me to upgrade without any major headaches.

2. Create a station type of 'external database'.  This might benefit others by allowing weewx to run fully as written.  It would just look at the external db (PiWx in this case) for it's data instead of a USB connected station such as a Davis, and then plug along as intended.  This might help others who have a non supported station, or just data in a database, and would also allow for upgrades to weewx, since I could submit it as a supported 'station'.

But, I do not have any idea which path to take.  Since you know the code better than anyone, point me in the right direction and I will take off running head first, hoping there are fewer brick walls in my path....

[Thomas Keffer]

Hard to say which is easier without knowing more about your homebrew station. However, if it fits in the general pattern of requiring a polling query at regular intervals, it's probably a lot easier to write a device driver and let weewx deal with the other details of putting things into databases, as well as regularly running a reporting service.

At its minimum, writing a device driver for weewx is dirt simple: you need only supply two functions and one of them is a one liner. I'd give it a try. 

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Christopher McDonald]

I've attached a block diagram of the design.  My Python code just listens on the GPIO pins for pulses from the anemometer or the rain bucket.  It has a ADC for the speed direction and an I2C interface for the temp/humidity/pressure sensor pack, so I get them digitally.  The code is very specific for the RPi since it uses the GPIO pins, but I'll post it anyway if anyone is curious.  It is documented inline.  The code is still fairly young as I am still building the sensor pack and have not yet finished the wind direction, but that will just be reading the value from the ADC and matching it to one of 16 possible values for direction.

Once my programs get the data, they dump it into a db called PiWx.  I though initially that would be easier than having it store in dictionaries or tuples to be read by weewx.  Also it would keep the data collecting from using too much memory and having to monitor the collection processes for performance.  I can just poll at whatever sensor I want, dump it to the PiWx db and then kill the polling program.  If it fails, no data, but the system doesn't hang.  The data process then fires again at the next interval.  I'm hoping that will make it more stable.

Could a device driver be an SQL query?  If so, I could just right a driver to pull the last n seconds from each table and feed that to weewx?

PiWx.pdf is the block diagram

temp.py is the temperature/pressure/humidity polling process

ppm_counter.py is the process for counting pulses from the anemometer. 

Thanks for all the help and insight Tom.

[Christopher McDonald]

I may be on to something.  I have read more and missed that 'stats' was the archive data and 'archive' was the current data.  I am now trying to push the data into the weewx.archive table, but I'm getting errors:

INSERT INTO weewx.archive (datetime, outTemp, outHumidity, pressure) VALUES ('1370639546.7','24.2','45.6','29.9236222435')

./temp.py:85: Warning: Field 'usUnits' doesn't have a default value

  cursor.execute("INSERT INTO weewx.%s (datetime, outTemp, outHumidity, pressure) VALUES ('%s','%s','%s','%s')" % (dbwx, curtime, temp, humidity, pressurein))

./temp.py:85: Warning: Field 'interval' doesn't have a default value

  cursor.execute("INSERT INTO weewx.%s (datetime, outTemp, outHumidity, pressure) VALUES ('%s','%s','%s','%s')" % (dbwx, curtime, temp, humidity, pressurein))

What are the usUnits and interval columns used for?  I searched the documentation and didn't find a field listing..... 8^/

[Andrew Milner]

1. usunits says if the values are US standard units (=1) or metric (=16).and interval gives the length of time period covered by the archive record (starting at datetime)

2. archive database contains values relating to current archive period, averaged over the period

3. stats database contains statistical data such as maximums and minimums accumulated/calculated from current loop packets.  If stats database is deleted a new one will be created which will still contain the stats data but accumulated/calculated from the archived data records before restarting to process loop packets.  The stats database provides quick access to max and min data without continually retrieving all archived records.  There is normally one stats record per day and many archived records per day

 

On a more general note it is obviously more efficient from the computer's point of view to pass loop data to weewx via a simple memory buffer/ variable mechanism rather than inserting into a database - with all the overheads involved in database access.

 

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TdbIIG_P49g/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

I wouldn't phrase it quite that way. At the end of the day, the archive database is the definitive database. You can think of the stats database as being an optimization strategy for producing the daily / weekly / monthly / yearly summaries. Indeed, I am working on an experimental version of weewx that does away with the stats database completely. It would do all queries out of the archive database. I haven't gotten far enough to tell if the performance will be acceptable and if it's going to work.

Oh, and one other thing would be given up: because the stats database is normally built from the LOOP data, it has the resolution of the value and time of mins and maxes down to the LOOP update frequency (normally a few seconds) rather than the archive update frequency (a few minutes). I'm interested in opinions whether that makes any difference to people.

Other points are as Andrew described them.

-tk

[Christopher McDonald]

On a more general note it is obviously more efficient from the computer's point of view to pass loop data to weewx via a simple memory buffer/ variable mechanism rather than inserting into a database - with all the overheads involved in database access.

Hmmmm..  I can see your logic in doing it that way, the only problem is... that is WAY past my programming level.  I can't even think of a starting point to start doing the block diagrams.  I don't suppose you know of an example (based on my previous posts with my code) that is similar that I could use as a starting point?  Are you saying something like run my polling python program and then ?  feed it to weewx?  On a scale of 1 - 10 of programmers, I consider myself a 2 or maybe a 3 on a good day.  What I'm thinking you are saying is that I need to be a 6 or a 7....

 

[vds]

On Saturday, June 8, 2013 9:43:02 AM UTC-7, Tom Keffer wrote:

because the stats database is normally built from the LOOP data, it has the resolution of the value and time of mins and maxes down to the LOOP update frequency (normally a few seconds) rather than the archive update frequency (a few minutes). I'm interested in opinions whether that makes any difference to people.

I tend to like the resolution we have now, especially for wind gust and rain rate, FWIW. 

[copi...@gmail.com]

Hello, I'm trying to do something very similar, how did the solution? thanks

El miércoles, 15 de mayo de 2013 22:02:52 UTC+2, Christopher McDonald escribió:

I am building my own station from scratch.  The unit will be a Raspberry Pi, with sensors going directly into the GPIO ports, and into an MPC2008 ADC.  Because of this, I will just dump the data directly into a database, so I don't 'need' any of the front end code for reading the sensors (Davis, Oregon Scientific, etc).  I just need to pull the data from the DB and display it.  All of my own code will take care of the data logging.  

Has anyone else done this?  I always like to learn by the way others have done it instead of reinventing the wheel.  Any suggestions or nudges in the right direction would be appreciated.

[Christopher McDonald]

Well.....  I finished writing my backend to collect the data, built the sensor pack, created my own databases and wrote my own front end.  I actually got everything mounted outside and up and running yesterday.  I'm nowhere near done with the frontend though.  

http://piwx.hamshadow.com

[copi...@gmail.com]

Thanks, I will be attentive to your web

[Louis Koffer]

Hi Christopher, 

I hope you don't mind I've picked up your ppm_counter code to experiment with an anemometer using python, and will hopefully log it to  database later.  I'm totally new to python though and I'm getting feedback saying invalid syntax at "if DEBUG = 0" line.  I would appreciate any support if you can, I realise this is going back a while.  Yours is the most useful information I've found yet, as a newbie.  I've checked your updated weather page too, looks great!  

Thanks, Louis

[Christopher McDonald]

I think you might have an older version if you are using the one I originally posted to this thread.  Try using this attached one.