Vantage Pro 2 output to CSV

[Todd Weiland]

Hi,

I was wondering if anyone has made weewx output to CSV file instead of a database.  I'm looking to just ingest the data into Splunk, but I can't seem to wrap my head around what looks to be binary output from the console.  I'm unable to have a SQL database in my environment :\ 

Any suggestions would be greatly appreciated.

Thanks,
Todd

[Todd Weiland]

This is what I get from the console.  Is this what others see?

LAMPS 0

OK
LOOP 1
LOOìu<ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ$»Åz
?

[mwall]

On Thursday, March 19, 2015 at 4:03:43 PM UTC-4, Todd Weiland wrote:

I was wondering if anyone has made weewx output to CSV file instead of a database.  I'm looking to just ingest the data into Splunk, but I can't seem to wrap my head around what looks to be binary output from the console.  I'm unable to have a SQL database in my environment :\ 

Any suggestions would be greatly appreciated.

by default weewx saves data using sqlite, and that is part of the standard python/weewx install.  is that not working for you?

fwiw, here is a weewx service that will spit out each new archive record as a row in a csv file.  but you have to have weewx working first.

put this code into a file csvwriter.py in the user directory.

import os
import time
import weewx
import weewx.engine

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):
        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):
            self.write_header(event.record)
        self.write_row(event.record)

    def write_header(self, record):
        with open(self.filename, "w") as f:
            f.write(','.join(self.sort_keys(record)))

    def write_row(self, record):
        with open(self.filename, "a") as f:
            f.write(','.join(str(self.sort_data(record))))

    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

    def sort_data(self, record):
        fields = [record['dateTime']]
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(record[k])
        return fields

put something like this in weewx.conf:

[CSVWriter]
    filename = /var/tmp/foobar.csv

[Engine]
    [[Services]]
        archive_services ..., user.csvwriter.CSVWriter

then restart weewx:

sudo /etc/init.d/weewx stop
sudo /etc/init.d/weewx start

or you could write less code and use the python csv module.

beware that this will not work properly with hardware that reports partial packets such as acurite or oregon scientific weather stations.

m

[Todd Weiland]

Hi mwall,

Thank you for responding.  My environment is such that I can't have a database (sqllite or mysql.)  I was trying to muddle my way through writing a driver to pull the data out and put it in a CSV file, but I'm having problems with binary being outputted and I'm unsure how to convert this over to ASCII.  I'll try the install of weewx again and see what kind of footprint is created. 

Thanks again!
Todd

[mwall]

On Thursday, March 19, 2015 at 5:24:16 PM UTC-4, Todd Weiland wrote:

Thank you for responding.  My environment is such that I can't have a database (sqllite or mysql.) 

todd,

could you tell us what kind of environment can run python programs but cannot have a sqlite database?

m

[Todd Weiland]

I can have them, it's just a regulatory pain.  I was trying to avoid it at all cost. I really only needed to output to CSV, but just couldn't make heads or tails of how to change the information coming out of the console into human readable.  Weewx is really cool and excellent programming, just didn't need all the bells and whistles.

[Todd Weiland]

Is it safe to say that the sqllite doesn't open ports and allow access from the outside?

[mwall]

On Friday, March 20, 2015 at 12:00:36 PM UTC-4, Todd Weiland wrote:

Is it safe to say that the sqllite doesn't open ports and allow access from the outside?

todd,

sqlite is built in to python.  a sqlite database is simply a file.  there is no network access to data in a sqlite database unless you add it.

fwiw, weewx does not listen on an ports either.

please post this 'no sql allowed' policy so we can all have a snarky laugh :)

m

[vince]

On Friday, March 20, 2015 at 9:00:36 AM UTC-7, Todd Weiland wrote:

Is it safe to say that the sqllite doesn't open ports and allow access from the outside?

A sqlite3 database is just a file in a particular binary format.

No processes running. No ports to open/close/worry about.

[Todd Weiland]

heh, thanks m :)  I'm in a govt environment which makes having any services a pain.  Everything has to be "locked" down and recorded in triplicate.  Sometimes it's just better not to have than have.  It sounds like this may work out well though.  I'm going to try and set it up next week and use the script you provided. 

[Todd Weiland]

So, I'm getting the following error:

Mar 23 11:19:44 holmes weewx[64063]: engine: Caught unrecoverable exception in engine:
Mar 23 11:19:44 holmes weewx[64063]:     ****  'CSVWriter' object has no attribute 'engine'
Mar 23 11:19:44 holmes weewx[64063]:     ****  Traceback (most recent call last):
Mar 23 11:19:44 holmes weewx[64063]:     ****    File "/usr/share/weewx/weewx/engine.py", line 831, in main
Mar 23 11:19:44 holmes weewx[64063]:     ****      engine = EngineClass(config_dict)
Mar 23 11:19:44 holmes weewx[64063]:     ****    File "/usr/share/weewx/weewx/engine.py", line 77, in __init__
Mar 23 11:19:44 holmes weewx[64063]:     ****      self.loadServices(config_dict)
Mar 23 11:19:44 holmes weewx[64063]:     ****    File "/usr/share/weewx/weewx/engine.py", line 141, in loadServices
Mar 23 11:19:44 holmes weewx[64063]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Mar 23 11:19:44 holmes weewx[64063]:     ****    File "/usr/share/weewx/user/csvwriter.py", line 10, in __init__
Mar 23 11:19:44 holmes weewx[64063]:     ****      self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)
Mar 23 11:19:44 holmes weewx[64063]:     ****    File "/usr/share/weewx/weewx/engine.py", line 277, in bind
Mar 23 11:19:44 holmes weewx[64063]:     ****      self.engine.bind(event_type, callback)
Mar 23 11:19:44 holmes weewx[64063]:     ****  AttributeError: 'CSVWriter' object has no attribute 'engine'
Mar 23 11:19:44 holmes weewx[64063]:     ****  Exiting.


I added csvwriter.py to /usr/share/weewx/user
I added the engine information and CSVWriter stanza.

Am I missing something?

[mwall]

On Monday, March 23, 2015 at 12:23:11 PM UTC-4, Todd Weiland wrote:

Am I missing something?

you're not missing anything, but apparently my coding skills are. i neglected to initialize the service properly.  add this line:

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):

        super(CSVWriter, self).__init__(engine, config_dict)

[Todd Weiland]

That worked perfectly.  Is there anyway to get more frequent updates, like 5 minutes apart, or is that a limitation of the archive feature vs the loop?

[Todd Weiland]

Nevermind, found it in the station information to change it.

[Todd Weiland]

To make new lines for the header and records, where would I put the '\n'.  I'm sorry for all the questions.  This is exactly what I needed.

Thanks,
Todd

On Thursday, March 19, 2015 at 4:13:24 PM UTC-5, mwall wrote:

[mwall]

On Monday, March 23, 2015 at 2:31:36 PM UTC-4, Todd Weiland wrote:

To make new lines for the header and records, where would I put the '\n'.  I'm sorry for all the questions.  This is exactly what I needed.

whoops!  another oversight.  try this:

    def write_header(self, record):
        with open(self.filename, "w") as f:
            f.write(','.join(self.sort_keys(record)))

            f.write("\n")

    def write_row(self, record):

        with open(self.filename, "a") as f:
            f.write(','.join(str(self.sort_data(record))))

            f.write("\n")

[mwall]

todd,

this is a bit more compact (i'm sure there are some python gurus who could do the sort_keys/sort_data in a single line):

import os
import time
import weewx
import weewx.engine

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):

        super(CSVWriter, self).__init__(engine, config_dict)

        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):

            self.write_line(sort_keys(event.record))
        self.write_line(sort_data(event.record))

    def write_line(self, fields):

        with open(self.filename, "a") as f:

            f.write('%s\n' % ','.join(fields))

    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

    def sort_data(self, record):
        fields = [record['dateTime']]
        for k in sorted(record):
            if k != 'dateTime':

                fields.append(str(record[k]))
        return fields

[Todd Weiland]

Awesome, thanks mwall for your awesome help and guidance.

[Jan Commandeur]

m and todd,

I also want use this pythonscript, but it gives an error

NameError: global name 'sort_data' is not defined


REC:    2015-04-19 07:47:53 CEST (1429422473) {'UV': None, 'outHumidity': 92.0, 'ptr': 384, 'outTempBatteryStatus': 0, 'radiation': None, 'delay': 5, 'inTemp': 15.900000000000004, 'windGustDir': 0.0, 'status': 0, 'barometer': 1017.7477300357414, 'rain': 0.0, 'pressure': 1017.5, 'rxCheckPercent': 100, 'rainTotal': 263.76, 'usUnits': 17, 'interval': 5, 'dateTime': 1429422473, 'windDir': 0.0, 'outTemp': 7.6000000000000005, 'windSpeed': 4.799999999431681, 'inHumidity': 53.0, 'windGust': 5.799999999313281}

Traceback (most recent call last):

  File "/usr/bin/weewxd", line 63, in <module>
    weewx.engine.main(options, args)
  File "/usr/share/weewx/weewx/engine.py", line 836, in main
    engine.run()
  File "/usr/share/weewx/weewx/engine.py", line 159, in run
    self.dispatchEvent(weewx.Event(weewx.STARTUP))
  File "/usr/share/weewx/weewx/engine.py", line 220, in dispatchEvent
    callback(event)
  File "/usr/share/weewx/weewx/engine.py", line 519, in startup
    self._catchup(self.engine.console.genStartupRecords)
  File "/usr/share/weewx/weewx/engine.py", line 634, in _catchup
    origin='hardware'))
  File "/usr/share/weewx/weewx/engine.py", line 220, in dispatchEvent
    callback(event)
  File "/usr/share/weewx/user/csvwriter.py", line 20, in handle_new_archive
    self.write_line(sort_data(event.record))
NameError: global name 'sort_data' is not defined


Op maandag 23 maart 2015 20:09:47 UTC+1 schreef mwall:

[Gary Roderick]

I think perhaps you have a typo, sort_data in line 20 should in fact be self.sort_data

Gary

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Gary

[Jan Commandeur]

Gary,

You gives me a good answer, but now I get this error:

TypeError: sequence item 0: expected string, int found

I use wh1080 driver
mysql database

Traceback (most recent call last):
  File "/usr/bin/weewxd", line 63, in <module>
    weewx.engine.main(options, args)
  File "/usr/share/weewx/weewx/engine.py", line 836, in main
    engine.run()
  File "/usr/share/weewx/weewx/engine.py", line 159, in run
    self.dispatchEvent(weewx.Event(weewx.STARTUP))
  File "/usr/share/weewx/weewx/engine.py", line 220, in dispatchEvent
    callback(event)
  File "/usr/share/weewx/weewx/engine.py", line 519, in startup
    self._catchup(self.engine.console.genStartupRecords)
  File "/usr/share/weewx/weewx/engine.py", line 634, in _catchup
    origin='hardware'))
  File "/usr/share/weewx/weewx/engine.py", line 220, in dispatchEvent
    callback(event)

  File "/usr/share/weewx/user/csvwriter.py", line 24, in handle_new_archive
    self.write_line(self.sort_data(event.record))
  File "/usr/share/weewx/user/csvwriter.py", line 29, in write_line

    f.write('%s\n' % ','.join(fields))

TypeError: sequence item 0: expected string, int found
jan@jan-901:~$


I used this script;

import os
import os.path

import time

import weewx
import weewx.engine

import weewx.archive

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):
        super(CSVWriter, self).__init__(engine, config_dict)

        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):

            self.write_line(self.sort_keys(event.record))
        self.write_line(self.sort_data(event.record))

    def write_line(self, fields):

        with open(self.filename, "a") as f:
            f.write('%s\n' % ','.join(fields))


    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

    def sort_data(self, record):
        fields = [record['dateTime']]
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(str(record[k]))
        return fields

Op zondag 19 april 2015 10:01:05 UTC+2 schreef gjr80:

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Gary

[mwall]

On Sunday, April 19, 2015 at 10:42:26 AM UTC-4, Jan Commandeur wrote:

Gary,

You gives me a good answer, but now I get this error:

TypeError: sequence item 0: expected string, int found

replace line 29 from this:

    def write_line(self, fields):
        with open(self.filename, "a") as f:
            f.write('%s\n' % ','.join(fields))

to this:

    def write_line(self, fields):
        with open(self.filename, "a") as f:

            f.write('%s\n' % ','.join([str(x) for x in fields]))

m

[Jan Commandeur]

This I found by Stack_overflow:

http://stackoverflow.com/questions/10880813/typeerror-sequence-item-0-expected-string-int-found

 f.write('%s\n' % ','.join(fields))
 

Replace

values = ",".join(value_list)

with

values = ','.join([str(i) for i in value_list])

OR

values = ','.join(str(value_list)[1:-1])


 
 be aware that some values could be unicode strings and therefore may cause the str to throw a UnicodeEncodeError error. In that case, replace the function str by the function unicode.
 
 values = ','.join([unicode(i) for i in value_list])


This is the output now it works

1429456680;None;1018.84459962;5;48.0;19.8;5;71.0;11.5;0;1018.6;480;None;None;263.76;100;0;17;90.0;2.69999999968;90.0;1.6999999998
1429456800.0;None;1018.7445756;0.5;48.0;19.8;5;71.0;11.5;0.0;1018.5;496.0;None;0.0;263.76;100.0;0.0;17;77.2921546908;2.6999999972;67.5;1.19999999876

this is the script, tanks mathew, gary

import os
import os.path
import time

import weewx
import weewx.engine

# import weewx.archive

i = 0

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):
        super(CSVWriter, self).__init__(engine, config_dict)

        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):
            self.write_line(self.sort_keys(event.record))
        self.write_line(self.sort_data(event.record))

    def write_line(self, fields):

        with open(self.filename, "a") as f:

#            f.write('%s\n' % ','.join(fields))
            f.write('%s\n' % ';'.join([str(i) for i in fields]))

    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

Op zondag 19 april 2015 16:42:26 UTC+2 schreef Jan Commandeur:

[Jan Commandeur]

The script, I hope now without errors.

import os
import os.path
import time

import weewx
import weewx.engine

i = 0

class CSVWriter(weewx.engine.StdService):
    def __init__(self, engine, config_dict):
        super(CSVWriter, self).__init__(engine, config_dict)

        d = config_dict.get('CSVWriter', {})
        self.filename = d.get('filename', '/var/tmp/data.csv')
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)

    def handle_new_archive(self, event):
        delta = time.time() - event.record['dateTime']
        if delta > event.record['interval'] * 60:
            return
        if not os.path.exists(self.filename):
            self.write_line(self.sort_keys(event.record))
        self.write_line(self.sort_data(event.record))

    def write_line(self, fields):

        with open(self.filename, "a") as f:
#            f.write('%s\n' % ','.join(fields))

            f.write('%s\n' % ','.join([str(i) for i in fields]))

    def sort_keys(self, record):
        fields = ['dateTime']
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(k)
        return fields

    def sort_data(self, record):
        fields = [record['dateTime']]
        for k in sorted(record):
            if k != 'dateTime':
                fields.append(str(record[k]))
        return fields

JanC






Op zondag 19 april 2015 17:30:54 UTC+2 schreef Jan Commandeur:

[mwall]

there is now a 'csv' extension:

https://github.com/weewx/weewx/wiki/csv