FilePile help please

88 views

Skip to first unread message

Colin Larsen

unread,

May 6, 2019, 11:05:29 PM5/6/19

to weewx-user

Hi all

I'm trying to get Filepile working but have come across an error that I don't understand. Any help appreciated. The database has been extended with fields to match those below in the data file - am I just missing spaces in the layout? Should it be AQI25 = 1.60 etc etc

Many thanks

This is my data file;

AQI25=1.60

AQI100=3.30

AQIIndex=6

AQICO2=683

This is my stanza in weewx.conf

##############################################################################

# This section is for FilePile

[FilePile]

filename = /home/pi/AQIData.txt

unit_system = METRIC

##############################################################################

This is my filepile.py (modified from original to suit)

import syslog

import weewx

import weewx.units

from weewx.wxengine import StdService

from weeutil.weeutil import to_float

weewx.units.USUnits['group_gas_concentration'] = 'ppm'

weewx.units.MetricUnits['group_gas_concentration'] = 'ppm'

weewx.units.MetricWXUnits['group_gas_concentration'] = 'ppm'

weewx.units.default_unit_format_dict['ppm'] = '%.0f'

weewx.units.default_unit_label_dict['ppm'] = ' ppm'

weewx.units.USUnits['group_dust'] = 'microgramm_per_meter_cubic'

weewx.units.MetricUnits['group_dust'] = 'microgramm_per_meter_cubic'

weewx.units.MetricWXUnits['group_dust'] = 'microgramm_per_meter_cubic'

weewx.units.default_unit_format_dict['microgramm_per_meter_cubic'] = '%.1f'

weewx.units.default_unit_label_dict['microgramm_per_meter_cubic'] = ' \xce\xbcg/m\xc2\xb3'

class FilePile(StdService):

"""WeeWX service for augmenting a record with data parsed from a file."""

def __init__(self, engine, config_dict):

# Initialize my superclass:

super(FilePile, self).__init__(engine, config_dict)

# Extract our stanza from the configuration dicdtionary

filepile_dict = config_dict.get('FilePile', {})

# Get the location of the file ...

self.filename = filepile_dict.get('filename', '/var/tmp/filepile.txt')

# ... and the unit system it will use

unit_system_name = filepile_dict.get('unit_system', 'METRICWX').strip().upper()

# Make sure we know about the unit system. If not, raise an exception.

if unit_system_name not in weewx.units.unit_constants:

raise ValueError("FilePile: Unknown unit system: %s" % unit_system_name)

# Use the numeric code for the unit system

self.unit_system = weewx.units.unit_constants[unit_system_name]

# Mapping from variable names to weewx names

self.label_map = filepile_dict.get('label_map', {})

syslog.syslog(syslog.LOG_INFO, "filepile: Using %s with the '%s' unit system"

% (self.filename, unit_system_name))

syslog.syslog(syslog.LOG_INFO, "filepile: Label map is %s" % self.label_map)

# Bind to the NEW_ARCHIVE_RECORD event

self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

def new_archive_record(self, event):

new_record_data = {}

try:

with open(self.filename, 'r') as fd:

for line in fd:

eq_index = line.find('=')

# Ignore all lines that do not have an equal sign

if eq_index == -1:

continue

name = line[:eq_index].strip()

value = line[eq_index + 1:].strip()

new_record_data[self.label_map.get(name, name)] = to_float(value)

# Supply a unit system if one wasn't included in the file

if 'usUnits' not in new_record_data:

new_record_data['usUnits'] = self.unit_system

# Convert the new values to the same unit system as the record

target_data = weewx.units.to_std_system(new_record_data, event.record['usUnits'])

# Add the converted values to the record:

event.record.update(target_data)

except IOError as e:

syslog.syslog(syslog.LOG_ERR, "FilePile: Cannot open file. Reason: %s" % e)

This is the syslog error;

May 7 14:50:16 raspberrypi weewx[6605]: engine: Caught unrecoverable exception in engine:

May 7 14:50:16 raspberrypi weewx[6605]: **** invalid literal for float(): 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683

May 7 14:50:16 raspberrypi weewx[6605]: **** Traceback (most recent call last):

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 890, in main

May 7 14:50:16 raspberrypi weewx[6605]: **** engine.run()

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 202, in run

May 7 14:50:16 raspberrypi weewx[6605]: **** self.dispatchEvent(weewx.Event(weewx.POST_LOOP))

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

May 7 14:50:16 raspberrypi weewx[6605]: **** callback(event)

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 574, in post_loop

May 7 14:50:16 raspberrypi weewx[6605]: **** self._software_catchup()

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup

May 7 14:50:16 raspberrypi weewx[6605]: **** self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, record=record, origin='software'))

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

May 7 14:50:16 raspberrypi weewx[6605]: **** callback(event)

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record

May 7 14:50:16 raspberrypi weewx[6605]: **** new_record_data[self.label_map.get(name, name)] = to_float(value)

May 7 14:50:16 raspberrypi weewx[6605]: **** File "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float

May 7 14:50:16 raspberrypi weewx[6605]: **** return float(x) if x is not None else None

May 7 14:50:16 raspberrypi weewx[6605]: **** ValueError: invalid literal for float(): 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683

May 7 14:50:16 raspberrypi weewx[6605]: **** Exiting.

Thomas Keffer

unread,

May 6, 2019, 11:13:43 PM5/6/19

to weewx-user

Hi, Colin

Does your text file use newlines ('\n') as line delineators? Perhaps it is using the MS-DOS standard of '\n\r'? Examine the file carefully and make sure it contains what you think it contains.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Colin Larsen

unread,

May 6, 2019, 11:23:28 PM5/6/19

to weewx-user

Hi Thomas

Thanks for the quick reply. It was just the format, it needs the spaces so each line in the text file is

AQI25 = 1.60

AQI100 = 3.30

AQIIndex = 6

AQICO2 = 683

Once I did that and checked the database after an archive period the fields were populated

Does my section with the units look valid? I've borrowed that from a similar service but I can't wrap my head around how the relationship between those and the data in the txt file get joined together

Thanks again

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com.

Colin Larsen

unread,

May 7, 2019, 2:56:46 AM5/7/19

to weewx-user

Well this is a little strange. This went fine for a while then stopped again with pretty much the same error. It appears to be missing the beginning of the first line in the file which is AQI25 = then gets the reading which is 4.55

invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863

Is the #015 a clue of any sorts?

The file looks ok

AQI25 = 4.55

AQI100 = 9.09

AQIIndex = 18

AQICO2 = 863

Error;

May 7 18:50:16 raspberrypi weewx[11960]: engine: Main loop exiting. Shutting engine down.

May 7 18:50:16 raspberrypi weewx[11960]: engine: Caught unrecoverable exception in engine:

May 7 18:50:16 raspberrypi weewx[11960]: **** invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863

May 7 18:50:16 raspberrypi weewx[11960]: **** Traceback (most recent call last):

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 890, in main

May 7 18:50:16 raspberrypi weewx[11960]: **** engine.run()

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 202, in run

May 7 18:50:16 raspberrypi weewx[11960]: **** self.dispatchEvent(weewx.Event(weewx.POST_LOOP))

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 574, in post_loop

May 7 18:50:16 raspberrypi weewx[11960]: **** self._software_catchup()

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup

May 7 18:50:16 raspberrypi weewx[11960]: **** self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, record=record, origin='software'))

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record

May 7 18:50:16 raspberrypi weewx[11960]: **** new_record_data[self.label_map.get(name, name)] = to_float(value)

May 7 18:50:16 raspberrypi weewx[11960]: **** File "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float

May 7 18:50:16 raspberrypi weewx[11960]: **** return float(x) if x is not None else None

May 7 18:50:16 raspberrypi weewx[11960]: **** ValueError: invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863

May 7 18:50:16 raspberrypi weewx[11960]: **** Exiting.

Colin Larsen

unread,

May 7, 2019, 3:01:19 AM5/7/19

to weewx-user

Apparently #015 is a numeric representation of \r or 'return' to we can rule that out?

Colin Larsen

unread,

May 7, 2019, 3:05:27 AM5/7/19

to weewx-user

And this is the code the writes the txt file - capturing a string via UDP

#!/usr/bin/env python

import socket

from shutil import copyfile

UDP_IP = "0.0.0.0"

UDP_PORT = 9886

fname = "/home/pi/AQIData.txt"

sock = socket.socket(socket.AF_INET, # Internet

socket.SOCK_DGRAM) # UDP

sock.bind((UDP_IP, UDP_PORT))

while True:

data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes

print "Received message:", data

dataSplit = data.split(',')

f = open(fname,'w')

for ds in dataSplit:

f.write(ds + "\r")

f.write('\n')

f.close()

gjr80

unread,

May 7, 2019, 4:23:49 AM5/7/19

to weewx-user

A few thoughts. If it works sometimes and not others there must be a problem with either the consistency of the format of the data or the algorithm being used to parse the data. Can't say I am a big fan of slicing in a situation like this, I think it can be a bit rigid at times. Maybe consider using split. To to obtain your name, value pair maybe something like:

            ...
            for line in fd:
                data = line.split("=")
                if len(data) != 2:
                    continue
                name,value = data
                new_record_data[self.label_map.get(name, name)] = to_float(value)
                ...

or cut out the middle man if you like:

            ...
            for line in fd:
                data = line.split("=")
                if len(data) != 2:
                    continue
                new_record_data[self.label_map.get(data[0], data[0])] = to_float(data[1])
                ...

Gary

On Tuesday, 7 May 2019 17:05:27 UTC+10, Colin Larsen wrote:

And this is the code the writes the txt file - capturing a string via UDP

#!/usr/bin/env python

import socket

from shutil import copyfile

UDP_IP = "0.0.0.0"

UDP_PORT = 9886

fname = "/home/pi/AQIData.txt"

sock = socket.socket(socket.AF_INET, # Internet

socket.SOCK_DGRAM) # UDP

sock.bind((UDP_IP, UDP_PORT))

while True:

data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes

print "Received message:", data

dataSplit = data.split(',')

f = open(fname,'w')

for ds in dataSplit:

f.write(ds + "\r")

f.write('\n')

f.close()

On Tue, May 7, 2019 at 7:01 PM Colin Larsen <colin...@gmail.com> wrote:

Apparently #015 is a numeric representation of \r or 'return' to we can rule that out?

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

Colin Larsen

unread,

May 7, 2019, 6:35:52 AM5/7/19

to weewx-user

Thanks Gary appreciate the thoughts, I'll try those out tomorrow

Colin

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com.

Thomas Keffer

unread,

May 7, 2019, 9:59:24 AM5/7/19

to weewx-user

You should terminate your lines with '\n', not '\r' when you write your file.

while True:

data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes

print "Received message:", data

with open(fname, 'w') as f:

for ds in data.split(','):

f.write(ds + "\n")

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CACjxfUtAX4SeZxkJWPZdtOSKFAOVaZY6-xJhUPmaAmTth5ATDA%40mail.gmail.com.

Colin Larsen

unread,

May 7, 2019, 2:13:01 PM5/7/19

to weewx-user

Hi Thomas,

Done :) and it's been stable overnight, thanks again for the advice.

Colin

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEA%3DdCv_XjMyW_9%2BmhX-gTmfH6FPwU7w5SpRMCZTLNQJMQ%40mail.gmail.com.

Reply all

Reply to author

Forward

0 new messages