Weewx crashes on FileParse errors

[Ron Walker]

I am using fileparse to capture incoming weather data from a file.  Watching syslog I notice that after a while fileparse starts to throw errors where it seems to be mixed up as to what is a name and what is a value.  For instance:

This is what the input record stream looks like:

Time=1560979190

windDir=225

windSpeed=0.7

windGust=6.7

windGustDir=0

outHumidity=100.0

outTemp=70.0

rain=0.02

pressure=29.43

supplyVoltage=4.35

lightLevel=0.00

Time=1560979192

windDir=248

windSpeed=0.0

windGust=6.7

windGustDir=0

outHumidity=100.0

outTemp=70.0

rain=0.02

pressure=29.43

supplyVoltage=4.35

lightLevel=0.00

After weewx runs for a while errors like this start appearing:

Jun 19 16:55:30 WeatherPi weewx[32206]: fileparse: cannot read value for '': could not convert string to float: l

Jun 19 16:55:41 WeatherPi weewx[32206]: fileparse: cannot read value for '': could not convert string to float: l

Jun 19 16:55:53 WeatherPi weewx[32206]: fileparse: cannot read value for '': could not convert string to float: l

Jun 19 16:56:52 WeatherPi weewx[32206]: fileparse: cannot read value for 'wind': could not convert string to float: windS

Eventually, weewx crashes with the following message:

Jun 19 16:56:52 WeatherPi weewx[32206]: engine: Main loop exiting. Shutting engine down.

Jun 19 16:56:52 WeatherPi weewx[32206]: engine: Shutting down StdReport thread

Jun 19 16:56:52 WeatherPi weewx[32206]: engine: StdReport thread has been terminated

Jun 19 16:56:52 WeatherPi weewx[32206]: engine: Caught unrecoverable exception in engine:

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****  'NoneType' object is not iterable

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****  Traceback (most recent call last):

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/engine.py", line 890, in main

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      engine.run()

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/engine.py", line 191, in run

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      callback(event)

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/engine.py", line 546, in new_loop_packet

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      self.accumulator.addRecord(event.packet, add_hilo=self.loop_hilo)

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/accum.py", line 256, in addRecord

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      func(self, record, obs_type, add_hilo, weight)

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/accum.py", line 314, in add_value

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      self[obs_type].addHiLo(val, record['dateTime'])

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****    File "/usr/share/weewx/weewx/accum.py", line 168, in addHiLo

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****      speed, dirN = val

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****  TypeError: 'NoneType' object is not iterable

Jun 19 16:56:52 WeatherPi weewx[32206]:     ****  Exiting.

In this case it is apparently seeing the value of the 'wind' as the name 'windSpeed'.  Can anyone help me understand why this would happen?  I've looked through my data file and I cannot see an obvious errors in the record structure.  Thanks in advance@

[gjr80]

Hi,

I would be looking very carefully at your data file, the first three errors you listed are a result of trying to convert an empty string to a float. The wind observation is a special vector type observation that WeeWX synthesises from a wind speed and wind direction. Given that ‘wind’ has also appeared in the errors I would be looking closely at your windSpeed and windDir values. Are they always numeric or can one or both at times be blank?

Also, have you defined a [[label_map]] under [FileParse] in weewx.conf? If so what does it contain?

Gary

[Ron Walker]

Hi Gary,

Thanks for your reply!  I have a filed map stanza in the csv.conf file as below:

[[FieldMap]]

        dateTime    = Time, unix_epoch

usUnits     = 

interval    = 

        pressure    = pressure, inHg

        outTemp     = outTemp, degree_F

        outHumidity = outHumidity, percent

        windSpeed   = WindSpeed, mile_per_hour

        windDir     = WindDir, degree_compass

        windGust    = WindGust, mile_per_hour

        windGustDir = WindGustDir, degree_compass

        rainRate    = rain, inch_per_hour

        rain        = DailyRain, inch

supplyVoltage = supplyVoltage, volt

radiation   = lightLevel, watt_per_meter_squared

I did not previously have a Label_map section int he weewx.conf file, but added it after your suggestion.  It is:

[[FieldMap]]

        dateTime    = Time, unix_epoch

usUnits     = 

interval    = 

        pressure    = pressure, inHg

        outTemp     = outTemp, degree_F

        outHumidity = outHumidity, percent

        windSpeed   = WindSpeed, mile_per_hour

        windDir     = WindDir, degree_compass

        windGust    = WindGust, mile_per_hour

        windGustDir = WindGustDir, degree_compass

        rainRate    = rain, inch_per_hour

        rain        = DailyRain, inch

supplyVoltage = supplyVoltage, volt

radiation   = lightLevel, watt_per_meter_squared

It just appears to me that the fileparse is getting out of synch in that it confuses labels for values and vice versa as in the example above where the value that is sees for 'wind' is actually the partial label for windSpeed.  There are no blank values in the data file.

Any other ideas?

[Ron Walker]

Sorry.  I pasted the FiledMap twice.  Here is the Label_Map.

[[Label_Map]]

Time = dateTime

pressure = pressure

outTemp = outTemp

outHumidity = outHumidity

windSpeed = windSpeed

windDir = windDir

windGust = windGust

windGustDir = windGustDir

supplyVoltage = supplyVoltage

lightLevel = radiation

rain = rainRate

DailyRain = rain

[gjr80]

Ron,

Since your data file field names are in fact WeeWX field names there is no need for a field map, I only asked because if you had one and it had some issues then it could be messing things up.

On looking again at the fileparse code I realise I mis-read it, though I am still sure that the issue is an occasional malformed data file. The fileparse driver is pretty basic, it splits each line at the '=', the string on the left of the '=' is the field name and the string on the right is the data. The data is converted to a float. If the data cannot be converted to a float WeeWX logs the offending field name and the error message. So in the case of this line:

Jun 19 16:55:30 WeatherPi weewx[32206]: fileparse: cannot read value for '': could not convert string to float: l

it tells us that fileparse tried to process the field '' (ie an empty string) but it could not convert the field data to a float with the error message being 'could not convert string to float: l'. That particular error message displays the offending data after the : and for some reason it has been truncated. Since it is a l (lower case L) I suspect that somehow the data file has been malformed and 'lightLevel' has somehow ended up on the right hand side of an '=' and since the corresponding field name is an empty string. Either that or the data file is badly mangled. I would say that at some stage your data file has a line:

=lightLevel

Have you been monitoring the data file when the error occurs?  You could try adding some debug code to fileparse.py, edit fileparse.py and add the highlighted line shown below:

def _get_as_float(d, s):
    v = None
    if s in d:
        try:
            v = float(d[s])

        except ValueError as e:
            loginf("d=%s" %(d, ))  
            logerr("cannot read value for '%s': %s" % (s, e))
return v

save the file then restart WeeWX. Next time WeeWX encounters that particular error it will dump the entire dictionary of data that it read from the file. Ideally we would like to see the data file but to do so would take a bit more code, this is a simple one liner that will give us he next best thing and may give us a clue as to what is going on with the data file.

Gary

[Ron Walker]

Hi Gary,

You are correct as far as the data file being the issue.  I confirmed this by creating a data file and then making sure that every record was complete.  Ran weewx using this data file and it proceeded with no issues.  I am using an arduino uno to collect the sensor data and then sending it out to a raspberry pi via serial port and saving it to a file.I have noticed that when the arduino starts sending data, it may send two or three lines of 'garbage" to start off.  Is there a way to ensure that those lines are discarded?  I am a newbie to python.

Also, I put that line into the fileparse.py file, but I did not notice any more info being given regarding the error in the syslog.

Ron 

[Ron Walker]

I have an update on this issue.  I modified the code that I'm using to retrieve the info from the arduino and write it to the file that fileparse is reading.  It now checks the length of the retrieved line and discards it if it is less than a certain length.  That seems to have eliminated the issue of writing incomplete or malformed records.  This has not eliminated the errors.  If I run the code to write to the file and then start weewx, approximately 560 so records are written to the file before the error starts showing up.  Is it possible that the reading and writing are somehow colliding?

[gjr80]

File access contention could be occurring, one would like to think RPi file system would handle that and not allow one or the other to access the file if it is in use. How likely is it that contention could occur, I guess it would depend on how often you write the file and how often you read it. I don't know anything of how arduinos write to a file or how your arduino code works but you might want to look at the structure of the code to make sure the file is 'written' as quickly as possible, eg rather than opening the file, polling a sensor, processing a one data point, writing that data point, polling the next sensor, etc do all the data gathering/polling up front and then open the file and write the data in one go. Another technique is to do an atomic write whereby you write the data to a temporary file and then once the temporary file is complete you copy the temporary file over the real data file.

You say your changes 'seem to have eliminated the issue of writing incomplete or malformed records', you might want to check that is indeed the case. You at least want to be able to see what the file was like when you get the error (what error do you get, 'cannot read value' or WeeWX exiting?). If the data file is written by the arduino so frequently that you cannot see the state of the file when the error occurs alter your arduino code to append the same data to another file. That way you can backtrack and find the state of the data file as it would have been when the error occurred. Remember the fileparse drive is fairly basic, whilst it will detect and deal with lines that have no data, malformed field names or a missing '=' it can still be tripped up.

One other thing, don't be afraid to post log extracts to show errors or other activity. It would have been helpful to see a log extract from the leadup to the 'approximately 560 records' error to the actual error. Often the logs may look the same or may appear to have no useful information but they often do. The good thing about the logs is their content lacks human interpretation/description.

Gary

[Ron Walker]

Hi Gary,

You were correct!!  The issue was with errors in my input file.  The main problem was how the data was being written to the file.  It was being committed to disk in chunks of what amount to about 360 lines, but it did not always write complete lines at the last record in the chunk.  I added a file.flush() command after each file.write() command to ensure that the data is written line-by-line. I was also able to add some code to check the incoming data and skip any faulty records.  Together those changes seem to resolve the issue.  It has been up and running for hours now.  Thanks again for your help!  You rock!

[gjr80]

Ron,

Good to see you got it fixed.

Gary