Stdcalibrate, UV index, and mqtt data

[Jim Munro]

Can stdcalibrate be used to correct UV index? Weewx is getting the UV index from a mqtt subscription. Wxmesh is running as a service and my UV data shows up attached to the REC
data. I have the equation UV = UV - (UV * .16). Kind of a large correction but thats what my sensor is returning. Part of my weewx.conf

StdCalibrate]

[[Corrections]]
# For each type, an arbitrary calibration expression can be given.
# It should be in the units defined in the StdConvert section.
# Example:
#foo = foo + 0.2
UV = UV - (UV * .16)
...

data_services = user.wxMQTT.wxMeshService,
process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate,
archive_services = weewx.engine.StdArchive

REC data shows my UV reading but with no correction applied. Data is an index of 0 to 12
Guess the other option is to apply the correction directly to the sensor but it is easier to apply in weewx if it works. Any ideas or suggestions?

Thanks
Jim

[gjr80]

Hi Jim,

Correcting UV via StdCalibrate should certainly be working. One common cause of calibrations 'not working' is that something comes along after the calibration is applied and either adds alters the obs concerned. Does not appear to be so in your case. Could I ask for a couple of things. Could you do a wee_debug run and post the resulting report, just make sure to check the report for sensitive info/passwords before posting it, wee_debug does a reasonable job at obfuscating such info but it's not perfect. Could you also give us a link or such to the exact wxMQTT.py file you are using - not much fun troubleshooting an issue only to find you are on a different version/file :)

Gary

[Jim Munro]

Hi Gary,

Here is the wee-debug output. attached

Here is a link to my WxMQTT,py 

https://groups.google.com/forum/#!topic/weewx-user/rOGqDHFymMY

 

Just for interest sake, I tried to apply a calibration to my 1wire sensor and it appears to have worked.  I tried to apply a calibration to another MQTT OBS and it didn'r do anything

Jim

[Jim Munro]

I think the problem maybe in the driver. DateTime seems to get returned as a float. Should likely be integer. Can anybody suggest code to fix this?
Likely this is offending line in the driver.

_packet[self.label_map.get(vname, vname)] = _get_as_float(data, vname)

Thanks
Jim

[gjr80]

I'm not sure if this is the problem or not Jim (I presume you mean the service since you are using the wxMeshService rather than the wxMesh driver though both have the same line). I am more inclined to think that having the wxMeshService update dateTime and usUnits is more likely to cause a problem. That being said I think it may be prudent to make a few changes to the wxMeshService, I'll come back to that later. First a bit of background as to what should be happening, then we can see what is happening and hopefully work out where the issue is. As you are no doubt aware weeWX is designed to work with a single driver (specified in weewx.conf through the station_type config option and the subsequent driver config option in the 'station_type' stanza. To allow data to be added to loop packets and archive records from additional sensors/sources, weeWX allows any number of services to be run as 'data services' where, ideally, these data services add observational data to the loop packet or archive record. The remaining weeWX services that act on the augmented loop packets and archive records as if the packet and record data had come from a single driver.

A driver should be emitting packets and/or records with a number of set fields and any number of observational data fields. The set fields are dateTime (which holds the timestamp of the packet/record) and usUnits (which holds unit system info for the packet/record). The observational data fields can really be anything, they do not have to be part of the db schema (though only fields in the db schema will be saved to db) nor do they have to be numeric. Data services normally augment these packets/records. Whilst there is no technical reason why a service cannot alter fields already in the packet/record (just as StdCalibrate does), data services normally just add fields to the existing packet/record. Ideally data services should not alter the packet/record dateTime field, nor would they normally alter the packet/record usUnits field. Strictly speaking there is nothing wrong with altering the usUnits field but unless the data service also takes care of converting the existing packet/record observational data to the new unit system you run a very high chance of getting mixed units. The normal convention is for the data service to take note of the existing packet/record usUnits value, convert the data generated by the data service to the correct unit system (if required) and then augment the packet/record with this converted data.

If there is more than one data service then each is run sequentially and the resulting packet/record is then passed onto the remaining services for processing. So the likes of StdConvert, StdCalibrate etc get to work on the augmented packet/record with StdArchive accumulating loop packet data and ultimately saving archive records to db.

So in your case we should see the OWFS driver emit a loop packet, data service wxMeshService should then add some additional fields from the MQTT broker and StdCalibrate should adjust the UV value as per your formula. At the moment you have found the REC data (ie the archive record) UV value is uncalibrated. To get a better picture of what is going on I would suggest running weeWX directly with debug set to 1. This will give us visibility of the data coming from wxMeshService as well as showing us the resulting loop packets and archive records. To run weeWX directly with debug set to 1:

1. edit weewx.conf and set debug=1
2. save weewx.conf
3. stop weeWX if it is running and then run it directly
4. let weeWX run for at least 5 minutes
   
5. stop weeWX
6. weeWX should have been outputting LOOP and REC data to screen, LOOP is the resulting LOOP packet after all the services have done with it, REC is the archive record once everyone is done with it. Capture the last REC and a handful of LOOP packets leading up to the REC (more if you want) and post it back here
7. weeWX should also have been logging the data it obtained from your MQTT broker. Have a look at the log and take an extract covering the same time period as the captured LOOP and REC data. Post the log extract here.
8. Go back through the log to when you started weeWX directly and capture and post the entire startup sequence.

We should now be able to see exactly what is coming out of the wxMeshService as well as the resulting loop packets and archive records. Hopefully this will add to the picture.

I said I would come back to a few changes to wxMeshService. I would look at the following:

1. avoid overwriting dateTime in event.packet or event.record. Either don't include them in the dictionary you use to update event.packet or event.record or update event.packet/event.record manually.
2. make sure wxMeshService is usUnits aware. At the moment wxMeshService obtains a dictionary from the MQTT broker and the usUnits value is set to METRIC, presumably the MQTT broker emits data in weeWX METRIC units. Nothing wrong with that, but before the wxMeshService data is added to event.packet or event.record you should make sure event.packet/event.record and the wxMeshService data dictionary have the same usUnits values. If they are different I would convert the wxMeshService data to the same units as event.packet/event.record and then add the data. You may be fortunate and your OWFS may emit weeWX METRIC data but there is no guarantee that will be the case for others.

You have a complex setup and I guess the 'updating' of the dateTime field by wxMeshService could be causing some odd effect. I would try change 1. Change 2 is unlikely to have any effect on your current issue as UV index is unitless (or the same UV units are used in all weeWX unit systems), it just good coding to make the wxMeshService usable by a wider audience.

Somewhat long winded, sorry, but yours is a complex setup.

Gary

[Jim Munro]

Hi Gary,

Here are some debug/output files.  I will try some reconfigurations.  Present situation is StdCalibrate works on owfs sensor not mqtt ones.

Here are attached syslog and output records.

[Jim Munro]

Just a followup.  When I reconfigure my system StdCalibrate works on the loop data.  eg MQTT calibrations are working and owfs (running as a service) is broken.  Maybe my understanding of Std Calibrate is wrong. Does it work on both loop data and archive data.  The loop data the was obtained successfully appears in the archive data but the data obtained from running owfs as a service does not.  I realize that different code paths in the drivers are occuring but it just seems odd.  Maybe my understanding is wrong.

Jim

[gjr80]

Hi Jim,

When you say 'reconfigure my system' what changes did you make? Looking at you LOOP/REC data it is clear that MQTT data is only being added to archive records (ie the REC entries). This morning I suspected that that reason for StdCalibrate not being applied to the archive record/REC data was that record_generation is set to 'software' in weewx.conf but in fact it is set to 'hardware' (when using software record generation StdCalibrate is skipped on archive records as the archive records are based on accumulated loop data which has already had StdCalibrate applied - to do so again would be wrong). I am wondering if there are a number of configuration errors here that are compounding to cause some strange effects (hence my question as to what reconfig did you do).

If it were me I would reconfigure/setup as follows:

1. set record_generation = software under [StdArchive] in weewx.conf
2. use OWFS as a driver with a loop binding with whatever polling_interval you want, 60 seconds sounds good (Forgetting about MQTT, if you ran your system like this you should have a working system producing loop packets based on your OWFS sensors and software generated archive records. Now we add in the MQTT data as a data service.)
3. set [wxMesh] in weewx.conf as follows (assumes you are using username=test, password=password and clientid=weewx_mqttc, if different set them as well):

[wxMesh]
    host = localhost
    topic = weewx
    binding = loop
    [[label_map]]
        outsideTemp = outTemp
        outsideHumidity = outHumidity
        insideTemp = extraTemp1
        pressure2 = pressure
        UV = UV

See how that goes, again run weeWX directly so you can see the LOOP and REC data. If you still run with debug=1 you will see the raw MQTT data in the log. Overwriting the loop packet dateTime in the wxMeshService may still be an issue, it is something I would remove, but let's see what this does. I also think you will have a units issue (MQTT appears to be emitting US customary units yet the packet usUnits field is being set to METRIC), but lets get the data flows working properly then we can worry about the correctness of the numbers.

Gary

[Jim Munro]

Thanks Gary,

   I can confirm your recommended configuration settings work.  The setting that got MQTT working with StdCalibrate  was binding = loop.   I had binding = archive.  The driver still needs more work as I was still getting mixed results with it.

Thanks again

Jim