how to add new data from MQTT and include it into db

[IceNov]

Just started experimenting with weewx - and using the simulator with lighttpd serving up the data to a great looking web page on my rPi.

While I don't have a weather station yet, I've been trying to incorporate data from an mqtt service and have had partial success. I am using the MQTTSubscribeService.

I can see the mqtt data when running weewxd from the command line - it is one of the fields showing up in the terminal output - but it's not being added to the database.

I've also tried importing a csv file with timestamp and value - which reported success - but still no new column in the db.

How do I get a new measurement into the standard database?

Thanks

Tony

 

[gjr80]

For WeeWX to save data to the database two conditions must be met (1) the data must be present in a field in a WeeWX generated archive record (lines starting with REC: when running WeeWX directly) and (2) the same field must exist in the database. If these conditions are not met then data for that field will not be saved to database, whether the data is from a service/driver or an import. Getting data from new fields into the database is typically achieved by either re-purposing an existing, unused database field (ie you rename the archive record field to match the repurposed database field) or you add a new type to the database with the same name as the field in your archive record. 

I suggest you have a read of the Customising the database section of the Customization Guide, in particular the Modifying an existing database section.

Gary

[Peter Fletcher]

Your condition 2) is, of course, an absolute requirement. Your condition 1) isn't, strictly speaking. It is perfectly possible (and fairly common) for a Service triggered by the Archive record event to add (e.g.) a calculated value to the archive record being processed, which is then inserted in the database.

[gjr80]

Hmm, 'the data must be present in a field in a WeeWX generated archive record (lines starting with REC: when running WeeWX directly)', that includes any calculated/derived/added fields by any WeeWX service last time I checked. I stand by my previous post.

Gary

[vince]

On Sunday, July 16, 2023 at 12:17:47 AM UTC-7 IceNov wrote:

I can see the mqtt data when running weewxd from the command line - it is one of the fields showing up in the terminal output - but it's not being added to the database.

I've also tried importing a csv file with timestamp and value - which reported success - but still no new column in the db.

How do I get a new measurement into the standard database?

Fields don't magically appear in the database.  You have to create them yourself somehow.

Simplest way is to map the incoming MQTT topic to an existing field in the database. See the example at https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/individual-example for help.  Notice how topic/temp1 is mapped to extraTemp1 in the db ?

[IceNov]

Thank you for the responses. I was able to get the MQTT data into the database by creating a new field in the database as advised. But still no success with importing the CSV. I suspect it is a datetime/interval issue as I noticed lots of these in the logs:

Jul 18 13:51:00 newmqtt wee_import[17507] INFO weewx.manager: Non-positive value for record field 'interval': 0
Jul 18 13:51:00 newmqtt wee_import[17507] INFO weewx.manager: *** record ignored
Jul 18 13:51:00 newmqtt wee_import[17507] INFO weewx.manager: Added record 2023-07-18 13:44:30 AEST (1689651870) to database 'weewx.sdb'

I'll keep trying...

Tony

[gjr80]

The error says it all. WeeWX field interval cannot be 0. What is in your source data and your import config file?

Gary

[IceNov]

The source data is csv with file head:

"dateTime", "SoC"
2023-07-18 06:32:00, 38.0
2023-07-18 06:32:10, 38.0
2023-07-18 06:32:20, 38.0
2023-07-18 06:32:30, 38.0
2023-07-18 06:32:40, 37.9
2023-07-18 06:32:50, 37.9
2023-07-18 06:33:00, 37.9
2023-07-18 06:33:10, 37.9
2023-07-18 06:33:30, 37.9
2023-07-18 06:33:40, 37.9

and import config file attached, which was modified from the example file.

thanks

[gjr80]

The problem you are having is the short time between successive records. The interval field in the WeeWX database is the interval between archive records in integer minutes and it must be 1 or greater. In your case wee_import calculates the interval between successive records by calculating the difference in timestamps between successive records. Your records are 10 or 20 seconds apart and wee_import rounds this to zero. Hence the error message.

The records you are attempting to import look very much like loop packets (they occur frequently and at possibly different intervals) rather than archive records which are less frequent but regular (eg every five minutes). wee_import was designed to import archive records not loop packets. I'm guessing your source data is captured MQTT data that for one reason or another was not accepted by the MQTTSubscribeService. If you want to use wee_import to import your data you are going to have to thin out the records. I do not know much thinning out you need to do; maybe > 30 seconds between records will work, maybe you need no more than one record in any given minute. Like I said wee_import was not designed for such imports so consequently it was never tested against such data.   

Gary

[IceNov]

OK, thanks. I can thin the data out as it doesn't change that quickly. I should have twigged when I saw the comment in the config file that said interval in minutes!