Some missing data in Current Conditions on Standard skin

[bgra...@umw.edu]

Hello,

With lots of good help from the group, I have my Seeed CO2 monitor (LoRa) sending data to my weewx.sdb archive. I have weewx 4.5.1 (setup.py) on Focus Ubuntu.

The sensor sends data only every 60 minutes and my archive update is every15 minutes.

Looking at weewx.sdb, I have a reading recorded every hour but a null every 15 minutes when there is no new data.

co2 column

417.0

null

null

null

390.0

null

null

null

394.0

My settings in (Standard) index.html.tmpl are:

Current Conditions

      <td class="stats_label">CO2 level</td>

      <td class="stats_data">$current.co2</td>

Sometimes I get a data reading and others just "N/A"

My settings in weewx.conf:

[MQTTSubscribeService]

    # Turn the service on and off.

    enable = true    

    binding = loop

 [[[mytopic]]]

    ignore = true

 [[[[CO2_Value_PPM]]]]

     name = co2

     ignore = false

     contains total = false

     conversion type = float

Since everything seems to be working, the "$current.co2" may not be the correct form to use here as the display needs to retain the last value until it's replaced with a new reading coming every 60 minutes from the sensor.

Much appreciated if someone could point out my mistake in configuration here.

The display can be seen at: http://grattans.org/wx

Cheers,

Bob

[vince]

See if $last rather than $current helps (untested)

[storm...@gmail.com]

If Vince's suggestion does not help, try adding the following to your weewx.conf:

[Accumulator]

    [[co2]]

        accumulator = firstlast

        extractor = last

[bgra...@umw.edu]

I tried changing $current.co2 to $last.co2 but it only printed "$last.co2" on the web page.

Adding the  [Accumulator] section to weewx.conf only caused an error and exit when I restarted weewx. I tried adding this in several places but none seems to let weewx restart without an exit error.  Is there a special place to add this? Couldn't find anything in the doc about placement.  Thanks

[vince]

On Saturday, December 4, 2021 at 7:16:00 AM UTC-8 bgra...@umw.edu wrote:

I tried changing $current.co2 to $last.co2 but it only printed "$last.co2" on the web page.

Adding the  [Accumulator] section to weewx.conf only caused an error and exit when I restarted weewx. I tried adding this in several places but none seems to let weewx restart without an exit error.  Is there a special place to add this? Couldn't find anything in the doc about placement.  Thanks

Usual answers apply.

Unless you post the error messages verbatim and/or the changes you made to the skin, we are not going to be able to help you.

 

[bgra...@umw.edu]

Sorry, here is some info. This time, I re-inserted the [Accumulator] paragraph at a different place in weewx.conf and it restarted without an exit. The web data is still showing "$last.co2"
It ran for a while and then stopped. Debug file attached. Thanks

weewx.conf

# Options for 'MQTTSubscribeService'

[Accumulator]

    [[co2]]

        accumulator = firstlast

        extractor = last

[MQTTSubscribeService]

    # This section is for the MQTTSubscribe service.

    # Turn the service on and off.

    # Default is: true

    # Only used by the service.

    enable = true    # false

==================================================

index.html.tmpl

<tr class = "even">

                <td class="stats_label">105 Crawl Temperature</td>

                <td class="stats_data">$current.extraTemp1 / $current.extraTemp1.degree_C</td>

              </tr>

                <tr class = "even">

                <td class="stats_label">CO2 level</td>

                <td class="stats_data">$last.co2</td>

              </tr>

==================================================

Standard web page reads: (http://grattans.org/wx)

105 Crawl Temperature 72.0°F / 22.2°C

CO2 level $last.co2

[gjr80]

The issue here is that the ‘co2 field’ receives data only once per hour, so based on a 15 minute archive period only one in four archive records will have co2 data. Changing the accumulator type and extraction policy will not change this. I would leave the co2 accumulator settings out of weewx.conf. I doubt whether $last will be much help either; $last displays data from the last known good record in the database as opposed to the current record used by $current. If there is no co2 data  in most records then $last will provide a similar result.

You may have better luck with some sort of aggregate over the past hour, say something like (untested) $span($hour_delta=1).co2.avg. The actual aggregate is probably meaningless since you only have one value per hour so min be mx will work just as well.

Gary

[gjr80]

The other approach is to give WeeWX at least one co2 value per archive period (it can be the same value) but that is a function of your data source and the driver being used and I don’t know the limitations/capabilities of each.

Gary

[vince]

Bob - suggest you think about having your node-red (?) that is capturing the once/hour co2 data write out an intermediate file whenever that happens, and have weewx always read from that scratch file.  That should get you something that should look like a step function where you'd have no nulls, just four 15-minute periods with the same value in it, then four with the next value, and so on.

There are lots of ways for weewx to ingest data from external files nowadays that might be worth a look.  Write a scratch file out periodically, read it always.

Might be worth a try.

[storm...@gmail.com]

Bob

Do you have this sensor --- "SenseCAP Wireless CO2 Sensor – LoRaWAN"   By default it is set to upload data once an hour according to the manufacturer.  Can you change the data upload to 15 or 5 minutes for the sensor? 

[bell...@gmail.com]

Bob,

As noted earlier, now that you have a base MQTTSubscribe configuration up and running, you probably want to look at its 'archive record cache capability'. Details are here,

https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Configuring-additional-options#expires_after

I think that setting expires_after to none would be the simplest. So you would end up with something like this.

[MQTTSubscribeService]

[[topics]]

[[[mytopic]]]

ignore = true

[[[[CO2_Value_PPM]]]]

name = co2

ignore = false

expires_after = none

rich

[bgra...@umw.edu]

I've tried a few suggestions. The sensor output time can be changed to a shorter period but, at this point, I prefer to keep it at 60 minutes since the internal battery looks impossible to change easily. At 60 minutes it is supposed to last a little over 3 years.

I added the suggestion " $span($hour_delta=1).co2.avg" which seems to bring in data more often but there are still some periods of "N/A."

I added " expires_after = none" but for now that doesn't seem to do anything.

The suggestion of writing data to a scratch file may be the only way to go if I can't configure it another way.

I have produced a daily graph on the main page

The sensor is in the house for now and the peaks seem to come from using the oven and burners on the gas stove.

Thanks for all the help. I'll give this a day or so to accumulate some data.

[gjr80]

Suggesting use of $hour_delta=1 wasn’t wise on my part as data arriving once per 60 minutes will likely lead to some occasions where no data is displayed depending on exactly when the data arrives. Conceivably data arriving or being processed a few seconds late could fall outside of $hour_delta=1. 

Actually, thinking some more, you my be able to use the aggregate ‘last’ to get the last value in the aggregation period, so try $span($hour_delta=2).co2.last. I may be mistaken but I think the aggregates exclude the None values so this tag may give you the most recent non-None value in the last two hours. Refer to http://weewx.com/docs/customizing.htm#Tag_$span and http://weewx.com/docs/customizing.htm#aggregation_types.

Failing that you may like to try $span($time_delta=3900).co2.avg or 4200 to display the average over the last 70 or 80 minutes. Be aware though that just as $hour_delta=1 missed some records going the other way to > 1 hour will likely see more than one record be included in some aggregates, probably not an issue but something to be aware of.

Gary

[Tom Keffer]

Take the accumulator stuff out.

Why not just use $hour.co2.last?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/60583ac3-73f3-479c-91d0-34fcdf1c0865n%40googlegroups.com.

[bell...@gmail.com]

I'd go with Tom's advice, but if you want to get the cache working I'd need a log file with debug set to 2. It needs to have a few archive records before the first CO2 reading arrives via MQTT and a few records after. Warning, the file is going to large, very large.
rich

[bgra...@umw.edu]

It looks like " $hour.co2.last" did it, at least the last four hours are updating correctly with no N/As mixed in.

Many thanks to all who contributed help on this, and much appreciated is all the hard work that goes into weewx maintenance and updates by many!

Bob

http://grattans.org/wx

[bgra...@umw.edu]

Ok, I seem to have jumped the gun :(   

Things are much better but I'm still getting the N/A in place of data every few archive records (current data). My suspicion here is that the sensor is not sending on exactly 60 minute intervals. Looking at the actual times in the LoRa server, it's 60, 61 or 62 minute intervals--not always the same. Is it possible, using the basic  " $hour.co2.last" syntax, to change the "hour" to say 65 minutes? This would probably catch the changes from one hour. I have attached a file of what is in the weewx.sdb showing the missing data in certain instances. If the sensor has updated just after the archive time (15 minutes) then it misses recording new data.

I'm just guessing here at what might be going on. The graph seems to be picking up the data with no problem.

Thanks.

Bob

[gjr80]

All you can add to the $hour tag is an offset in hours ($hours_ago) but all this does is offset the one hour window, you will still likely see missing data as the window is still exactly one hour. 

Did you try my earlier suggestion to try $span with a slightly longer than one hour span? This increases the window beyond one hour.

Gary

[vince]

I'm not seeing a problem worth working at this point, but if it's supported (I don't know) you might try $day.co2.last and see if that makes any difference.

But personally I'd call it as "it is what it is" given the limitations of your setup providing updates so rarely.

[bell...@gmail.com]

My offer to debug why the cache is not working for you still stands.
rich

[bgra...@umw.edu]

Gary,

When I was trying various suggestions, I probably missed yours the other day. I have it installed now and, so far, it has gone through two cycles successfully. Before I call it this time, I’ll let it run for a day or so. The sensor has an internal clock or timer which may not be completely accurate for exactly 60 minutes. Or, there may be some other reason that causes the minute or so delay.

Much appreciated for the help—I’ll let you know how it goes.

Cheers,

Bob

[Tom Keffer]

No reason why you can't use $day.co2.last. It will give you the last valid value for the day.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/57504a16-555a-4123-a5dd-ae7b39861befn%40googlegroups.com.

[bgra...@umw.edu]

I'm sure everyone is tired of me and this topic but I'm trying to understand what is happening here.

I have switched to using " $day.co2.last" which should work for all hourly data sent except for maybe something very close to midnight.

 Looking at the contents of weewx.sdb I still notice gaps in the db. With weewx.log, I can see that a couple of missing data were sent correctly but:

 Dec  9 12:08:11 n4mrv weewx[448959] INFO user.MQTTSubscribe: (Service) TopicManager ignoring record outside of interval 1639069690.000000 1639069692.000000 1639069689.979975 co2: 423.0, dateTime: 1639069689.9799752, usUnits: 1

Dec  9 13:09:25 n4mrv weewx[448959] INFO user.MQTTSubscribe: (Service) TopicManager ignoring record outside of interval 1639073364.000000 1639073366.000000 1639073363.761206 co2: 415.0, dateTime: 1639073363.7612064, usUnits: 1

This happens for a couple of hours and the data is not included in the db.

On the third hour, the data is sent and correctly put into the db

Dec  9 14:10:37 n4mrv weewx[448959] DEBUG user.MQTTSubscribe: (Service) TopicManager data-> outgoing accumulated mytopic: co2: 400.0, dateTime: 1639077038.0, usUnits: 1

The attached file shows what is happening.

Thanks in advance for any suggestion where I'm going wrong here. There may be something I have incorrect in MQTTSubscribe.  I'll be happy to send more information if needed.

Bob

[vince]

I see this on my system all the time and just ignore it.  Sometimes it will work for 18 hours straight.  Sometimes there will be bursts where I get those messages for a few hours then they go away.

My gut feel is that it's related to my MQTT publishing system (an old model-B with a wifi dongle) only talking 2.4GHz and my local spectrum here just being a mess since I'm surrounded by multiple misconfigured neighbors.  Fix for me would be to either drop in a modern pi that does 5GHz wifi or go wired ethernet, but I'm not willing to devote a 'good' pi to a 'nice to have' outside sensor setup.

So I just ignore it rather than go crazy trying to never miss a reading.  It's not that important (here, for me).

[bell...@gmail.com]

Hi,
I think you have gotten bit by some code that needs to be improved. The code attempts to make sure that data arriving is neither too old nor to 'new'. Since your data does not have a timestamp, this check is a bit silly. To turn it off you can use the ignore_start_time = True and ignore_end_time = True options.  See, https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Date-Time-processing
So, you would end up with something like:

[MQTTSubscribeService]
    [[topics]]
        ignore_start_time = True
        ignore_end_time = True

        [[[mytopic]]]
            ignore = true
        [[[[CO2_Value_PPM]]]]
            name = co2
            ignore = false
            expires_after = none

rich

[vince]

Thanks Rich - I took a look at mine and my differences were literally a fraction of a second slow in some cases.  Pretty amazing.

Since then I have 'one' missed measure in 16 hours after ignoring the start/end times, so much much better so far :-)

Just so I can find it via search later on, here's the query I run to search for my external sensors which are mapped to extraTemp1 and the like.  The dateTime maps to around 6PM PST yesterday....

echo "select datetime(dateTime, 'unixepoch', 'localtime'),dateTime,extraTemp1,extraTemp2 from archive where (extraTemp1 is null and dateTime > 1639101300);" | sqlite3 /home/weewx/archive/weewx.sdb