Data continuity through power outage

[morr...@gmail.com]

We're just recovering from Hurricane Fiona here. My weewx server was off for 14 hours becasue of the power outage.

I'm thinking of caching readings in my DIY Moteino based weather station, or the base station which accepts data from it via 915MHz radios, and then bursting them to weewx when it has recovered.

Both weather station and base station are relatively low power. The weather station is already battery powered. I should be able to hack up a UPS for the base station so it will continue to receive data.

If I send time-stamped records to weewx well after the observation time has passed, will genLoopPacket() put the packets in the right archive record?

For example, my station sends a data record every 15 seconds. If weewxd is unavailable for 14*3600=50400 seconds, it will cache about 3000 packets. When weewx comes back, and runs genLoopPacket() on those 3000 packets as fast as possible, will the resulting archive records span the 14 hour gap in data? Or wil the data be squished into (depending on processing speed) one archive?

[Tom Keffer]

Off the top of my head, the one area of concern is that on startup the archiving process does check the wall time clock to figure out the bounds of the archive record it is working on. It might be possible to change that to using the timestamp of the first LOOP packet, but there's also likely to be lots of unattended consequences to doing so. 

Still, if the effort to set things up is not too expensive, I'd say, "Try it!" 

The normal way of doing a catch up is through genArchiveRecords() but, of course, that would require that you do your own archive record generation --- not an easy task.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/55b54cc3-051d-45a9-bcf0-f5c5b5ffb11an%40googlegroups.com.

[Bill Morrow]

On Sat, Sep 24, 2022, 20:47 Tom Keffer, <tke...@gmail.com> wrote:

Still, if the effort to set things up is not too expensive, I'd say, "Try it!" 

Ok! I should be able to set up a test system without too much trouble.

Just to be sure I understand correctly: the archive process' idea of timestamping is internal - it doesn't pay attention to the timestamp in a loop packet?

[Greg Troxel]

"morr...@gmail.com" <morr...@gmail.com> writes:

> We're just recovering from Hurricane Fiona here. My weewx server was off
> for 14 hours becasue of the power outage.
>
> I'm thinking of caching readings in my DIY Moteino based weather station,
> or the base station which accepts data from it via 915MHz radios, and then
> bursting them to weewx when it has recovered.
>
> Both weather station and base station are relatively low power. The weather
> station is already battery powered. I should be able to hack up a UPS for
> the base station so it will continue to receive data.

Beware that a UPS with no load doesn't necessarily last all that long,
but a battery with low drain will, compared to one running an unloaded
inverter.

Overall my impression is that you are heading down the path of
reinventing the Davis data logger. That's not a bad thing, but I think
realizing it and trying to have a similar scheme would enable at least
conceptual reuse.

I don't think you need to generate archive records on the moteino side,
but I think it is important to have it send something that looks like

time, temp, humdity, ....

and then have the code that receives it make rcords for that time,
rather than for now. This leads to needing clock sync in the moteino,
maintained when the computers are down.


I was only out 1h, which feels very lucky.

[Tom Keffer]

That's correct.

Before weewx starts asking for LOOP packets it figures out what the start and end of the archive interval will be. It uses this information to figure out when to stop asking for LOOP packets and, instead, ask for an archive record. See the call to _get_console_time() on line 594 in the function pre_loop(), which is expected to return a wall clock time.

An alternative would be to delay figuring this information out until the first LOOP packet has been received.

[morr...@gmail.com]

On Saturday, September 24, 2022 at 9:31:33 p.m. UTC-3 Tom Keffer wrote:

That's correct.

Before weewx starts asking for LOOP packets it figures out what the start and end of the archive interval will be. It uses this information to figure out when to stop asking for LOOP packets and, instead, ask for an archive record. See the call to _get_console_time() on line 594 in the function pre_loop(), which is expected to return a wall clock time.

An alternative would be to delay figuring this information out until the first LOOP packet has been received.

Gah, on further thought, this concept seems to be fighting against the design of weewx. Might be better to put  everything, including weewx, on a machine that is backed up by battery.

Move the last 3 bubbles from PC to a raspberry pi or old screenless laptop?

[Tom Keffer]

Absolutely. I use a 550 VA UPS, which lasts for about 4 hours. Over the 20 years I've been posting to the internet, I've hit that limit maybe 3 or 4 times. In those 3 or 4 cases, the archive ability of the VantagePro worked flawlessly.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/8a02f1c1-be66-45de-b1b2-f1f58d754f41n%40googlegroups.com.

[Greg Troxel]

"morr...@gmail.com" <morr...@gmail.com> writes:

> Gah, on further thought, this concept seems to be fighting against the
> design of weewx. Might be better to put everything, including weewx, on a
> machine that is backed up by battery.

I don't really see it as fighting the design. Yes it makes sense to
have a UPS, but if it's feasible to have batteries for a
microcontroller, for weeks even, and not a computer, having non-computer
storage makes a lot of sense. It's also helpful while you upgrade the
os and weewx, especially if you have trouble.

[Vince Skahan]

I guess I see a lot of issues starting with expecting wifi working, meaning yet more ups need.  Theoretically a ups backed m1 macbook air might do the job given its long battery life, but at that point just buy a vantage vue and data logger for similar price and rely on the vue console and logger to keep you collecting data through the outage.

Davis faq says a vue can run months with a usb or serial logger and C batteries, and a couple days with a weatherlinkip.  https://support.davisinstruments.com/article/9ombkjkvvv-how-long-does-the-battery-last-on-my-vantage-vue-or-vantage-pro-2-console-faq

.

[morr...@gmail.com]

I am leaning towards agreeing with you, Vince. So thinking of enhancing the software in my DIY remote station and driver to create archive records. When the outage is over, and the remote station and the base are talking again, load all archives required to fill in the gap.

I have no budget for new hardware. :)

[Vince Skahan]

Yep.  Blasted budgets.

If it was me, I'd try to move everything the esp8266 does to one pi zeroW with a nice big/fast SD card for storage and just do the read-radio-then-emit-mqtt-then-subscribe-in-weewx on one small computer. Quick google search found one library for the pi that might be worth a look for the radio (link).  MQTT is of course easy.

If you have (or can find) a zeroW you can get over a day with a 5000 ma usb power brick (link) or even a Adafruit power booster and any battery it supports.  Be careful though to get ones that can charge while discharging.  I bought the wrong Adafruit card a few years ago when I started fiddling with powering via battery.  That gets you down under $25 if you have a pi of course, given the supply chain issues.

[morr...@gmail.com]

On Tuesday, September 27, 2022 at 3:20:14 p.m. UTC-3 vince wrote:

If it was me, I'd try to move everything the esp8266 does to one pi zeroW with a nice big/fast SD card for storage and just do the read-radio-then-emit-mqtt-then-subscribe-in-weewx on one small computer. Quick google search found one library for the pi that might be worth a look for the radio (link).  MQTT is of course easy.

I actually originally used a Pi 2 for the radio receiver - MQTT publisher.  The radio code was a bit flaky, so I moved to the Arduino ecology. Going back to Pi and also running weewx on the Pi might work out OK.

I also have hundreds of Arietta single board computers from a surplus donation, which are basically a Pi ZeroW. I ran one of them off a decrepit 12V 10 AH LiPo battery pack during Fiona. It ran for 36+ hours with no drop in supply voltage.

Rethinking this problem as "weewx and everything upstream must be uninterruptible" moves it back to being a (simpler?) hardware problem.