Ecowitt Gateway Driver, WH57 and lightning data

[michael.k...@gmx.at]

How are lightning strikes and lightning distance calculated?

- How does the WH57 do it?

- How does the Ecowitt Gateway Driver do it?

Lightning distance and lightning strike count aren't independent values when you don't look at the events individually, but over a observation period:

- WH57 has an Interval of 79 seconds. Within 79 seconds many lightning events can occur. When the Sensors emits a new value, is the distance weighted? If not, any further calculations are pointless anyway. (Maybe Rainer Lang knows the answer or knows someone who knows the answer)

- weewx has an archive_interval, does the driver weight the distance/count or is the value for the interval just the average of the individual distance values?

For the basic topic see also https://github.com/weewx/weewx/pull/734 

[Rainer Lang]

That's rather a question for the wxforum https://www.wxforum.net/index.php?board=111.0
it's not really a weewx question

In a way the Ecowitt gateway driver is dumb - it just asks the console (GW1x00, GW2000, WH2650, WN19x0) for information, receives it and puts it into the weewx loop.
Depending on some settings in weewx.conf, weewx does some own calculation but nothing regarding the WH57.

If you want to know what the driver receives in detail, look up https://www.wxforum.net/index.php?topic=40730.0
and read the section about the Ecowitt Gateway driver.

Basically, in a nutshell, the WH57 itself already provides all this info to the console - the console just records it and hands it over to the weewx Ecowitt Gateway driver on request.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9aa91a43-55e5-4065-9039-529d33464609n%40googlegroups.com.

[Rainer Lang]

Google is your friend (most of the time  😎)

read https://www.wxforum.net/index.php?topic=40093.msg460073#msg460073

It's generally good to look for answers regarding Fine Offset/Ecowitt (clone) hardware in the wxforum https://www.wxforum.net/index.php?board=111.0
(and for German speaking people - gmx.at looks like Austria - also in https://www.wetterstationsforum.info/index.php and their related WiKi for Ecowitt stations) 😉

[gjr80]

I really don't see the point of trying to weight lightning strike distance from a WH57. In essence the WH57 reports three things; the number of lightning strike detected today, the time of the last detected lightning strike and the distance of the last lightning strike. This is clearly stated in the WH57 manual. The Ecowitt Gateway API is not so specific (it omits the word 'last'), but the API reports the same data as displayed by the Ecowitt's WSView Plus app so I think it is safe to assume the API reports time and distance of last strike. So if a given number of lightning strikes occur in a period the Ecowitt gateway driver and any connected console/app will only see time and distance data for the last detected strike, nothing for any earlier strikes.

The referenced PR does not state what type of lighting detector is used, but the code looks suspiciously Ecowitt gateway driver'ish and for the reasons outlined above I really don't see any value in weighting light strike distance.

Gary

[Rainer Lang]

Hi Gary,

I think you know that by saying that your driver were "dump" I didn't want to discredit your work. 😉 - Just that the work is done by the console (or earlier in the chain).

The WH57 provides only the number of strikes since the last reporting including the distance of that last strike. In an interval of 79 seconds which starts with the first detection.
It's the console which does the counting and adds the EPOCH time stamp as the WH57 has no internal clock which is gauged against current real time.
From there the API response from the console is number of strikes, time stamp of last strike, distance of last strike (which will be saved in the console after a thunderstorm until the next lightning detection).
The number of strikes (strike count) is reset to zero by the console at midnight. That's why it keeps on sending, count 0, distance last strike, time stamp last strike after a day with lightnings and no new lightning detected past midnight.
The detection, counting etc. is done by the AS3935 IC inside the WH57 (https://www.sciosense.com/products/wireless-sensor-nodes/as3935-franklin-lightning-sensor-ic/ ).
As for weighting, I don't see a reason for that either.

Rainer

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/bcd2d06b-bfb6-48bd-b290-d8710a869ddan%40googlegroups.com.

[gjr80]

The issue is not whether the driver is 'dumb' or what the console does nor over what period. The fact of the matter is that WH57 only provides the distance of the last detected strike, not the average, not the sum nor any other aggregate. That distance has no relationship whatsoever with any earlier strikes so there is no value in weighting the reports distance value. It is simply the distance of the last detected strike, nothing more.

Gary  

[michael.k...@gmx.at]

So for the WH57, any further calculations are pointless.

The referenced PR is for a Tempest, as far as I know, the Tempest reports every singly lightning detection as an event immediately, as an addition to the packet containing all the values every minute.

[michael.k...@gmx.at]

I am just now in the middle of this Storm with stroboscopic-like lightning flashing.

I recognized my HP2551 is obviously showing lightning strikes in realtime. Every one or the other second.

Then I checked back in the manual and now disagree with gjr80, when he states:

gjr80 schrieb am Mittwoch, 21. Juni 2023 um 22:32:42 UTC+2:

In essence the WH57 reports three things; the number of lightning strike detected today, the time of the last detected lightning strike and the distance of the last lightning strike. This is clearly stated in the WH57 manual. 

Gary

From the manual:

2.1 Features Lightning Detector 

• Detects lightning bolts and storms within 25 miles (40 kilometers) 
• High or low sensor sensitivity selectable to meet different requirements.
• Long wireless range up to 330 feet (100 meters) in open areas
• Transmits readings every 79 seconds
• Easy installation includes hanging hole  

When paired with a GW1000 Wi-Fi Gateway:

• Monitor number of strikes daily, and the time & distance of the last strike detected within a 25-mile radius of your location on the Live Data page of the WS View app 6 (requires the gateway and your phone is using the same Wi-Fi network)
• Battery power level display on the WS View App

When paired with a Weather Station Console (HP2551/HP3500/HP3501):

• View lightning data in real-time on the Display
•  Get alerted to lightning strikes with the flashing lightning icon

When uploaded to Ecowitt Weather Server: 

• View lightning data & history records & graph on the website
• Receive email alerts from the server
• Remote monitoring with smart phone

So, there is different behaviour depending on the hardware the sensor data is received, which cannot be since the sensor doesn't know who is listening to. The described features are not sensor features but features on the receiving side. Also, there is a contradiction in "Transmits readings every 79 seconds" and "View lightning data in real-time on the Display". So, whatever is stated in the manual is not very clear, nor is it matching the witnessed reality.

If there is a possibility to extract a singled detected lightning event and it's data with standard devices, is another story. If so, I'd see the point of doing so. At least the minimal distance and weighted average distance are interesting values to me. I doubt it is possible doing this with the "Ecowitt Gateway Driver", I have no idea if it is possible with the interceptor driver and a WH2551, it should be possible using SDR.

[Greg Troxel]

"michael.k...@gmx.at" <michael.k...@gmx.at> writes:

> I am just now in the middle of this Storm with stroboscopic-like lightning
> flashing.
>
> I recognized my HP2551 is obviously showing lightning strikes in realtime.
> Every one or the other second.
>
> Then I checked back in the manual and now disagree with gjr80, when he
> states:
>
> gjr80 schrieb am Mittwoch, 21. Juni 2023 um 22:32:42 UTC+2:
>
> In essence the WH57 reports three things; the number of lightning strike
> detected today, the time of the last detected lightning strike and the
> distance of the last lightning strike. This is clearly stated in the WH57
> manual.
> Gary

This sounds very much like the chip that is in the Acurite 6045M

> If there is a possibility to extract a singled detected lightning event and
> it's data with standard devices, is another story. If so, I'd see the point
> of doing so. At least the minimal distance and weighted average distance
> are interesting values to me. I doubt it is possible doing this with the
> "Ecowitt Gateway Driver", I have no idea if it is possible with the
> interceptor driver and a WH2551, it should be possible using SDR.

Certainly it would be great to log the bits and study them. The 6045M
only transmits periodically, with a longer period without lightning and
then faster in 'active mode'. I suspect the chip reports immediately,
and there is a different radio in the WH57. There is no reason why it
couldn't be set up to send promptly, other than battery life.

[michael.k...@gmx.at]

My best guess is that the Sensor reports every 79s "Hello, I'm there" and every detected lightning on top of that, immediately. If you don't live at Catatumbo River mouth, this shouldn't drain the battery too much. Let's see if Rainer Lang knows more Details. 

[michael.k...@gmx.at]

Probably the only way getting reasonable data is with SDR.

[Rainer Lang]

"If there is a possibility to extract a singled detected lightning event and
> it's data with standard devices, is another story. If so, I'd see the point
> of doing so. At least the minimal distance and weighted average distance
> are interesting values to me. I doubt it is possible doing this with the
> "Ecowitt Gateway Driver", I have no idea if it is possible with the
> interceptor driver and a WH2551, it should be possible using SDR."

what part of transmitting every 79 seconds has not been understood ?
And the 79 seconds start with the first detected discharge.

Why would SDR receive more and more often ? Magic SDR ?
SDR is nothing else but a radio receiver as is a console.
Just by changing the console the sensor won't post faster.

And it's not an Acurite chipset (maybe some Acurite device uses the same chip);
it's an AS3935 - before starting wild speculations I would read the specifications and see if the chip can be animated to provide data more frequently.
Then you can try your luck modifying the sensor firmware if that's possible and not limited by the chip itself ...

And have a look what exactly the chip provides/can provide ...
check if it provides such things like minimal distance, weighted distance etc.
Nice wishlist, but as far as I have understood the specifications, it doesn't do that.
Admittedly, I'm not an expert here - somebody else may be able to extract more information

Google can be your friend - look up the AS3935 specifications/data sheet etc.

But as for the WH57 as manufactured, it will transmit every 79 seconds ....

And it has nothing to do with the Ecowitt Gateway driver or any other driver - you can have it poll every two seconds.
But it can get only what the console has available to send. And the console only has what it receives from what was transmitted.

Maybe it would be helpful to understand how weewx works ...

Classically the data received from a driver or extension is collected in a loop where data gets accumulated.
That doesn't seem to be what you want. You seem to want every single data-item to be recorded separately.
Then you will also have to write the code which does this and save the data in a way you can reuse it - i.e. write your own extension.
You'd probably need your own separate lightning database for that too.
In the typical way weewx works, collects and archives data that won't work as far as I can see.

By the way - if there are no more discharges within 79 seconds, your request above will be fulfilled 😉

all the sensor related information could have been found and read in the Ecowitt WiKi
https://meshka.eu/Ecowitt/dokuwiki

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/437ddc4e-27e4-409c-952d-4555f8a855d7n%40googlegroups.com.

[michael.k...@gmx.at]

Rainer, the WH57 sends lightning data as the lighning is detected, the console shows this data immediately.

I observed this mutliple times:

You see the lightning out there, the LED of the WH57 flashes red, the counter on the console goes up, the distance is also updated. If there is a lightning just a few seconds later, the same game again. Even when polling the console API in an interval < 79s, you get the lightning data updates more frequently than 79s.

Wait for the next storm and see. I have no doubt that with SDR you can receive these data, as it's sent by the WH57. 

Your information does not fit the observed reality. Whats true is that you need to store lightning data in a different format, to calculate a meaningful value for the archive record. Something like the count per interval and a kind of weighted average for the distance. It exists for the Tempest, which transmits lightning data using UDP as it detects a lightning. Apart from the other data which is transmitted in a constant interval.

[vince]

To me the most likely explanation is that the display displays lightning hits in realtime as events, and also includes aggregated lightning (over that observation period) when it periodically reports all data from all sensors.

This sounds like it is the same as the Tempest which reports lightning 'events' in realtime (evt_strike) and only updates aggregate observations (obs_st) including lightning totals/averages for that observation period once per minute.   See https://weatherflow.github.io/Tempest/api/udp/v171/ for their API description.

As a recovered former WeatherFlow field tester I can say a lot about their build quality and magic math data massaging (which is why I sold/gifted my gear) but I definitely love the open API they have that clearly describes what they're doing....

[michael.k...@gmx.at]

I think the main difference is that the Ecowitt consoles can only be polled, or configured to push data a defined interval. As far as I know, there is no such event mechanism from the console to other devices, messages queues, whatever. But I'd be very happy, if someone would prove me wrong.

[Rainer Lang]

this is all clear and long stated in the WiKi - the 79 interval gets restarted once a discharge is detected.
But thinking that a SDR would receive values more often and faster than a normal Ecowitt console (that was the statement I was referring to) is an illusion.

That's at least how the statement reads to me - maybe it was not properly formulated.

saying " If there is a possibility to extract a singled detected lightning event and

> it's data with standard devices, is another story. If so, I'd see the point
> of doing so. At least the minimal distance and weighted average distance
> are interesting values to me."

and

"I doubt it is possible doing this with the
> "Ecowitt Gateway Driver", I have no idea if it is possible with the
> interceptor driver and a WH2551, it should be possible using SDR."

means that a SDR can do what an Ecowitt console or Gary's driver can't do.

The text says it's doubted that the standard devices can detect this (assuming "standard devices" are Ecowitt consoles with the local Ecowitt API), and this doubt is not justified by anything. Of course they can.

And why Gary's driver should not be able to retrieve these readings is also a mystery to me. Of course it can, probably down to one second.
But the retrieval is from the console, not from the sensor, and the console produces added value to the sensor transmissions.

That a SDR should be superior here is by no means true - but of course a SDR can be used.
But it won't get anything faster and more than the Ecowitt console.

and then

"I think the main difference is that the Ecowitt consoles can only be polled, or configured to push data a defined interval. As far as I know, there is no such event mechanism from the console to other devices, messages queues, whatever. But I'd be very happy, if someone would prove me wrong."

What else should the console be able to do but receiving sensor data, (pre-)processing them and responding to an API request or posting data ?
Nothing more is possible and makes sense as the data provider only sends data - there is no bi-directional communication.
At least not in the usual unidirectional sensor world.

The console already provides additional data which the rtl_433 software probably doesn't provide. rtl_433 only decodes the sensor post (afaik).

And the sensor is dumb. It posts only the event distance which its chip has calculated and assigned to the 14 possible values.
Summary count and time-stamp are added by the console.

"Your information does not fit the observed reality."

I guess you didn't read the WiKi properly. And in my mails I didn't say that the interval isn't interrupted and as a result the discharge is immediately transmitted. On the contrary.
You probably misread or misunderstood my text.

I was only referring to the insinuation that a SDR should be able to do something what the console (and via the console) Gary's driver cannot do.

All data is available - but the processing in the way you may like it is not. This needs to be programmed and the data stored following a suitable data model.

"As far as I know, there is no such event mechanism from the console to other devices, messages queues, whatever."

There is - in the context of Ecowitt IoT devices.
There you can trigger IoT devices based on sensor readings, single or combined.
This general process could be used ...

And, there may be something more in the bush - originally meant for the communication with IoT devices.
As I don't know the full API description yet, it's too early to make reliable statements.

I think what would be helpful (at least for me to understand better - maybe I simply don't get the point ... - maybe it was already all clearly presented but that portion got cut off from the thread) to have a draft architecture of what exactly you want to do - not only verbalized but also drawn in a picture.
And then describe for which steps/areas/segments you think you already have tools/solutions and for what not.
1. to see the gaps
2. maybe to revise the whole architecture in a target oriented manner

and then see what's already available and what needs to be added

Having been working as an IT and network architect, I'm not so much into a Lego block approach but in a more structured approach.
Big picture first - and then drill down in a result- and technology-open mindset

A picture is worth 1000 words they say - and in my experience this is very true

On 03.06.2024 17:56, 'michael.k...@gmx.at' via weewx-user wrote:

Rainer, the WH57 sends lightning data as the lighning is detected, the console shows this data immediately.

I observed this mutliple times:

You see the lightning out there, the LED of the WH57 flashes red, the counter on the console goes up, the distance is also updated. If there is a lightning just a few seconds later, the same game again. Even when polling the console API in an interval < 79s, you get the lightning data updates more frequently than 79s.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/59dc1609-6909-471e-9e83-9b7036a58249n%40googlegroups.com.

[vince]

For others lost in the arguing :-) the wiki page is at https://meshka.eu/Ecowitt/dokuwiki/doku.php?id=start#wh57 and the discussion about their APIs is at https://meshka.eu/Ecowitt/dokuwiki/doku.php?id=start#apis_application_programming_interfaces

It was interesting to see some discussion in that wiki suggesting the console displaying in realtime (or not) depends on which console you have - https://meshka.eu/Ecowitt/dokuwiki/doku.php?id=start#ws3800_and_ws3900_ws3910 as one example.

[michael.k...@gmx.at]

Let's boil it down to this sentence:

>  That a SDR should be superior here is by no means true - but of course a SDR can be used.

If the sensor emits every single lightning event, and the console can only be polled, or set up to send data in a fixed interval: it is by all means true!

It is polling vs. event based data. It would be ridiculous, to try to get every lightning event, with polling the device every second. Even if you did so, there would be occasions, where multiple lightnings could be detected within the second. SDR would receive the events as they happen. For such event driven data, sampling with fixed intervals is just not the right solution. And that is what the weatherflow guys do much better, than the ecowitt people.

[michael.k...@gmx.at]

Today I recorded this: https://www.youtube.com/watch?v=SaAj23eqhh4
I have no other explanation than the WH57 reports lightning strikes more often than every 79s.