AS3935 database issue

[Mikael Fredriksson]

Hi.

Trying to get the AS3935 lightning detector module to work in Weewx.

Seems like the module sends data to Weewx but there seems to be some issue with the database. I´m stuck!

Got this in syslog, as3935: callback failed: Missing value for record field 'interval'

Don´t know how to fix it, hope someone can help me out here!

/Micael

[DS]

I am seeing the same thing.  It looks like the error is coming from manager.py here:

def _calc_weight(self, record):
     if 'interval' not in record:
         raise ValueError("Missing value for record field 'interval'")
     elif record['interval'] <= 0:
         raise ValueError("Non-positive value for record field 'interval': %s" % (record['interval'], ))
     weight = 60.0 * record['interval'] if self.version >= '2.0' else 1.0
 return weight

Which seems to me like the as3935 service isn't adding it to the record?

[gjr80]

The issue is indeed to do with missing field ‘interval’. Some background. The AS3935 service was written back around 2015 when WeeWX was somewhat more tolerant of the the existence or otherwise of field ‘interval’. Move forward to WeeWX v3.7.0 and later and WeeWX is much more stringent in insisting that databases managed by the WeeWX API contain interval data.

The AS3935 service has the ability to store individual lightning strike data in a separate database (this is done by setting the config option ‘data_binding’ in the [AS3935] stanza in weewx.conf, if the ‘data_binding’ option is omitted or set to None saving of lightning data to an external database is disabled). In this mode the AS3935 service attempts to save a record that does not contain the field ‘interval’ to the database specified by the ‘data_binding’ config option. The WeeWX API rejects this and thus the error you are seeing. The fix for this issue will require modification of the AS3935 service to add an ‘interval’ field to the record being saved to the external database or for the data to be saved to an external database using other than the WeeWX API.

The AS3935 service also adds fields ‘avg_distance’ and ‘lightning_strikes’ to loop packets and archive records (whether this extra data is saved to the WeeWX archive depends on whether the user has modified the schema for the WeeWX archive or not). This occurs independently of lightning strike data being saved to an external database. Folks who are using the AS3935 service without issue almost certainly do not save individual lightning strike data to an external database, ie they have not set the ‘data_binding’ config option under [AS3935].

So to a solution, I guess it is hope that Matthew or some other kind person will update the AS3935 service to operate properly with the WeeWX API. In the interim the AS3935 service can be used by removing/disabling ‘data_binding’ config option and making use of the average distance and lightning strike data being added to loop packets and archive records (remembering whether this extra data is saved to the WeeWX archive depends on whether the user has modified the schema for the WeeWX archive or not).

Gary

[Mikael Fredriksson]

Hi!

Now I've got the database correct. Added two new observations in weewx.sdb, lightning_strike and avg_distance. Removes my "data_binding = wx_binding" in [AS3935] also. And modified schema. Followed the weewx documantations.

When I use $day.avg_distance.last in my htmlpage I see the values from my database, but I can't sort this new issue out! Spent hours trying to get it right but, no....

First I want to get the readings in Metric. No matter what I change in skin.conf the values don't show  up in Metric on my page...

Second, The values have lots of decimals, and maybe that changes when I get the readings in Metric, but the lightning_strikes count shows like for example 35.000000  on my page when it would look much better with just 35. But also this I can't find any solution for....brain overheated now.

So a few steps forward but then this happens :) Hope for some help here!

Thanks,

Mikael

[Mikael Fredriksson]

Also I can get the unit "km" to show after the value "avg distance"

[dan Forster]

Micael,

 

While watching TV last night (well, trying more like), this whole AS3935 thing was laying on my mind, and after reading the replies to this post, I decided to remove the separate database bindings and do things properly and modify the existing weewx schema. I did this using the weewx documentation which says it all very clearly. This is now running fine.

 

I am about to read some more and then try to alter the Seasons home page in order that I can see any storm action… I feel then I will be at the same point as you.

 

Interestingly, I have been running a Davis station using weewx for 4 years now (less this dreaded lighting module), keeping it updated on a regular basis and it has been faultless for all that time. It is a lovely piece of programming…

 

Let me know if you have any tips.

 

Dan

[gjr80]

When you add a new field (observation) to the archive WeeWX needs to know how to handle unit conversion and formatting of data in the field. This is done using WeeWX unit groups; for example, temperature based fields are normally assigned the WeeWX unit group group_temperature. In the case of a distance based field it may be appropriate to use the group group_distance. There are a number of ways to make such an assignment but in this case the easiest approach is to add the following code to the bottom of extensions.py (located in /home/weewx/bin/users if WeeWX was installed using setup.py otherwise in /usr/share/weewx/users):

import weewx.units
weewx.units.obs_group_dict['avg_distance'] = 'group_distance'

Save extensions.py and then restart WeeWX. WeeWX tags using the avg_distance field should now have correct formatting and unit conversion (note default formats, units and labels for a report are set in either skin.conf or the weewx.conf [StdReport) stanza).

You can read further about adding fields and setting unit groups in the Customization Guide - http://weewx.com/docs/customizing.htm#customizing_units

In this case we were able to assign an appropriate existing unit group, if no suitable unit group existed we would need to create the appropriate unit conversion formulae as well as default labelling and formatting settings.

Gary

[Mikael Fredriksson]

Hi Gary,

indeed this solved my issue with the conversion! Great! I've read about it in the documentations but was not able to figure it out all the way..

Can you please explain how to solve the issue with too many numbers after a comma in lightning_strikes? It counts strikes as it should but the output in my html always shows for example 35,000000 when it should be more appropriate with only 35. Strange because in the cell of the database it shows with only one decimal....

Thanks for fixing the conversion issue, helped alot!

Mikael

[gjr80]

Hi Mikael,

Ah, I thought the only issue was with distance. Just as you did for avg_distance you also need to do the same for lightning_strikes. In this case there are no unit conversions that need to be defined (lightning_strikes is just a count so there are no other units to convert to/between) but you do need to give WeeWX the ability to format and label tags using lightning_strikes. This time we want to use group_count, so add something like the following to the bottom of extensions.py:

weewx.units.obs_group_dict['lightning_strikes'] = 'group_count'

Again restart after saving the changes.

Gary

[Mikael Fredriksson]

Gary,

that didn't change anything unfortunally. Still several decimals...

This is how the last lines in my extension.py file looks like now:

import schemas.wview
schema_with_lightning_strikes = schemas.wview.schema + [('avg_distance', 'REAL'), ('lightning_strikes', 'REAL')]

import weewx.units
weewx.units.obs_group_dict['avg_distance'] = 'group_distance'

weewx.units.obs_group_dict['lightning_strikes'] = 'group_count'

Is this how it's supposed to look now?

Do I have to change anything in the weewx.conf file? Or in units.py maybe?

Mikael

[gjr80]

Mikael,

I had assumed that WeeWX had a default format for ‘count’ fields but it appears that one does not exist, so we will have to define one. There is no default label either, though not that one is really required (as hPa is required for pressure) but we might as well add one. Try adding the following to the end of extensions.py:

weewx.units.default_unit_format_dict['count'] = '%.0f'
weewx.units.default_unit_label_dict['count'] = ''

Again save and restart WeeWX.

Note that the above format and label can be overridden (as with any other observation) from skin.conf or the [StdReport] stanza of weewx.conf.

Gary

[Mikael Fredriksson]

Gary,

now things looks like I want it to look! I'm sure I hadn't solved this myself so a big thank you for that solution :)

This forum is just great for people like me who isn't very good at this stuff.

But I know I'm learning a lot when things don't go as I want and I get to read much about it. I think many solutions are in the weewx docs but sometimes it gets to difficult for a beginner!

Once again, thank you Gary for helping me!

Mikael

[dan Forster]

Mikael,

When you get some time, it might be nice to document the whole procedure you took to get everything working perfectly, in simple steps (which you can copy and paste), so new and old can use easily. There is much interest in those new inexpensive storm detectors sold on eBay and lots of weewx'ers out there...

Take care

Dan

[Steve2Q]

Mikael..yes, I agree with Dan. I tried a long time ago to get my sensor working, but never was successful. If you could make a "cookbook" that would be very helpful

Steve

[Mikael Fredriksson]

Well, as I'm rather new to this I don't feel really confident writing a description, but I'll try the best I can!

1. First I got the CJMCU as3935 chip soldered and hooked up to my RPi3B+, connections as follows:
VCC -> 3.3V
GND -> GND
SCL -> PIN 5
MOSI -> PIN 3
SI -> 3.3V
A0 -> 3.3V
A1 -> 3.3V
Connected with a breadborad.

2. In terminal, run "sudo i2cdetect -y 1" to see if the chip is connected, it should show 0x03.

3.Now install http://lancet.mit.edu/mwall/projects/weather/releases/weewx-as3935-0.6.tgz following the instructions.
(I did install another one also but I assume this one is the right one)

4. Open /usr/share/weewx/user/extensions.py in a text editor and add these lines at the bottom of the file:

import schemas.wview
schema_with_lightning_strikes = schemas.wview.schema + [('avg_distance', 'REAL'), ('lightning_strikes', 'REAL')]
import weewx.units
weewx.units.obs_group_dict['avg_distance'] = 'group_distance'
weewx.units.obs_group_dict['lightning_strikes'] = 'group_count'

weewx.units.default_unit_format_dict['count'] = '%.0f'
weewx.units.default_unit_label_dict['count'] = ''

5. My settings in weewx.conf:

[AS3935]
bus = 1
indoors = false # set to True if indoors
noise_floor = 0
calibration = 6
pin = 17
address = 3
binding = archive

Under [Databindings]
[[wx_binding]]

Add this: schema = user.extensions.schema_with_lightning_strikes
And put a # in front of schema = schemas.wview.schema (guess this replace the schema to use the new one.)

6. Stop WeeWX.
7. Make a backup of the weewx database file. Located under: /var/lib/weewx
8.Run sudo wee_database weewx.conf --reconfigure - You will need to specify the full path to weewx.conf (e.g. /home/weewx/weewx.conf or /etc/weewx/weewx.conf)
9. Replace the old database file with the new one.
10. Run sudo wee_database --rebuild-daily
11. Restart WeeWX.

(To add the new observation type I followed:
https://github.com/poblabs/weewx-belchertown/wiki/Adding-a-new-observation-type-to-the-WeeWX-database)

I think this is how I got running, with alot of help from Gary(gjr80)
And maybe this isn't the correct way to make a description, but now it's done!

Please feel free to make any corrections to my descriptions, I'm might have got somethings wrong and maybe forgot some step :)
And remember to backup your database file and maybe your weewx setup before doing any changes!

Let me know if there are some questions and I'll see if I remember something leftout!

//Mikael

[Mikael Fredriksson]

And use it like this for example in a web page:

<tr><td>Lightning strikes today</td><td>$day.lightning_strikes.sum</td><td></td></tr>

<tr><td>Distance (Last strike)</td><td>$year.avg_distance.last</td><td>$year.avg_distance.lasttime</td></tr>

Will result in:

Lightning strikes (<40km)

Lightning strikes today	0
Distance (Last strike)	20 km	2019-07-21 19:50:00

[dan Forster]

Mikael,

Superb.

In order for me to add this to the Seasons skin I did this: -

Edit /etc/weewx/skins/Seasons/current.inc and add the labels and data: -

<tr>
        <td class="label">$obs.label.rainRate</td>
        <td class="data">$current.rainRate</td>
      </tr>
      <tr>
        <td class="label">$obs.label.rain_today</td>
        <td class="data">$day.rain.sum</td>
      </tr>
        <tr>
        <td class="label">$obs.label.lightning_strikes</td>
        <td class="data">$day.lightning_strikes.sum</td>
      </tr>
        <tr>
        <td class="label">$obs.label.avg_distance</td>
        <td class="data">$year.avg_distance.last</td>
      </tr>

I also edited (in the same directory) skin.conf and added the following to get the correct labels to show in the skin: -

[Labels]

    # Override some labels. No need to identify these as a "Battery"
    # or "Voltage", because that is included in the template.
    [[Generic]]
        txBatteryStatus      = Transmitter
        windBatteryStatus    = Wind
        rainBatteryStatus    = Rain
        outTempBatteryStatus = Outside Temperature
        inTempBatteryStatus  = Inside Temperature
        consBatteryVoltage   = Console
        heatingVoltage       = Heating
        supplyVoltage        = Supply
        referenceVoltage     = Reference

        # Some specialized labels
        rain_today           = Rain Today
        wind                 = Wind
        lightning_strikes    = Lightning Strikes Today
        avg_distance         = Distance (Last Strike)

I have added a screenshot to show how the results display.

Cheers

Dan

[Peter Leban]

I did this in another way. My AS3935 is connected to another RPi which reports the number of strokes in the recent minute. The "weewx" RPi downloads the file from AS3935 RPi every minute and stores the value into the new database entry (either one unused or create a new entry). Very simple and very effective. I keep the raw log (timestamp, energy, distance) in the AS3935 RPi if I ever want to check it.

Dne nedelja, 07. julij 2019 18.27.16 UTC+2 je oseba Mikael Fredriksson napisala:

[Mikael Fredriksson]

Would be nice to have a list or like a "spreadsheet" on the webpage that shows today's strikes and avg distance to each strike. This should also be emptied at midnight.
Can someone accomplish the code to implement in the index.html.tmpl file?
Would be great!

Mikael

[p q]

Why not try it yourself? - at least the layout of how you want it to look. There are a lot of good HTML tools out there like Coffee Cup.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ebfec5ef-2c83-415f-afaf-8f02c0f150ce%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[peter]

I don't use weewx AS3935 database but keep daily log files in my RPi. Every minute, a python script analyses the file and counts the number of strokes in the last minute. It writes a txt file with a single digit that weewx reads every minute and stores the value in the database under a new column (e.g. strikes). I don't read distance but I could if I wanted. It would mean you'd have to either average the distance over the last minute's strikes or decide another algorithm.

Weewx does the ploting for me but for another project, I simply aggregate the strikes over a period of time (an hour or 10 minutes) and use plotly to show the plot. Herebelow is a most recent example (running out of time) of the small storm we had (aggregate period: 10 minutes).

I prefer to keep a daily log separately since there could be hundreds to thousands of strikes detected every storm and I don't want a single strike to be in the database.

[Mikael Fredriksson]

Ok so now I made my first table!

But can't figure out how to get the 4-5 last strikes to fill out the table in order....and that it updates the table as new strikes appears.

I use Sqlite and have these two observations in weewx.sdb, Lightning strikes and Avg distance

The very advanced code:

<div class="card" >
             <h1><i class="fas fa-bolt" style="color:#E00202" ><!-- m-grau --></i> Last strikes (<40km)</h1>
            <table>
            <tr>
               <th>Strike</th>
               <th>Distance</th>
               <th>Time</th>
            </tr>
             <tr>
               <td>a</td>
               <td>b</td>
               <td>c</td>
            </tr>
            <tr>
               <td>d</td>
               <td>e</td>
               <td>f</td>
            </tr>
            <tr>
               <td>g</td>
               <td>h</td>
               <td>i</td>
            </tr>
            <tr>
               <td>j</td>
               <td>k</td>
               <td>l</td>
            </tr>
            </table>
            </div>

Gives me this:

So how could I get it populated with the numbers from my database file?

Mikael