To many data :-)

[Hajo Harms]

Hi

i use weewx and mesowx.

I have moved my server from remote to local.

I changed weewx and meso to the latest version include 

https://groups.google.com/d/msg/weewx-user/eAUsTqR8yYQ/UVMxAAtgAAAJ

So it seems to work basicly, but weewx put every 2 second one record to mysql.....

Oct  9 22:13:41 NASTS219PII user.notice weewx[4253]: engine: 2017-10-09 22:13:41 CEST (1507580021) LOOP value 'barometer' 1020.08170508 outside limits (26.0, 32.5)

Oct  9 22:13:41 NASTS219PII user.notice weewx[4253]: manager: Added record 2017-10-09 22:13:41 CEST (1507580021) to database 'meso'

Oct  9 22:13:41 NASTS219PII user.info weewx[4253]: raw: deleted rawdata prior to 2017-10-08 22:10:00 CEST (1507493400)

Oct  9 22:13:41 NASTS219PII user.notice weewx[4253]: engine: 2017-10-09 22:13:42 CEST (1507580022) LOOP value 'barometer' 1020.08170508 outside limits (26.0, 32.5)

Oct  9 22:13:41 NASTS219PII user.notice weewx[4253]: manager: Added record 2017-10-09 22:13:42 CEST (1507580022) to database 'meso'

Oct  9 22:13:42 NASTS219PII user.notice weewx[4253]: engine: 2017-10-09 22:13:43 CEST (1507580023) LOOP value 'barometer' 1020.08170508 outside limits (26.0, 32.5)

Oct  9 22:13:42 NASTS219PII user.notice weewx[4253]: manager: Added record 2017-10-09 22:13:43 CEST (1507580023) to database 'meso'

Oct  9 22:13:44 NASTS219PII user.notice weewx[4253]: engine: 2017-10-09 22:13:45 CEST (1507580025) LOOP value 'barometer' 1020.08170508 outside limits (26.0, 32.5)

Oct  9 22:13:44 NASTS219PII user.notice weewx[4253]: manager: Added record 2017-10-09 22:13:45 CEST (1507580025) to database 'meso'

Oct  9 22:13:46 NASTS219PII user.notice weewx[4253]: engine: 2017-10-09 22:13:47 CEST (1507580027) LOOP value 'barometer' 1020.08170508 outside limits (26.0, 32.5)

Oct  9 22:13:46 NASTS219PII user.notice weewx[4253]: manager: Added record 2017-10-09 22:13:47 CEST (1507580027) to database 'meso'

is it a problem with the raw.py driver ?

Regards from Germany :-)

[Glenn McKechnie]

Hi,

If your station generates LOOP packets at 2 second intervals then
mesowx (raw.py) is working as expected.
That's the Real Time data for the first meso graph, and the left panel
which are updated continuously, ie: as fast as the LOOP packets are
supplied (from memory, you may be able to tweak mesowx's display, but
the data will continue at that pace.)

What is your station? I'm not familiar with the capabilities of the
various units but 2 seconds sounds like a Davis?

On side note - you need to configure your Standard Quality service
[StdQC] in weewx.conf as it appears to be set for US values, you are
generating Metric - hPa and its rejecting those values as outside it
limits.

2017-10-09 22:13:47 CEST (1507580027) LOOP value 'barometer'
1020.08170508 outside limits (26.0, 32.5)

Yours, is probably set at the defaults...
[StdQC]

[[MinMax]]
barometer = 26, 32.5, inHg

When it should be something like (as a direct conversion) ...

[StdQC]
····
[[MinMax]]
barometer = 880, 1100, hPa

Or commented out.


Cheers
Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[gjr80]

Just to clarify the StdQC operation, specifying units different to the database units is quite acceptable as the StdQC service will automatically convert the limits to the database units. The danger comes when units are not specified in [StdQC] as the limits are then taken to be in native database units.

Since 26.0inHg and 32.5inHg are approximately 880.5hPa and 1100.6hPa respectively, I suspect the issue here is that the barometer reading of 1020.08170508 is in fact being treated as inHg and not mbar/hPa. Setting the StdQC barometer limits to 800, 1100 (with no units) will prevent the StdQC limit warning but it will not fix the underlying issue.

Gary

[Glenn McKechnie]

On 10 October 2017 at 10:36, gjr80 wrote:
> Just to clarify the StdQC operation, specifying units different to the
> database units is quite acceptable as the StdQC service will automatically
> convert the limits to the database units. The danger comes when units are
> not specified in [StdQC] as the limits are then taken to be in native
> database units.

Clarity is always welcomed. :-)

I see the line in weewx.conf that says...
# This section is for quality control checks. If units are not specified,
# values must be in the units defined in the StdConvert section.
[StdQC]

Taking that onboard and with what you're saying then...

Regardless of the database units being used (US, METRIC, METRICWX),
weewx will convert the [[MinMax]] entry, for any field, to whatever is
required, but only so long as units are specified in that line?
If there are no units specified it's assumed that they are the same
units as the database [StdConvert].

So Hajos [StdQC] service entry for barometer is missing the units
designation, or there is no entry at all?
(Looking at a default install of 3.7.1 config, the barometer entry
exists as inHg, with the units label after it.)
Or, it is a tad more complicated and involves database types...

> Since 26.0inHg and 32.5inHg are approximately 880.5hPa and 1100.6hPa
> respectively, I suspect the issue here is that the barometer reading of
> 1020.08170508 is in fact being treated as inHg and not mbar/hPa.

Because their database is in USunits, and the barometer value being
presented to the database is therefore still being recognized as inHg?
ie: the driver is not notifying weewx of the incoming units (mbar/hPa)
- and the database type it's intended for. (such as owfs.py which
defaults to METRIC and if you send it to another type (say METRICWX)
then conversion happens automatically - cm become mm, kph become m/s)

> Setting the
> StdQC barometer limits to 800, 1100 (with no units) will prevent the StdQC
> limit warning but it will not fix the underlying issue.

So, even if it's specified as ...

[[MinMax]]
barometer = 880, 1100, hPa

Or revert to the default installations setting complete entry of
(which would be converted automatically anyway)...

[[MinMax]]
barometer = 26, 32.5, inHg

neither will truly fix it? Until the database unit mismatch is sorted?

Or have I lost the thread entirely? And if I have missed it, what then
is the suspected underlying issue? (To recap: I'm assuming it's the
notification from driver to weewx of the units/database it's expecting
on handover. Failing that it's the entry in [StdQC]?)

Coffee time!

>
> Gary
>
> On Tuesday, 10 October 2017 08:04:54 UTC+10, Glenn McKechnie wrote:
>>
>>
>> On side note - you need to configure your Standard Quality service
>> [StdQC] in weewx.conf as it appears to be set for US values, you are
>> generating Metric - hPa and its rejecting those values as outside it
>> limits.
>>
>> 2017-10-09 22:13:47 CEST (1507580027) LOOP value 'barometer'
>> 1020.08170508 outside limits (26.0, 32.5)
>>
>> Yours, is probably set at the defaults...
>> [StdQC]
>>
>> [[MinMax]]
>> barometer = 26, 32.5, inHg
>>
>> When it should be something like (as a direct conversion) ...
>>
>> [StdQC]
>> ····
>> [[MinMax]]
>> barometer = 880, 1100, hPa
>>
>> Or commented out.

[Andrew Milner]

I suspect your lack of coffee probably made you over complicate things

His StdQC section should either

have min and max values specified in units which match the database (metric values for METRIC or METRICWX and Imperial values for the US default) 

or

specify the units used for the min and max values given

Without seeing his weewx.conf the reason for the logged error is hard to say - but IMHO the cause is more likely to be not specifying the units correctly rather than any issue with drivers.

[Glenn McKechnie]

On 10 October 2017 at 15:59, Andrew Milner <andrew.s...@gmail.com> wrote:
> I suspect your lack of coffee probably made you over complicate things

Every chance of that, and I still may need more judging by this post.

> His StdQC section should either
> have min and max values specified in units which match the database (metric
> values for METRIC or METRICWX and Imperial values for the US default)
> or
> specify the units used for the min and max values given

Yep, I offered that in my first post as, Metric values with their units

    [[MinMax]]
        barometer = 880, 1100, hPa

What I didn't realize at the time was that Hajo may be missing the units on his barometer stanza, and that if he had them then the conversion would have happened automatically and there would be no need for different values / units, or concern about database types.

> Without seeing his weewx.conf the reason for the logged error is hard to say
> - but IMHO the cause is more likely to be not specifying the units correctly
> rather than any issue with drivers.

I agree, it's the simplest answer.

But Gary then says

I suspect the issue here is that the barometer reading of 1020.08170508 is in

fact being treated as inHg and not mbar/hPa. Setting the StdQC barometer limits to

 800, 1100 (with no units) will prevent the StdQC limit warning but it will not fix the
 underlying issue.

And that leaves me puzzled, as I gave the units in my answer so [StdQC] wouldn't be treating it as inHg - unless there's another reason. ?

Ahh, penny drops!  
Unless Gary's saying that the rejection (because it lies between the 2 extremes) confirms that it's NOT being converted from inHg to hPa.

The 'underlying issue' refers to - no units being on that line.

Hajo, can you put me out of my misery.   What's your [StdQC][ section say?

Maybe it's too much coffee?

[Andrew Milner]

"clink ... clink ..."

[gjr80]

I guess what I wanted to clarify was that replacing one set of minmax figures and units with another, equivalent set of minmax figures and units will not alter the QC outcome because when units are included in a minmax spec the valies are converted to the underlying database units.

Given the small log exact we have seen it is not possible to definitiely say what the issue is. The fact that 26.0 and 32.5 appear in the logs do not mean that those figures are actually in the barometer minmax spec as these could be converted or unconverted values. However, given that a default install uses a barometer minmax of barometer = 26.0, 32.5, inHg I suspect that is what is in Hajo's weewx.conf. If this is the case the log snippet suggests the database unit system is US and seeing a barometer value in the 1000s suggests that either the driver has supplied a packet with a barometer value that does not agee with the packet's usUnits field. Alternatively, the driver could well being providing correct data and weeWX is garbling it, i think it unlikely though. Of course if the minmax spec is not what I surmise then it could b something else. To be certain we need to see the minmax stanza and the raw loop packets coming from the driver.

Gary

[Andrew Milner]

Gary - need to see the whole weewx.conf to see if db is actually metric, metricwx or us, although it could have been changed on the fly and there may have been other attempts to get rid of log errors resulting in feverish adjustments to min max StdQC values before arriving at the current situation.

[Glenn McKechnie]

Well, hopefully Hajo's original question has been answered satisfactorily.

On that note. I think Hajo has been using Weewx and Mesowx for a while
and when there is mention of an upgrade I wonder if weewx's logging
behaviour has changed in that time?

Has manager.py always been so chatty in the logs or is this possibly a
recent change that makes raw.py's behaviour seem different? Maybe the
original raw.py was quieter?

I know from my setup that the LOOP packets are about 20 seconds apart.
I must admit that 2 seconds apart would seem rather ... busy.


Hajo,

When you post your weewx.conf file :-)

What version of weewx was your last version and were you using Luc's
raw.py with that (raw_0.4.x-lh.py from the linked weewx-user thread)?
Or was it the original that came with mesowx?

[Hajo Harms]

Hi Glenn

thanks for response. My english is basic, so some information will be lost in translation....

I have changed the StdQC.. barometer. So no more over/underflow... :-)

Tha database meso is only for one day, correct ? Because the datavolume...

The other thing is, my NAS is near by, so i have many noise (writing ervery 2 sec).. I think 5 or 10 sec will be also ok ?

Or is there a way to cache the data?

And what's the different between pressure and barometer ?

I have a Vantage pro 2 plus and a Qnap Nas 219P++

thanks for all the response :-)

Hajo

[Hajo Harms]

And the weewx.conf

[Glenn McKechnie]

Hi Hajo,

On 10 October 2017 at 21:58, Hajo Harms <hajo....@gmail.com> wrote:

Hi Glenn

thanks for response. My english is basic, so some information will be lost in translation....

Your english is fine.

I'm still curious, but only if you remember. What version of weewx was your last version and were you using Luc's raw.py with that (raw_0.4.x-lh.py from the linked weewx-user thread), or was it the original that came with mesowx?

I have changed the StdQC.. barometer. So no more over/underflow... :-)

Yes, That's good.  That means the Davis driver is fine! (Phew!) :-))

And was the original missing the units entry on that line?

 

Tha database meso is only for one day, correct ? Because the datavolume...

24 hours by default. That matches the time span for the 24-hour graphs.

 The [Raw] section in your weewx.conf file has the following

    # The max amount of raw data to retain specified in hours (set to 0 to retain all data)
    # This will in effect keep a rolling window of the data removing old data based on·
    # the time of the most recent record. It is recommended to set this to at least 24.
    #
    # NOTE: if increasing this value (or setting to unlimited), keep in mind that raw data·
    # may consume VERY large amounts of space!
   data_limit = 24

Once it's been running for a while (data_limit) you'll see the following style of message every 5 minutes, that's raw.py cleaning up the old data.

 raw: deleted rawdata prior to 2017-09-27 08:10:00 AEST (1506463800)

The other thing is, my NAS is near by, so i have many noise (writing ervery 2 sec).. I think 5 or 10 sec will be also ok ?

Or is there a way to cache the data?

Nothing simple that I know of, however.

Looking at the code (in my case that's raw_0.4.1-lh.py, around line 128) , you could tweak the time value in

def newLoopPacket(self, event):
[...]
if dateTime > (self.lastLoopDateTime + 2):

where that last figure, 2 refers to seconds. If you change that value to 10

if dateTime > (self.lastLoopDateTime + 10):

then it should ignore anything falling within that 10 second span. You'll get the timestamps that are > 10 seconds apart and the ones in-between will not make it into the database.

And, I've just tested it on mine and that appears to work.

If you go down this path, save a copy of the file first, then make your edits to the working copy. Better safe than sorry.

After those edits, you'll need to restart weewx

/etc/init.d/weewx restart

 

And what's the different between pressure and barometer ?

Probably best to visit the docs for that one.

 http://www.weewx.com/docs/usersguide.htm#Meteorological_problems

and I'll go and read the MinMax section

http://www.weewx.com/docs/usersguide.htm#StdQC

 

I have a Vantage pro 2 plus and a Qnap Nas 219P++

thanks for all the response :-)

:-))
 

Hajo

[Hajo Harms]

Hi Glenn

so I use weewx 3.7.1 and meso 0.4.0 with raw_0.4.1-lh.py.

But my section of def newLoop Packet looks different

 def newLoopPacket(self, event):

        packet = event.packet

        prune_period = 300

        # It's possible for records with duplicate dateTimes - this occurs when an archive packet

        # is processed since the LOOP packets are queued up and then returned immediately when

        # looping resumes, coupled with the fact that for Vantage Pro consoles the dateTime value is

        # added by weewx. So, for database storage, skip the duplicates until we get a new one to

        # avoid a duplicate key error, but publish them all to redis regardless.

        dateTime = packet['dateTime']

        if dateTime != self.lastLoopDateTime:

            self.dbm.addRecord(packet)

            self.lastLoopDateTime = dateTime

        if dateTime > (self.lastPrunedDateTime + prune_period):

            if self.dataLimit is not None:

                ts = ((dateTime - (self.dataLimit * 3600)) / prune_period) * prune_period  # preset on 5-min boundary

                self.prune_rawdata(self.dbm, ts, 2, 5)

            self.lastPrunedDateTime = dateTime

but this is not my main problem :-)

If I start index-example the come tools tell me this errors :

Element could not be found for fieldId: dayRain

meso.MesoConsole.MesoConsole._initializeFieldValueDisplayManager @ MesoConsole.js:69

(anonymous) @ MesoConsole.js:18

MesoConsole @ MesoConsole.js:16

MesoWxConsole @ MesoWxConsole.js:7

MesoWxApp.start @ MesoWxApp.js:20

(anonymous) @ index-example.html:32

l @ jquery-2.0.3.min.js:4

fireWith @ jquery-2.0.3.min.js:4

ready @ jquery-2.0.3.min.js:4

S @ jquery-2.0.3.min.js:4

AbstractHighstockChart.js:314 Uncaught TypeError: Cannot read property 'getData' of undefined

    at RawChart.meso.AbstractHighstockChart.AbstractHighstockChart._loadInitialDataAndCreate (AbstractHighstockChart.js:314)

    at RawChart.meso.AbstractHighstockChart.AbstractHighstockChart._create (AbstractHighstockChart.js:307)

    at RawChart.AbstractHighstockChart (AbstractHighstockChart.js:121)

    at new RawChart (RawChart.js:29)

    at createTodayChart (MesoWxApp.js:52)

    at selectChart (MesoWxApp.js:45)

    at MesoWxApp.start (MesoWxApp.js:28)

    at HTMLDocument.<anonymous> (index-example.html:32)

    at l (jquery-2.0.3.min.js:4)

    at Object.fireWith [as resolveWith] (jquery-2.0.3.min.js:4)

meso.AbstractHighstockChart.AbstractHighstockChart._loadInitialDataAndCreate @ AbstractHighstockChart.js:314

meso.AbstractHighstockChart.AbstractHighstockChart._create @ AbstractHighstockChart.js:307

AbstractHighstockChart @ AbstractHighstockChart.js:121

RawChart @ RawChart.js:29

createTodayChart @ MesoWxApp.js:52

selectChart @ MesoWxApp.js:45

MesoWxApp.start @ MesoWxApp.js:28

(anonymous) @ index-example.html:32

l @ jquery-2.0.3.min.js:4

fireWith @ jquery-2.0.3.min.js:4

ready @ jquery-2.0.3.min.js:4

S @ jquery-2.0.3.min.js:4

jquery-2.0.3.min.js:6 GET http://192.168.1.17/Wetter/mesowx/web/meso/data.php?entity_id=weewx_raw&data=outTemp%3A%3Ac%3A%2Cdewpoint%3A%3Ac%3A%2CrainRate%3A%3AmmHr%3A%2CoutHumidity%3A%3Aperc%3A%2Cheatindex%3A%3Ac%3A%2Cwindchill%3A%3Ac%3A%2Cbarometer%3A%3Amb%3A%2CdateTime%3A%3Ams%3A%2CwindDir%3A%3Adeg%3A%2CwindSpeed%3A%3Akph%3A%2C&order=desc&limit=1 500 (Internal Server Error)

send @ jquery-2.0.3.min.js:6

ajax @ jquery-2.0.3.min.js:6

x.(anonymous function) @ jquery-2.0.3.min.js:6

getJSON @ jquery-2.0.3.min.js:6

meso.AggregateDataProvider.AggregateDataProvider.getData @ AggregateDataProvider.js:13

meso.PollingRealTimeRawDataProvider.PollingRealTimeRawDataProvider._poll @ PollingRealTimeRawDataProvider.js:44

(anonymous) @ meso.js:14

setTimeout (async)

meso.PollingRealTimeRawDataProvider.PollingRealTimeRawDataProvider._schedulePoll @ PollingRealTimeRawDataProvider.js:40

meso.PollingRealTimeRawDataProvider.PollingRealTimeRawDataProvider._poll @ PollingRealTimeRawDataProvider.js:52

PollingRealTimeRawDataProvider @ PollingRealTimeRawDataProvider.js:19

(anonymous) @ Config.js:82

(anonymous) @ Config.js:678

MesoWxApp.js:14 An unexpected error occurred during and HTTP request (3) [{…}, "error", "Internal Server Error", callee: ƒ, Symbol(Symbol.iterator): ƒ]

I think i have a problem with my databases....???

[Glenn McKechnie]

Hi Hajo,

On 12 October 2017 at 08:06, Hajo Harms <hajo....@gmail.com> wrote:
> Hi Glenn
>

> so I use weewx 3.7.1 and meso 0.4.0 with raw_0.4.1-lh.py.
>
> But my section of def newLoop Packet looks different

[...]

Okay, use the attached version of Luc's ( raw_0.4.3-lh.py ) with that
change already made at line 132

> but this is not my main problem :-)
>
> If I start index-example the come tools tell me this errors :
>
> Element could not be found for fieldId: dayRain

There are a couple of references to dayRain in mesowx and it seems to
be something to do with Davis stations.

web/js/Config-example.js
web/index-example.html
web/meso/include/config-example.json
web/meso/js/SocketIoRealTimeRawDataProvider.js

Have you changed / uncommented any of the references to dayRain in
those mesowx files? By default it's disabled.

In the original raw.py the schema had dayRain listed. but we're not
using that one.

In all of Luc's variants of raw.py his schema has replaced dayRain
with rain. I believe he uses a Davis so I assume it worked for his
setup, as it is.
Anyone else with a working Davis installation and knowledge of what,
if anything, was tweaked with Mesowx?

Have you altered yours ie: changed it back to dayRain?

schema = [
...
('rain', 'REAL'), # rain, was: dayRain

If you, or anyone else, don't solve it then feel free to send me (
off-list, email as above) your files and I'll attempt a comparison
here.

web/js/Config-example.js (and / or Config.js)
web/index-example.html (and / or index.html)
web/meso/include/config-example.json (and / or config.json)
web/meso/js/SocketIoRealTimeRawDataProvider.js
also your version of raw.py

[Hajo Harms]

Hi

long btime ago :-)

But sometimes family and job must bbe done....

So I think in the moment the weewx things work, I will look later concretly.

At the moment I have an problem with  meso/html :

1. Request URL:
   http://192.168.1.17/Wetter/mesowx/web/meso/data.php?entity_id=weewx_raw&data=outTemp%3A%3Ac%3A%2Cdewpoint%3A%3Ac%3A%2CdayRain%3A%3Amm%3A%2CrainRate%3A%3AmmHr%3A%2CoutHumidity%3A%3Aperc%3A%2Cheatindex%3A%3Ac%3A%2Cwindchill%3A%3Ac%3A%2Cbarometer%3A%3Amb%3A%2CdateTime%3A%3Ams%3A%2CwindDir%3A%3Adeg%3A%2CwindSpeed%3A%3Akph%3A%2C&order=desc&limit=1
2. Request Method:
   GET
3. Status Code:
   500 Internal Server Error
4. Remote Address:
   192.168.1.17:80
5. Referrer Policy:
   no-referrer-when-downgrade

I think the data structure is correct or?