How to avoid problems with errors in weewx archive?

[Hrmeteohub Pljusak]

I am not positive that my questions have not been answered. If so, please point me to location where I can read how to mitigate the problems. Thank you!

Recently I have switched from wview to weewx 3.7.1. I must say I got rather disappointed with problems I have encountered.

First, Oregon Scientific WMR88/100/200 stations started to produce erratic graphs on WU and other web pages, with number of incomplete measurements. The weewx documentation gently describes that as "station problem". That is not acceptable, or nice, but just a dirty walk-around-the-problem.

Yes, I am aware of the way that stations "blasts data" from sensors to station, but wview did away with it quite gently. Also, other (read windows-based) software works nicely with OS stations. Weewx simply gives us nothing (null), and lets the user deal with it. Well guess what!

This should be corrected.

The graph below is from WU.

Just to ilustrate, this is an example of the data from weewx.sdb:

dateTime,ET,altimeter,appTemp,barometer,cloudbase,dewpoint,hourRain,humidex,inDewpoint,inHumidity,inTemp,inTempBatteryStatus,interval,maxSolarRad,outHumidity,outTemp,outTempBatteryStatus,pressure,rain,rain24,rainBatteryStatus,rainRate,rainTotal,usUnits,windBatteryStatus,windDir,windGust,windGustDir,windSpeed,windrun
1526378490.0,None,1008.47595669,18.7382035985,1008.41843898,1330.2378329,9.93516495581,0.0,20.4483894915,51.3461294561,46.0,23.0,0.0,1,None,55.0,19.2,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,337.5,0.700003479692,None,0.700003479692,0.0
1526378610.0,None,None,18.174962293,None,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,None,17,0.0,225.0,1.40000695938,None,1.40000695938,0.0420002087815
1526378670.0,None,1008.47595669,18.384962815,1008.41843898,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,988.0,17,0.0,315.0,1.10000546809,None,1.10000546809,0.126000626344
1526378730.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,225.0,0.900004473889,None,0.900004473889,0.19200095443
1526378850.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,315.0,1.10000546809,None,1.10000546809,0.246001222863
1526378910.0,None,1008.47595669,18.6649635109,1008.41843898,1364.37850406,9.66136186735,20.3234925844,None,1,None,54.0,19.2,0.0,988.0,17,0.0,225.0,0.700003479692,None,0.700003479692,0.312001550948
1526379030.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
1526379150.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
1526379210.0,None,1008.47595669,None,None,None,None,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,17,0.0,250.773234946,0.900004473889,None,0.750003728241,0.35400175973
1526379330.0,None,None,18.2996293732,None,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,None,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.40000695938,None,1.40000695938,0.399001983424
1526379390.0,None,1008.47595669,18.0196286772,1008.41138549,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.80000894778,None,1.80000894778,0.483002400987
1526379450.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,292.5,2.50001242747,None,2.50001242747,0.591002937854

Each "None" means that weewx did not get the data from station. Well, wview and every other software works around this, and so should weewx!

Furthermore, weewx tends to have simply wrong - erratic measurements in archive from Davis VantagePro, Pro2 and Vue stations. Same problem can be found on much cheaper and lower quality FineOffset WH1080/2080 stations. Note that this is data that is not seen on WU - I don't know why - Perhaps WU or weewx does some clean-up of the data later. This graph is drawn from data as it is copied from table "archive" in 5 minute intervals.

weewx 3.7.1. is installed on OpenWRT/LEDE 17.0.1 The data is read in 5 minute intervals from weewx database (archive, not loop), and written to database on separate server and (using weewx) sent to WU. Data is read from WMR and WH stations every 90 seconds, from Davis stations every 60 seconds (archive interval).

The database example is made using CVS export weewx extension.

The weewx.conf is as follows:

# WEEWX CONFIGURATION FILE
# Copyright (c) 2009-2015 Tom Keffer <tkeffer>
# See the file LICENSE.txt for your rights.
#

debug = 0
WEEWX_ROOT = /home/weewx
socket_timeout = 20
version = 3.7.1

[Station]
   
    location = STATIONNAME
    latitude = 432423412
    longitude = 3245143214
    altitude = 175, meter    # Choose 'foot' or 'meter' for unit
    station_type = WMR100
    rain_year_start = 1
    # Start of week (0=Monday, 6=Sunday)
    week_start = 0
    station_url = http://www.pljusak.com

##############################################################################

[WMR100]
    # This section is for the Oregon Scientific WMR100
   
    # The driver to use
    driver = weewx.drivers.wmr100
   
    # The station model, e.g., WMR100, WMR100N, WMRS200
    model = WMR100


[StdRESTful]
    # To guard against parsing errors, put your password(s) in quotes:
   
    [[StationRegistry]]
        # To register this weather station with weewx, set this to true
        register_this_station = false
    [[AWEKAS]]
        enable = false
        username = replace_me
        password = replace_me
    [[CWOP]]
        # and specify the station ID (e.g., CW1234).
        enable = false
        station = replace_me
    # If this is an APRS (radio amateur) station, uncomment
    # the following and replace with a passcode (e.g., 12345).
    #passcode = replace_me (APRS stations only)
    [[PWSweather]]
        enable = false
        station = replace_me
        password = replace_me
    [[WOW]]
        enable = false
        station = replace_me
        password = replace_me
    [[Wunderground]]
        enable = true
        station = ISTATIONNAME1
        password = PASSOWRD
        rapidfire = False

[StdReport]

    SKIN_ROOT = skins
    HTML_ROOT = /tmp

    # This module is not used

   [[simple]]
        HTML_ROOT = /tmp/simple
        skin = simple

[StdConvert]
    # DO NOT MODIFY THIS VALUE UNLESS YOU KNOW WHAT YOU ARE DOING!
    target_unit = METRICWX    # Options are 'US', 'METRICWX', or 'METRIC'

[StdCalibrate]
   
    [[Corrections]]
        # For each type, an arbitrary calibration expression can be given.
        # It should be in the units defined in the StdConvert section.
        # Example:
        foo = foo + 0.2

[StdQC]
   
    [[MinMax]]
        barometer = 900, 1400, mbar
        outTemp = -30, 50, degree_C
        inTemp = -30, 50, degree_C
        outHumidity = 0, 100
        inHumidity = 0, 100
        windSpeed = 0, 300, km_per_hour
        pressure = 900, 1100, mbar

[StdWXCalculate]
   
    [[Calculations]]
        # Derived quantities are calculated by this service. Possible values are:
        #  hardware        - use the value provided by hardware
        #  software        - use the value calculated by weewx
        #  prefer_hardware - use value provide by hardware if available,
        #                      otherwise use value calculated by weewx
       
        pressure = prefer_hardware
        barometer = prefer_hardware
        altimeter = prefer_hardware
        windchill = hardware
        heatindex = hardware
        dewpoint = prefer_hardware
        inDewpoint = prefer_hardware
        rainRate = hardware

[StdTimeSynch]
    clock_check = 14400
    max_drift = 5

[StdArchive]
    # If the station hardware supports data logging then the archive interval
    # will be downloaded from the station. Otherwise, specify it (in seconds).
    archive_interval = 90
    # If possible, new archive records are downloaded from the station
    # hardware. If the hardware does not support this, then new archive
    # records will be generated in software.
    # Set the following to "software" to force software record generation.
    record_generation = software
    # Whether to include LOOP data in hi/low statistics
    loop_hilo = True
    # The data binding used to save archive records
    data_binding = wx_binding

[DataBindings]
    [[wx_binding]]
        database = archive_sqlite
        table_name = archive
        manager = weewx.wxmanager.WXDaySummaryManager
        schema = schemas.wview.schema

[Databases]
    [[archive_sqlite]]
        database_type = SQLite
        database_name = weewx.sdb

[DatabaseTypes]
    [[SQLite]]
        driver = weedb.sqlite
        SQLITE_ROOT = /tmp

[Engine]
   
    [[Services]]
        prep_services = weewx.engine.StdTimeSynch
        data_services = ,
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.csv.CSV
        archive_services = weewx.engine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
        report_services = weewx.engine.StdPrint

[CSV]
    filename = /tmp/data.txt
    binding = archive

[vince]

On Wednesday, May 16, 2018 at 2:38:50 PM UTC-7, Hrmeteohub Pljusak wrote:

That is not acceptable, or nice, but just a dirty walk-around-the-problem.

 

Yes, I am aware of the way that stations "blasts data" from sensors to station, but wview did away with it quite gently. Also, other (read windows-based) software works nicely with OS stations. Weewx simply gives us nothing (null), and lets the user deal with it. Well guess what!

 

This should be corrected.

Well, wview and every other software works around this, and so should weewx!

Furthermore, weewx tends to have simply wrong - erratic measurements

wow - pretty demanding for a user of free software developed and supported by 'volunteers'....

 

[Andrew Milner]

Weewx does not invent data so if there is no reported data from a station within an archive interval weewx will CORRECTLY report None for that reading.  The 'simple' solution is to make sure that your archive interval is large enough to covers a period which will include at least one reading from each sensor.

[Hrmeteohub Pljusak]

Hi Vince,

It was not my intention to yell at anybody, but rather to express frustration about problems that I have encountered, and can not "put my finger" at the cause. It bothers me even more that I have about 40 volunteers on my back that would like to use this software, but can not due to these problems.

If you or anybody got it as demanding, I am sorry.

[Thomas Keffer]

​Your rant would be a lot more useful if it offered ​some constructive examples of where you think weewx is failing. Instead, there's just vague generalities like "erratic measurements from Davis VantagePro2..." 

I flatly do not believe this. The VP2 driver is nearly 10 years old, has been extremely well vetted, and used by thousands without any problems. Most likely the problem is in your configuration. 

The WMR100 driver is not as mature, but is still used by many people. I would not be surprised if it had some problems, but your inchoate rant isn't going to put us any closer to solving them.

I am sorry you are disappointed. If you had paid anything for Weewx, I would be happy to refund your money. But you didn't. 

Stop whining and offer a pull request if you think there's a problem.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hrmeteohub Pljusak]

I understand that, and to some extent that is always the problem when dealing with wildly different hardware, as stated in my post.

I tried setting  "record_generation = hardware" for Davis stations, as they do work better than others.

Furhtermore, I tried five different archive interval settings from 60 all the way to 300 second, but with same results for WMR88. WMR200 does tend to behave better, but not without problems

Also, I have noticed that driver for WMR88/100 calculates the dewpoint by itself, although I believe that WMR88 and 200 does it as it should? I am referring to line 350 in driver code, where it is stated that WMR stations do not calculate the dewpoint below 20F.

[Andrew Milner]

…. the choices seem to include:

1.  Use weewx database and make your graphing package do the smoothing (in the way I presume wview does)

2.  Use weewx database and weewx graphs - rather than making your own graphs

3.  Increase archive interval to ensure every archive record contains at least one reading for each sensor - I think some sensors only output every 15 minutes (=900 seconds)

4.  Explain what you feel to be wrong

or

5.  Return to wview

As a further point you seem to be jumping around many different station types and drivers.  Stations of different types emit different data and weewx will cope with each.  However the config file needs to be set up for each station type with regard to hardware/software preferences, archive interval, hardware/software record creation etc.  HOWEVER - it is overriding within weewx that an archive record contains the average value for a reading during an archive period.  If there is no reading there is no value to store.  In the hgardware manual there is information regarding each station type - and for example the config options for a fine offset station would involve periodic polling at about 60 second intervals (no point in more frequently as station does not emit any faster) and software record creation rather than hardware …. and so on.  The options to use for fine offset are not the same options you would use with a davis station, so it is not possible to just change drivers when changing station types.

My FineOffset has run for several years without issues.  Using QC values in your config helps reduce spikes

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Hrmeteohub Pljusak]

Thank you.

I will try extending the archive interval for stations other that Davis Vantage series. However, I believe that weewx uses archive interval from VP station, as it is set to "record_generation = hardware".

Is there some function that goes through archive data in weewx and does "the smoothing"? I ask that because I have compared local data with data that are on WU and our remote server, and the values with errors did not match on all three locations. At the end of the day, I could not see errors on WU while database dumps to CSV file contained 1-2 error per day. The csv file was done with extenstion set up:

[CSV]
    filename = /tmp/data.txt
    binding = archive

At first I thought that these errors are perhaps only in loop data, or errors on USB communication, but after several weeks I could not catch them in logs with matching timestamps.

[Andrew Milner]

You are probably bashing your head against a brick wall because you are possibly trying to do something weewx is not designed to do.  It is not a real time reporting system.

Try starting with an archive interval of 300 seconds and avoid things like 60 second archive intervals.

Seek help for getting one station type to work correctly before moving on to others - it may not be just the interval that needs 'tweaking' for differing station types.

weewx will use the vantage interval, but the interval should be ideally set to say 300 seconds (use wee_device to set hardware interval)

weewx should never invent data for an archive record, and should always report none when there has been no input during the archive interval.

[Andrew Milner]

WU is a world of its own.  I know that it reports readings for times when none have been submitted (eg I have been down) and so on - so matching times is probably impossible,  It certainly seems to always adjust times to 5 minute boundaries.

[Andrew Milner]

Nothing goes through the data and changes the data contained in the archive.

The plotting engine when drawing graphplots (creating the .png files) does, I believe, try and smooth (and scale) the data for the plots.

Have you compared your graphs with the standard weewx report graphs/plots??

[Hrmeteohub Pljusak]

Dear Andrew,

First, thank you and thank Thomas and everybody else for producing weewx. I apologize once more for coming harsh in my first post.

It appears that you are quite right that I am forcing weewx into something it was not made to do. Perhaps maybe not all of it, but I suspect it is the way I am doing thins more to be blame in some instances.

Also, we are doing that with several weather stations, at locations apart, with so many different co-factors. Seems like a good recipe for headache.

As luck will have it, weewx is now my tool of choice. You are very much correct when you say that I have many problems on a number of fronts simultaneously. Unfortunately. It looks a bit chaotic when I have presented it all at once. Please accept my exclamations as expression of anger onto my/our choices and problems. 

Thank you for giving me info on WU. I did not know that.

If nothing goes through the database, than the problem must be on "my side", CSV reporting and sending to pljusak.com server.

Some of the problems were expected, like intermittent sending of data from WMR stations to weewx. I was combing through data and testing parts of the process for Davis and FineOffset stations for weeks now, and simply can not put my finger on the problem. No such event was seen in weeks for WS2300, WMR100, WMR200 stations in weeks of testing.

Please do not read the rest of my post if you do not have too much time on your hands. As Thomas wrote, I tend to rant.

Let me give some perspective to HRmeteohub project.

The goal of the project is to send data from weather stations to crowd sourcing web page pljusak.com, as well as to WU, AWEKAS, and so on. The project was started by amateur meteorologists and enthusiasts in Croatia, and it runs for about ten years now. At first phase, all users have used some kind of windows based software. In that (first) phase, our job was to hand out the templates for PC software that users were using and users would use FTP to send the graphs and data to server. It worked, but what a mess (and security headaches).

In second phase we started using Open2300 in small devices such as NSLU and wifi routers. About six years ago this has transformed in customized OpenWRT firmware for 5-6 different weather station types. Hence my references to wview. At that time our team stabilized on one sysadmin, one programmer, and me in charge of building the firmware, testing, documening as well as supporting users together with sysadmin.

However, hardware producers have been changing their routers and weather stations, combined with changes in OpenWRT - making our task almost impossible.

At this time about 40 users use HRmeteohub firmware, and some would like to, but supported wifi routers are not on the market anymore. Pljusak.com is NGO run by volunteers, all the software, firmware and services are provided for free and all income from advertising is used to maintain the server and keep NGO functioning. The web page is now about 5-6 years old and ready for overhaul. Basically the server produces something like this:

And each station has got it's page:

This is where weewx comes on stage. We have been testing it on RaspberryPi with good results, but it is rather too complicated for our average meteo-enthusiast to install and setup R-pi with weewx and our scripts for sending the data to pljusak.com. Plus, R-pi (version 2B at that time) wirth power source, wifi card, box was a bit more expensive than wifi router. Our users could can buy second-hand PC with old windows on it for the same price. It (r-pi) also did not work as reliably as wifi routers. At that time, putting Python and weewx in small, cheap wifi router was simply not possible, but in time, OpenWRT merged with LEDE, routers got faster with more Flash and RAM, and suddenly it become a viable option.

After about six months of tries and errors, the firmware proved to be stable. I can not recall single crash of weewx or firmware in last two months.

Unfortunately, Python is big beast for wifi routers with 16MB flash, and 64Mb RAM, so we could not install the graphing, but settle down for archiving the data in devices RAM (default option) or USB stick (works well, but still under testing). Every two hours one script tests internet connection and how much RAM database occupies and if it reaches critical level the user is notified and the database is cleared. Yes, I know that is not nice thing, but the user has got it's data backed up on server. Generally this happens depending on archiving interval, how much RAM device has got and if there are other services needing more RAM.

We use weewx to read the measurements from weather station, and archive it in database. Sending data to WU, CWOP, AWEKAS and others is done using weewx extensions.We have not come up with extension for our pljusak.com so far as none of us is proficient in Python.

What we do with archived data in weewx.sdb is rather simple. CSV extension dumps last measurement to text. Every 5 minutes we send it to server, parse it, put it in database and make a graph. We have tried sending one last measurement from database or sending 2-3 of them, but above mentioned problems persisted. The (raw, unparsed) data sent from router is visible on the web page, so we can see if what arrived at server before it hits database and graphs). We can also see csv.txt on router through it's Luci web page and confirm that the data dumped from weewx.sdb and on server are a match.

OK. I guess I have taken you enough of your time.

I'll be back when I have some solutions for above mentioned problems.

[Andrew Milner]

I seriously recommend that you focus on one particular model of weather station first - eg FineOffset (use software records rather than hardware - see weewx hardware manual for more on the options) and maybe Davis - use hardware records and once again see weewx hardware manual for options).  In both cases I would suggest use a 5 minute archive interval.  For WU do not use rapid fire until you are happy things are ok.

For the rest of your setup it would probably be simplest for you to have arranged for your server to have accepted input via a restless API and built a restless service to upload directly to your site - like weewx does to WU and other hosts - but that is far more suitable for the development forum rather than this forum as it is outside the scope of current weewx.

[Hrmeteohub Pljusak]

Setting 5-min archiving interval made great difference on WMR88 - I had only one drop-out in last four hours. Already I have made it a default in the firmware, for all stations. I'll test this on WMR88 for several days and then see how it goes.
I the mean time I will focus probably on Davis stations. It is hard to say if archiving interval of 5 minutes made a difference so far, since the errors did happen only 1-2 times each day.

I would really like to work with just one type of station, but our volunteers have all kind of stations, from very old to brand new, and expect us to support them all.

You are right on the money with a need for new API.

Thank you for the advice.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

[Andrew Milner]

try 10 minutes even as the interval.  Be honest - since weewx is storing AVERAGES anyway, there is probably little difference between a 5 minute interval and a 10 minute interval except that your servers will have a fraction of the workload and the stations will suffer fewer dropouts and losses of data which to me would be a win-win situation!!  You may as well try it and I think you will be very pleasantly sur[rised at the result of using either 10 or even 15 minute archive intervals!!

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[HrMeteoHub]

Huh, I have waited two days to see how my WMR and Davis are going with
aforemntioned change. While WMR performs butifully, with not a single
drop-out, Davis VP still struggles. I have to dig into that next.
As far as lengthening the archive interval, this is not solely my
decision. There are many stations involved, and I have to see if others
agree with it. I'll try, and then the we'll make a decision.

One thing that bothers me are constraints of the router that runs the
installed software. Python is a beast, and after everything is set up,
there is not much memory left on the RAM. Think 400MHz processor, 60MB
total RAM. I suspect that is why archive interval is having a strong
impact on performance in Davis stations as they have large archive
capacity and may overload the router, either on the startup or collide
with some other housekeeping that OS is doing. The situation is very
different from e.g. Raspberry Pi with +10x RAM.
Also, I think I have to dig deeper in the process how I am reporting the
info to our page, as you have suggested. I am thin on Python knowledge
so I will need help from someone for that.

I'll keep you posted, and thank you very much!

> <http://pljusak.com> server.

>
> Some of the problems were expected, like intermittent
> sending of data from WMR stations to weewx. I was combing
> through data and testing parts of the process for Davis and
> FineOffset stations for weeks now, and simply can not put my
> finger on the problem. No such event was seen in weeks for
> WS2300, WMR100, WMR200 stations in weeks of testing.
>
> Please do not read the rest of my post if you do not have
> too much time on your hands. As Thomas wrote, I tend to rant.
>
>
> Let me give some perspective to HRmeteohub project.
>
> The goal of the project is to send data from weather
> stations to crowd sourcing web page pljusak.com

> <http://pljusak.com>, as well as to WU, AWEKAS, and so on.

> And each station has got it's page:
>

> This is where weewx comes on stage. We have been testing it
> on RaspberryPi with good results, but it is rather too
> complicated for our average meteo-enthusiast to install and
> setup R-pi with weewx and our scripts for sending the data

> to pljusak.com <http://pljusak.com>. Plus, R-pi (version 2B

> up with extension for our pljusak.com <http://pljusak.com>

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to a topic
> in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe

> <https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe>.

> To unsubscribe from this group and all its topics, send an email

> to weewx-user+...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+...@googlegroups.com

> <mailto:weewx-user+...@googlegroups.com>.

[Andrew Milner]

I did not quite follow your posting.  I do not see how the buffer size in the VP can overload weewx!!  Weewx is designed to run 24/7 and only do catchup processing/recovery in the event of power failures, so under normal usage I would not have thought there would be a problem.  Have you trierd ther VP with software record generation or only hardware?  When you change the archive interval in weewx did you also change it in the VP station.  What interval is weewx using (see log at startup of weewx)

If lengthening the archive interval solved the wmr problem then as I see it you have a simple choice - use the larger archive interval or don't use wmr hardware!!

I would have thought that only the stations which experience problems and issues would need to use the larger interval - not necessarily every station in your network.

As has been said many times in this forum comments like 'VP struggles' do not tell us anything - we (and you) need to look at the log, maybe run with debug enabled and so forth in order to understand and more importantly see what is going on within weewx - any error messages and so forth.

> <mailto:weewx-user+unsub...@googlegroups.com>.

[vince]

On Tuesday, May 22, 2018 at 5:27:23 AM UTC-7, Andrew Milner wrote:

As has been said many times in this forum comments like 'VP struggles' do not tell us anything - we (and you) need to look at the log, maybe run with debug enabled and so forth in order to understand and more importantly see what is going on within weewx - any error messages and so forth.

On a 60MB RAM system it is possible the computer simply is out of memory and oddities are happening.

Turn off the CSV extension and see if that helps.

Alternately turn off WU and run only the CSV extension and see if 'that' helps.

But ideally you'd have some data you could tell us about the ram/cpu usage on the system.

You haven't shown us your custom weewx skin either.  If that's too complicated the computer will work too hard when the output is generated.  We have had numerous occcasions where people trying to run a complicated skin makes the system unstable or unpredictable (my example - running Saratoga templates on a 128MB RAM Seagate Dockstar (pogoplug) here).

Ideally, set debug=1 and provide the syslog around the time(s) you see whatever 'struggles' you are experiencing.

I still think the long-term solution is to just tell people to buy a Raspberry Pi if they want to run weewx.   You're talking well under $50 US and it is a very known quantity.  You're doing all kinds of things like writing custom firmware for embedded routers and trying to fit too much stuff in too small a capacity computer.  That's guaranteed pain.

Or just keep running (unsupported/dead) wview on your (unsupported/hacked/legacy) old routers....

[HrMeteoHub]

Dear Vince,

I believe you are right on the money with your last remark. Dead
software works as long as you have hardware, so I tried with cramming
weewx in router, and strange things happened.
Creating embedded firmware with Python inside for 24/7 task is
problematic at best. Low RAM, slow processor and so on is all hitting me
in the head. Check the capture, it runs with only 20MB of free RAM. I
have to move database out of RAM to USB stic/thumbdrive/disk.
By the way this router is only $17. ...not so important in grand scheme
of things.

There is no skin (generating) at all. Only CVS is rewritten to one file
(in RAM). There is no room/processor power for graphing. Even with such
"naked" weewx the processor routinely runs at more than 30%.

So, let me recap. Syncing archiving interval with WMR and WH stations on
5 minutes helped, and it works ok now. My friends are testing other
stations.

However, if moving database to USB disk does not help, there are only
two way out of this - using embedded device with more RAM, or clearing
measurement on VP periodically to prevent this in the first place. I am
not sure the first one will help, and latter feels bad on so many
levels. If router with more RAM does not help, Raspberry pi, here I come
fo VP stations.

Thanx!

Nick

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+...@googlegroups.com

> <mailto:weewx-user+...@googlegroups.com>.

[Andrew Milner]

I am sure that using a Pi for VP stations will be a better answer.  VP's output loop data at I believe 2 second intervals - even if record interval is 5 minutes.  Fine offset on the other hand will only output fresh data at a minimum of every 54 seconds so is usually configured to be polled every minute - a big difference in method of operation and load on the processor.

> <mailto:weewx-user+unsub...@googlegroups.com>.