Timing of archiving and report generation

[marcu...@marcus.ax]

I've just installed Weewx on a Raspberry Pi connected to a FineOffset WH1080. Archive interval is set to 1800:

(wxengine: The archive interval in the configuration file (300) does not match the station hardware interval (1800).

Two questions:

1. Archive data seems to be generated at minutes 08 and 38. Reports are generated at 02 and 32. That means that when a report is generated, the data it's using is already 24 minutes old. Why this delay, and more importantly, is there a way to control this?

2. Is it possible to set archive interval to 300, to have new updates every five minutes, or should I stick with the hardware interval of 1800?

[mwall]

On Thursday, August 22, 2013 5:23:47 PM UTC-4, marcu...@marcus.ax wrote:

1. Archive data seems to be generated at minutes 08 and 38. Reports are generated at 02 and 32. That means that when a report is generated, the data it's using is already 24 minutes old. Why this delay, and more importantly, is there a way to control this?

the 30 minute (1800) interval is coming from the station hardware interval.  your weewx.conf says to use a 5 minute interval, but when weewx.conf conflicts with the station interval and hardware record generation is specified, weewx uses the station interval.

hardware record generation reads recorded data from the station console.

software record generation reads live data from the station console.
 

2. Is it possible to set archive interval to 300, to have new updates every five minutes, or should I stick with the hardware interval of 1800?

if you want updates every 5 minutes, you have two options:

1) set the station hardware interval to 300 using wee_config_fousb

wee_config_fousb --set-interval 300

2) tell weewx to use software record generation and leave 300 as the archive interval

[StdArchive]
    ...
    archive_interval = 300
    record_generation = software

 

[marcu...@marcus.ax]

Thanks! The "--set-interval" parameter is in minutes, though, so it's:

wee_config_fousb --set-interval 5

I've got four empty readings during the night, though (null vales for everything coming from the weather station, but inside temp and other console-values were correctly saved). I'm currently searching for causes. Trying to use software generated records, see if that helps...

[marcu...@marcus.ax]

Two more questions:

1. Will software record generation use the stations current value at the time of creation? Does this differ from hardware generation? We have a helicopter which flies quite close to our weather station sometimes, a single value at the time of flyover would cause create quite an interesting readout!

2. At the moment, we have a wind gust of 23 knots, which occured about 7 hours ago. When checking the archive database, the value of the gust at that time is 2.23 knots. It's not even the highest value in the database today! Any ideas?

[mwall]

On Friday, August 23, 2013 3:11:02 AM UTC-4, marcu...@marcus.ax wrote:

Two more questions:

1. Will software record generation use the stations current value at the time of creation? Does this differ from hardware generation? We have a helicopter which flies quite close to our weather station sometimes, a single value at the time of flyover would cause create quite an interesting readout!

hardware and software generation are different.  software record generation uses whatever the console is reading from the sensors at the time weewx queries the console.  the fineoffset consoles read from the sensors every 48 seconds (the fineoffset 30x0 reads uv information every 60 seconds).  hardware record generation reads logged values from the console.  the console accumulates sensor readings every 48 seconds then saves these (averaged?  there is no spec so we do not know exactly the algorithm) to console memory every n minutes, where n is the console logging interval.
 

2. At the moment, we have a wind gust of 23 knots, which occured about 7 hours ago. When checking the archive database, the value of the gust at that time is 2.23 knots. It's not even the highest value in the database today! Any ideas?

it is not unusual to miss high wind events on the fineoffset stations.  the long sensor query period (48 seconds) means you will miss many events.

if the gust is recorded in console memory and weewx is using hardware record generation, then you should see it in the weewx database.

if the gust is displayed on the console and weewx is using software record generation, you might see it in the weewx database.  this depends on how loop data are transformed to archive records - tom will have to chime in on the specifics of how this happens.

there might be some averaging/smoothing happening when the data are displayed, so if peak events are not showing up in your graphs, first look at the raw data in the database.

m

[Thomas Keffer]

With software generation, what goes into the database:

•   'windSpeed' : Average wind speed seen during the period.
  
•   'windDir' : Vector averaged wind direction seen during the period
  
•   'windGust': Max value seen in the loop packets during the archive period or, if the instrument emits wind gust data in its loop packets (like the Davis instruments) the max value seen there, whichever is greatest.
  
•   'windGustDir': The direction of the above.
  

There's no smoothing done under the covers in the plots. What's shown depends entirely on the aggregation (if any) you choose. If you choose to plot "avg" wind gust then, yes, you will get some smoothing. But, if you choose to plot "max" wind gust, then that's what you're going to get.

Make sense?

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[marcu...@marcus.ax]

The problem I had was a wind gust value in the generated web page which I could not find in the database of weewx, a windgust of 23 knots during the night, when average wind was 1-2 knots. Happened again at 02:05 today, a gust of 47 knots, where average wind is 3 knots. I can't find a gust value of 47 in the archive database. Is it stored in knots?

Other than this gust-problem, it's working marvelous!

[Thomas Keffer]

If you are using US Customary units, wind speed is stored in miles per hour. 

The maximum wind gust seen during an archive period should be in the archive database under the schema column 'windGust'. I'm not very familiar with the Fine Offset driver (fousb.py), but it appears to capture the reading and emitting it, so it should be appearing in the database.

Try this:

$ cd /home/weewx/archive

$ sqlite3 weewx.sdb

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), windGust from archive where dateTime>1377327600;

This should print out the time and wind gust values for all archive records since midnight yesterday (midnight 2013-8-25)

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[marcu...@marcus.ax]

Ooh, using US as target_unit, but I'm in Finland... Actually, I do not know if the system (Fine Offset) reports US or Metric. But everything seems to be correct, though, barometric pressure, wind, temperature... So that should'nt be a problem. But I display wind in knots, temperature in celsius and barometric pressure in mbar, so values on reports will not be the same as those in the database...

I ran this query on weewx archive database:

select datetime(dateTime, 'unixepoch', 'localtime'), windGust from archive where dateTime>1377378000 ORDER BY windGust DESC

Highest value reported is 13.64531137632 on 2013-08-25 10:09:54. At 02:04 value is NULL, and in the report the reported 47 knots is from 02:05.

However, the other databse, for stats, have a max record in table wind of 53.91016461792 (which results in 47 when converted from mph to kts) for that time. So, now the question is: where does this value come from?

As I said earlier, we do have a hell which sometimes flies more or less directly above our weather station. I'm uncertain if they fly during the night, though...

[Thomas Keffer]

The unit system used inside the database should not matter --- it is purely an "implementation detail." 

How long is your archive period? It seems to be very short, such as one minute. You say:

At 02:04 value is NULL, and in the report the reported 47 knots is from 02:05.

If your archive period is one minute, the gust should be in the 0205 slot (not 0204). Or, am I getting something wrong?

We're going to need a little more information here about what, exactly, is in the archive database and the stats database.

-tk

[marcu...@marcus.ax]

Archive period is 300 seconds. Next report is from 02:09...

The 47 knots have no corresponding value in archive database. How's the stats database data gathered, is it retreived from the archive database or from the station?

What data should I provide?

[Thomas Keffer]

So your five minute archive reports are not on five minute boundaries? Odd.

The stats database can be regenerated from the archive database, but it is actually created from the station data. So, first time through, it sees all the data coming from the station. In the case of wind gust this means it has a little better time resolution (it knows the exact time a gust came through instead of just which archive record it is in), but the gust strength should be the same.

I know I'm being repetitive, but before we go chasing down a possible bug, let's be sure we know what is exactly in the databases. I don't know what time zone you are in, so the only SQL query I can form is the following rather clumsy query done in local time:

$ cd/home/weewx

$ sqlite3 weewx.sdb

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), windSpeed, windGust from archive where datetime(dateTime,'unixepoch','localtime') > "2013-08-25 02:00:00" limit 10;

This will print wind speed and gust speed for the first 10 records after 0200 this morning. Adjust date / time as necessary.

Then something similar for the stats database:

$ sqlite3 stats.sdb

sqlite> select dateTime, max, datetime(maxtime,'unixepoch','localtime') from wind where date(dateTime, 'unixepoch','localtime') = "2013-08-25";

Again, adjust date as necessary.

Let us know what you get.

-tk

[AdamHI]

I found this thread after blundering around a bit.

I use 300 s intervals on my Ambient Weather WS2090 (Fine Offset). I had recently changed over to v2.4 and noticed that the timestamp at the top of the web pages was on the 04/09/14 ... minutes after the hour, instead of the 05/10/15 . . .  like it had been under v2.2  . I prefer the "round" numbers, so I tried software generation, and i got back to the 05/10/15. I do note that my station console clock is about 20 seconds ahead of the time on my Raspberry Pi. The Pi's time is pretty much spot on with ntp, but every 2 or 3 months i have to reset the console's clock because it runs fast.

Now that i found this thread, i'm thinking that i should go back to hardware generation because the station at least gets data every 48 seconds, even if we don't know the details of what the console is doing with that data during that time. Is that most of you would recommend? 

I will reset the console clock again later (a bit of a pain), and see if that corrects the 04/09/14 -vs- 05/10/15 issue.

--adam, wh6m
http://abkhome.dyndns.org:8003/wx

[mwall]

hi adam,

please see the Fine Offset section of the weewx user guide, specifically the section about operating modes:

http://weewx.com/docs/usersguide.htm#Operating_modes

some additional details:

weewx completely ignores the console clock of Fine Offset stations.  also, any records stored in the Fine Offset console have no timestamp associated with them!  instead, each contains a number, in seconds, that is relative to the previous record.  so we have to guess what the time is whenever we talk to a Fine Offset station.

the console documentation (such as it is) indicates that one *should* be able to set the console clock, but so far no one has found a Fine Offset station where this works.  take a look at the wee_config_fousb code if you want to learn more.

[mwall]

On Monday, September 2, 2013 2:17:13 PM UTC-4, AdamHI wrote:

I use 300 s intervals on my Ambient Weather WS2090 (Fine Offset). I had recently changed over to v2.4 and noticed that the timestamp at the top of the web pages was on the 04/09/14 ... minutes after the hour, instead of the 05/10/15 . . .  like it had been under v2.2  . I prefer the "round" numbers, so I tried software generation, and i got back to the 05/10/15. I do note that my station console clock is about 20 seconds ahead of the time on my Raspberry Pi. The Pi's time is pretty much spot on with ntp, but every 2 or 3 months i have to reset the console's clock because it runs fast.

btw, some Fine Offset consoles are *supposed* to set the time automatically using a radio clock (i think that is what distinguishes a 2090 from a 2080, or perhaps a 2080 from a 1080?). the thermo/hydro sensor unit is supposed to pick up the clock signal, then the console is supposed to get that from the thermo/hydro to set its clock.

we have had mixed results with these clocks - for the most part they seem to *not* work.

[AdamHI]

Thanks for the info. I'm "playing" with Adaptive -vs- Periodic reading, and Hardware -vs- Software archiving now.

I had gone back to Software archiving, then I had reset the console clock, so now it's about 10 seconds slow. When i went back to Hardware archiving, the timepoints are now 06/11/16 . . .  .  For now i'm going to keep Adaptive reading, and Software archiving, for a while.

There is supposedly a radio-controlled clock on the weather station, but it doesn't work. Very few, if any, WWVB based radio controlled clocks work well out here in the middle of the Pacific.

Take care 

--adam

[Rune Lægreid]

Hi,

I have a satelite network connection with quota limitations on my cabin uploading weewx images to a web-server (http://torolmen.no/vaerside-for-torolmen). 

My settings are:

Station-type = FineOffset USB

record_generation = software

archive_interval = 1200

I need to limit the uplink ftp connection and would like to change the interval, but regardless of what changes I make the report generator seems to be stuck at 5 minute intervals.

What is wrong?

[Simon Bové-Christensen]

Hi... I know this is an old topic, but the thing im looking for is mentioned here, so I thought it would be best to write here.

Im trying to change the archive interval from 5 minutes to 1 minute on my Rpi with weewx 3.1.0 running. My station is a WH1080 Fineoffset and it is mentioned both in the wiki and here in this thread that it should be possible to change the archive interval from 5 minutes to for example 1 minut, so the webiste is updated more frequent.

I have tried to use sudo wee_config_fousb --set-interval=1 but this command is not recognized by weewx, so i was thinking the protocol could have changed in the more recent versions of weewx?

If you need something specific from me, just write a wishlist and i'll try and grant the wishes :-P

Have a great day guys, love this software :-)

[Thomas Keffer]

Simon,

You did not say what version of weewx you are using, but if it is the latest, V3.2, you now use wee_config to configure. See the User's Guide.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simon Bové-Christensen]

I'm running Weewx 3.1.0
Is it the same thing there?

Thanks for the quick answer :-)

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/4aAtrfam0Gk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

See your User's Guide in the docs subdirectory.

-tk

[Simon Bové-Christensen]

Thanks...
It seems like my WH1080 dosen't allow the database interval to be changed

Here's the log:

pi@raspberrypi wee_config_device /home/weewx/weewx.conf --set-interval=1
Using configuration file /home/weewx/weewx.conf
Using FineOffsetUSB driver version 1.7 (weewx.drivers.fousb)
Traceback (most recent call last):
  File "/usr/bin/wee_config_device", line 43, in <module>
    main()
  File "/usr/bin/wee_config_device", line 40, in main
    device.configure(config_dict)
  File "/usr/share/weewx/weewx/drivers/__init__.py", line 65, in configure
    self.do_options(options, parser, config_dict, prompt)
  File "/usr/share/weewx/weewx/drivers/fousb.py", line 379, in do_options
    self.station = FineOffsetUSB(**config_dict[DRIVER_NAME])
  File "/usr/share/weewx/weewx/drivers/fousb.py", line 991, in __init__
    self.openPort()
  File "/usr/share/weewx/weewx/drivers/fousb.py", line 1060, in openPort
    raise weewx.WeeWxIOError(e)
weewx.WeeWxIOError: could not claim interface 0: Operation not permitted

[Andrew Milner]

Did you make sure weewx was not running at the same time|?

[Simon Bové-Christensen]

I didn't. Thanks mate.. I'll try that :-)

[Simon Bové-Christensen]

Didn't work either sadly...

I just have to live with the 5 min delay...

pi@raspberrypi ~ $ sudo service weewx stop

[ ok ] Stopping weewx weather system: weewx..

pi@raspberrypi ~ $ wee_config_device /home/weewx/weewx.conf --set-interval=1

Using configuration file /home/weewx/weewx.conf

Using FineOffsetUSB driver version 1.7 (weewx.drivers.fousb)

Traceback (most recent call last):

  File "/usr/bin/wee_config_device", line 43, in <module>

    main()

  File "/usr/bin/wee_config_device", line 40, in main

    device.configure(config_dict)

  File "/usr/share/weewx/weewx/drivers/__init__.py", line 65, in configure

    self.do_options(options, parser, config_dict, prompt)

  File "/usr/share/weewx/weewx/drivers/fousb.py", line 379, in do_options

    self.station = FineOffsetUSB(**config_dict[DRIVER_NAME])

  File "/usr/share/weewx/weewx/drivers/fousb.py", line 991, in __init__

    self.openPort()

  File "/usr/share/weewx/weewx/drivers/fousb.py", line 1060, in openPort

    raise weewx.WeeWxIOError(e)

weewx.WeeWxIOError: could not claim interface 0: Operation not permitted