Kindly clarify hand on the dependencies between archive_interval polling_interval & read_period:

[Theophanis Kontogiannis]

Hello,

After reading through documentation and forum I am still confused. 

Since my cognitive style is "work by example" I will describe what I try to do and what I think should be done.

My Fineoffset should read the met station every five minutes.

Then every 30 minutes weewx should ask the console for all the previous half our readings. So basically every 30 minutes it should fetch six (6) groups of readings (6x5=30)

Then those data should be archived in the mysql DB.

To my understanding all this is implemented as:

'wee_device --set-interval=5'

and then in the config file:

[FineOffsetUSB]

    # This section is for the Fine Offset series of weather stations.

    polling_mode = PERIODIC

    # The station model, e.g., WH1080, WS1090, WS2080, WH3081

    model = WS2080

    # How often to poll the station for data, in seconds

    polling_interval = 1800

    # The driver to use:

    driver = weewx.drivers.fousb

...............

[StdArchive]

   

    # If the station hardware supports data logging then the archive interval

    # will be downloaded from the station. Otherwise, specify it (in seconds).

    archive_interval = 1800

   

    # How long to wait (in seconds) before processing new archive data. Must

    # be greater than zero.

    archive_delay = 15

   

    # If possible, new archive records are downloaded from the station

    # hardware. If the hardware does not support this, then new archive

    # records will be generated in software.

    # Set the following to "software" to force software record generation.

    record_generation = hardware

    

    # Whether to include LOOP data in hi/low statistics

    loop_hilo = True

    # The data binding used to save archive records

    data_binding = wx_binding

Is this correct?

Thank you

[mwall]

you want this:

set hardware's archive interval to 5 minutes (300 seconds)

wee_device --set-interval=5

set weewx archive interval to match the hardware archive interval

[StdArchive]
    archive_interval = 300

poll the hardware every 60 seconds, use software record generation

[FineOffsetUSB]
    polling_interval = 60
    record_generation = software

this will get archive records from the hardware when weewx starts up, but once it is running weewx will generate the archive data.  if you configure weewx to get archive records from the hardware (record_generation = hardware), then you are very likely to experience the fine offset usb lockup.

m

[Theophanis Kontogiannis]

Thank you for your answer.
I assume it is meant to be polling_interval=1800 to poll every 30 minutes.

So now I have done all that, but.

It all started well. Polled every 30 minutes, downloaded in bulk the data from the console, created the reports, uploaded to WU.

After three-four iterations all that stopped.

It polls once every 30 minutes, and downloads only summary of the previous 30 minutes.

Also no reports have been created/updated in the last 12 hours.

My config file is here:  http://pastebin.com/fX57syuX

My logs are here:       http://pastebin.com/JgB9Drp6

Thank you

[mwall]

On Tuesday, March 14, 2017 at 6:26:02 AM UTC-4, Theophanis Kontogiannis wrote:

Thank you for your answer.
I assume it is meant to be polling_interval=1800 to poll every 30 minutes.

polling_interval is meant to be 60 seconds.  polling_interval is how often weewx will contact the hardware.

find offset 10xx/20xx/30xx stations have poorly designed firmware - if you try to talk to the firmware at the same time that the firmware is collecting data from the sensors, there is a good chance that the usb communications will lock up, and the only solution is to power cycle the station.  when you do that, you lose all data in the logger.

so you want polling_mode=PERIODIC, polling_interval=60 and record_generation=software

you probably want archive_interval=300, instead of the factory default of 1800
 
https://github.com/weewx/weewx/wiki/FineOffset%20USB%20lockup

or search the forum for fine offset usb lockup

m

[Theophanis Kontogiannis]

This is exactly what I do not want to happen.

I need weewx to contact the console every 30 minutes and download in bulk the individual 5 minute appart readings that were storred in the console from the station over the last 30 minutes.

So console <---> station  every 05 minutes

     weewx  <---> console every 30 minutes followed by reports update.

To my understanding of the documentation this means:

   wee_device --set-interval=5

   archive_interval=300

   polling_interval=1800

What do I get wrong?

[mwall]

On Tuesday, March 14, 2017 at 10:22:46 AM UTC-4, Theophanis Kontogiannis wrote:

I need weewx to contact the console every 30 minutes and download in bulk the individual 5 minute appart readings that were storred in the console from the station over the last 30 minutes.

is weewx supposed to be running during the 30 minutes?

the approach you describe is prone to causing usb lockups.

each time it initiates a read of the console memory, the driver does a bunch of short reads in order to synchronize with the  console (the console clock cannot be trusted, and the logger records do not have absolute timestamps).  it can take up to an archive interval to synchronize.  the problem is that every time you try to read the logger memory you increase the chance of locking up the usb, which results in losing all of the console data.

the weewx architecture is not designed for the operating mode that you describe.  it is designed to read logger memory at startup, or at every archive interval.

it is not designed to read the logger at arbitrary intervals greater than an archive interval.

that might be a useful approach for situations where the computer sleeps for a long period, the occasionally wakes to read some data and upload it.  it would be like the wee_reports utility, but for collecting data.

could you explain more why you want to use this approach?

you could write your own python script to do this.  one approach would be to import the fousb driver, then invoke the genArchiveRecords method in the fousb driver.  but i'd probably go with the direct query approach as implemented in the te923 driver.

m

[Theophanis Kontogiannis]

Hi and thank you for the follow up.

Exactly this is is the use case but goes down to hard disks standby.

The data base is installed on a RAID5 which during day time does various IOPS during about 60% of the time, but during night is idle almost 95% of the time, hence the disks get parked and the power consumption goes down to 25W total (compared to 70W when disks are spinning).

Running weewx every five minutes means that the data must be written in the data base.

During the day this is ok.

But during the night it means the disks will keep on spinning practically for ever at maximum power consumption. 

Let us assume for a moment that there is no lock up issue, so to move on with the discussion on parameter values and their practical meaning.

Using the last given values (archive=read=5 minutes & poll=30minutes) the expected behavior is not the actual.

I expect weewx to contact every 30 minutes the console and receive all the stored 5 minute apart entries of the last 30 minutes (ideally total 6 readings - 5x6=30)

The first time I run weewx this exactly what happened. In fact weewx download ALL the stored records updated the database, update WU with all records and created all reports.

Following that first run this is not what I see.

What I see is weewx waking up every 30 minutes, reading ONE stored record (is it averaging all the records in one record or is it reading only the last record??), updating WU with only that record (indeed I see only 30 minute apart single readings in the WU station's portal) and NEVER updating the local reports.

And this is the point I try to short out here. 

Maybe I am perceiving wrongly the document? 

Or I might have misconfigured something? 

But is my expectation correct against those specific settings?

Thank you.

---

Best regards,

ΜΦΧ,

Theophanis Kontogiannis

Στάλθηκε από το iPad μου

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/s1XmmtTP530/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mwall]

On Tuesday, March 14, 2017 at 2:08:39 PM UTC-4, Theophanis Kontogiannis wrote:

Exactly this is is the use case but goes down to hard disks standby.

The data base is installed on a RAID5 which during day time does various IOPS during about 60% of the time, but during night is idle almost 95% of the time, hence the disks get parked and the power consumption goes down to 25W total (compared to 70W when disks are spinning).

Running weewx every five minutes means that the data must be written in the data base.

thank you for the explanation.

when the driver polls the station, it asks only for the current values; polling_interval controls how often the driver queries for current data, not historical records.

weewx has no mechanism to say "read historical data every x minutes/hours/days"; the driver only reads historical data when weewx starts up.

fine offset 10xx/20xx/30xx stations send data no more frequently than 48 seconds (so you could use polling_interval of 48 seconds instead of 60, if you want)

on the hardware, archive_interval determines how often the hardware aggregates 48-second readings into records.  in weewx, archive_interval does the same thing, but it also determines how often reports will be generated.

services that upload data from weewx may be tied to loop data (the 48-second readings) or archive data.

so if you do not need weewx to be running continuously during the time when you want the hard disks to sleep, you could start weewx in the morning, let it run normally during the day, then stop weewx when you want the hard disks to spin down.  that would make weewx read the archive data from the station from overnight, then weewx would do record generation during the day, then the weather station would record data overnight until weewx starts up again the next day.

that would mean you only get updated reports during the time that weewx is running (during the day).

m

[mwall]

theophanis,

you might consider having weewx run all the time, but send its data to a sqlite database on a tmpfs (ramdisk).

then schedule a cron job to run every 30 minutes (or 60 minutes or whatever) that reads the latest records from the sqlite database and inserts them to the database that is running on RAID.

m

[Theophanis Kontogiannis]

On Tuesday, March 14, 2017 at 11:03:54 PM UTC+2, mwall wrote:

On Tuesday, March 14, 2017 at 2:08:39 PM UTC-4, Theophanis Kontogiannis wrote:

Exactly this is is the use case but goes down to hard disks standby.

The data base is installed on a RAID5 which during day time does various IOPS during about 60% of the time, but during night is idle almost 95% of the time, hence the disks get parked and the power consumption goes down to 25W total (compared to 70W when disks are spinning).

Running weewx every five minutes means that the data must be written in the data base.

thank you for the explanation.

when the driver polls the station, it asks only for the current values; polling_interval controls how often the driver queries for current data, not historical records.

weewx has no mechanism to say "read historical data every x minutes/hours/days"; the driver only reads historical data when weewx starts up.

And this makes everything crystal clear.

What I though was happening: "run every polling_interval and fetch all previously non fetched data.

So thankful for your continuous follow up that shorted things out.

 

fine offset 10xx/20xx/30xx stations send data no more frequently than 48 seconds (so you could use polling_interval of 48 seconds instead of 60, if you want)

on the hardware, archive_interval determines how often the hardware aggregates 48-second readings into records.  in weewx, archive_interval does the same thing, but it also determines how often reports will be generated.

services that upload data from weewx may be tied to loop data (the 48-second readings) or archive data.

so if you do not need weewx to be running continuously during the time when you want the hard disks to sleep, you could start weewx in the morning, let it run normally during the day, then stop weewx when you want the hard disks to spin down.  that would make weewx read the archive data from the station from overnight, then weewx would do record generation during the day, then the weather station would record data overnight until weewx starts up again the next day.

The second solution you suggested bellow looks more preferable in my environment. You see the root filesystem is on high end CF cards running on RAID 1, so it is fast, with steady low consumption, and available continuously day and night.

"you might consider having weewx run all the time, but send its data to a sqlite database on a tmpfs (ramdisk)."

Any guidance on what to look for to implement it?

[mwall]

On Tuesday, March 14, 2017 at 5:56:39 PM UTC-4, Theophanis Kontogiannis wrote:

The second solution you suggested bellow looks more preferable in my environment. You see the root filesystem is on high end CF cards running on RAID 1, so it is fast, with steady low consumption, and available continuously day and night.

"you might consider having weewx run all the time, but send its data to a sqlite database on a tmpfs (ramdisk)."

Any guidance on what to look for to implement it?

the first part is trivial.

1) create a mount point

sudo mkdir /var/weewx

2) create a tmpfs by adding an entry to fstab:

echo "weewx /var/weewx tmpfs=100M,noexec,nosuid,nodev 0 0" | sudo tee -a /etc/fstab

3) mount the filesystem

sudo mount -a

4) modify weewx.conf to put the weewx database on the tmpfs

[DatabaseTypes]
    [[SQLite]]
        driver = weedb.sqlite
        SQLITE_ROOT = /var/weewx

the second part will take a bit more work.  you'll have to find/write something that periodically queries the sqlite database and inserts the results into the database on the RAID filesystem.

m

[Theophanis Kontogiannis]

Thank you.

Anything more specific on point 5 (schema matching between SQLite / MySQL - special converting software needed)?  :)

Seriously, should I look for conversion between SQLite and MySQL?

[Theophanis Kontogiannis]

Actually I found this: https://groups.google.com/d/msg/weewx-user/SYn7UIpSqKo/Zs_gAz4C-bcJ
And this:  https://github.com/weewx/weewx/wiki/Transfer%20from%20sqlite%20to%20MySQL

@Tom is it still valid for the new 3.7?

PS: I think we are heading out of the original context. This will be my last post in the thread. Will wait for your answers but I will continue in a different new/existing thread.

Thank you for you great help.

[Thomas Keffer]

On Tue, Mar 14, 2017 at 4:34 PM, Theophanis Kontogiannis <tko...@gmail.com> wrote:

Actually I found this: https://groups.google.com/d/msg/weewx-user/SYn7UIpSqKo/Zs_gAz4C-bcJ

@Tom is it still valid for the new 3.7?

​Did you see the Wiki entry Transfer from sqlite to MySQL?​

-tk