SyncError: Weewx and Mesowx - weewx dies as a response to a dead link reported(?) by sync.py

Glenn McKechnie

unread,

Feb 27, 2016, 10:58:55 PM2/27/16

to weewx-user

Greetings all,

weewx version 3.3.1
Mesowx version mesowx-0.4.0
sync.py version - 2015-11-02 Modified by Luc Heijst to work with weewx version 3.2.1 - copy attached
raspian wheezy and jessie - deb installations for weewx - manual installations for mesowx
logs attached - (hopefully!) Yes, remembered!
weatherpi debug =1
freezerpi debug = 1

This is possibly more about Mesowx than weewx, however I used Luc's file that was posted here so it's probably the best starting point --- and as I work through this post I realize it may well include weewx's engine.py?

Hardware, software, background, history ...
2 off raspberry pi 2 (weatherpi [raspian wheezy 7.8] and freezerpi [raspian Jessie 8.0]) running weewx version 3.3.1 (deb install)
weatherpi runs an owfs weather station
freezerpi runs 9 DS18B20 1-wire sensors that monitor a coolroom and surrounds.
These 2 pi stations sync to seperate remote databases on a more grunty machine, whitebeard, which is running Mesowx https://bitbucket.org/lirpa/mesowx mesowx-0.4.0 with Luc Heijst's sync.py (lh4) sourced from this group https://groups.google.com/forum/#!topic/weewx-user/DnaWsMpC9vE , specifically https://groups.google.com/group/meso-user/attach/4091dbb6b1381/sync_lh4.py?part=0.1
Mysql is used as the database on all machines and works as expected, when all machines are up!

After much peddling and tweaking both weewx instances and Mesowx installations (seperate directories and databases on whitebeard) are dutifully co-operating and sharing their toys - all is good, there's always more tweaking to do but everything works as intended, database is populating, graphs are displaying. The sun is shining.

The problem...

The problem occurs when whitebeard is suspended (it's actually a mixed server/desktop) and weewx or mesowx's sync.py on weatherpi (only weatherpi) can't handle the lost connection. This results in weewx ungracefully falling over when it tries to start from cold. From cold because weatherpi is (was) restarted with a 2 hourly cronjob that handled an sqlite backup script (a remnant of the pre-mesowx sqlite database config) and it's during that restart that it decides it's had enough with an unreponsive whitebeard (192.168.0.100) and everything should die, as follows...

weatherpi log snippet...
[...]
Feb 28 02:17:03 weatherpi weewx[4343]: engine: Finished loading service weewx.restx.StdWOW
Feb 28 02:17:03 weatherpi weewx[4343]: engine: Loading service user.retain.RetainLoopValues
Feb 28 02:17:03 weatherpi weewx[4343]: RetainLoopValues: excluding fields: set(['rain'])
Feb 28 02:17:03 weatherpi weewx[4343]: engine: Finished loading service user.retain.RetainLoopValues
Feb 28 02:17:03 weatherpi weewx[4343]: engine: Loading service user.sync.SyncService
Feb 28 02:17:03 weatherpi weewx[4343]: sync archive: requesting latest dateTime from http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:15 weatherpi weewx[4343]: sync: failed http request attempt #1 to http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:15 weatherpi weewx[4343]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:15 weatherpi weewx[4343]: sync: retrying again in 0 seconds
Feb 28 02:17:27 weatherpi weewx[4343]: sync: failed http request attempt #2 to http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:27 weatherpi weewx[4343]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:27 weatherpi weewx[4343]: sync: retrying again in 0 seconds
Feb 28 02:17:39 weatherpi weewx[4343]: sync: failed http request attempt #3 to http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:39 weatherpi weewx[4343]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/weatherpi/meso/data.php
Feb 28 02:17:39 weatherpi weewx[4343]: restx: Shut down WOW thread.
Feb 28 02:17:39 weatherpi weewx[4343]: engine: Caught unrecoverable exception in engine:
Feb 28 02:17:39 weatherpi weewx[4343]:     **** sync archive: Failed to invoke http://192.168.0.100/weatherpi/meso/data.php after 3 tries
Feb 28 02:17:39 weatherpi weewx[4343]:     **** Traceback (most recent call last):
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/weewx/engine.py", line 842, in main
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      engine = EngineClass(config_dict)
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/weewx/engine.py", line 75, in __init__
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      self.loadServices(config_dict)
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/weewx/engine.py", line 136, in loadServices
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/user/sync.py", line 109, in __init__
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      self.back_fill()
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/user/sync.py", line 177, in back_fill
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      last_datetime_synced = self.fetch_latest_remote_datetime()
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/user/sync.py", line 229, in fetch_latest_remote_datetime
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      http_response = self.make_http_request(self.latest_url, postdata)
Feb 28 02:17:39 weatherpi weewx[4343]:     ****    File "/usr/share/weewx/user/sync.py", line 282, in make_http_request
Feb 28 02:17:39 weatherpi weewx[4343]:     ****      raise SyncError, "sync archive: Failed to invoke %s after %d tries" % (url, self.http_max_tries)
Feb 28 02:17:39 weatherpi weewx[4343]:     **** SyncError: sync archive: Failed to invoke http://192.168.0.100/weatherpi/meso/data.php after 3 tries
Feb 28 02:17:39 weatherpi weewx[4343]:     **** Exiting.
[...]

and that's the end of weewxd until a manual restart when I log in again, when whitebeards awake.

On the otherhand, freezerpi is allowed to keep running and it handles whitebeard's suspend gracefully. When whitebeard is in suspend mode, freezerpi tries to connect, can't, and accepts defeat graciously and allows weewx to continue. When whitebeard wakes up freezerpi goes into busy mode and plays catch up , as per the following...

freezerpi log snippet...
[...]
Feb 27 08:04:33 freezerpi weewx[24936]: reportengine: Running reports for latest time in the database.
Feb 27 08:04:40 freezerpi weewx[24936]: reportengine: Running report CoolroomReport
Feb 27 08:04:40 freezerpi weewx[24936]: reportengine: Found configuration file /etc/weewx/skins/Coolroom/skin.conf for report CoolroomReport
Feb 27 08:04:40 freezerpi weewx[24936]: cheetahgenerator: Generated 1 files for report CoolroomReport in 0.13 seconds
Feb 27 08:04:49 freezerpi weewx[24936]: genimages: Generated 7 images for CoolroomReport in 8.21 seconds
Feb 27 08:04:49 freezerpi weewx[24936]: reportengine: copied 0 files to /var/www/weewx
Feb 27 08:04:49 freezerpi weewx[24936]: sync: failed http request attempt #1 to http://192.168.0.100/freezerpi/meso/updateData.php
Feb 27 08:04:49 freezerpi weewx[24936]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/freezerpi/meso/updateData.php (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host')))
Feb 27 08:04:49 freezerpi weewx[24936]: sync raw: unable to sync record, skipping
Feb 27 08:04:49 freezerpi weewx[24936]:    **** Reason: sync: Failed to invoke http://192.168.0.100/freezerpi/meso/updateData.php after 1 tries
Feb 27 08:04:49 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:36:32 AEDT (1456515392)
Feb 27 08:05:14 freezerpi weewx[24936]: sync: failed http request attempt #1 to http://192.168.0.100/freezerpi/meso/updateData.php
Feb 27 08:05:14 freezerpi weewx[24936]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/freezerpi/meso/updateData.php (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host')))
Feb 27 08:05:14 freezerpi weewx[24936]: sync raw: unable to sync record, skipping
Feb 27 08:05:14 freezerpi weewx[24936]:    **** Reason: sync: Failed to invoke http://192.168.0.100/freezerpi/meso/updateData.php after 1 tries
Feb 27 08:05:14 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:36:43 AEDT (1456515403)
Feb 27 08:05:27 freezerpi weewx[24936]: reportengine: Running reports for latest time in the database.
Feb 27 08:05:35 freezerpi weewx[24936]: reportengine: Running report CoolroomReport
Feb 27 08:05:35 freezerpi weewx[24936]: reportengine: Found configuration file /etc/weewx/skins/Coolroom/skin.conf for report CoolroomReport
Feb 27 08:05:36 freezerpi weewx[24936]: cheetahgenerator: Generated 1 files for report CoolroomReport in 0.12 seconds
Feb 27 08:05:41 freezerpi weewx[24936]: sync: failed http request attempt #1 to http://192.168.0.100/freezerpi/meso/updateData.php
Feb 27 08:05:42 freezerpi weewx[24936]:    **** Reason: HTTPConnectionPool(host='192.168.0.100', port=None): Max retries exceeded with url: http://192.168.0.100/freezerpi/meso/updateData.php (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host')))
Feb 27 08:05:42 freezerpi weewx[24936]: sync raw: unable to sync record, skipping
Feb 27 08:05:42 freezerpi weewx[24936]:    **** Reason: sync: Failed to invoke http://192.168.0.100/freezerpi/meso/updateData.php after 1 tries
Feb 27 08:05:42 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:37:03 AEDT (1456515423)
Feb 27 08:05:44 freezerpi weewx[24936]: genimages: Generated 7 images for CoolroomReport in 8.81 seconds
Feb 27 08:05:44 freezerpi weewx[24936]: reportengine: copied 0 files to /var/www/weewx
Feb 27 08:06:02 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:37:29 AEDT (1456515449)
Feb 27 08:06:02 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:37:41 AEDT (1456515461)
Feb 27 08:06:02 freezerpi weewx[24936]: sync raw: send record 2016-02-27 06:38:08 AEDT (1456515488)
[...]

Not a real solution, but...
I've fixed the immediate problem and have stopped the cronjob which means that there is no restart of weewx on weatherpi, thus no need for it to call for meso/data.php and fall over. But it does show a weakness - somewhere - that could be fixed by dealing with the broken link much as freezerpi weewx does when calling meso/updateData.php while it's deep in the loop; freezerpi weewx just allows it to quickly time out and goes about its business, when the link comes backup then it gets busy catching up.

Weatherpi weewx on the other hand can't get up from its knees before it's made to fall over again.

Problem area...?
From my limited knowledge of python and decoding error messages, the problem appears to be in /usr/share/weewx/user/sync.py but the exception is generated/handled by perhaps 2 different weewx modules? The one that results in full failure is the /usr/share/weewx/weewx/engine.py   /usr/share/weewx/user/sync.py combination. Is the required fix in engine.py or sync.py? From the log snippets above there seems to be 2 different routines within sync.py ; sync raw and sync archive with the later one being the failure point?

Anyway, mutterings aside, In sync.py is the code that seems to be the starting point
Starting at line 273...

except (socket.error, urllib3.exceptions.MaxRetryError), e:
                syslog.syslog(syslog.LOG_ERR, "sync: failed http request attempt #%d to %s" % (count+1, url))
                syslog.syslog(syslog.LOG_ERR, "   **** Reason: %s" % (e,))
                retry = True
            if retry and count+1 < self.http_max_tries:
                # wait a bit before retrying, ensuring that we exit if signaled
                syslog.syslog(syslog.LOG_DEBUG, "sync: retrying again in %s seconds" % (self.http_retry_interval,))
                self._wait(self.http_retry_interval)
        else:
            raise SyncError, "sync archive: Failed to invoke %s after %d tries" % (url, self.http_max_tries)

    def _wait(self, duration):
        if duration is not None:
            if self.exit_event.wait(duration):
                syslog.syslog(syslog.LOG_DEBUG, "sync: exit event signaled, aborting")
                raise AbortAndExit

Is this the spot to catch this non-fatal error?

Thanks in advance to everyone that may respond. I'm no coder but I can kludge around a bit. I can certainly test and respond.

Should I post this to meso-users? The fullest response to Luc's original post with his sync.py was here. I'm happy to take it elsewhere though.

cheers
Glenn

raw.py
As an aside. Perhaps I should be using the raw.py installation if I want to suspend a server (or sudo-server). I'll do some more reading and get my head around what that setup actually does but in the meantime getting this setup more robust with a fix to the above is probably in others interests as well.

freezerpisyslog.1

sync_lh4.py

weatherpisyslog.1

Luc Heijst

unread,

Feb 29, 2016, 12:33:12 AM2/29/16

to weewx-user

Hi Glenn,

Your situation needs a bit extra coding in the sync module. When raw (loop) data can't be sent, sync just skips it after x retries. But for archive records you don't want that. I think sync has to do the following scenario when it can't talk to the other side.
Retry with a longer pause. Currently this is 0 seconds, maybe 60 seconds. Then instead of posting it's original message with archive data again it should empty it's internal message queue and ask the remote computer the datetime of the latest record ( the same as done at startup of sync). When this fails, sync could repeat this with intervals of one minute until the remote computer is available again. Then all can proceed as normal.
I wll have a look at the code to see if I can change it as described.

Luc

Glenn McKechnie

unread,

Feb 29, 2016, 1:48:51 AM2/29/16

to weewx-user

Hi Luc,

I'm still getting my head around the raw | real | sync differences. I haven't quite nailed them into position in my mind.

I do however understand what you're saying with the problem outlined above and your solution makes good sense; fallback to the proven behaviour - as shown by freezerpi - and start from that known, and stable, point.
This may help with new installs too as weewx won't die while we fiddle around setting up all the related bits correctly - ie: flaky wireless links and setting up of the remote web server. Either way one less hurdle on the path.

Your help is much appreciated Luc. For this modded sync file in the first instance and for offering to do what you can with this hiccup, amongst all the other work you've done/ and or doing. And here's hoping you have success with this one. ;-)

cheers
Glenn

Luc Heijst

unread,

Feb 29, 2016, 6:37:36 AM2/29/16

to weewx-user

Hi Glenn,

When you run the Highstock charts of MesoWX on a remote (web) server AND you want to show both archived AND real-time data of your weather station, you will need on the web server two database tables with archive and raw data respectively.

Here you need module sync.py which sends both raw and archive data to te remote machine (when configured so).

When you have the MesoWX files on the same machine as weewx (with the archive data base -which then has to be MySQL-), you only need a local raw table as weewx doesn't provide such database.

Then you need module raw.py which synchronizes the loop data with a local raw MySQL table.

When each loop packet doesn't contain all loop weather data (in the same packet), but only partial data (as is seen with the meteostick driver), you can use module retain.py which completes each raw packet with the history of elder loop packets to get a 'complete' raw packet.

In your situation - when the web server is not always available- having a database with one or more days with raw weather data seems to me not usefull. The 'missed' raw data is not stored anywhere, so your raw database will show gaps during the periods the web server was 'offline'.

Luc

On Monday, 29 February 2016 03:48:51 UTC-3, Glenn McKechnie wrote:

I'm still getting my head around the raw | real | sync differences. I haven't quite nailed them into position in my mind.

Glenn

Glenn McKechnie

unread,

Mar 1, 2016, 3:55:59 AM3/1/16

to weewx-user

Hi Luc,

Okay, I had a look in the remote servers database and I see the 2 tables, archive and raw. Thats' the 2 required databases when setup as a remote server.

If it's all done locally, then the archive (table) is accessed directly from the weewx db and as you point out it just needs the raw table, thus the seperate database that just includes the raw (without installing and looking for myself). That makes a lot more sense now. (raw table vs database raw)

retain.py , I webt by the description and installed that as, by the description it was a fallback - good to have but not necessary. Reading between the lines I heard "won't break existing setup" so I went with it :-)

Ha! I just realized that the Archive link on the mesowx page refers to the archive table, I'm a bit slow but I do catch up eventually. So the Real-time and 24-hours links are the raw table in action. Got it.

Yes. I agree - it is a psuedo server and, if I want the best results, I'll need to look at options for a real (24hr) server that meets low power requirements. I do like the feedback that the mesowx gives with its pop-ups (although sometimes annoying). The information is presented well for in-depth feedback - Hi's lows per time scale and rolling data via that mouse activated window. I'm finding it particularly good for all those temp sensors from the coolroom.
There is still a place for the other skins for the quick overview and feel (yet to decide on the one that suits, or I should tweak to suit here), but mesowxo is a must for data freaks!

Thanks again Luc. Your explanation has certainly helped.

L.J.M. Heijst

unread,

Mar 1, 2016, 4:44:35 AM3/1/16

to weewx...@googlegroups.com

Hi Glenn,

You should have a look at meteotemplate too. Also uses its own remote archive database and has lots and lots of real time Highchart graphs to choose from. See the meteotemplate section on my website http://www.lucdesign.nl.

Luc

PS I have written a modified sync.py that populates the remote meteotemplate database in the same way as sync.py does for MesoWX. Only Meteotemplate doesn't use raw data.

Verstuurd vanaf mijn iPad

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/Z4sW4oDDboo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Luc Heijst

unread,

Mar 1, 2016, 3:03:28 PM3/1/16

to weewx-user

Hi all,

I'm testing a new version of sync.py for weewx v3xx.

Release notes:

No more fatal errors when remote computer or remote routines can't be found
No more fatal errors when the remote computer detected an error in the call
In case of a 'remote' error, no immediate retries are initiated (it has no use to repeat the call immediately)
When needed, 'immediate' retries are possible though.
In case of a 'remote' error, wait 15 minutes (= 900 s) before trying again.

More to come...

Luc

Luc Heijst

unread,

Mar 5, 2016, 1:19:37 PM3/5/16

to weewx-user

Hi all,

My latest sync.py version is now working OK with remote servers that will go off-line for a while.

To overcome weewx 'freezes' at startup because the catchup program can't talk with the off-line remote computer, I had to put this part in a separate 'catchup-thread'. The main (startup) thread then can continue and end normally, followed by the start of other weewx parts.

The result of this is that this version won't work anymore for systems with a LOCAL sqlite weewx database. Sqlite won't allow database calls in more than one thread.

So it will be:

a) Local weewx database is sqlite and fatal errors will occur when the remote database is off-line.

b) Local weewx database is mysql and no more fatal errors will occur when the remote database is (going) off-line

When I find the time I will publish these driver versions on github, so be patient.

Luc

Edu GG

unread,

May 20, 2016, 9:56:37 AM5/20/16

to weewx-user

Hello Luc,

thank you very much for your support and your time.

I'm trying to solve this problem since a week ago, and after install your versions of sync and raw.py, now the problem is "Failed to invoke..."

May 20 15:52:35 raspberrypi weewx[3521]: engine: Caught unrecoverable exception in engine:

May 20 15:52:35 raspberrypi weewx[3521]: **** sync archive: Failed to invoke ...../web/meso/data.php after 3 tries

May 20 15:52:35 raspberrypi weewx[3521]: **** Traceback (most recent call last):

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/weewx/engine.py", line 853, in main

May 20 15:52:35 raspberrypi weewx[3521]: **** engine = EngineClass(config_dict)

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May 20 15:52:35 raspberrypi weewx[3521]: **** self.loadServices(config_dict)

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/weewx/engine.py", line 136, in loadServices

May 20 15:52:35 raspberrypi weewx[3521]: **** self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/user/sync.py", line 108, in __init__

May 20 15:52:35 raspberrypi weewx[3521]: **** self.back_fill()

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/user/sync.py", line 176, in back_fill

May 20 15:52:35 raspberrypi weewx[3521]: **** last_datetime_synced = self.fetch_latest_remote_datetime()

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/user/sync.py", line 228, in fetch_latest_remote_datetime

May 20 15:52:35 raspberrypi weewx[3521]: **** http_response = self.make_http_request(self.latest_url, postdata)

May 20 15:52:35 raspberrypi weewx[3521]: **** File "/usr/share/weewx/user/sync.py", line 281, in make_http_request

May 20 15:52:35 raspberrypi weewx[3521]: **** raise SyncError, "sync archive: Failed to invoke %s after %d tries" % (url, self.http_max_tries)

May 20 15:52:35 raspberrypi weewx[3521]: **** SyncError: sync archive: Failed to invoke ..../web/meso/data.php after 3 tries

May 20 15:52:35 raspberrypi weewx[3521]: **** Exiting.

Where can I download this new version of sync.py?

Thank you very much.

Glenn McKechnie

unread,

May 20, 2016, 10:27:21 AM5/20/16

to weewx...@googlegroups.com

On 20 May 2016 at 23:56, Edu GG <edu.g...@gmail.com> wrote:
> Hello Luc,
> thank you very much for your support and your time.
> I'm trying to solve this problem since a week ago, and after install your
> versions of sync and raw.py, now the problem is "Failed to invoke..."

[...]

> "sync archive: Failed to invoke %s after %d tries" % (url,
> self.http_max_tries)
> May 20 15:52:35 raspberrypi weewx[3521]: **** SyncError: sync archive:
> Failed to invoke ..../web/meso/data.php after 3 tries
> May 20 15:52:35 raspberrypi weewx[3521]: **** Exiting.

I think that error is on your mesowx server end - ie it's the path in
your mesowx install.
Is your web server directory named web, because that's where yours is
looking for the data.php file - /web/meso/data.php

on my local mesowx server that file is located at...
-rw-rw-r-- 1 root www-data 3099 May 10 2014
/var/www/html/freezerpi/meso/data.php

ie my web server root is at /var/www/html/
and my mesowx install is in a server directory named freezerpi
and the meso files are in that directory ... freezerpi/meso/data.php

and the corresponding entry for that setup, in the remote
/etc/weewx/weewx.conf installation is

" remote_server_url = http://192.168.0.110/freezerpi/meso/ "

where the above URL equates to the directory path
/var/www/html/freezerpi/meso/ on the local server

I hope that makes sense!?

p.s. I've labelled them as local and remote as that's my setup - they
can be the same machine , the directory/path/URL logic still applies

> Where can I download this new version of sync.py?

I'll leave that one to Luc, but I thought it was buried in the
previous messages; perhaps not though? sync_lh7.py is the one you're
looking for (or later if there have been further updates)

--

Cheers
Glenn

Edu GG

unread,

May 20, 2016, 10:41:05 AM5/20/16

to weewx-user

Thank you Glenn for quickly respond,

I'm using sync_lh4.py, and yes, mi data.php file is in /var/www/web/meso/, I try the private IP, the public IP, the real domain, localhost, absolute path, relative... and the result is the same error 500, or 400 if I not put the http://

What can I do?. Thank you very much.

Edu.

Luc Heijst

unread,

May 20, 2016, 11:28:55 AM5/20/16

to weewx-user

Hi Edu,

Did you set a (the same!) security_key on both sides?

Also look in the error log of your web server; you might find a reported error there.

Luc

Edu GG

unread,

May 20, 2016, 3:09:15 PM5/20/16

to weewx-user

Thank you Luc for reply,

apache2/error.log

[Fri May 20 20:50:46 2016] [error] [client 212.104.164.214] PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Unable to parse json config file: include/config.json' in /var/www/web/meso/include/JsonConfig.class.php:18\nStack trace:\n#0 /var/www/web/meso/include/JsonConfig.class.php(24): JsonConfig->__construct()\n#1 /var/www/web/meso/data.php(48): JsonConfig::getInstance()\n#2 {main}\n thrown in /var/www/web/meso/include/JsonConfig.class.php on line 18

[Fri May 20 20:50:47 2016] [error] [client 212.104.164.214] PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Unable to parse json config file: include/config.json' in /var/www/web/meso/include/JsonConfig.class.php:18\nStack trace:\n#0 /var/www/web/meso/include/JsonConfig.class.php(24): JsonConfig->__construct()\n#1 /var/www/web/meso/data.php(48): JsonConfig::getInstance()\n#2 {main}\n thrown in /var/www/web/meso/include/JsonConfig.class.php on line 18

Yes Luc, the security_key is the same at weewx.conf and :

my config.json:

"weewx_archive" : { // the entity ID

// the entity type

"type" : "table",

// the dataSource value should be the ID of a data source defined above in the dataSource section

"dataSource" : "weewx_mysql",

// the database table name

"tableName" : "archive",

// section for access control configuration

"accessControl" : {

// for allowing remote updating of data

"update" : {

// set 'allow' to true to allow this entity's data to be updated

"allow" : true,

// the securityKey that must be sent along with each request to update

// it's recommended to generate a random key, i.e.

// http://www.random.org/strings/?num=1&len=20&digits=on&upperalpha=on&loweralpha=on&unique=on&format=html&rnd=new

// An empty key is not allowed.

"securityKey" : "ZVMO58Opa2pSCH8ESIRb"

.....

"weewx_raw" : {

"type" : "table",

"dataSource" : "raw_mysql",

"tableName" : "raw",

"accessControl" : {

"update" : {

"allow" : true,

"securityKey" : "lO0JLvt0wmpvPsCgcQ99"

and weewx.conf:

[RemoteSync]

#

# This section is for configuration of the MesoWx remote sync service/plugin. This service

# will synchronize archive and/or raw/LOOP data to a Meso web server instance over HTTP.

#

# The base URL of your Meso instance

remote_server_url = http://www.meteolebrija.es/web/meso/

# The Meso entity ID for archive data

archive_entity_id = weewx_archive

# The Meso security required to update the archive data entity

archive_security_key = ZVMO58Opa2pSCH8ESIRb

# The Meso entity ID for raw data

raw_entity_id = weewx_raw

# The Meso security key required to update the raw data entity

raw_security_key = lO0JLvt0wmpvPsCgcQ99

thank you very much for your attention,

Edu.

Luc Heijst

unread,

May 20, 2016, 3:37:56 PM5/20/16

to weewx-user

Hi Edu,

There seems to be a problem somewhere in your config.json file. This is very hard to debug. When you forgot one comma, ) or } you are in trouble (also when you have one extra where there shouldn't be any). Don't use any tabs, only spaces.

May be you start all over with a new example file and change as little as possible each time. Also create new security keys, because they use to be 'secret' and now we all now yours!

Luc

Edu GG

unread,

May 20, 2016, 3:49:12 PM5/20/16

to weewx-user

Thank you Luc for quickly respond,

don't worry for securekeys, I'll change now.

"Don't use tabs..." upss!, I'm sure that the config.json have many tabulations added.

I'll cp the original config-example.json and like you recommend, start again.

I will discuss the results.

Luc Heijst

unread,

May 20, 2016, 3:54:36 PM5/20/16

to weewx-user

Edu,

My config-example.json file doesn't have any tabs, only spaces!

Luc

Edu GG

unread,

May 20, 2016, 4:30:27 PM5/20/16

to weewx-user

good choice!,

the error not appear, but appear other:

May 20 22:20:41 raspberrypi weewx[3150]: manager: added record 2016-05-20 22:20:42 CEST (1463775642) to database 'mesowx'

May 20 22:20:42 raspberrypi weewx[3150]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463775642' for key 'PRIMARY'

May 20 22:23:51 raspberrypi weewx[3150]: manager: added record 2016-05-20 22:23:51 CEST (1463775831) to database 'mesowx'

May 20 22:23:51 raspberrypi weewx[3150]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463775831' for key 'PRIMARY'

May 20 22:24:03 raspberrypi weewx[3150]: manager: added record 2016-05-20 22:24:03 CEST (1463775843) to database 'mesowx'

May 20 22:24:03 raspberrypi weewx[3150]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463775843' for key 'PRIMARY'

... each record on mesowx returns an duplicate entry.

but its great!!! ... And I can see data on www.meteolebrija.es/web ... on US units, I'd like to see in metric units.

Luc Heijst

unread,

May 20, 2016, 5:02:12 PM5/20/16

to weewx-user

Edu,

You shouldn't get any duplicate keys in the raw database unless (for some reason) the raw data is inserted twice.

If you want on your web site metric data, you should check the units in files config.json and Config.js.

I have on both sides the database in METRIC units (thus rain in cm!)

config.json:

"columns" : {

"dateTime" : {"type" : "number", "unit" : "s"},

"interval" : {},

"barometer" : {"unit" : "inHg"},

"inTemp" : {"unit" : "f"},

"outTemp" : {"unit" : "f"},

"inHumidity" : {"unit" : "perc"},

"outHumidity" : {"unit" : "perc"},

"windSpeed" : {"unit" : "mph"},

"windDir" : {"unit" : "deg"},

"windGust" : {"unit" : "mph"},

"windGustDir" : {"unit" : "deg"},

"rainRate" : {"unit" : "inHr"},

"rain" : {"unit" : "in"},

"dewpoint" : {"unit" : "f"},

"windchill" : {"unit" : "f"},

"heatindex" : {"unit" : "f"}

},

into:

"columns" : {

"dateTime" : {"type" : "number", "unit" : "s"},

"interval" : {},

"barometer" : {"unit" : "hPa"},

"outTemp" : {"unit" : "c"},

"outHumidity" : {"unit" : "perc"},

"windSpeed" : {"unit" : "kph"},

"windDir" : {"unit" : "deg"},

"windGust" : {"unit" : "kph"},

"windGustDir" : {"unit" : "deg"},

"rainRate" : {"unit" : "cmHr"},

"rain" : {"unit" : "cm"},

"dewpoint" : {"unit" : "c"},

"windchill" : {"unit" : "c"},

"heatindex" : {"unit" : "c"},

},

Also look at the units in file Config.js in folder web/js:

Config.js

    Config.fieldDefaults = {

        'dateTime':             new meso.FieldDef('dateTime',             meso.Unit.ms,           0,      meso.Agg.avg),

        'outTemp':              new meso.FieldDef('outTemp',              mesowx.Unit.c,          1,      meso.Agg.avg,   'Temp'),

        'dewpoint':             new meso.FieldDef('dewpoint',             mesowx.Unit.c,          1,      meso.Agg.avg,   'Dewpoint'),

        'rain':                 new meso.FieldDef('rain',                 mesowx.Unit.mm,         1,      meso.Agg.sum,   'Rain'),

        'rainRate':             new meso.FieldDef('rainRate',             mesowx.Unit.mmHr,       1,      meso.Agg.max,   'Rain Rate'),

        'dayRain':              new meso.FieldDef('dayRain',              mesowx.Unit.mm,         1,      meso.Agg.max,   'Rain Today'),

        'windSpeed':            new meso.FieldDef('windSpeed',            mesowx.Unit.mps,        1,      meso.Agg.avg,   'Wind Speed'),

        'windDir':              new meso.FieldDef('windDir',              mesowx.Unit.deg,        0,      meso.Agg.avg,   'Wind Dir'),

        'windGust':             new meso.FieldDef('windGust',             mesowx.Unit.mps,        1,      meso.Agg.avg,   'Gust Speed'),

        'windGustDir':          new meso.FieldDef('windGustDir',          mesowx.Unit.deg,        0,      meso.Agg.avg,   'Gust Dir'),

        'outHumidity':          new meso.FieldDef('outHumidity',          mesowx.Unit.perc,       0,      meso.Agg.avg,   'Humidity'),

        'barometer':            new meso.FieldDef('barometer',            mesowx.Unit.hPa,        1,      meso.Agg.avg,   'Barometer'),

        'heatindex':            new meso.FieldDef('heatindex',            mesowx.Unit.c,          1,      meso.Agg.avg,   'Heat Index'),

		'rxCheckPercent':       new meso.FieldDef('rxCheckPercent',       mesowx.Unit.perc,       0,      meso.Agg.avg,   'Rx Check'),

		'txBatteryStatus':      new meso.FieldDef('txBatteryStatus',      mesowx.Unit.perc,       0,      meso.Agg.avg,   'tx BattStat'),

		'ConsBatteryVoltage':   new meso.FieldDef('ConsBatteryVoltage',   mesowx.Unit.perc,       2,      meso.Agg.avg,   'Cons BattVolt'),

		'windBatteryStatus':    new meso.FieldDef('windBatteryStatus',    mesowx.Unit.perc,       0,      meso.Agg.avg,   'Wind BattStat'),

		'rainBatteryStatus':    new meso.FieldDef('rainBatteryStatus',    mesowx.Unit.perc,       0,      meso.Agg.avg,   'Rain BattStat'),

		'outTempBatteryStatus': new meso.FieldDef('outTempBatteryStatus', mesowx.Unit.perc,       0,      meso.Agg.avg,   'outTemp BattStat'),

		'inTempBatteryStatus':  new meso.FieldDef('inTempBatteryStatus',  mesowx.Unit.perc,       0,      meso.Agg.avg,   'inTemp BattStat'),

		'humidex':              new meso.FieldDef('humidex',              mesowx.Unit.c,          1,      meso.Agg.avg,   'Humidex'),

		'appTemp':              new meso.FieldDef('appTemp',              mesowx.Unit.c,          1,      meso.Agg.avg,   'App Temp')

Luc

Edu GG

unread,

May 20, 2016, 5:56:57 PM5/20/16

to weewx-user

Thank you Luc, thank you very much....

... and display metric units, ... I'm still investigating.
Thank you very much again, Glenn and Luc.

Edu.

Glenn McKechnie

unread,

May 20, 2016, 7:10:40 PM5/20/16

to weewx...@googlegroups.com

Hi Edu, Luc,

I see the night shift has sorted this problem out :-) The graphs look
good and appear to be updating okay and I see the units are now
metric.

Regarding the duplicate keys.
I haven't had my morning coffee yet but as I understand it that's on
the mesowx end, in its database? The duplicate keys are very recent (
1463775642 <--> 21 May 06:20:42 AEST 2016 ) and there doesn't appear
to be a lot of history to those records so I'd be reconfiguring
weewx.conf to write to a new mesowx database (keep your existing
[backup paranoia], just point it to fresh, new name ) and restart
weewx. With a bit of luck it will fill that one nicely and the
'duplicate entry' hiccups were just a reflection of your previous
set-up attempts.

As Luc says, and I think you've confirmed, The mesowx config files are
very touchy regarding style / format. One misplaced comma gives you a
blank screen and no useful log entries anywhere. Definitely a case for
making changes slowly (and saving the working versions regularly).

Good to see another success.

--

Cheers
Glenn

Edu GG

unread,

May 20, 2016, 7:23:22 PM5/20/16

to weewx-user

Hello Glenn,

thank you for recommend, now here is 01:20am... sleep hour for me, but at wake up I try to do this with de db mesowx.

For now, each 10sec returns the duplicate errors:

May 21 01:20:50 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786450' for key 'PRIMARY'

May 21 01:21:05 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786464' for key 'PRIMARY'

May 21 01:21:21 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786481' for key 'PRIMARY'

May 21 01:21:29 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786460' for key 'PRIMARY'

May 21 01:21:35 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786494' for key 'PRIMARY'

May 21 01:21:46 raspberrypi weewx[2747]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463786506' for key 'PRIMARY'

tomorrow continue. Thank you!

Edu.

Glenn McKechnie

unread,

May 20, 2016, 7:37:40 PM5/20/16

to weewx...@googlegroups.com

That's a lot of duplicates.
I'd almost suspect there's two instances of sync.py running, although
how I don't know. Check your weewx.conf to make sure you're not
calling it twice --ie: only one entry in restful_services, if it's
remote and sync.py

Good night, or Good morning

--

Cheers
Glenn

Luc Heijst

unread,

May 20, 2016, 7:49:54 PM5/20/16

to weewx-user

As Glenn said, check if sync.py isn't called twice. And if you use retain, make sure the call is placed in front of the call to sync.
Don't use raw.py when you use sync.py. Sync.py will handle both archive and raw data.

Luc

Edu GG

unread,

May 21, 2016, 6:54:58 AM5/21/16

to weewx-user

Thanks for your support,

I checked my weewx.conf and the restful_services section there is only one instance of user.sync.SyncService.

I deleted configurations [RetainLoopValues] and [Raw], and still the same.

I will create a new db mesowx02 and modify the call on the weewx.conf files [Databases] and config.json "raw_mysql"...

To see results give.

Edu.

Edu GG

unread,

May 21, 2016, 7:11:11 AM5/21/16

to weewx-user

mmmh... after create the db and reconfigure calls in weewx.conf and config.json, the log returns

May 21 13:09:10 raspberrypi weewx[5628]: engine: Caught database OperationalError: (1054, "Unknown column 'usUnits' in 'field list'")

May 21 13:09:10 raspberrypi weewx[5628]: **** Waiting 2 minutes then retrying...

after 2 minutes, the same.

Luc Heijst

unread,

May 21, 2016, 9:17:54 AM5/21/16

to weewx-user

There was no need to delete section [RetainLoopValues], but as you have done: check the [Engine] section if there is somewhere a call to user.retain.RetainLoopValues. If so, delete this call as well.

Luc

Edu GG

unread,

May 21, 2016, 9:45:14 AM5/21/16

to weewx...@googlegroups.com

thanks,

now my archive_services = weewx.engine.StdArchive, user.weewxwd3.WdArchive, user.raw.RawService but the error continues:

May 21 15:42:34 raspberrypi weewx[6557]: manager: added record 2016-05-21 15:42:35 CEST (1463838155) to database 'mesowx'

May 21 15:42:35 raspberrypi weewx[6557]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463838155' for key 'PRIMARY'

May 21 15:42:57 raspberrypi weewx[6557]: manager: added record 2016-05-21 15:42:57 CEST (1463838177) to database 'mesowx'

May 21 15:42:57 raspberrypi weewx[6557]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463838177' for key 'PRIMARY'

¿it's the first time that you see this? Thank you Luc and Glenn.

Edu.

--

Luc Heijst

unread,

May 21, 2016, 9:53:28 AM5/21/16

to weewx-user

As I wrote before: don't use raw.py, so delete the call to user.raw.RawService from the Engine section.
Luc

Edu GG

unread,

May 21, 2016, 10:07:12 AM5/21/16

to weewx...@googlegroups.com

ok, I forgot to delete the raw call at Engine section,

after that, data continue refreshing every 10secs at web. Well, however the error continue but instead every second, every minute returns:

May 21 16:00:29 raspberrypi weewx[6752]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463839200' for key 'PRIMARY'

May 21 16:01:34 raspberrypi weewx[6752]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463839260' for key 'PRIMARY'

May 21 16:02:25 raspberrypi weewx[6752]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463839320' for key 'PRIMARY'

May 21 16:03:26 raspberrypi weewx[6752]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463839380' for key 'PRIMARY'

this is very strange for me, my friends. The mysql logs returns no error.

Greetings.

Edu.

2016-05-21 15:53 GMT+02:00 Luc Heijst <ljm.h...@gmail.com>:

As I wrote before: don't use raw.py, so delete the call to user.raw.RawService from the Engine section.
Luc

Luc Heijst

unread,

May 21, 2016, 10:45:20 AM5/21/16

to weewx-user

For some reason the call to your webserver to insert new data in the database seems to be executed twice. The first time the data is stored, the second time mysql refuses to store the data with the same time stamp which is the primary key. Is your archive interval set to one minute? This could explain the 1 minute interval.
Please include a copy of your [Engine] section of weewx.conf. I like to check your current settings.
Luc

Andrew Milner

unread,

May 21, 2016, 11:05:51 AM5/21/16

to weewx-user

make sure that you stop and restart weewx when you remove services to make sure they are truly stopped from running and not left hanging on to an old thread or something similar .. and ensure there is no other weewx running by issuing the stop command twice (the second time should say there is no process to stop).

Edu GG

unread,

May 21, 2016, 11:29:04 AM5/21/16

to weewx...@googlegroups.com

thank you Andrew for recommend, I always type stop twice, for receive de not running notification.

I'll tell my history... before weewx installing, was using weather-display since year 2010, but wanted to use the Raspberry Pi to save power consumption costs, then I have the extension weewxwd that can be seen in meteolebrija.es/weewx/WD installed, for this my wewewx.conf is very extense, but the data that shown on WeatherDisplayLuve is not showing like I want, and I found mesowx and the Luc adaptation for v.3x, great!!.

Luc, this is my complete weewx.conf (users, passwords and keys hidden) for your review:

# WEEWX CONFIGURATION FILE

#

# See the file LICENSE.txt for your rights.

##############################################################################

# This section is for general configuration information.

# Set to 1 for extra debug info, otherwise comment it out or set to zero

debug = 0

# Root directory of the weewx data file hierarchy for this station

WEEWX_ROOT = /

# How long to wait before timing out a socket (FTP, HTTP) connection

socket_timeout = 20

# Do not modify this. It is used when installing and updating weewx.

version = 3.5.0

##############################################################################

# This section is for information about the station.

[Station]

# Description of the station location

location = "Lebrija, Sevilla"

# Latitude and longitude in decimal degrees

latitude = 36.91

longitude = -6.07

# Altitude of the station, with unit it is in. This is downloaded from

# from the station if the hardware supports it.

altitude = 40, meter

# Set to type of station hardware. There must be a corresponding stanza

# in this file with a 'driver' parameter indicating the driver to be used.

station_type = WS23xx

# If you have a website, you may specify an URL

station_url = http://www.meteolebrija.es

# The start of the rain year (1=January; 10=October, etc.). This is

# downloaded from the station if the hardware supports it.

rain_year_start = 1

# Start of week (0=Monday, 6=Sunday)

week_start = 0

##############################################################################

[WS23xx]

# This section is for the La Crosse WS-2300 series of weather stations.

# Serial port such as /dev/ttyS0, /dev/ttyUSB0, or /dev/cuaU0

port = /dev/ttyUSB0

# The station model, e.g., 'LaCrosse WS2317' or 'TFA Primus'

model = LaCrosse WS-2357

# The pressure calibration offset, in hPa (millibars)

pressure_offset = 0

# The driver to use:

driver = weewx.drivers.ws23xx

##############################################################################

# This section is for uploading data to Internet sites

[StdRESTful]

[[StationRegistry]]

# To register this weather station with weewx, set this to true

register_this_station = true

[[AWEKAS]]

# This section is for configuring posts to AWEKAS.

# If you wish to do this, set the option 'enable' to true,

# and specify a username and password.

enable = true

username = **************************

password = **************************

[[CWOP]]

# This section is for configuring posts to CWOP.

# If you wish to do this, set the option 'enable' to true,

# and specify the station ID (e.g., CW1234).

enable = true

station = DW4956

# If this is an APRS (radio amateur) station, uncomment

# the following and replace with a passcode (e.g., 12345).

passcode = **************************

[[PWSweather]]

# This section is for configuring posts to PWSweather.com.

# If you wish to do this, set the option 'enable' to true,

# and specify a station and password.

enable = true

station = LEBRIJA

password = **************************

[[WOW]]

# This section is for configuring posts to WOW.

# If you wish to do this, set the option 'enable' to true,

# and specify a station and password.

enable = false

station = replace_me

password = replace_me

[[Wunderground]]

# This section is for configuring posts to the Weather Underground.

# If you wish to do this, set the option 'enable' to true,

# and specify a station (e.g., 'KORHOODR3') and password.

enable = true

station = ISEVILLA12

password = **************************

# Set the following to True to have weewx use the WU "Rapidfire"

# protocol. Not all hardware can support it. See the User's Guide.

rapidfire = True

[[WeatherCloud]]

id = **************************

key = **************************

##############################################################################

# This section specifies what reports, using which skins, to generate.

[StdReport]

# Where the skins reside, relative to WEEWX_ROOT

SKIN_ROOT = /etc/weewx/skins

# Where the generated reports should go, relative to WEEWX_ROOT

HTML_ROOT = /var/www/weewx

# The database binding indicates which data should be used in reports.

data_binding = wx_binding

# Each of the following subsections defines a report that will be run.

[[StandardReport]]

# See the customizing guide to change the units, plot types and line

# colors, modify the fonts, display additional sensor data, and other

# customizations. Many of those changes can be made here by overriding

# parameters, or by modifying templates within the skin itself.

# The StandardReport uses the 'Standard' skin, which contains the

# images, templates and plots for the report.

skin = Standard

[[[Units]]]

[[[[Groups]]]]

group_altitude = meter

group_speed2 = km_per_hour2

group_pressure = mbar

group_rain = mm

group_rainrate = mm_per_hour

group_temperature = degree_C

group_degree_day = degree_C_day

group_speed = km_per_hour

[[wdTesttags]]

HTML_ROOT = /var/www/weewx/WD

skin = Testtags

[[[Units]]]

[[[[Groups]]]]

group_altitude = meter # Options are 'foot' or 'meter'

group_degree_day = degree_C_day # Options are 'degree_F_day' or 'degree_C_day'

group_pressure = mbar # Options are 'inHg', 'mbar' or 'hPa'

group_rain = mm # Options are 'inch' or 'mm'

group_rainrate = mm_per_hour # Options are 'inch_per_hour' or 'mm_per_hour'

group_speed = km_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = km_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

group_temperature = degree_C # Options are 'degree_F' or 'degree_C'

[[[[TimeFormats]]]]

date_f = %d/%m/%Y # Options are '%m/%d/%Y' or '%d/%m/%Y'

date_time_f = %d/%m/%Y %H:%M # Options are '%m/%d/%Y %H:%M' or '%d/%m/%Y %H:%M'

[[wdPWS]]

HTML_ROOT = /var/www/weewx/WD

skin = PWS

[[[Units]]]

[[[[Groups]]]]

group_pressure = mbar # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'

group_rain = mm # Options are 'inch', 'cm', or 'mm'

group_rainrate = mm_per_hour # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'

group_speed = km_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = km_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

group_temperature = degree_C # Options are 'degree_F' or 'degree_C'

[[wdClientraw]]

HTML_ROOT = /var/www/weewx/WD

skin = Clientraw

[[[Units]]]

[[[[Groups]]]]

group_pressure = mbar # Options are 'inHg', 'mmHg', 'mbar', or 'hPa'

group_rain = mm # Options are 'inch', 'cm', or 'mm'

group_rainrate = mm_per_hour # Options are 'inch_per_hour', 'cm_per_hour', or 'mm_per_hour'

group_speed = km_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = km_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

group_temperature = degree_C # Options are 'degree_F' or 'degree_C'

[[wdStackedWindRose]]

HTML_ROOT = /var/www/weewx/WD

skin = StackedWindRose

[[[Units]]]

[[[[Groups]]]]

group_speed = km_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = km_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

[[[[TimeFormats]]]]

date_f = %d/%m/%Y # Options are '%m/%d/%Y' or '%d/%m/%Y'

date_time_f = %d/%m/%Y %H:%M # Options are '%m/%d/%Y %H:%M' or '%d/%m/%Y %H:%M'

[[wdSteelGauges]]

HTML_ROOT = /var/www/weewx/WD

skin = SteelGauges

[[[Units]]]

[[[[Groups]]]]

group_pressure = mbar # Options are 'inHg', 'mbar' or 'hPa'

group_rain = mm # Options are 'inch' or 'mm'

group_rainrate = mm_per_hour # Options are 'inch_per_hour' or 'mm_per_hour'

group_speed = km_per_hour # Options are 'mile_per_hour', 'km_per_hour', 'knot', or 'meter_per_second'

group_speed2 = km_per_hour2 # Options are 'mile_per_hour2', 'km_per_hour2', 'knot2', or 'meter_per_second2'

group_temperature = degree_C # Options are 'degree_F' or 'degree_C'

[[FTP]]

# FTP'ing the results to a webserver is treated as just another report,

# albeit one with an unusual report generator!

skin = Ftp

# If you wish to use FTP, uncomment and fill out the next four lines.

#user = replace with the ftp username

#password = replace with the ftp password

#server = replace with the ftp server name, e.g, www.threefools.org

#path = replace with the ftp destination directory (e.g., /weather)

# Set to True for a secure FTP (SFTP) connection. Not all servers

# support this.

secure_ftp = False

# To upload files from something other than what HTML_ROOT is set

# to above, specify a different HTML_ROOT here.

#HTML_ROOT = /var/www/weewx

# Most FTP servers use port 21

port = 21

# Set to 1 to use passive mode, zero for active mode

passive = 1

[[RSYNC]]

# rsync'ing to a webserver is treated as just another report

skin = Rsync

# If you wish to use rsync, you must configure passwordless ssh using

# public/private key authentication from the user account that weewx

# runs as to the user account on the remote machine where the files

# will be copied.

#

# The following three lines determine where files will be sent.

#server = replace with the rsync server name, e.g, www.threefools.org

#path = replace with the rsync destination directory (e.g., /weather)

#user = replace with the rsync username

# Rsync can be configured to remove files from the remote server if

# they don't exist under HTML_ROOT locally. USE WITH CAUTION: if you

# make a mistake in the remote path, you could could unintentionally

# cause unrelated files to be deleted. Set to 1 to enable remote file

# deletion, zero to allow files to accumulate remotely.

delete = 0

##############################################################################

# This service acts as a filter, converting the unit system coming from

# the hardware to a unit system in the database.

[StdConvert]

# The target_unit affects only the unit system in the database. Once

# chosen it cannot be changed without converting the entire database.

# Modification of target_unit after starting weewx will result in

# corrupt data - the database will contain a mix of US and METRIC data.

#

# The value of target_unit does not affect the unit system for

# reporting - reports can display US, Metric, or any combination of units.

#

# In most cases, target_unit should be left as the default: US

#

# In particular, those migrating from a standard wview installation

# should use US since that is what the wview database contains.

# DO NOT MODIFY THIS VALUE UNLESS YOU KNOW WHAT YOU ARE DOING!

target_unit = US # Options are 'US', 'METRICWX', or 'METRIC'

##############################################################################

# This section can adjust data using calibration expressions.

[StdCalibrate]

[[Corrections]]

# For each type, an arbitrary calibration expression can be given.

# It should be in the units defined in the StdConvert section.

# Example:

foo = foo + 0.2

pressure = pressure + 0.057

rain = rain * 1.4

##############################################################################

# This section is for quality control checks. If units are not specified,

# values must be in the units defined in the StdConvert section.

[StdQC]

[[MinMax]]

barometer = 26, 32.5, inHg

outTemp = -40, 120, degree_F

inTemp = 10, 120, degree_F

outHumidity = 0, 100

inHumidity = 0, 100

windSpeed = 0, 120, mile_per_hour

pressure = 24, 34.5, inHg

##############################################################################

# This section controls the origin of derived values.

[StdWXCalculate]

# Derived quantities are calculated by this service. Possible values are:

# hardware - use the value provided by hardware

# software - use the value calculated by weewx

# prefer_hardware - use value provide by hardware if available,

# otherwise use value calculated by weewx

pressure = prefer_hardware

barometer = prefer_hardware

altimeter = prefer_hardware

windchill = prefer_hardware

heatindex = prefer_hardware

dewpoint = prefer_hardware

inDewpoint = prefer_hardware

rainRate = prefer_hardware

##############################################################################

# For hardware that supports it, this section controls how often the

# onboard clock gets updated.

[StdTimeSynch]

# How often to check the weather station clock for drift (in seconds)

clock_check = 14400

# How much it can drift before we will correct it (in seconds)

max_drift = 5

##############################################################################

# This section is for configuring the archive service.

[StdArchive]

# If the station hardware supports data logging then the archive interval

# will be downloaded from the station. Otherwise, specify it (in seconds).

archive_interval = 60

# How long to wait (in seconds) before processing new archive data. Must

# be greater than zero.

archive_delay = 15

# If possible, new archive records are downloaded from the station

# hardware. If the hardware does not support this, then new archive

# records will be generated in software.

# Set the following to "software" to force software record generation.

record_generation = software

# Whether to include LOOP data in hi/low statistics

loop_hilo = True

# The data binding used to save archive records

data_binding = wx_binding

##############################################################################

# This section binds a data store to a database.

[DataBindings]

[[wx_binding]]

# The database must match one of the sections in [Databases].

# This is likely to be the only option you would want to change.

#database = archive_sqlite

database = archive_mysql

# The name of the table within the database

table_name = archive

# The manager handles aggregation of data for historical summaries

manager = weewx.wxmanager.WXDaySummaryManager

# The schema defines the structure of the database.

# It is *only* used when the database is created.

schema = schemas.wview.schema

[[wd_binding]]

manager = weewx.manager.DaySummaryManager

schema = user.weewxwd3.schema

table_name = archive

database = weewxwd_sqlite

[[raw_binding]]

database = raw_mysql

table_name = raw

manager = weewx.manager.Manager

schema = user.raw.schema

##############################################################################

# This section defines various databases.

[Databases]

# A SQLite database is simply a single file

[[archive_sqlite]]

database_type = SQLite

database_name = weewx.sdb

# MySQL

[[archive_mysql]]

database_type = MySQL

database_name = weewx

[[weewxwd_mysql]]

database_name = weewxwd

database_type = MySQL

host = localhost

user = **************************

password = **************************

[[weewxwd_sqlite]]

database_name = weewxwd.sdb

root = %(WEEWX_ROOT)s

database_type = SQLite

[[raw_mysql]]

database_name = mesowx

host = localhost

user = **************************

password = **************************

database_type = MySQL

##############################################################################

# This section defines defaults for the different types of databases.

[DatabaseTypes]

# Defaults for SQLite databases

[[SQLite]]

driver = weedb.sqlite

# Directory in which the database files are located

SQLITE_ROOT = /var/lib/weewx

# Defaults for MySQL databases

[[MySQL]]

driver = weedb.mysql

# The host where the database is located

host = localhost

# The user name for logging in to the host

user = **************************

# The password for the user name

password = **************************

##############################################################################

# This section configures the internal weewx engine.

[Engine]

[[Services]]

# This section specifies the services that should be run. They are

# grouped by type, and the order of services within each group

# determines the order in which the services will be run.

prep_services = weewx.engine.StdTimeSynch

data_services = ,

process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.weewxwd3.WdWXCalculate

archive_services = weewx.engine.StdArchive, user.weewxwd3.WdArchive

restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.wcloud.WeatherCloud, user.sync.SyncService

report_services = weewx.engine.StdPrint, weewx.engine.StdReport

##############################################################################

# Options for extension 'Weewx-WD'

[Weewx-WD]

data_binding = wd_binding

[[Averages]]

[[Rainfall]] Jan =70,8 mm

Feb =30,7 mm

Mar =28,5 mm

Apr =59,0 mm

May =

Jun =

Jul =

Aug =

Sep =

Oct =

Nov =

Dec =

############################################################################################

[RemoteSync]

#

# This section is for configuration of the MesoWx remote sync service/plugin. This service

# will synchronize archive and/or raw/LOOP data to a Meso web server instance over HTTP.

#

# The base URL of your Meso instance

remote_server_url = http://www.meteolebrija.es/web/meso/

# The Meso entity ID for archive data

archive_entity_id = weewx_archive

# The Meso security required to update the archive data entity

archive_security_key = **************************

# The Meso entity ID for raw data

raw_entity_id = weewx_raw

# The Meso security key required to update the raw data entity

raw_security_key = **************************

############################################################################################

Luc Heijst

unread,

May 21, 2016, 11:55:53 AM5/21/16

to weewx-user

Edu,

I see nothing in your weewx.conf file what should point to the duplicate key error you have. May be as a last try: reboot your PI, then we know for sure no 'forgotten' instances are running somewhere.

Also I ran out of ideas where to look! :-)

Luc

PS. The numbers in the weewx-WD section: should they contain comma's? Most of the time we use the US standard with a decimal point (just a remark, do not know if it is good or wrong)

Edu GG

unread,

May 21, 2016, 12:12:54 PM5/21/16

to weewx...@googlegroups.com

thank you Luc, the wd data, and clientraw.txt (the file that interests me most.) is running with commas, weather-display is so.

after rebooting the error continues:

May 21 18:10:26 raspberrypi weewx[2753]: manager: added record 2016-05-21 18:10:00 CEST (1463847000) to daily summary in 'weewxwd.sdb'

May 21 18:10:27 raspberrypi weewx[2753]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463847000' for key 'PRIMARY'

is not a critical error, but I see in the log, creaks a bit.

Thank you.

Glenn McKechnie

unread,

May 21, 2016, 12:43:09 PM5/21/16

to weewx...@googlegroups.com

On 22 May 2016 at 01:29, Edu GG <edu.g...@gmail.com> wrote:
> thank you Andrew for recommend, I always type stop twice, for receive de not
> running notification.

[...]

>
> # This section binds a data store to a database.
>
> [DataBindings]
>

[...]

>
> [[raw_binding]]
> database = raw_mysql
> table_name = raw
> manager = weewx.manager.Manager
> schema = user.raw.schema
>
> ##############################################################################
>
> # This section defines various databases.
>
> [Databases]

[...]

>
> [[raw_mysql]]
> database_name = mesowx
> host = localhost
> user = **************************
> password = **************************
> database_type = MySQL

These are remnants from raw.py aren't they? and you're using sync.py.

What's your "dataSource" entry in config.json? If it's the same as the
above then I suspect that's your double entry.
It might pay to post that file here (less keys and other secrets)

--

Cheers
Glenn

Edu GG

unread,

May 21, 2016, 1:15:06 PM5/21/16

to weewx-user

here it is:

{

/*

* Define your database connection(s). Multiple can be defined separated by a comman.

* The key is the data source ID and is used to reference the it elsewhere in the config. The

* ID must be unique. Each data source must define a 'type' property; the only supported type

* currently is 'mysql'. The rest of the properties required depend on the type

* (see examples below).

*/

"dataSource" : {

"weewx_mysql" : { // the data source ID

"type" : "mysql",

"host" : "localhost",

"user" : "*************",

"password" : "*************",

"database" : "weewx"

},

"raw_mysql" : { // the data source ID

"type" : "mysql",

"host" : "localhost",

"user" : "************",

"password" : "**************",

"database" : "mesowx"

}

},

/*

* Define your data entities. Entities are effectively database tables containing columns

* of your data. The key is the ID of the entity which is used to reference the entity in

* the HTTP API. Each entity must define a 'type' property; the only supported type is

* currently 'table' (see examples below).

*/

"entity" : {

// This entity definition is for a typical Weewx archive table.

"weewx_archive" : { // the entity ID

// the entity type

"type" : "table",

// the dataSource value should be the ID of a data source defined above in the dataSource section

"dataSource" : "weewx_mysql",

// the database table name

"tableName" : "archive",

// section for access control configuration

"accessControl" : {

// for allowing remote updating of data

"update" : {

// set 'allow' to true to allow this entity's data to be updated

"allow" : true,

// the securityKey that must be sent along with each request to update

// it's recommended to generate a random key, i.e.

// http://www.random.org/strings/?num=1&len=20&digits=on&upperalpha=on&loweralpha=on&unique=on&format=html&rnd=new

// An empty key is not allowed.

"securityKey" : "***********************"

}

},

.......

// This example shows a configuration with a retention policy

"weewx_raw" : {

"type" : "table",

"dataSource" : "raw_mysql",

"tableName" : "raw",

"accessControl" : {

"update" : {

"allow" : true,

"securityKey" : "*****************"

}

},

...........

Greetings,

Edu.

Glenn McKechnie

unread,

May 21, 2016, 8:21:13 PM5/21/16

to weewx...@googlegroups.com

I think mesowx is trying to update the weewx database, it should be
using the archive table in the mesowx database

I'm struggling to get my head around it as I keep losing my place in
all these messages and associated snippets; so to clarify for me, are
the weewx daemon and the mesowx installation on the same machine?

I originally thought that they were seperate, thus the use of sync.py.
But with raw.py artifacts and the language in /config.json I now
suspect that they are both together?

I see weewx talking to a mysql database on localhost named weewx

> [DataBindings]
>
> [[wx_binding]]
[...]
> database = archive_mysql

mesowx is talking to 2 mysql databases on local host - weewx and mesowx

"dataSource" : {
"weewx_mysql" : { // the data source ID

[...]
"database" : "weewx"

"raw_mysql" : { // the data source ID

[...]
"database" : "mesowx"

mesowx should be accessing its own mysql database named - mesowx - on localhost.
That database should consist of 2 tables archive and raw. These are
the two tables that sync should be updating, using weewx's database
wewwx as the data source.
This is the database that meso should working from.

That should explain the duplicate entries - IF - I'm making sense of it.

Do the above changes first , then look at this next section

===============================================

The 2 weewx.conf entries (from a raw.py installation?) I mentioned in
my earlier mail (see copy below) are, I believe, redundant, try
commenting them out and I believe nothing will break - probaly nothing
will change either as nothing should be accessing them anyway - but
who knows?

[...]

>
> # This section binds a data store to a database.
>
> [DataBindings]
>

[...]

>
> [[raw_binding]]
> database = raw_mysql
> table_name = raw
> manager = weewx.manager.Manager
> schema = user.raw.schema
>
> ##############################################################################
>
> # This section defines various databases.
>
> [Databases]

[...]

>
> [[raw_mysql]]
> database_name = mesowx
> host = localhost
> user = **************************
> password = **************************
> database_type = MySQL

> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

Cheers
Glenn

Glenn McKechnie

unread,

May 21, 2016, 8:34:54 PM5/21/16

to weewx...@googlegroups.com

Forgot to add - as a sanity check - have a look at the mesowx database
and see if there are two tables in it archive and raw, and that they
have recent data in them.

eg:-

mysql -uroot -p mesowx
Enter password:
mysql> show tables;
[...]
+---------------------+
| archive |
| raw |
+---------------------+

and if they do exist that they have recent data in them eg:-

mysql> select * from archive where dateTime > 1463876000;
mysql> select * from raw where dateTime > 1463876000;

or use install/use phpmyadmin from a browser

--

Cheers
Glenn

Luc Heijst

unread,

May 21, 2016, 10:01:09 PM5/21/16

to weewx-user

Sync.py has a lot of debug messages in it. When you set debug=1 in weewx.conf and then restart weewx you might get a clue what sync is doing.
Luc

Edu GG

unread,

May 22, 2016, 7:03:58 AM5/22/16

to weewx-user

hello Glenn,

yes, the databases are in the same machine. All is in the same machine.

my mesowx database only have raw table:

mysql> show tables;

+------------------+

| Tables_in_mesowx |

+------------------+

| raw |

+------------------+

1 row in set (0.00 sec)

commenting the raw databinding and databases section, the error continues:

May 22 12:48:22 raspberrypi weewx[7712]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463914080' for key 'PRIMARY'

Luc, after debugging on level 1, this is the log returns:

May 22 12:53:11 raspberrypi weewx[7829]: sync raw: send record 2016-05-22 12:53:11 CEST (1463914391)

May 22 12:53:25 raspberrypi weewx[7829]: sync raw: send record 2016-05-22 12:53:25 CEST (1463914405)

May 22 12:53:26 raspberrypi weewx[7829]: manager: added record 2016-05-22 12:53:00 CEST (1463914380) to database 'weewx'

May 22 12:53:26 raspberrypi weewx[7829]: manager: added record 2016-05-22 12:53:00 CEST (1463914380) to daily summary in 'weewx'

May 22 12:53:26 raspberrypi weewx[7829]: restx: StationRegistry: wait interval (226 < 604800) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: CWOP: wait interval (60 < 600) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: AWEKAS: wait interval (226 < 300) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: WeatherCloud: wait interval (226 < 600) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: get record 2016-05-22 12:53:00 CEST (1463914380); last synced 2016-05-22 12:49:00 CEST (1463914140)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: send record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: put record in queue 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: reportengine: Running reports for latest time in the database.

May 22 12:53:26 raspberrypi weewx[7829]: reportengine: Running report StandardReport

May 22 12:53:26 raspberrypi weewx[7829]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463914380' for key 'PRIMARY'

May 22 12:53:26 raspberrypi weewx[7829]: reportengine: Found configuration file /etc/weewx/skins/Standard/skin.conf for report StandardReport

May 22 12:53:27 raspberrypi weewx[7829]: restx: PWSWeather: Published record 2016-05-22 12:53:00 CEST (1463914380)

after grep the 1463914380 dateTime that is the Duplicate entry in this example, the log give me:

May 22 12:53:26 raspberrypi weewx[7829]: manager: added record 2016-05-22 12:53:00 CEST (1463914380) to database 'weewx'

May 22 12:53:26 raspberrypi weewx[7829]: manager: added record 2016-05-22 12:53:00 CEST (1463914380) to daily summary in 'weewx'

May 22 12:53:26 raspberrypi weewx[7829]: restx: StationRegistry: wait interval (226 < 604800) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: CWOP: wait interval (60 < 600) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: AWEKAS: wait interval (226 < 300) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: restx: WeatherCloud: wait interval (226 < 600) has not passed for record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: get record 2016-05-22 12:53:00 CEST (1463914380); last synced 2016-05-22 12:49:00 CEST (1463914140)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: send record 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: sync archive: put record in queue 2016-05-22 12:53:00 CEST (1463914380)

May 22 12:53:26 raspberrypi weewx[7829]: sync: http request failed (500 Internal Server Error): Unable to update entity due to unexpected error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1463914380' for key 'PRIMARY'

May 22 12:53:27 raspberrypi weewx[7829]: restx: PWSWeather: Published record 2016-05-22 12:53:00 CEST (1463914380)

the "not passed for record" appears because the time for report it's shorter than expecting for server... I think.

Cheers and happy sunday!

Edu.

Andrew Milner

unread,

May 22, 2016, 7:15:57 AM5/22/16

to weewx-user

I don't know the full ins and outs of sync - as I do not use sync, only raw. I have a fine offset which generates loop records every 48 seconds and archive records are created every 5 minutes. However I do recall that meso used to put into the raw table at the time of an archive record both the loop record and a resulting archive record. I got around this apparent duplication in the raw table (I only wanted loop data) by dropping records if the new record was within 2 seconds of a previous loop record.

I suspect that what you are seeing is a result of these apparent duplicate records. I forget the 'fix' |I made to meso/raw to prevent the duplication, but I do remember doing it!!

Hope this may help. Check your raw table in fine detail and see if it contains close/repeat records.

Luc Heijst

unread,

May 22, 2016, 7:19:25 AM5/22/16

to weewx-user

Hi Edu,

Well, Glen found your problem! Good catch, Glenn!

In a situation you have both weewx and mesowx on the same machine, you should NOT use module sync.py (which synchronises both archive and raw data), but use instead module raw.py (which only synchronises raw data).

In such a case the weewx database must be a MySQL database.

MesoWX will use the data of the weewx MySQL database and the raw MySQL database which is filled by module raw.py.

See my thread: https://groups.google.com/forum/#!searchin/weewx-user/raw/weewx-user/eAUsTqR8yYQ/UVMxAAtgAAAJ how to use raw.py

Luc

Message has been deleted

Luc Heijst

unread,

May 22, 2016, 7:37:07 AM5/22/16

to weewx-user

Hi Andrew,

raw.py takes care of loop data with the same timestamp as the last sent record as you can see in the code snipped below.

However, it won't skip loop data with a time stamp since last send of less than 2 seconds.

In such a case line:

    if dateTime != self.lastLoopDateTime:

should be change into:

    if dateTime > (self.lastLoopDateTime + 2):

Luc

Part of raw.py - version 0.4
===================================================================

def newLoopPacket(self, event):
    packet = event.packet
    prune_period = 300
    # It's possible for records with duplicate dateTimes - this occurs when an archive packet
    # is processed since the LOOP packets are queued up and then returned immediately when
    # looping resumes, coupled with the fact that for Vantage Pro consoles the dateTime value is
    # added by weewx. So, for database storage, skip the duplicates until we get a new one to
    # avoid a duplicate key error, but publish them all to redis regardless.
    dateTime = packet['dateTime']
    if dateTime != self.lastLoopDateTime:
        self.dbm.addRecord(packet)
        self.lastLoopDateTime = dateTime

===================================================================

Glenn McKechnie

unread,

May 22, 2016, 7:58:38 AM5/22/16

to weewx...@googlegroups.com

Thanks Luc, I felt like I was fumbling the ball rather than catching it though.

So, to clarify - Edu's meso/include/config.json is configured
correctly for a raw.py installation.
The current, mesowx, database is also a database that was created by
raw.py at some stage and therefore is correct to leave as it is.

The only thing Edu needs to do is switch out weewx/user/sync.py and
replace it with weewx/user/raw.py AND reinstate the raw_mysql entries
in weewx.conf that I said to comment out. ( I don't think either the
user or password need to be included in the [[raw_mysql]] section
either, but if they are left in nothing should break, they'll just be
ignored as the config.json has them hardcoded in - tho I'm not sure
which is coded for precedence?)

[DataBindings]

[...]

[[raw_binding]]
database = raw_mysql
table_name = raw
manager = weewx.manager.Manager
schema = user.raw.schema

##############################################################################

# This section defines various databases.

[Databases]

[...]

[[raw_mysql]]
database_name = mesowx
host = localhost
user = **************************
password = **************************
database_type = MySQL

> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to weewx-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

Cheers
Glenn

Andrew Milner

unread,

May 22, 2016, 7:58:54 AM5/22/16

to weewx-user

Luc - the change I made was to the older version of raw - not the LH version!!! Just had a look at the LH I am running and see it is indeed as you describe. Out of curiosity was the +2 version one I sent to you????? I am not presently dropping the close records since I went to v3 weewx and started using your version of raw!!! Would be nice to get the 'feature' back again though!!

Andrew

Luc Heijst

unread,

May 22, 2016, 8:34:22 AM5/22/16

to weewx-user

On your request version 0.4.3-lh where loop records are skipped within 2 seconds from the sent previous one.

Luc

raw_0.4.3-lh.py

Luc Heijst

unread,

May 22, 2016, 8:43:43 AM5/22/16

to weewx-user

Glenn,

Module raw.py only uses two parameters:

data_binding = raw_binding

data_limit = 24

The database username and password are found via the data_binding parameter.

No need to put them in the raw section; these settings are ignored.

Luc

Andrew Milner

unread,

May 22, 2016, 9:47:06 AM5/22/16

to weewx-user

Thanks Luc - have installed it. What was in 0.4.2 by the way - since I have just gone from 0.4.1 to 0.4.3 and could not find any other line changes when I eyeballed 4.1 v 4.3 so must have missed the 4.2 alteration!!!

Andrew

Edu GG

unread,

May 22, 2016, 9:57:16 AM5/22/16

to weewx-user

sorry but I don't undestand that what I do...

I would have to use only raw?, deactivating sync?

Deleting user.sync.SyncService from restful_services, loses data refresh every 10 seconds on the web.

Greetings to all!

Edu.

Luc Heijst

unread,

May 22, 2016, 9:58:19 AM5/22/16

to weewx-user

Andrew,

In verssion 4.2 line 117:

raise Exception('raw: prune failed after %d attemps' % max_tries)

was changed into:

# Don't raise an exception; it's not worth stopping weewx for this

syslog.syslog(syslog.LOG_ERR, 'raw: prune failed after %d attemps' % max_tries)

Luc

Andrew Milner

unread,

May 22, 2016, 10:26:06 AM5/22/16

to weewx-user

Is your web server also on the same machine??

Andrew Milner

unread,

May 22, 2016, 10:40:37 AM5/22/16

to weewx-user

raw.py will update the raw table in mesowx database when every loop record is received. How often does your station provide a loop record?

Edu GG

unread,

May 22, 2016, 11:22:22 AM5/22/16

to weewx-user

hello Andrew,

yes, the apache2 also running on the same raspberrypi.

in the post above you can see my weewx.conf in [StdArchive] section,

archive_interval = 60 (1 minute)

and js/Config.js:

Config.realTimePollingInterval = 10000; (10 seconds)

Andrew Milner

unread,

May 22, 2016, 1:13:19 PM5/22/16

to weewx-user

Meso may well poll the raw database every 10 seconds, but the records will only be put in the raw database when loop records are emitted by the station's driver (assuming the station emits loop records and not just archive records) - otherwise you will only get updates every archive interval regardless of how often you 'poll' the database!!!

Andrew Milner

unread,

May 22, 2016, 1:22:28 PM5/22/16

to weewx-user

..... and as a further question - have you used wee-device to change the archive interval on the station from the factory default of 60 minutes to also be 1 minute?

Edu GG

unread,

May 22, 2016, 1:37:04 PM5/22/16

to weewx-user

hello Andrew,

yes, I used wee_device --set-interval=1

the station base receive data from sensor every 8 seconds:

after restart for you review:

May 22 19:31:24 raspberrypi weewx[3653]: ws23xx: connection changed from unknown to cable

May 22 19:31:24 raspberrypi weewx[3653]: ws23xx: using 8 second polling interval for cable connection

Greetings,

Edu.

Edu GG

unread,

May 22, 2016, 3:53:34 PM5/22/16

to weewx...@googlegroups.com

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/Z4sW4oDDboo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Edu GG

unread,

May 22, 2016, 3:57:25 PM5/22/16

to weewx...@googlegroups.com

you advise me cancel the sync.py tasks and work only with raw?.

thus also it is operative the update in real time?. Thank you.

Edu.

L.J.M. Heijst

unread,

May 22, 2016, 4:04:00 PM5/22/16

to weewx...@googlegroups.com

Hi Edu,

Did you have a look at the thread I sent you in this message? It contains a doc file too with examples how to set up raw.py.

Luc

Verstuurd vanaf mijn iPad

--

Luc Heijst

unread,

May 22, 2016, 4:10:01 PM5/22/16

to weewx-user

The link seems not correct. Search in the weewx forum for: Upgraded raw plugin for MesoWx v0.4.0
Luc

Edu GG

unread,

May 22, 2016, 4:17:04 PM5/22/16

to weewx...@googlegroups.com

Yes Luc,

this is the raw.py version that's installed... thanks for recommend carefully read the examples. I'll do so.

Greetings.

2016-05-22 22:10 GMT+02:00 Luc Heijst <ljm.h...@gmail.com>:

The link seems not correct. Search in the weewx forum for: Upgraded raw plugin for MesoWx v0.4.0
Luc

Andrew Milner

unread,

May 22, 2016, 9:37:33 PM5/22/16

to weewx-user

Edu - three questions to try and help see what is working as expected, and what is not:

Is weather station and driver supplying the expected data ?

a) When you run weewx with debug=1 set in weewx.conf do the logs show you receiving a LOOP record every 8 seconds and a REC record every minute?

Is raw.py getting that data ?

b) Does the log show raw.py adding a record to mesowx every 8 seconds?

Is raw.py adding the data to the mesowx database raw table ?

c) If you look at your mesowx raw table does that show a new raw record has been added every 8 seconds?

Glenn McKechnie

unread,

May 22, 2016, 10:14:53 PM5/22/16

to weewx...@googlegroups.com

Luc,
the notes ( installation instructions raw_0.4.2-lh.txt ) in that
thread need to have the note regarding [[Services]] added to them in
the " - Edit weewx.conf " section.

[...]

edit weewx.conf to add

user.raw.RawService to archive services, as in...

[Engine]
[[Services]]
archive_services = weewx.engine.StdArchive , user.raw.RawService

[...]

Also Luc, it may well be time to publish all these Luc.py's for Mesowx
in the one place, warts and all? It's getting to be a bit of a
convoluted chase trying to get the most recent updates together from
their various locations; I'm finding it a chore and I'm at least
semi-familiar with the process and files. I'd expect newcomers to be
having an even harder time of it.
You mentioned GitHub for the files. I'd suggest the weewx/wiki for the
installation specific notes (If they are there, I and others could add
to them?) All providing that's okay with Tom, Matthew and other
involved parties.

Edu,
if you haven't done so already, you'll need to remove "
user.sync.SyncService " from restful_services (or where ever it is in
your config file ) and add the user.raw.RawService call to
archive_services (as per snippet above)
I see the your installation at http://www.meteolebrija.es/web/ is
running okay so hopefully that is with raw.py, and there are no
unwanted messages in your logs?

On 23 May 2016 at 06:10, Luc Heijst <ljm.h...@gmail.com> wrote:
> The link seems not correct. Search in the weewx forum for: Upgraded raw plugin for MesoWx v0.4.0
> Luc
>

> --

> You received this message because you are subscribed to the Google Groups "weewx-user" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

Cheers
Glenn

Glenn McKechnie

unread,

May 22, 2016, 10:19:13 PM5/22/16

to weewx...@googlegroups.com

Argh, sorry for the noise folks

That link for "Upgraded raw plugin for MesoWx v0.4.0" is
https://groups.google.com/forum/#!searchin/weewx-user/raw.py/weewx-user/eAUsTqR8yYQ/UVMxAAtgAAAJ

with the latest upgrade notes posted by Luc on the 26th March

--

Cheers
Glenn

Luc Heijst

unread,

May 22, 2016, 11:22:42 PM5/22/16

to weewx-user

Glenn,

First of all the weewx wiki should only contain one descriptive line of text with a link to the (not created yet) sources and documentation on github.
I have two things to solve first:

I don't want to include the original files and description of mesowx because the files are not mine and also i have no intention to maintain those files.
So, I need a way to tell the user which files they should use from the original sources and which files need to be replaced / added from my sources.

Secondly I want better names for the modules raw and sync, because new users often don't know which module / modules they should use.
The names should reflect their use:
Sync raw-only on local mesowx machine
Sync both archive and raw on remote mesowx machine
Any suggestions for those module names?

When we don't use anymore the module names 'raw' and 'sync' the user won't mix up anymore the original installation description of mesowx and our weewx installation instruction for mesowx. It even doesn't matter if the original mesowx raw and sync modules are copied by a new user to the weewx user directory (they simply won't be used anymore).

Currently I'm still very busy with developing formula's for translating the raw values of our new meteostick driver to their iso values.
I learned recently how to put my files on github, but how the version control works and how to deal with a pull request is a complete other chapter. I don't have the time right now to sort this all out.

Luc

Andrew Milner

unread,

May 22, 2016, 11:25:02 PM5/22/16

to weewx-user

..... and an apology from me also to Edu - as I also have now seen that your site appears to be ok - it wasn't when I had looked earlier and thus my message trying to be helpful. So, also from me, apologies to all for my unwanted additional noise!!

Andrew

Andrew Milner

unread,

May 22, 2016, 11:34:01 PM5/22/16

to weewx-user

Luc

From my viewpoint, as I am not syncing anything anywhere - raw CREATES a raw table containing the LOOP data whilst SYNC syncs one or more databases (and maybe one or more tables but as I do not use sync I do not know) with copies held elsewhere - locally or remote. Thus, in my simple understanding, raw can remain being called raw and sync can remain sync. However, I am unsure where the 'sync' part comes from in your version of raw - as I am using raw but not syncing anything anywhere, and have no sync parameters established anywhere - and I am too lazy to go and re-read the original meso writeup again!!

Andrew

Glenn McKechnie

unread,

May 23, 2016, 12:12:29 AM5/23/16

to weewx...@googlegroups.com

On 23 May 2016 at 13:22, Luc Heijst <ljm.h...@gmail.com> wrote:
> Glenn,
>
> First of all the weewx wiki should only contain one descriptive line of text with a link to the (not created yet) sources and documentation on github.

Fair enough, so long as there is something there that points to a
weewx addon / plugin. I'm still finding add-ons tucked away in
various places that I haven't found via more direct weewx links.

> I have two things to solve first:
>
> I don't want to include the original files and description of mesowx because the files are not mine and also i have no intention to maintain those files.
> So, I need a way to tell the user which files they should use from the original sources and which files need to be replaced / added from my sources.

I see a clear separation between your scripts and Mesowx even now.
Yours are the links between weewx and mesowx. Mesowx and all its stand
alone java stays and is maintained over there. No confusion in my
mind.

It's a shame that the original site seems to have a disconnect with
the weewx side. Perhaps it's because of the direction of discovery.
Weewx will always come first being the parent program, the add on /
child comes a natural second. I know you've posted duplicate info
their without response. I also feel that a lot of the recent posts
seem to have nothing to do with the original concept of mesowx,
meso-weewx

> Secondly I want better names for the modules raw and sync, because new users often don't know which module / modules they should use.
> The names should reflect their use:
> Sync raw-only on local mesowx machine
> Sync both archive and raw on remote mesowx machine
> Any suggestions for those module names?
> When we don't use anymore the module names 'raw' and 'sync' the user won't mix up anymore the original installation description of mesowx and our weewx installation instruction for mesowx. It even doesn't matter if the original mesowx raw and sync modules are copied by a new user to the weewx user directory (they simply won't be used anymore).

Roll the 3 scripts into one and give that a distinct, unique name -
mesolink.py ??. That combined script is then installed under
weewx/user and the only sections that light up (are used) are the ones
that have their individual configuration sections named in weewx.conf.
That should cut down on the mental / physical gymnastics regarding
user installations and concentrate the installation decisions into the
commented (ie: documented) weewx.conf sections. Comment out [[Raw]]
and it no longer runs, add [[Sync]] and that section of mesolink.py
passes the if test and runs.

If you want to stick to individual scripts,

raw becomes mesoloc.py ( for meso local)
sync becomes mesorem.py (for meso remote)

It's clear that they belong to mesowx, it's also a little clearer
which one may be needed?

or some other combination of parent program (mesowx) and usage (local / remote)

Whichever way it goes, I'd also give some thought into merging the 2
databases into 1. Create them both with archive and raw tables
included.
The old raw.py can use the sync generated database if it was created
first, and is local.
The old sync.py can also use the raw generated database it it was
created first, and is local.

> Currently I'm still very busy with developing formula's for translating the raw values of our new meteostick driver to their iso values.
> I learned recently how to put my files on github, but how the version control works and how to deal with a pull request is a complete other chapter. I don't have the time right now to sort this all out.

I'm with you there, and it was only a suggestion intended to give you
an encouraging push, if you needed one :-)

I'll help where I can with mesowx (documentation, testing of raw,
retain, sync mesolink?), I can't with meteostick as I have no need /
use for one. :-)

--

Cheers
Glenn

Luc Heijst

unread,

May 23, 2016, 12:18:42 AM5/23/16

to weewx-user

Andrew,
It's not that straitforward as you mention: module sync syncs just one database: the weewx archive database. At startup of weewx sync adds all missed archive records to its remote archive database. Module sync just writes loop data to a remote raw database, no syncing. When weewx is stopped for a while the remote raw database will have a data gap.
You could use sync to sync to a local machine, but then you would end up with two archive databases, or when you are unlucky, get duplicate record error messages among others. You could technically use both raw and sync modules on one machine. I did it a while to have both a local and a remote mesowx system for testing.
You could use a section 'raw' in weewx.conf thinking you control the writing of the remote raw database with these parameters.
So, really, I need better descriptive names!
Luc

Glenn McKechnie

unread,

May 23, 2016, 12:22:07 AM5/23/16

to weewx...@googlegroups.com

More noise.
Where I said...

The old raw.py can use the sync generated database if it was created
first, and is local.
The old sync.py can also use the raw generated database it it was
created first, and is local.

The thought is more along the lines of

old raw.py or old sync.py could only deal with its own...
the new mesoloc.py / mesorem.py or combined mesolink.py can handle
the one for whichever case it encounters.

??

--

Cheers
Glenn

Luc Heijst

unread,

May 23, 2016, 12:33:04 AM5/23/16

to weewx-user

Glenn,

You made my day! ( or is it night? )
One combined module which handles all! But not all at the same time; one has to choose!

The weewx.conf section for the combined module could have a parameter which asks if mesowx is / will be on the localhost yes or no.
Depending on this answer other parameters are needed which can be explained in the documentation.

I will give this a thought.
Luc

Edu GG

unread,

May 23, 2016, 2:32:03 AM5/23/16

to weewx-user

Hello Andrew,

a) When you run weewx with debug=1 set in weewx.conf do the logs show you receiving a LOOP record every 8 seconds and a REC record every minute?

May 23 08:21:23 raspberrypi weewx[6561]: ws23xx: station archive interval is 1 minutes

May 23 08:21:31 raspberrypi weewx[6561]: ws23xx: using 8 second polling interval for cable connection

Is raw.py getting that data ?
b) Does the log show raw.py adding a record to mesowx every 8 seconds?

May 23 08:24:01 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:01 CEST (1463984641)

May 23 08:24:14 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:15 CEST (1463984655)

May 23 08:24:23 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:23 CEST (1463984663)

May 23 08:24:32 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:32 CEST (1463984672)

May 23 08:24:46 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:46 CEST (1463984686)

May 23 08:24:55 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:55 CEST (1463984695)

May 23 08:25:04 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:25:04 CEST (1463984704)

Is raw.py adding the data to the mesowx database raw table ?
c) If you look at your mesowx raw table does that show a new raw record has been added every 8 seconds?

+------------+---------+-----------------+--------+---------+------------+-------------+----------------+---------+----------+------+----------+-----------+-----------+-------+-----------+------+------------+------------+------------+-----------+-----------+-----------+-----------+-----------+-----------+-------------+-------------+------------+------------+------------+------------+----------+----------+-----------------+--------------------+

+------------+---------+-----------------+--------+---------+------------+-------------+----------------+---------+----------+------+----------+-----------+-----------+-------+-----------+------+------------+------------+------------+-----------+-----------+-----------+-----------+-----------+-----------+-------------+-------------+------------+------------+------------+------------+----------+----------+-----------------+--------------------+

Greetings,

Edu.

Edu GG

unread,

May 23, 2016, 2:42:46 AM5/23/16

to weewx-user

Hello Glenn,

making this changes...

the " - Edit weewx.conf " section.

[...]

edit weewx.conf to add

user.raw.RawService to archive services, as in...

[Engine]
[[Services]]
archive_services = weewx.engine.StdArchive , user.raw.RawService

[...]

if you haven't done so already, you'll need to remove "

user.sync.SyncService " from restful_services...

the daemon exiting:

May 23 08:39:24 raspberrypi weewx[6756]: ws23xx: close LinuxSerialPort

May 23 08:39:24 raspberrypi weewx[6756]: engine: Caught unrecoverable exception in engine:

May 23 08:39:24 raspberrypi weewx[6756]: **** Unit system of incoming record (0x01) differs from 'raw' table in 'mesowx' database (0x00)

May 23 08:39:24 raspberrypi weewx[6756]: **** Traceback (most recent call last):

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/engine.py", line 859, in main

May 23 08:39:24 raspberrypi weewx[6756]: **** engine.run()

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/engine.py", line 185, in run

May 23 08:39:24 raspberrypi weewx[6756]: **** self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/engine.py", line 218, in dispatchEvent

May 23 08:39:24 raspberrypi weewx[6756]: **** callback(event)

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/user/raw.py", line 128, in newLoopPacket

May 23 08:39:24 raspberrypi weewx[6756]: **** self.dbm.addRecord(packet)

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/manager.py", line 235, in addRecord

May 23 08:39:24 raspberrypi weewx[6756]: **** self._addSingleRecord(record, cursor, log_level)

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/manager.py", line 258, in _addSingleRecord

May 23 08:39:24 raspberrypi weewx[6756]: **** self._check_unit_system(record['usUnits'])

May 23 08:39:24 raspberrypi weewx[6756]: **** File "/usr/share/weewx/weewx/manager.py", line 640, in _check_unit_system

May 23 08:39:24 raspberrypi weewx[6756]: **** self.std_unit_system))

May 23 08:39:24 raspberrypi weewx[6756]: **** UnitError: Unit system of incoming record (0x01) differs from 'raw' table in 'mesowx' database (0x00)

May 23 08:39:24 raspberrypi weewx[6756]: **** Exiting.

Greetings.

Edu.

Andrew Milner

unread,

May 23, 2016, 3:08:02 AM5/23/16

to weewx-user

Edu - I am now getting totally confused. In response to the question about "is raw adding data to the database" you posted log lines which have come, I think, from sync not from raw, so I am still nopt sure if you have removed all references to sync and are using only raw since everything is on the same machine - as advised by Luc.

I would have expected the log to be something like:

May 23 09:51:05 raspberrypi weewx[15449]: restx: PWSWeather: Published record 2016-05-23 09:50:00 EEST (1463986200)
May 23 09:51:05 raspberrypi weewx[15449]: restx: Wunderground-PWS: Published record 2016-05-23 09:50:00 EEST (1463986200)
May 23 09:51:05 raspberrypi weewx[15449]: restx: WOW: Published record 2016-05-23 09:50:00 EEST (1463986200)
May 23 09:51:24 raspberrypi weewx[15449]: genimages: Generated 6 images for BigImages in 19.94 seconds
May 23 09:51:32 raspberrypi weewx[15449]: genimages: Generated 6 images for SmallImages in 7.92 seconds
May 23 09:51:39 raspberrypi weewx[15449]: GaugeGenerator: Generated 7 images for HTMLPages in 6.57 seconds
May 23 09:51:40 raspberrypi weewx[15449]: historygenerator.pyc: Generated 6 tables in 0.92 seconds
May 23 09:51:48 raspberrypi weewx[15449]: cheetahgenerator: Generated 12 files for report HTMLPages in 8.73 seconds
May 23 09:51:48 raspberrypi weewx[15449]: reportengine: copied 0 files to /home/weewx/public_html/Bootstrap
May 23 09:52:03 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:52:03 EEST (1463986323) to database 'mesowx'
May 23 09:52:56 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:52:57 EEST (1463986377) to database 'mesowx'
May 23 09:53:50 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:53:50 EEST (1463986430) to database 'mesowx'
May 23 09:54:43 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:54:44 EEST (1463986484) to database 'mesowx'
May 23 09:54:43 raspberrypi weewx[15449]: raw: deleted rawdata prior to 2016-05-16 08:50:00 EEST (1463377800)
May 23 09:55:37 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:55:37 EEST (1463986537) to database 'mesowx'
May 23 09:55:37 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:55:00 EEST (1463986500) to database 'archive'
May 23 09:55:38 raspberrypi weewx[15449]: manager: added record 2016-05-23 09:55:00 EEST (1463986500) to daily summary in 'archive'
May 23 09:55:38 raspberrypi weewx[15449]: restx: PWSWeather: Published record 2016-05-23 09:55:00 EEST (1463986500)
May 23 09:55:38 raspberrypi weewx[15449]: restx: Wunderground-PWS: Published record 2016-05-23 09:55:00 EEST (1463986500)

where you see the manager adding the loop records to the mesowx database (in my case every 48 seconds), and raw purging the raw table.

Can you confirm that you have stopped using sync and are using only raw?

Glenn McKechnie

unread,

May 23, 2016, 4:17:43 AM5/23/16

to weewx...@googlegroups.com

In your last post ..

>
>
>May 23 08:24:01 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:01 CEST (1463984641)
>May 23 08:24:14 raspberrypi weewx[6561]: sync raw: send record 2016-05-23 08:24:15 CEST (1463984655)

As Andrew notes they are sync.py entries

The table output in that post is also useful as it shows the 2nd field
header and its contents, which are the ones being reported in the
exception / error message in your next post, when you change to the
raw.py module

>May 23 08:39:24 raspberrypi weewx[6756]: **** UnitError: Unit system of incoming record (0x01) differs from 'raw' table in 'mesowx' database (0x00)

We are looking at the usUnits field which has a value of 0 -- |
dateTime | usUnits | <--> | 1463984641 | 0 |
The error message points to this

>+------------+---------+-----------------+--------+---------+------------+-------------+snip
>| dateTime | usUnits | barometer | inTemp | outTemp [...]
>+------------+---------+-----------------+--------+---------+------------+-------------+snip
>| 1463984641 | 0 | 30.051404569252 | 74.84 | 59.54 | [...]

What you have is a change in the unit type see,
http://www.weewx.com/docs/customizing.htm#units

It should be one of 3 values - decimal 1 (US), 16 (METRIC) or
17(METRICWX). I don't know what the 0 is from, or where it came from
but you're trying to add 1 (0x01 or UsUnits) to that field and weewx,
or raw.py doesn't want to do it. Changing Units mid stream is a
problem for weewx, understandably as it affects every measurement so
changing that one field needs to be reflected in every field and
that's a huge task. Looking back at the weewx.conf file you posted,
you are indeed using US units for the weewx database.
I recall you wanted Metric units for your output, yet weewx is sending
UsUnits to your database (0x01) which appears to be confirmed by the
high values for inTemp (74.84) and outTemp (59.54) that your mesowx
database shows.

It would be nice to know why that field is zero and Luc may be able to
shed some light there.

How to fix it?
How much data do you have, how important is it to keep it? I'm talking
about the meso database, NOT your weewx database.

If you, or some other magician, can conjure up some SQL magic then you
could change all those usUnits values from 0 to 1 and that should
pacify the beast.

The other less elegant way is to create a new database. Move your old
one aside and allow raw.py to create a new on and allow that to be
populated - hopefully correctly
.
Before you rush of and do that though, you may want to wait and see if
there is other advice that's perhaps less drastic, or more helpful?

Edu GG

unread,

May 23, 2016, 4:18:05 AM5/23/16

to weewx...@googlegroups.com

Hello Andrew,

no, like I mention above, if stop sync and active raw, the log returns error with units and exit.

Andrew Milner

unread,

May 23, 2016, 4:32:23 AM5/23/16

to weewx-user

Since Luc is the man who has sorted out sync and raw to work with weewx v3 perhaps it would be a good idea to do what he says - and then let us try to see what is wrong and why you get errors when you run raw instead of sync!!! As he has said raw is for local use and sync is for remote use. You have both archive and meso databases on your local machine, so should be using raw.py

Edu GG

unread,

May 23, 2016, 4:46:27 AM5/23/16

to weewx-user

Thank you Glenn for your attention and support...

really the US units, with inch and knots, it has always been a nightmare for me.

when make the first weewx installation, I change target_unit = US to target_unit = METRIC... and the problems started with incongruities, and I saw that this issue it could be a problem, always it has been.

well, now the units are US, but the reports has metrics units:

[[[Units]]]

[[[[Groups]]]]

group_altitude = meter

group_speed2 = km_per_hour2

group_pressure = mbar

group_rain = mm

group_rainrate = mm_per_hour

group_temperature = degree_C

group_degree_day = degree_C_day

group_speed = km_per_hour

but the max-min has US units:

[[MinMax]]

barometer = 26, 32.5, inHg

outTemp = -40, 120, degree_F

inTemp = 10, 120, degree_F

outHumidity = 0, 100

inHumidity = 0, 100

windSpeed = 0, 120, mile_per_hour

pressure = 24, 34.5, inHg

and config.json in US units too, because if I change to metrics, the data is displayed, incomprehensibly, in US units:

"columns" : {

"dateTime" : {"unit" : "s"},

"barometer" : {"unit" : "inHg"},

"inTemp" : {"unit" : "f"},

"outTemp" : {"unit" : "f"},

"inHumidity" : {"unit" : "perc"},

"outHumidity" : {"unit" : "perc"},

"windSpeed" : {"unit" : "mph"},

"windDir" : {"unit" : "deg"},

"rainRate" : {"unit" : "inHr"},

"dayRain" : {"unit" : "in"},

"dewpoint" : {"unit" : "f"},

"windchill" : {"unit" : "f"},

"heatindex" : {"unit" : "f"}

but js/Config.js are in metric units:

Config.fieldDefaults = {

'dateTime': new meso.FieldDef('dateTime', meso.Unit.ms, 0, meso.Agg.avg),

'outTemp': new meso.FieldDef('outTemp', mesowx.Unit.c, 1, meso.Agg.avg, 'Temperatura'),

'dewpoint': new meso.FieldDef('dewpoint', mesowx.Unit.c, 1, meso.Agg.avg, 'Rocio'),

'rain': new meso.FieldDef('rain', mesowx.Unit.mm, 2, meso.Agg.sum, 'Lluvia'),

'rainRate': new meso.FieldDef('rainRate', mesowx.Unit.mmHr, 2, meso.Agg.max, 'Tasa Lluvia'),

'dayRain': new meso.FieldDef('dayRain', mesowx.Unit.mm, 2, meso.Agg.max, 'Lluvia Hoy'),

'windSpeed': new meso.FieldDef('windSpeed', mesowx.Unit.kph, 1, meso.Agg.avg, 'Velocidad Viento'),

'windDir': new meso.FieldDef('windDir', mesowx.Unit.deg, 0, meso.Agg.avg, 'Direccion Viento'),

'windGust': new meso.FieldDef('windGust', mesowx.Unit.kph, 1, meso.Agg.avg, 'Rachas'),

'windGustDir': new meso.FieldDef('windGustDir', mesowx.Unit.deg, 0, meso.Agg.avg, 'Direccion Rachas'),

'outHumidity': new meso.FieldDef('outHumidity', mesowx.Unit.perc, 0, meso.Agg.avg, 'Humedad'),

'barometer': new meso.FieldDef('barometer', mesowx.Unit.mb, 1, meso.Agg.avg, 'Presion'),

'windchill': new meso.FieldDef('windchill', mesowx.Unit.c, 1, meso.Agg.avg, 'Viento Frio'),

'heatindex': new meso.FieldDef('heatindex', mesowx.Unit.c, 1, meso.Agg.avg, 'Indice Calor'),

'inTemp': new meso.FieldDef('inTemp', mesowx.Unit.c, 1, meso.Agg.avg, 'Temp Int'),

'inHumidity': new meso.FieldDef('inHumidity', mesowx.Unit.perc, 1, meso.Agg.avg, 'Hum Int')

};

... I think this is gibberish. :-(

Greetings.

Glenn McKechnie

unread,

May 23, 2016, 5:37:16 AM5/23/16

to weewx...@googlegroups.com

I believe the usUnits is all to do with compatibility with existing
software (Weather Display) and its choice of databases, and the
dominant weather station, at the time, Davis Instruments.
I'm reminded about the saying "Engineering is a compromise", the older
I get the more I understand its meaning.

Yes it can be confusing, for the most part it's invisible and can be
fixed in software although, as you note, it gets a bit weird at times.
I got caught the first time, thankfully I clued into it early enough
to not lose too much history in the database change.

That aside, the problem with your database is that 0 entry in the unit
field doesn't belong to any system that weewx recognises. That needs
to be fixed at your end, and whatever generated it in the first place
needs to be found and fixed for others. Luc will hopefully have some
insight into that.

Edu GG

unread,

May 23, 2016, 6:00:16 AM5/23/16

to weewx-user

I hate to lose data. in fact, what I want is to import all the data I have since 2010 from the database of weather-display software.

But if the solution is to change all units and create new databases, I find a way to import the data later.

However, as it is now running relatively well, and all I have is a duplicate error log every minute, I can leave it while I make a completely clean install another RaspberryPi-2 I have here in stand-by, and use metric units from the beginning, and also use only raw.py from the beginning.

Thank you very much Luc, Glenn and Andrew for your invaluable help.

Greetings,

Edu.

Luc Heijst

unread,

May 23, 2016, 7:46:47 AM5/23/16

to weewx-user

Edu,

Fact is: your weewx database is in US units

Fact is: you want use mesowx on the same machine as weewx.

You have in your situation still two choices.

a. use module raw.py to populate a raw database which units also have to be US (otherwise I foresee problems with reports and graphs when both databases don't have the same units).

b. use module sync.py to copy the loop and archive data over the network to another machine. I will explain later what that has to do with your situation.

On that 'other' machine you can have a choice to have both 'remote' archive and raw databases in METRIC units. mesowx file config.json is the place where to describe which units you want in the 'remote' archive and raw databases.

The conversion from the received data in US units to the METRIC values to be stored in the 'remote' databases will be done by mesowx.

The funny (and also difficult to understand) part is the fact that mesowx doesn't care about the name of the 'remote' machine. You can use any machine for that as long as it is reachable via the internet and you have enough rights to do what must be done.

Suppose you choose to use the same name and/or ip-address as that of the machine where weewx runs. It is allowed, right?

Then module sync.py and the modules on mesowx work together to copy all weewx archive and loop data over the net to ... the same machine (who cares?) and store this data after to be converted to metric in two mesowx databases.

No problem with that. The network traffic wil go to your router and back to your local machine (a very reliable network you have here!)

Now, there is ONE thing to take in mind: the weewx archive database (in US units) and the synchronised mesowx archive database (in your case in METRIC units) may not have the same name, right? Otherwise you are in BIG TROUBLE!

In the mesowx configuration file config.json you can give the mesowx archive database any name you like (as long as MySQL and mesowx agree with that name).

"dataSource" : {

"weewx_mysql" : { // the data source ID

"type" : "mysql",

"host" : "localhost",

"user" : "your MySQL user name for the mesowx database",

"password" : "your MySQL password for the mesowx database",

"database" : "fill in the name you like, as long as it is NOT THE SAME NAME as your weewx database. You could use here the name 'mesowx_weewx' for instance"

The mesowx table names can remain 'archive' and 'raw' because they belong to database 'mesowx_weewx'.

In short I will give the steps to be done:

1. Configure config.json for a database with name: mesowx_weewx

2. Configure the fields in both arhive and raw section in config.json in the metric units you want (for example you can use hPa but also mbar if you like that more)

3. Put in a section [RemoteSync] in weewx.conf all parameters needed for mesowx. In fact you have allready done that before; you can use the same settings as you used before.

4. Search for the line starting with 'restful_services = ' in weewx.conf.

5. Put at the end of that line: ', user.sync.SyncService'

6. Restart weewx.

7. Now pay attention and be very, very patient! If all goes well, mesowx will create a new database 'mesowx_weewx' on your local machine with the tables 'archive' and 'raw' (both in METRIC units).

Then mesowx will ask weewx what the time stamp is of its most recent archive record and then will catchup all 'missed' weewx archive data.

Because the mesowx_weewx archive table is empty right now, mesowx will request ALL weewx archive records it has. In your case probably over a million records! So this can take a few hours to complete! Mesowx reports a debug message per 1000 read records.

This process wil take several hours, may be more than a day! When for some reason the process is stopped before it was finished, it is not too much of a trouble. When you restart weewx again mesowx will continue with its catchup task and continues with the last saved mesowx_archive record.

8. When the catchup is finished mesowx will take care of its other two jobs: handling the archive and loop data what is send to it via the net, converting its data to metric units and store the data to the mesowx_weewx tables archive and raw tables respectively.

9. Each time weewx is restarted, mesowx will do a catchup again to make sure both archive databases remain fully synchronised. Typically none or only a few records need to be read in such cases.

Success!

Something else:

You mentioned the section [[MinMax]] in weewx.conf.

You can put your data in metric units as well when you put the proper metric unit names at the end of the line, like:

inTemp = -20, 50, degree_C

windSpeed = 0, 90, km_per_hour

Luc

Edu GG

unread,

May 23, 2016, 9:19:20 AM5/23/16

to weewx-user

hello Luc,

after modify the dataSource in config.json and the units in both section, back to the begining error:

May 23 15:06:45 raspberrypi weewx[8563]: sync archive: requesting latest dateTime from http://www.meteolebrija.es/web/meso/data.php

May 23 15:06:46 raspberrypi weewx[8563]: sync archive: http request failed (500 Internal Server Error):

May 23 15:06:46 raspberrypi weewx[8563]: sync: retrying again in 0 seconds

May 23 15:06:46 raspberrypi weewx[8563]: sync archive: http request failed (500 Internal Server Error):

May 23 15:06:46 raspberrypi weewx[8563]: sync: retrying again in 0 seconds

May 23 15:06:46 raspberrypi weewx[8563]: sync archive: http request failed (500 Internal Server Error):

May 23 15:06:46 raspberrypi weewx[8563]: ws23xx: close LinuxSerialPort

May 23 15:06:46 raspberrypi weewx[8563]: engine: Caught unrecoverable exception in engine:

May 23 15:06:46 raspberrypi weewx[8563]: **** sync archive: Failed to invoke http://www.meteolebrija.es/web/meso/data.php after 3 tries

May 23 15:06:46 raspberrypi weewx[8563]: **** Traceback (most recent call last):

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/weewx/engine.py", line 853, in main

May 23 15:06:46 raspberrypi weewx[8563]: **** engine = EngineClass(config_dict)

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May 23 15:06:46 raspberrypi weewx[8563]: **** self.loadServices(config_dict)

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/weewx/engine.py", line 136, in loadServices

May 23 15:06:46 raspberrypi weewx[8563]: **** self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/user/sync.py", line 108, in __init__

May 23 15:06:46 raspberrypi weewx[8563]: **** self.back_fill()

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/user/sync.py", line 176, in back_fill

May 23 15:06:46 raspberrypi weewx[8563]: **** last_datetime_synced = self.fetch_latest_remote_datetime()

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/user/sync.py", line 228, in fetch_latest_remote_datetime

May 23 15:06:46 raspberrypi weewx[8563]: **** http_response = self.make_http_request(self.latest_url, postdata)

May 23 15:06:46 raspberrypi weewx[8563]: **** File "/usr/share/weewx/user/sync.py", line 281, in make_http_request

May 23 15:06:46 raspberrypi weewx[8563]: **** raise SyncError, "sync archive: Failed to invoke %s after %d tries" % (url, self.http_max_tries)

May 23 15:06:46 raspberrypi weewx[8563]: **** SyncError: sync archive: Failed to invoke http://www.meteolebrija.es/web/meso/data.php after 3 tries

May 23 15:06:46 raspberrypi weewx[8563]: **** Exiting.

returning to initial dataSource but with US units in both section, the data con web dissapears, it shows nothing (screenshot attached).

Greetings.

Screenshot_2016-05-23-15-14-41.png

Andrew Milner

unread,

May 23, 2016, 9:44:57 AM5/23/16

to weewx-user

Which of Luc's suggestions are you trying to do - option a or option b?

Luc Heijst

unread,

May 23, 2016, 9:47:07 AM5/23/16

to weewx-user

Edu,

Don't give up to quick!

Maybe your RPI can't translate 'www.meteolebrija.es' to a proper IP address. Just fill in your ip address instead of http://www.meteolebrija.es/web/meso/data.php. In your case that will be http://212.104.164.214/web/meso/data.php I think.

Also have a look (again) at your Apache error log; you might have a parse error or other error again.

Luc

Edu GG

unread,

May 23, 2016, 10:04:38 AM5/23/16

to weewx...@googlegroups.com

Andrew, option b.

Luc, I tried public IP, private IP, localhost... and the error return.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/Z4sW4oDDboo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Luc Heijst

unread,

May 23, 2016, 10:11:35 AM5/23/16

to weewx-user

Sorry Edu,

I give up! I have tried several times to explain it as clearly as possible. Can't do it any better than I did so far.

My advice is: look for a local 'computer-geek' buddy and read with him/her all the mails we sent to you and probably you two together will sort this out.

Succes!

Luc

Andrew Milner

unread,

May 23, 2016, 10:12:37 AM5/23/16

to weewx-user

A silly question perhaps - but does the page actually exist, with the correct permissions etc???

Luc Heijst

unread,

May 23, 2016, 10:19:33 AM5/23/16

to weewx-user

Andrew,

We have been there numerous messages ago! Then there appeared to be a syntax error in config.json. We are just running in circles!

Luc

Andrew Milner

unread,

May 23, 2016, 10:32:54 AM5/23/16

to weewx-user

Luc - the simplest answer is to get raw working as it shou]d. The raw table that exists now can safely be deleted (it is only transient anyway as it is continually being purged) - and then we could start to see why raw does not work in Edu's situation.

I think that messing with sync on the same machine (whether or not it should theoretically work) is probably not the way to go and is effectively adding another layer of complexity as he does not really need/want a synced copy of the archive database as far as \I can tell, and he only went using sync because raw did not appear to work as he wanted. There is absolutely no reason why we should not be able to resolve raw and meso's display of raw data, and I am sure that is what we should be trying to persuade Edu to do!! My meso uses raw for up to 7 days loop data display and weewx archive for older data.

Andrew

Andrew Milner

unread,

May 23, 2016, 10:34:09 AM5/23/16

to weewx-user

Maybe it would help if Edu posted the actual files he is using - would end the guesswork!!

Edu GG

unread,

May 23, 2016, 12:00:23 PM5/23/16

to weewx...@googlegroups.com

well,

thank you very much to all, this is the first...

I'm trying again since last Luc's recommendation.

1.- Overwritted the original sync_lh4.py (renamed sync.py for use) by raw_0.4.1-lh.py (renamed sync.py for use) - don't understand until now.

sudo cp raw_0.4.1-lh.py /usr/share/weewx/user/sync.py

2.- weewx.conf modified for metric units (file attached)

3.- js/Config.js in metric units too (file attached)

4.- config.json dataSource modified like Luc recommend (file attached)

and log returns:

May 23 17:32:13 raspberrypi weewx[10430]: engine: Caught unrecoverable exception in engine:

May 23 17:32:13 raspberrypi weewx[10430]: **** Module 'user.sync' has no attribute 'SyncService' when searching for 'user.sync.SyncService'

May 23 17:32:13 raspberrypi weewx[10430]: **** Traceback (most recent call last):

May 23 17:32:13 raspberrypi weewx[10430]: **** File "/usr/share/weewx/weewx/engine.py", line 853, in main

May 23 17:32:13 raspberrypi weewx[10430]: **** engine = EngineClass(config_dict)

May 23 17:32:13 raspberrypi weewx[10430]: **** File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May 23 17:32:13 raspberrypi weewx[10430]: **** self.loadServices(config_dict)

May 23 17:32:13 raspberrypi weewx[10430]: **** File "/usr/share/weewx/weewx/engine.py", line 136, in loadServices

May 23 17:32:13 raspberrypi weewx[10430]: **** self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

May 23 17:32:13 raspberrypi weewx[10430]: **** File "/usr/share/weewx/weeutil/weeutil.py", line 1023, in _get_object

May 23 17:32:13 raspberrypi weewx[10430]: **** raise AttributeError("Module '%s' has no attribute '%s' when searching for '%s'" % (mod.__name__, part, module_class))

May 23 17:32:13 raspberrypi weewx[10430]: **** AttributeError: Module 'user.sync' has no attribute 'SyncService' when searching for 'user.sync.SyncService'

May 23 17:32:13 raspberrypi weewx[10430]: **** Exiting.

if I use sync_lh4.py like sync.py instead raw_0.4.1-lh.py like sync.py, returns the begining error:

May 23 17:54:50 raspberrypi weewx[10594]: engine: Caught unrecoverable exception in engine:

May 23 17:54:50 raspberrypi weewx[10594]: **** sync archive: Failed to invoke http://www.meteolebrija.es/web/meso/data.php after 3 tries

May 23 17:54:50 raspberrypi weewx[10594]: **** Traceback (most recent call last):

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/weewx/engine.py", line 853, in main

May 23 17:54:50 raspberrypi weewx[10594]: **** engine = EngineClass(config_dict)

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May 23 17:54:50 raspberrypi weewx[10594]: **** self.loadServices(config_dict)

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/weewx/engine.py", line 136, in loadServices

May 23 17:54:50 raspberrypi weewx[10594]: **** self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/user/sync.py", line 108, in __init__

May 23 17:54:50 raspberrypi weewx[10594]: **** self.back_fill()

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/user/sync.py", line 176, in back_fill

May 23 17:54:50 raspberrypi weewx[10594]: **** last_datetime_synced = self.fetch_latest_remote_datetime()

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/user/sync.py", line 228, in fetch_latest_remote_datetime

May 23 17:54:50 raspberrypi weewx[10594]: **** http_response = self.make_http_request(self.latest_url, postdata)

May 23 17:54:50 raspberrypi weewx[10594]: **** File "/usr/share/weewx/user/sync.py", line 281, in make_http_request

May 23 17:54:50 raspberrypi weewx[10594]: **** raise SyncError, "sync archive: Failed to invoke %s after %d tries" % (url, self.http_max_tries)

May 23 17:54:50 raspberrypi weewx[10594]: **** SyncError: sync archive: Failed to invoke http://www.meteolebrija.es/web/meso/data.php after 3 tries

May 23 17:54:50 raspberrypi weewx[10594]: **** Exiting.

If I continue make changes I'm sure that I can´t back to the version that was running ok (with the duplicate error) until now.

I try to restore the version that running ok (with the duplicate error), and leave it like this until install clean installation in new raspi.

Thanks a lot.

weewx.conf.txt

Config.js.txt

config.json.txt

Edu GG

unread,

May 23, 2016, 12:15:31 PM5/23/16

to weewx...@googlegroups.com

well... wait a moment.

after create the db mesowx_weewx, I thought that sync create automagically... and re-overwrite the sync.py by sync_lh4.py, the log returns:

May 23 18:10:40 raspberrypi weewx[10829]: engine: Loading service user.sync.SyncService

May 23 18:10:41 raspberrypi weewx[10829]: sync archive: requesting latest dateTime from http://www.meteolebrija.es/web/meso/data.php

May 23 18:10:42 raspberrypi weewx[10829]: sync archive: 7929 records to sync since last synced record with dateTime: ******* N/A ******* ( N/A )

May 23 18:10:42 raspberrypi weewx[10829]: sync archive: back_filling 7929 records

May 23 18:10:48 raspberrypi weewx[10829]: sync archive: back_filled 200 records; timestamp last record: 2016-05-17 21:02:00 CEST (1463511720)

May 23 18:10:51 raspberrypi weewx[10829]: sync archive: back_filled 400 records; timestamp last record: 2016-05-18 00:27:00 CEST (1463524020)

May 23 18:10:53 raspberrypi weewx[10829]: sync archive: back_filled 600 records; timestamp last record: 2016-05-18 03:49:00 CEST (1463536140)

May 23 18:10:55 raspberrypi weewx[10829]: sync archive: back_filled 800 records; timestamp last record: 2016-05-18 07:09:00 CEST (1463548140)

May 23 18:10:57 raspberrypi weewx[10829]: sync archive: back_filled 1000 records; timestamp last record: 2016-05-18 10:29:00 CEST (1463560140)

........

May 23 18:12:15 raspberrypi weewx[10829]: sync archive: back_filled 7800 records; timestamp last record: 2016-05-23 15:10:00 CEST (1464009000)

May 23 18:12:17 raspberrypi weewx[10829]: sync archive: back_filled 7929 records; timestamp last record: 2016-05-23 17:21:00 CEST (1464016860)

May 23 18:12:18 raspberrypi weewx[10829]: sync archive: done back_filling 7929 records

May 23 18:12:18 raspberrypi weewx[10829]: sync archive: will sync archive records

May 23 18:12:18 raspberrypi weewx[10829]: sync archive: waiting for new records

May 23 18:12:18 raspberrypi weewx[10829]: sync raw: will sync raw records

May 23 18:12:18 raspberrypi weewx[10829]: engine: Finished loading service user.sync.SyncService

May 23 18:12:18 raspberrypi weewx[10829]: engine: Loading service weewx.engine.StdPrint

May 23 18:12:18 raspberrypi weewx[10829]: engine: Finished loading service weewx.engine.StdPrint

May 23 18:12:18 raspberrypi weewx[10829]: sync raw: waiting for new records

May 23 18:12:18 raspberrypi weewx[10829]: engine: Loading service weewx.engine.StdReport

May 23 18:12:18 raspberrypi weewx[10829]: engine: Finished loading service weewx.engine.StdReport

May 23 18:12:18 raspberrypi weewx[10829]: engine: Starting up weewx version 3.5.0

May 23 18:12:18 raspberrypi weewx[10829]: engine: Station does not support reading the time

May 23 18:12:18 raspberrypi weewx[10829]: ws23xx: gen_records: since_ts=1464016860 count=0 clock=True

May 23 18:12:18 raspberrypi weewx[10829]: ws23xx: using computer clock with latest_ts of 2016-05-23 18:11:18 CEST (1464019878)

May 23 18:12:18 raspberrypi weewx[10829]: ws23xx: count is 51 to satisfy timestamp of 2016-05-23 17:21:00 CEST (1464016860)

May 23 18:12:18 raspberrypi weewx[10829]: ws23xx: downloading 51 records from station

May 23 18:12:38 raspberrypi weewx[10829]: engine: Main loop exiting. Shutting engine down.

May 23 18:12:38 raspberrypi weewx[10829]: sync archive: exit event signaled, exiting queue loop

May 23 18:12:38 raspberrypi weewx[10829]: sync: thread shutting down

May 23 18:12:38 raspberrypi weewx[10829]: sync raw: exit event signaled, exiting queue loop

May 23 18:12:38 raspberrypi weewx[10829]: sync: thread shutting down

May 23 18:12:38 raspberrypi weewx[10829]: sync: Shut down syncing thread: RawSyncThread

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down WeatherCloud thread.

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down AWEKAS thread.

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down CWOP thread.

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down PWSWeather thread.

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down Wunderground-RF thread.

May 23 18:12:38 raspberrypi weewx[10829]: restx: Shut down StationRegistry thread.

May 23 18:12:38 raspberrypi weewx[10829]: ws23xx: close LinuxSerialPort

May 23 18:12:38 raspberrypi weewx[10829]: engine: Caught unrecoverable exception in engine:

May 23 18:12:38 raspberrypi weewx[10829]: **** Unit system of incoming record (0x10) differs from 'archive' table in 'weewx' database (0x01)

May 23 18:12:38 raspberrypi weewx[10829]: **** Traceback (most recent call last):

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 859, in main

May 23 18:12:38 raspberrypi weewx[10829]: **** engine.run()

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 154, in run

May 23 18:12:38 raspberrypi weewx[10829]: **** self.dispatchEvent(weewx.Event(weewx.STARTUP))

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 218, in dispatchEvent

May 23 18:12:38 raspberrypi weewx[10829]: **** callback(event)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 519, in startup

May 23 18:12:38 raspberrypi weewx[10829]: **** self._catchup(self.engine.console.genStartupRecords)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 624, in _catchup

May 23 18:12:38 raspberrypi weewx[10829]: **** origin='hardware'))

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 218, in dispatchEvent

May 23 18:12:38 raspberrypi weewx[10829]: **** callback(event)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/engine.py", line 595, in new_archive_record

May 23 18:12:38 raspberrypi weewx[10829]: **** dbmanager.addRecord(event.record)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/manager.py", line 235, in addRecord

May 23 18:12:38 raspberrypi weewx[10829]: **** self._addSingleRecord(record, cursor, log_level)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/manager.py", line 1147, in _addSingleRecord

May 23 18:12:38 raspberrypi weewx[10829]: **** super(DaySummaryManager, self)._addSingleRecord(record, cursor, log_level=log_level)

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/manager.py", line 258, in _addSingleRecord

May 23 18:12:38 raspberrypi weewx[10829]: **** self._check_unit_system(record['usUnits'])

May 23 18:12:38 raspberrypi weewx[10829]: **** File "/usr/share/weewx/weewx/manager.py", line 640, in _check_unit_system

May 23 18:12:38 raspberrypi weewx[10829]: **** self.std_unit_system))

May 23 18:12:38 raspberrypi weewx[10829]: **** UnitError: Unit system of incoming record (0x10) differs from 'archive' table in 'weewx' database (0x01)

May 23 18:12:38 raspberrypi weewx[10829]: **** Exiting.

We will lose sanity with this. hahahaha

Greetings.

Edu GG

unread,

May 23, 2016, 12:24:53 PM5/23/16

to weewx...@googlegroups.com

after make a new weewx_metric db and restart sync is updating this new database but I see nothing in log about mesowx_weewx database...

Like Luc's prediction, this task take some hours... but for now, the data not shown in www.meteolebrija.es/web ...

I'll wait, I think.

Edu GG

unread,

May 23, 2016, 12:37:51 PM5/23/16

to weewx...@googlegroups.com

the data already is shown....

there is an error in config.js pressure in "mbar" not fuction... must be "mb".

well, now the data on screen in metric, and database in metric, and "duplicate error" has gone...

but in mesowx_weewx database only see 'archive' table...

and other issue, now, the old data appears in US units but with metrics simbols... this is hilarious, hehehe.

I think I should delete such data, or to treat them and convert them to metric units with hard work.

Thank you friends.

Greetings.

Edu.

Andrew Milner

unread,

May 23, 2016, 1:06:12 PM5/23/16

to weewx-user

I give up ...... raw.lh... should become raw.py not sync.py .... and sync.lh..... becomes sync.py

why on earth have you renamed raw.lh..... to sync.....- you seem determined to do your own thing whatever anyone says!!!!

I give up

Edu GG

unread,

May 23, 2016, 1:28:40 PM5/23/16

to weewx-user

sorry Andrew, but some posts above, Glenn says something that confuse me:

Glenn wrote:

"The only thing Edu needs to do is switch out weewx/user/sync.py and
replace it with weewx/user/raw.py..."

and after more than 15-20 config settings I did not quite understand what he meant.

Now is running, without logerrors and synchronizing databases. And all is in metric units. Now I must change the old data recovered by sync that is in US units.

Your help was essential for me to understand better how it works weewx and mesowx. Thank you so much everyone.

Count me for what you need.

Greetings.

Edu.

Edu GG

unread,

May 23, 2016, 2:17:30 PM5/23/16

to weewx...@googlegroups.com

well, some UPDATEs mysql sentences and the old data is in metric, I have not lost any data.

Regards.

Edu.

--

Edu GG

unread,

May 24, 2016, 9:41:28 AM5/24/16

to weewx-user

sorry my friends, but what could have happened to yesterday?, no wind gusts are painted.

since the databases source was changed, the windgust not recorded, and evidently, WU and others not uploaded...

now only record NULL in windGust and windGustDir fields in weewx db (weewx_metric for me).

Thank you very much, and please, sorry for the inconvenience.

Greetings,

Edu.

Andrew Milner

unread,

May 24, 2016, 9:52:52 AM5/24/16

to weewx...@googlegroups.com

Are you saying that new records in the standard weewx archive database no longer contain values for gust and gust dir - or that you have lost values that you used to have?

Are these hardware or software generated in your system? Do wind and windDir have values - or are you talking about the results of your sql importing of old data perhaps? - otherwise I was not aware that you had changed the source of the data for the weewx archive and stats tables in the weewx database!!

Forget meso for now - if you have a problem with the basic weewx then that should be sorted before you then then try and add the mesowx extension - which is not a part of standard weewx.

Without logs or snippets from your database tables it is sometimes very hard to actually understand exactly what your problem is!!

--