Debian installation fail with Seagate Dockstare Debian Lenny (SQL pb ?)

[Philippe Amram]

Hi,
I'm from Paris,
 just started installing Weewx on my Seagate Dockstare (Debian Lenny)
I have a Vantage Pro 2 with IP datalogger.

I have made all upgrades described in http://www.weewx.com/docs/sheeva.htm,
then installed weewx as described in
http://www.weewx.com/docs/debian.htm.

When I tail the syslog file, I got a tone of errors messages like this :

Mar 24 08:16:24 debian python: wxengine: Using configuration file /home/weewx/weewx.conf.
Mar 24 08:16:24 debian weewx[23922]: wxengine: Starting up weewx version 1.13.0.
Mar 24 08:16:34 debian weewx[23922]:     ****  Traceback (most recent call last):
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 645, in main
Mar 24 08:16:34 debian weewx[23922]:     ****      engine = EngineClass(options, args)
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 85, in __init__
Mar 24 08:16:34 debian weewx[23922]:     ****      self.setupServices(config_dict)
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 128, in setupServices
Mar 24 08:16:34 debian weewx[23922]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc, self, config_dict))
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weeutil/weeutil.py", line 656, in _get_object
Mar 24 08:16:34 debian weewx[23922]:     ****      obj = mod(*args, **kwargs)
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 369, in __init__
Mar 24 08:16:34 debian weewx[23922]:     ****      self.setupArchiveDatabase(config_dict)
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 408, in setupArchiveDatabase
Mar 24 08:16:34 debian weewx[23922]:     ****      weewx.archive.config(archiveFilename)
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weewx/archive.py", line 476, in config
Mar 24 08:16:34 debian weewx[23922]:     ****      if weeutil.dbutil.schema(archiveFilename):
Mar 24 08:16:34 debian weewx[23922]:     ****    File "/home/weewx/bin/weeutil/dbutil.py", line 32, in schema
Mar 24 08:16:34 debian weewx[23922]:     ****      with sqlite3.connect(database) as _connection:
Mar 24 08:16:34 debian weewx[23922]:     ****  AttributeError: 'sqlite3.Connection' object has no attribute '__exit__'

This is quite disturbing.
Would you have any suggestion to guide me and help getting weewx up and running ?
Thanks so much !
Cheers.

Philippe

[Thomas Keffer]

Hello, Philippe and welcome to weewx!

It seems that the version of pysqlite that came with your Dockstar is an old one and does not support the Python "with" statement, introduced with Python 2.5.

The code used to have a workaround to deal with this problem, but other users talked me out of including it (it had unfortunate side effects) in version 1.12.4.

Try upgrading to a more recent version of pysqlite:

  sudo apt-get install python-pip

  sudo pip install pysqlite

Then running the older weewx v1.12.3.

That should get you up and running.

In the meantime, I have an idea for a better workaround. I should get it out in a day or two. 

-tk

Philippe

--
You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/TAi721Rg5PAJ.
To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

--
Thomas Keffer

* kef...@threefools.org
( +1 541-386-8891 (h)
( +1 541-490-9507 (c)

Skype: tkeffer

[Thomas Keffer]

OK, I've put up a beta version of v1.13.1 that has a workaround.

First install a more recent version of pysqlite:

  sudo apt-get install python-pip

  sudo pip install pysqlite

Then install the beta:

http://sourceforge.net/projects/weewx/files/betas/weewx-1.13.1b1.tar.gz/download

What v1.13.1 does is check the version of sqlite3 that comes with Python to see if it supports the "with" statement. If it does, it uses that version. If it does not, then it falls back to a user-supplied version. For the Python enthusiasts, here is what the test looks like:

import sqlite3

if not hasattr(sqlite3.Connection, "__exit__"):

    del sqlite3

    from pysqlite2 import dbapi2 as sqlite3

With your Debian Lenny system it will fail the first test, but if you install the more recent version of pysqlite, it will succeed with the second.

Let me know how it works!

-tk

[Philippe Amram]

Thanks Tom,

hereafter, the result of my work :

apt-get install python-pip : OK (I do not use sudo because I'm root and also, my plug does not have it)

pip install pysqlite : I get : "-bash: pip: command not found"

still some work to do ...

cheers
Philippe

[Thomas Keffer]

Hello, Philippe

From your email, it is not clear what went wrong. Were you unable to install pip? This is a tool that makes it easier to install python tools, such as pysqlite, but it is not essential.

The goal here is to get a newer version of pysqlite. Any method that works is OK. 

In addition to the method you tried, here are some others:

1. Try simply upgrading:

  sudo apt-get install python-pysqlite2

2. If that doesn't work, you can try downloading pysqlite from their website and following the build instructions included (see file install-source.txt). There are dependencies that must be satisfied first. Specifically, you must have the gcc compiler and the sqlite development package:

  sudo apt-get install python-dev

  sudo apt-get install libsqlite3-dev

3. Rather than do #2, it might still be easier to install pip, then use it to install pysqlite. See instructions here on how to install pip. Once installed, installing pysqlite is easy:

  sudo pip install pysqlite

Let me know how this goes!!

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/-j4SiL-5miMJ.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe Amram]

Thanks Tom for your efforts to help me,

"1. Try simply upgrading:

  "sudo apt-get install python-pysqlite2"
My plug says that I already have the latest available version.

"2. If that doesn't work, you can try downloading pysqlite from their website and following the build instructions included (see file install-source.txt). There are dependencies that must be satisfied first. Specifically, you must have the gcc compiler and the sqlite development package:

  sudo apt-get install python-dev" :
My plug says that I already have the latest available version

  "sudo apt-get install libsqlite3-dev" :

My plug says that I already have the latest available version

"3. Rather than do #2, it might still be easier to install pip, then use it to install pysqlite. See instructions here on how to install pip. Once installed, installing pysqlite is easy:" :

some steps before succed:

Install curl : done, (http://fredmac.blogspot.fr/2011/01/installer-curl-dans-debian-lenny.html)
then install pip : done (http://www.pip-installer.org/en/latest/installing.html)
then "pip install pysqlite"

Then (my_new_env)debian:/# /etc/init.d/weewx start
Then (my_new_env)debian:/# tail -f /var/log/syslog
 ---> answer of plug : "tail: Can't open`/var/log/syslog' reading mode: No file or directory
tail: no remaining file"

I'm getting lost.

Looking forward...
cheers
Philippe

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Philippe Amram]

Quite late here..
after a reboot and a (re)start of weewx + tail ,

I get the following message from the tail :

Mar 25 00:06:37 debian python: wxengine: Using configuration file /home/weewx/weewx.conf.
Mar 25 00:06:37 debian weewx[2093]: wxengine: Starting up weewx version 1.13.0.
Mar 25 00:06:40 debian weewx[2093]: VantagePro: successfully woke up console
Mar 25 00:06:45 debian weewx[2093]: wxengine: Loading service weewx.wxengine.StdCalibrate
Mar 25 00:06:45 debian weewx[2093]: wxengine: Finished loading service weewx.wxengine.StdCalibrate
Mar 25 00:06:45 debian weewx[2093]: wxengine: Loading service weewx.wxengine.StdQC
Mar 25 00:06:45 debian weewx[2093]: wxengine: Finished loading service weewx.wxengine.StdQC
Mar 25 00:06:45 debian weewx[2093]: wxengine: Loading service weewx.wxengine.StdArchive
Mar 25 00:06:45 debian weewx[2093]: weeutil: Not able to instantiate module 'weewx.wxengine.StdArchive'
Mar 25 00:06:47 debian weewx[2093]: VantagePro: successfully woke up console
Mar 25 00:06:47 debian weewx[2093]: wxengine: Caught unrecoverable exception in wxengine:
Mar 25 00:06:47 debian weewx[2093]:     ****  'sqlite3.Connection' object has no attribute '__exit__'
Mar 25 00:06:47 debian weewx[2093]:     ****  Traceback (most recent call last):
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 645, in main
Mar 25 00:06:47 debian weewx[2093]:     ****      engine = EngineClass(options, args)
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 85, in __init__
Mar 25 00:06:47 debian weewx[2093]:     ****      self.setupServices(config_dict)
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 128, in setupServices
Mar 25 00:06:47 debian weewx[2093]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc, self, config_dict))
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weeutil/weeutil.py", line 656, in _get_object
Mar 25 00:06:47 debian weewx[2093]:     ****      obj = mod(*args, **kwargs)
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 369, in __init__
Mar 25 00:06:47 debian weewx[2093]:     ****      self.setupArchiveDatabase(config_dict)
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 408, in setupArchiveDatabase
Mar 25 00:06:47 debian weewx[2093]:     ****      weewx.archive.config(archiveFilename)
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weewx/archive.py", line 478, in config
Mar 25 00:06:47 debian weewx[2093]:     ****      if weeutil.dbutil.schema(archiveFilename):
Mar 25 00:06:47 debian weewx[2093]:     ****    File "/home/weewx/bin/weeutil/dbutil.py", line 34, in schema
Mar 25 00:06:47 debian weewx[2093]:     ****      with sqlite3.connect(database) as _connection:
Mar 25 00:06:47 debian weewx[2093]:     ****  AttributeError: 'sqlite3.Connection' object has no attribute '__exit__'
Mar 25 00:06:47 debian weewx[2093]:     ****  Exiting.

still some work to do...
Cheers

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

You need to use the beta version of weewx that I posted earlier.

I know this is a little confusing. There are two version of pysqlite. The version that came with your plug, and the one that you just installed using pip. Weewx v1.13.0, the one you are running below, always uses the former. The beta that I posted tries the former. If that fails, then it uses the latter.

You can find it at: http://sourceforge.net/projects/weewx/files/betas/weewx-1.13.1b1.tar.gz/download

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/DZ00Xb7j1IQJ.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe Amram]

Thank you so much Tom,

it works !!!!!
... and I have new tricks...
Every 10 to 20 reading archives in the console (IP), I get the following message :
"Unable to pass CRC16 check while getting data"
then it goes back to the " ****  Waiting 60 seconds then retrying..."

and again...
Shoud I tune some timeouts ?
or reduce the electrical noise somewhere ?
or ?

Cheers Tom
Philippe

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

Try setting debug=1 in the configuration file (weewx.conf), running it until you get one of these errors, then send me the relevant section of the system log.

You are using the IP datalogger, correct?

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/WBoE_xAisWgJ.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Thomas Keffer]

Oh, one other thing: be sure to restart weewx after setting debug=1.

 -tk

[Philippe Amram]

Hi Tom again,

I have reloaded the conf then started again.
This is what the tail of syslog file gives :

wxengine: Using configuration file /home/weewx/weewx.conf.

Mar 25 15:09:51 debian weewx[2021]: wxengine: Starting up weewx version 1.13.1b1.
Mar 25 15:09:51 debian weewx[2595]: VantagePro: successfully woke up console
Mar 25 15:09:52 debian weewx[2595]: VantagePro: Unable to wake up console
Mar 25 15:09:52 debian weewx[2595]: wxengine: Caught WeeWxIOError: Socket read error
Mar 25 15:09:52 debian weewx[2595]:     ****  Waiting 60 seconds then retrying...
Mar 25 15:09:53 debian weewx[2021]: VantagePro: successfully woke up console
Mar 25 15:09:53 debian python: wxengine: Using configuration file /home/weewx/weewx.conf.
Mar 25 15:09:53 debian weewx[2783]: wxengine: Starting up weewx version 1.13.1b1.
Mar 25 15:09:56 debian weewx[2783]: VantagePro: successfully woke up console
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdCalibrate
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdCalibrate
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdQC
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdQC
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdArchive
Mar 25 15:10:01 debian weewx[2783]: stats: Backfilling stats database.
Mar 25 15:10:01 debian weewx[2783]: stats: stats database up to date.
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdArchive
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdTimeSynch
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdTimeSynch
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdPrint
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdPrint
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdRESTful
Mar 25 15:10:01 debian weewx[2783]: weeutil: Not able to instantiate module 'weewx.restful.Ambient'
Mar 25 15:10:01 debian weewx[2783]: wxengine: Data will not be posted to Wunderground
Mar 25 15:10:01 debian weewx[2783]: weeutil: Not able to instantiate module 'weewx.restful.Ambient'
Mar 25 15:10:01 debian weewx[2783]: wxengine: Data will not be posted to PWSweather
Mar 25 15:10:01 debian weewx[2783]: weeutil: Not able to instantiate module 'weewx.restful.CWOP'
Mar 25 15:10:01 debian weewx[2783]: wxengine: Data will not be posted to CWOP
Mar 25 15:10:01 debian weewx[2783]: wxengine: No RESTful upload sites. Thread not started.
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdRESTful
Mar 25 15:10:01 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdReportService
Mar 25 15:10:01 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdReportService
Mar 25 15:10:01 debian weewx[2783]: VantagePro: Getting archive packets since 2012-03-25 15:05:00 CEST (1332680700)
Mar 25 15:10:02 debian weewx[2021]: VantagePro: Max retries exceeded while getting EEPROM data at address 29
Mar 25 15:10:02 debian weewx[2021]: weeutil: Not able to instantiate module 'weewx.VantagePro.VantagePro'
Mar 25 15:10:02 debian weewx[2021]: wxengine: Unable to open WX station hardware: While getting EEPROM data value at address 29
Mar 25 15:10:02 debian weewx[2021]: wxengine: Caught WeeWxIOError: While getting EEPROM data value at address 29
Mar 25 15:10:02 debian weewx[2021]:     ****  Waiting 60 seconds then retrying...
Mar 25 15:10:04 debian weewx[2783]: VantagePro: Unable to wake up console
Mar 25 15:10:04 debian weewx[2783]: VantagePro: Unable to wake up console
Mar 25 15:10:04 debian weewx[2783]: wxengine: Caught WeeWxIOError: Unable to wake up VantagePro console
Mar 25 15:10:04 debian weewx[2783]:     ****  Waiting 60 seconds then retrying...
^C
debian:~# vi /home/weewx/weewx.conf
debian:~# tail -f /var/log/syslog
Mar 25 15:11:11 debian weewx[2783]: wxengine: Data will not be posted to CWOP
Mar 25 15:11:11 debian weewx[2783]: wxengine: No RESTful upload sites. Thread not started.
Mar 25 15:11:11 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdRESTful
Mar 25 15:11:11 debian weewx[2783]: wxengine: Loading service weewx.wxengine.StdReportService
Mar 25 15:11:11 debian weewx[2783]: wxengine: Finished loading service weewx.wxengine.StdReportService
Mar 25 15:11:11 debian weewx[2783]: VantagePro: Getting archive packets since 2012-03-25 15:05:00 CEST (1332680700)
Mar 25 15:11:14 debian weewx[2783]: VantagePro: Unable to wake up console
Mar 25 15:11:14 debian weewx[2783]: VantagePro: Unable to wake up console
Mar 25 15:11:14 debian weewx[2783]: wxengine: Caught WeeWxIOError: Unable to wake up VantagePro console
Mar 25 15:11:14 debian weewx[2783]:     ****  Waiting 60 seconds then retrying...

(etc...)

PS : I have tested the "regular" version an a desktop PC running Ubuntu. It works more or less (communications errors with the console, but data arrive soon or later).

Cheers

Philippe
Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

I believe you have two daemon instances of weewx running at the same time. At least, there are two different process IDs listed in the system log (2021 and 2783).

Stop all of the daemons (using "kill -9" if you have to) and run weewx from the command line. It will be much clearer what is happening.

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/qDrC0GTS370J.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe Amram]

Tom,

you were right !
I had some rermaining old processes from my old wview sw. (I have tested it for a while and decided a few days aggo to move to weewx !)

Killed all of them,
Weewx and my plug have then started running together smoothly.
It's such a great pleasure Tom, many many thanks for your help.

I'm very proud to switch off now my old pc (the one I have been keeping for 15 years... running Weatherlink since 2007).
Now, I have to fine tune my templates and import my data from eatherlink.

I will keep you posted.
Enjoy your Sunday

cheers from the sunny paris.

Philippe :-)

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

You're welcome!

If you use weewx to post to a website, be sure to share it with us.

Cheers from not-so-sunny Oregon.

-tk

[Jason Rennie]

On Sat, Mar 24, 2012 at 12:11 PM, Thomas Keffer <tke...@gmail.com> wrote:

What v1.13.1 does is check the version of sqlite3 that comes with Python to see if it supports the "with" statement. If it does, it uses that version. If it does not, then it falls back to a user-supplied version. For the Python enthusiasts, here is what the test looks like:

import sqlite3

if not hasattr(sqlite3.Connection, "__exit__"):

    del sqlite3

    from pysqlite2 import dbapi2 as sqlite3

Tom, great idea!

Jason

[Thomas Keffer]

By the way, Philippe, I found a small bug in that version I sent you. The plot colors will not be correct. 

It is a beta, after all!

-tk

On Sun, Mar 25, 2012 at 7:58 AM, Philippe Amram <ph.a...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/AmUd1s1BT08J.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe AMRAM]

Thanks Tom for the advice,
Anyway, that is not a very important bug .

I keep you posted.

cheers.


Philippe AMRAM

[Philippe Amram]

Hi Tom again,

weewx is running on my dockstar with  Debian
and I'm very proud of it !
I have noticed that the communication with the console (IP datalogger) breaks quite often.
That leads to a very slow upload process :

"Mar 27 22:41:20 debian weewx[6607]: VantagePro: successfully woke up console
Mar 27 22:41:40 debian weewx[6607]: VantagePro: Retrieving 131 page(s); starting index= 0
Mar 27 22:41:50 debian weewx[6607]: Archive: added archive record 2012-03-25 16:30:00 CEST (1332685800)
Mar 27 22:41:53 debian weewx[6607]: Archive: added archive record 2012-03-25 16:35:00 CEST (1332686100)
Mar 27 22:41:56 debian weewx[6607]: Archive: added archive record 2012-03-25 16:40:00 CEST (1332686400)
Mar 27 22:41:58 debian weewx[6607]: Archive: added archive record 2012-03-25 16:45:00 CEST (1332686700)
Mar 27 22:42:01 debian weewx[6607]: Archive: added archive record 2012-03-25 16:50:00 CEST (1332687000)
Mar 27 22:44:03 debian weewx[6607]: VantagePro: Unable to pass CRC16 check while getting data
Mar 27 22:44:23 debian weewx[6607]: VantagePro: successfully woke up console
Mar 27 22:44:23 debian weewx[6607]: wxengine: Caught WeeWxIOError: Unable to pass CRC16 check while getting data
Mar 27 22:44:23 debian weewx[6607]:     ****  Waiting 60 seconds then retrying...

What would you suggest to improve ?

Cheers
Philippe

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

Hi, Philippe,

I actually think the problem is with your database, not the connection to the VantagePro. It's taking 11 seconds to write 5 records (the size of a VantagePro 'page'). That's much too long. By the time it's done and ready for another page, the socket connection has timed out. By contrast, my desktop machine (an average Optiplex) can do 60+ records in that time.

You could increase the socket timeout (option timeout, in the [VantagePro] section). That should help, but it will take an excruciatingly long time to download all the data and store it in the database.

This is a plug with a solid-state disk, right? How slow is the disk? Do you know anything about it? It shouldn't take this long for these writes. They are pretty straightforward.

Gnome comes with a disk utility that can perform read and write benchmarks. Unfortunately, the write benchmark requires the disk to be empty, but you could probably devise something yourself say, copying a file.

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/baR8c-GTkPEJ.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

--
Thomas Keffer

kef...@threefools.org
+1 541-386-8891 (h)

[Philippe AMRAM]

Hi Tom,

I have checked my flash disk using hdparm -tT /dev/hda.

/dev/sda1:
 Timing cached reads:   516 MB in  2.00 seconds = 258.00 MB/sec
 Timing buffered disk reads:   54 MB in  3.10 seconds =  17.44 MB/sec

Cheers.
Philippe

[Philippe AMRAM]

Tom,

I have increased the timeout up to 20 (!)
The system is getting much slowwwwwwwwwwwwwwwer,
and I still get :
Mar 28 07:56:17 debian weewx[7260]: VantagePro: Retrieving 58 page(s); starting index= 0
Mar 28 07:56:27 debian weewx[7260]: Archive: added archive record 2012-03-27 08:05:00 CEST (1332828300)
Mar 28 07:56:30 debian weewx[7260]: Archive: added archive record 2012-03-27 08:10:00 CEST (1332828600)
Mar 28 07:56:33 debian weewx[7260]: Archive: added archive record 2012-03-27 08:15:00 CEST (1332828900)
Mar 28 07:56:35 debian weewx[7260]: Archive: added archive record 2012-03-27 08:20:00 CEST (1332829200)
Mar 28 07:56:37 debian weewx[7260]: Archive: added archive record 2012-03-27 08:25:00 CEST (1332829500)
Mar 28 08:00:09 debian weewx[7260]: VantagePro: Unable to pass CRC16 check while getting data


Cheers
Philippe

[Philippe Amram]

Hi Tom,

my god feeling tells me that there could be a communication problem with the console.
The loops fail often (more than 10 times, my limit)
The wake-up of the console fails often.
The pages download fails after 5 lines of download on my plug and 20 times on my desktop PC.

I have noticed that the firmware of my console is quite old (1.6 or so...)
I should update it and then maybe, get an improvement of the complete process.
As soon as I find a way to ugrade WITHOUT using Windows and WITHOUT using a serial port, I let you know...

Cheers.

Philippe

Le samedi 24 mars 2012 08:24:33 UTC+1, Philippe Amram a écrit :

[Thomas Keffer]

It's possible that you have a console firmware problem, but that's not the first place I would look.

The reason why the download fails after 5 records is that is the size of a VantagePro "page". There are 512 pages on the device, each with 5 records, for a total of 2560 records.

Weewx is reading a page, then extracting the five records, writing each to the database. It is taking so long to write a record that by the time it gets done with all 5, the console has figured that the program is not getting back to it, so it stops listening. A CRC error results. 

Your test using hdparm is not conclusive because it was a read test. You really need a write test.

The one thing that is inconsistent with my theory is that you say the same thing happens on your desktop machine, albeit after 20 records. 

I have included a file that will test how long it takes to do 100 database writes. To use it, pick a directory where you have write privileges, download the file into it, then run it:

  python write_test.py

Let us know what the result is. For comparison, my desktop takes 7.2 seconds, my fit-PC, 7.8 seconds. I'm betting your plug will take much longer, but I could be wrong!

Finally, you say that your firmware is quite old. If you run configure with the '--info' option, what does it say at the top for "FIRMWARE DATE"?

  cd /home/weewx

  ./bin/configure.py weewx.conf --info

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/FKJsqRZT6SYJ.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe AMRAM]

Tom,

nervertheless, This ilooks more or less as a communcation pb, no ?

debian:/home/weewx# ./bin/configure.py weewx.conf --info
Querying...
(several 10th of secounds later)

Traceback (most recent call last):

  File "./bin/configure.py", line 150, in <module>
    main()
  File "./bin/configure.py", line 90, in main
    getattr(station_mod, stationType).runOptions(config_dict, options, args)
  File "/home/weewx/bin/weewx/VantagePro.py", line 1324, in runOptions
    info(config_dict)
  File "/home/weewx/bin/weewx/VantagePro.py", line 1344, in info
    console_time = time.strftime("%x %X", station.getTime())
  File "/home/weewx/bin/weewx/VantagePro.py", line 608, in getTime
    raise weewx.RetriesExceeded("While getting console time")
weewx.RetriesExceeded: While getting console time


Cheers
Philippe AMRAM

[Thomas Keffer]

It certainly does.

I believe you can get the same information from the display on the console through the setup menu.

-tk
________________
Blame my Android for any typos.

[Philippe AMRAM]

Hi Tom,

I finaly got it :

VantagePro EEPROM settings:
       
        CONSOLE FIRMWARE DATE: <Unavailable>
       
        CONSOLE SETTINGS:
          Archive interval: 300 (seconds)
          Altitude:         328 (meter)
          Wind cup type:    large
          Rain bucket type: 0.2 MM
          Rain year start:  11
          Onboard time:     03/28/12 21:56:31
         
        CONSOLE DISPLAY UNITS:
          Barometer:   hPa
          Temperature: degree_10F
          Rain:        mm
          Wind:        km_per_hour
         
        RECEPTION STATS:
          Total packets received:       528
          Total packets missed:         16
          Number of resynchronizations: 0
          Longest good stretch:         105
          Number of CRC errors:         15
         
        BAROMETER CALIBRATION DATA:
          Current barometer reading:    30.028 inHg
          Altitude:                     328 feet
          Dew point:                    46 F
          Virtual temperature:          61 F
          Humidity correction factor:   20
          Correction ratio:             1.012
          Correction constant:          -0.279 inHg
          Gain:                         0.000
          Offset:                       15.000

Cheers
Philippe

[Philippe AMRAM]

and then :

debian:/home/weewx# python write_test.py
Performing 100 writes to ./junk.sdb
Number of writes:   100
Total elapsed time: 4.6


does not look that bad ?
does it ?

Philippe

[Philippe AMRAM]

While it's performing
"Mar 28 22:06:46 debian weewx[2291]: Archive: added archive record 2012-03-28 22:00:00 CEST (1332964800)"

I get

forming 100 writes to ./junk.sdb
Number of writes:   100

Total elapsed time: 14.4


Philippe

[Thomas Keffer]

It seems that the connection failed while getting the firmware date (that's why it says <Unavailable>)!

Very frustrating.

Let me think of a simple test we can perform to pinpoint where the problem is.

By the way, you have not confirmed that this is uses a WeatherLink IP connection. Again, I assume it does? 

One final request: please send your configuration file, weewx.conf.

-tk

[Thomas Keffer]

That's an excellent time. So clearly, I was wrong --- the problem is not with the database writes.

Again, I'll try and think of a simple test for the connection. The VantagePro has a number of test functions that could be invoked through the socket connection.

Again, please send you weewx.conf.

-tk

[Philippe AMRAM]

Yes Tom, I use an IP datalogger (Weatherlink IP).

Philippe AMRAM
57 rue Camille Pelletan
92290 F-Chatenay Malabry
Gsm : 06 85 40 42 11
Fixe : 01 46 60 01 18
Fax :  09 56 70 75 09

[Thomas Keffer]

And the copy of weewx.conf? You can send it in a private email, if you wish. 

tkeffer at gmail dot com.

-tk

[Philippe AMRAM]

I 'll send you my weewx.conf file tonight. (when I'm back home).

Philippe AMRAM
57 rue Camille Pelletan
92290 F-Chatenay Malabry
Gsm : 06 85 40 42 11
Fixe : 01 46 60 01 18
Fax :  09 56 70 75 09

[Steve]

Hi Philippe
Thanks for the offer
Regards
Steve
--
Sent from my Galaxy Note

[Thomas Keffer]

I've attached another test you can try. It will issue a DMP command to your station. This command simply requests all records from the VP, one after another, very fast. If the problem is your connection, or the firmware, it will show it.

I've tried it with my USB station and it took 72 seconds. I do not have access to a WeatherLink IP. Hopefully it will work!

To use it:

  $ python test_ip.py host-ip

substitute your host ip for host-ip. For example,

  $ python test_ip.py 192.168.0.41

It assumes port 22222 (the norm for the WeatherLink IP).

-tk

[Philippe AMRAM]

Excellent test !

Result :
Timestamps received:  1212
Nulls received:       106
Elapsed time (secs):  258.166978836

Cheers
Philippe

[Thomas Keffer]

I'm actually amazed that it worked without a problem! :-)

This is a very long elapsed time.

Are you sending me everything it printed out? It should say "Loop terminated normally" or, alternatively, that an exception was thrown. I suspect the latter because the number of timestamps (1318) is not adding up to the total memory of the logger (2560).

-tk

[Philippe AMRAM]

I should have added the following :

****** N/A ******** (    N/A   )
... (many of them)
****** N/A ******** (    N/A   )
****** N/A ******** (    N/A   )
^CException: Socket write error
Timestamps received:  1212

I had to "CTRL C" because it was displaying tens of "****** N/A ******** (    N/A   )"

cheers.

Philippe AMRAM

[Philippe AMRAM]

I do the test again, and do not "CTRL C" this time...

Philippe AMRAM

[Philippe AMRAM]

I got it !

Loop terminated normally
Timestamps received:  1222
Nulls received:       2676
Elapsed time (secs):  518.033209801
:

Philippe AMRAM

[Thomas Keffer]

That is very slow.

One other thing to try: reset the VP2.

To reset:

  # cd /home/weewx

  # ./bin/configure.py --clear weewx.conf

Then, unplug, take out the batteries, wait 20 seconds, then put everything back together.

Then try the test again.

If it is still very slow, then contact Davis and ask if they have seen this problem before.

-tk

[Thomas Keffer]

One other thing: be sure to change tcp_send_delay to 0.5. Did you do this before running the test?

0tk

[Philippe AMRAM]

I have done a reset (clear + remove the batteries)
then started the reading test again.
same result : slowwwwwwww

Cheers Tom
Philippe

[Philippe AMRAM]

U had asked me to : "./bin/configure.py weewx.conf --info"
Now the result is :

VantagePro EEPROM settings:
       
        CONSOLE FIRMWARE DATE: Dec  1 2006

       
        CONSOLE SETTINGS:
          Archive interval: 300 (seconds)
          Altitude:         328 (meter)
          Wind cup type:    large
          Rain bucket type: 0.2 MM
          Rain year start:  11

          Onboard time:     03/29/12 22:55:10

         
        CONSOLE DISPLAY UNITS:
          Barometer:   hPa
          Temperature: degree_10F
          Rain:        mm
          Wind:        km_per_hour
         
        RECEPTION STATS:

          Total packets received:       84
          Total packets missed:         4
          Number of resynchronizations: 0
          Longest good stretch:         33
          Number of CRC errors:         3
         
        BAROMETER CALIBRATION DATA:
          Current barometer reading:    29.640 inHg
          Altitude:                     328 feet
          Dew point:                    43 F
          Virtual temperature:          59 F
          Humidity correction factor:   17
          Correction ratio:             1.001

          Correction constant:          -0.279 inHg
          Gain:                         0.000
          Offset:                       15.000

Cheers
Philippe

[Thomas Keffer]

That is a very recent Firmware.

Having cleared the memory (and therefore having no need to download old data), try just restarting weewx. Might work.

-tk

[Peter Nock]

I'll chime in here for reference from another weatherlink IP:

> python write_test.py
Performing 100 writes to ./junk.sdb
Number of writes:   100

Total elapsed time: 0.2

>python test_ip.py <ip>
....
Loop terminated normally
Timestamps received:  2560
Nulls received:       0
Elapsed time (secs):  258.526223898

[Thomas Keffer]

That is great information, Peter!

OK, Philippe, it looks like you are getting half the communication speeds that Peter is getting. That's slow, but probably not so slow that it can't work.

Peter, do you have any suggestions for how to tune tcp_send_delay or other parameters?

That must be one hulking machine you've got there, Peter, to do that write test in 0.2 seconds!

-tk

[Peter Nock]

That was with tcp_send_delay set at 0.5. With 1.0 I get almost exactly the same results as Philippe:

Loop terminated normally
Timestamps received:  2560
Nulls received:       0

Elapsed time (secs):  516.278360844

I've tried the test_ip.py script with 0.1 but it errors quite quickly (CRC error), 0.2 gives

Loop terminated normally
Timestamps received:  2560
Nulls received:       0

Elapsed time (secs):  104.45986414

As a note I'm running the station with tcp_send_delay set at 1.0. The Weatherlink IP is not happy with more than one concurrent connection at a time either; I had to stop the main weewx daemon before test_ip would run regardless of the tcp_send_delay.

Yes I'd have expected sqlite to take much longer to write 100 records to disk too! I noticed that since 3.7 they've added write-ahead logging which presumably is responsible for the much improved speed. See: http://www.sqlite.org/draft/wal.html .

[Thomas Keffer]

Very useful information!  I think I'll leave the default delay alone (1.0 seconds).  Philippe, you could try to reduce it a bit, maybe to 0.5 or 0.75. Experiment using the test_ip.py script.

Write Ahead Logging must explain the difference. I am running v3.6.22.

Thanks,. Peter!

-tk

[Philippe Amram]

Hi Tom,

it's a long time we have been in touch !
My Weewx Debian Dockstar Plug has been running for weeks.
Unfortunatelay, this afternoon, when I was about to check my website (we have a storm currently in Paris),
I have noticed that the site was not uploaded anylonger (since this morning 10 am local time).

Got from the syslog :
Apr 15 21:02:13 debian weewx[2027]: VantagePro: successfully woke up console
Apr 15 21:02:15 debian weewx[2027]: VantagePro: Retrieving 0 page(s); starting index= 0
Apr 15 21:02:15 debian weewx[2027]: wxengine: Starting main packet loop.
Apr 15 21:02:18 debian weewx[2027]: VantagePro: successfully woke up console
Apr 15 21:02:19 debian weewx[2027]: VantagePro: Clock error is -4.72 seconds (positive is fast)
Apr 15 21:02:19 debian weewx[2027]: VantagePro: Requesting 200 LOOP packets.
Apr 15 21:02:21 debian weewx[2027]: VantagePro: successfully woke up console
Apr 15 21:03:04 debian weewx[2027]: VantagePro: LOOP #21; read error. Try #1
Apr 15 21:03:04 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:09 debian weewx[2027]: VantagePro: LOOP #22; read error. Try #2
Apr 15 21:03:09 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:14 debian weewx[2027]: VantagePro: LOOP #23; read error. Try #3
Apr 15 21:03:14 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:19 debian weewx[2027]: VantagePro: LOOP #24; read error. Try #4
Apr 15 21:03:19 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:24 debian weewx[2027]: VantagePro: LOOP #25; read error. Try #5
Apr 15 21:03:24 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:29 debian weewx[2027]: VantagePro: LOOP #26; read error. Try #6
Apr 15 21:03:29 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:34 debian weewx[2027]: VantagePro: LOOP #27; read error. Try #7
Apr 15 21:03:34 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:39 debian weewx[2027]: VantagePro: LOOP #28; read error. Try #8
Apr 15 21:03:39 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:44 debian weewx[2027]: VantagePro: LOOP #29; read error. Try #9
Apr 15 21:03:44 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:49 debian weewx[2027]: VantagePro: LOOP #30; read error. Try #10
Apr 15 21:03:49 debian weewx[2027]:       ****  Socket read error
Apr 15 21:03:49 debian weewx[2027]: VantagePro: Max retries (10) exceeded.
Apr 15 21:03:50 debian weewx[2027]: VantagePro: Unable to wake up console
Apr 15 21:03:50 debian weewx[2027]: wxengine: Caught WeeWxIOError: Max retries exceeded while getting LOOP data.
Apr 15 21:03:50 debian weewx[2027]:     ****  Waiting 60 seconds then retrying...

and again and again...
Of course, there is no FTP update.

Any suggestion Tom ?

Looking forward to reading from you.
Cheers

[Thomas Keffer]

Hi, Philippe.

Two things to try:

• First, did you disable the "Upload to weatherlink.com" checkbox in the weatherlink IP setup page? To do this: use a web browser to go to http://a.b.c.d where a.b.c.d is the IP address of your weatherlink IP. A very simple configuration webpage should appear. Uncheck the box "Upload to weatherlink.com" box, then click save.
• The only other thing I can suggest is to unplug the console, take out the batteries, wait a few seconds, then put it back together and try again.

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/weewx-user/-/peVRGcWT8R4J.

To post to this group, send email to weewx...@googlegroups.com.
To unsubscribe from this group, send email to weewx-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/weewx-user?hl=en.

[Philippe AMRAM]

Thanks Tom for the answer.

1 - The datalogger has been set to "no upload" a few weeks aggo. I have checked again, still off.
2 - I have allready done the ''off, then on'' of my console yesterday.

I'am affraid we have to find something else...

Cheers
Philippe

[Philippe AMRAM]

Tom,

I have checked my archive files.
Is the size weewx.sdb not to large ?

-rw-r--r-- 1 root root    59392 avr 16 08:01 stats.sdb
-rw-r--r-- 1 root root    40960 mar 25 19:29 stats.sdb.old
-rw-r--r-- 1 root root 50006016 avr 16 08:01 weewx.sdb

Thanks a lot Tom.

Philippe

[Philippe AMRAM]

Tom,

I am so sick and tired with this IP datalogger that I'm going to test the direct USB connection between the Dockstar and my console.
I keep you posted about the results.

Cheers.

Philippe

[Thomas Keffer]

That sounds like an excellent idea.

Regarding your archive, after 5 years, mine is about 80 MB big. How many years of data is in yours?

-tk

[Philippe AMRAM]

Actually, I have noticed that after a hardware network encident (switch off of the router for instance), the console gets in trouble.
Even if you hardware reset the console, it will recover only bit by bit.
That means that the data will also be retrieved bit by bit.
--> the next FTP update will most probably happen hours after the crash...

That enforces my motivation to go back to an USB link instead of an IP.

News very soon.

Cheers


Philippe

[Philippe AMRAM]

I haved finanely moved from a Weatherlink IP connection to a direct USB connection.
After a few tricks to make it run (several ON and OFF switches , some reboots,..) all errors have disappeared...
Now lets focus on the HTML layouts, and other  sensors addin (other 1-wire temp sensors and an electricity meter connection)

Thanks a lot Tom for your precious help !
Keep in touch
Cheers

Philippe