vueiss.py driver not writing to database (RPi)

[Loden Hos]

Good morning,

I am trying to run weewx on a raspberry pi 2b (Rev 1.1) with a custom driver named vueiss.py.
this driver can fetch data from davis vantage vue without the davis console.
Its installation and use is not very good documented, see the authors ressources:
https://buwx.de/index.php/technik
https://buwx.de/index.php/technik/79-logger2
https://github.com/buwx/
Sorry, the links are only in german.

But with a little effort, everything works fine.
The Simulator is writing to the mariadb and reports are showing up.
The pi is receiving data from the Vue ISS on its konsole.
The vueiss driver is successfully installed and activated in weewx.
Though i noted a massive clock error with vueiss-driver:
--Aug 16 00:32:46 wetterbot2 weewx[21924] INFO weewx.engine: Clock error is -1629066766.19 seconds (positive is fast)
compared to simulater driver:
--Aug 16 02:10:46 wetterbot2 weewx[1477] INFO weewx.engine: Clock error is -0.22 seconds (positive is fast)

Only problem: With the vueiss driver no data is written into the database. no reports.

Please see my logs:
For wee_debug --info --verbosity=2, see https://pastebin.com/0N3m5cJP
For tail -f /var/log/syslog after a restart of weewx service, see https://pastebin.com/67F02BTG

Finally I am attaching my install notes, as suggested by the forum guidelines.

Looking forward to your hints,

best, Loden

-----install os-----

# sd-card: image 2021-05-07-raspios-buster-armhf-lite.zip
# flashed with balena etcher
#
# file ssh created in /boot
#
# raspi booted with new sd

sudo raspi-config
# changes in:
# -local
# -timezone
# -keyboard
# -spi
# -hostname

sudo apt update
sudo apt dist-upgrade

sudo shutdown -r now

-----setup python-----
# https://www.techcoil.com/blog/how-to-setup-raspbian-stretch-lite-on-raspberry-pi-3-to-run-python-3-applications/

sudo apt-get install python3-pip python-pip virtualenv
pip3 list

python -V
python3 -V
sudo update-alternatives --list python
sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.7 2


-----install nginx, mariadb, php7, phpmyadmin-----
# https://www.raspifun.de/viewtopic.php?t=24
sudo apt-get install nginx

sudo apt-get install mariadb-server mariadb-client
sudo mariadb
CREATE USER 'davis'@'localhost' IDENTIFIED BY 'davis';
GRANT ALL ON *.* TO 'davis'@'localhost';
FLUSH PRIVILEGES;
exit

sudo apt-get install php7.3 php7.3-fpm php7.3-mysql libzip4 php-pear php7.3-curl php7.3-gd php7.3-intl php7.3-xml php7.3-zip php7.3-mbstring
sudo nano /etc/nginx/sites-available/default
    # pass PHP scripts to FastCGI server
    #
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    #
    #    # With php-fpm (or other unix sockets):
        fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;
    #    # With php-cgi (or other tcp sockets):
    #    fastcgi_pass 127.0.0.1:9000;
    }

    # Add index.php to the list if you are using PHP
    index index.php index.html index.htm;

sudo /etc/init.d/php7.3-fpm reload && sudo /etc/init.d/nginx reload

sudo apt-get install phpmyadmin
sudo nano /etc/nginx/sites-available/default
# add:
server {
    listen          81;
    server_name     wetterbot2;
    root        /usr/share/phpmyadmin;
    index       index.php index.html index.htm;
    if (!-e $request_filename) {
        rewrite ^/(.+)$ /index.php?url=$1 last;
        break;
    }
    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }
}
sudo /etc/init.d/php7.3-fpm reload && sudo /etc/init.d/nginx reload
# find phpmyadmin at https://192.168.1.164:81

-----install weewx-----
# https://www.bitblokes.de/wetterstation-mit-raspberry-pi-sense-hat-und-weewx-schritt-fuer-schritt/

wget -qO - http://weewx.com/keys.html | sudo apt-key add -
wget -qO - http://weewx.com/apt/weewx.list | sudo tee /etc/apt/sources.list.d/weewx.list
sudo apt-get update
sudo apt-get install weewx
Die folgenden NEUEN Pakete werden installiert:
  libimagequant0 liblcms2-2 libwebpdemux2 libwebpmux3 python-cheetah python-configobj python-olefile python-pil python-serial python-usb
  weewx
  lat, long: 52.492580,13.428213
sudo service weewx status

echo "weewx_reports /var/weewx/reports tmpfs size=20M,noexec,nosuid,nodev 0 0" | sudo tee -a /etc/fstab
sudo mkdir -p /var/weewx/reports
sudo mount -a
    ggf.
    sudo cp -r /var/www/html/weewx/ /var/weewx/reports/
    sudo rm -r /var/www/html/weewx
sudo sed -i -e 's%HTML_ROOT =.*%HTML_ROOT = /var/weewx/reports%' /etc/weewx/weewx.conf
sudo service weewx restart
sudo ln -s /var/weewx/reports /var/www/html/weewx
sudo chmod -R 755 /var/www/html/weewx
# find weewx at http://192.168.1.164/weewx/

# https://github.com/weewx/weewx/wiki/Raspberry%20Pi
sudo apt-get purge fake-hwclock
sudo apt-get install ftp

# https://github.com/weewx/weewx/wiki/Minimize-writes-on-SD-cards
sudo dphys-swapfile swapoff

# http://www.weewx.com/docs/usersguide.htm#configuring_mysql
sudo apt install python3-mysqldb python-mysqldb
    [[wx_binding]]
        # The database should match one of the sections in [Databases]
        database = archive_mysql
    [[MySQL]]
        driver = weedb.mysql
        host = localhost
        user = davis
        password = davis
sudo service weewx restart

# up to here: success. simulator data is saved in mariadb and shown in reports.

-----install RaspiRFM II radio module software-----
# http://www.seegel-systeme.de/produkt/raspyrfm-ii/
# http://www.seegel-systeme.de/2015/09/02/ein-funkmodul-fuer-den-raspberry-raspyrfm/#Installation
sudo apt-get install git-core python-dev python-pip
cd git
git clone https://github.com/Phunkafizer/RaspyRFM.git
cd RaspyRFM
sudo pip3 install -e .
sudo pip install -e .
cd

-----setup VueISS device software and driver-----
# https://buwx.de/index.php/technik
# https://buwx.de/index.php/technik/79-logger2
# https://github.com/buwx/
sudo apt install git
mkdir git
cd git
# driver:
git clone https://github.com/buwx/weewx
# software:
git clone https://github.com/buwx/logger
cd
pi@wetterbot2:~ $ python3 git/logger/raspi_rfm.py
2021-08-15 22:06:43,649    INFO    Starting Davis-ISS logging
2021-08-15 22:06:47,949    INFO    I 100 50 00 7A FF 71 00 AE 41 -51.5   -9  1 rr=0mm/h     w(0km/h, 172)
2021-08-15 22:06:50,512    INFO    I 101 80 00 7A 2E 09 00 D9 A1 -50.5  -12  1 t=23.1C      w(0km/h, 172)
2021-08-15 22:06:53,074    INFO    I 102 70 00 7A 02 C1 80 B5 CF -51.0  -10  1 sol=11       w(0km/h, 172)
2021-08-15 22:06:58,199    INFO    I 104 50 00 7A FF 71 00 AE 41 -50.5  -13  2 rr=0mm/h     w(0km/h, 172)
^C
# at this point the ISS is sending data, pi is receiving it.

Setup weewx to use the driver named vueiss.py
---------------------------------------------
# add tables to database weewx (phpmyadmin):
CREATE TABLE sensor (
  dateTime    BIGINT NOT NULL,
  data        VARCHAR(80),
  description VARCHAR(80),
  INDEX (dateTime)
);

CREATE TABLE last_sensor (
  dateTime BIGINT NOT NULL
);

INSERT INTO last_sensor VALUES(0);

cd
sudo cp git/weewx/bin/user/drivers/vueiss.py /usr/share/weewx/user/vueiss.py
sudo cp git/weewx/skins/Standard/* /etc/weewx/skins/Standard/

sudo nano /etc/weewx/weewx.conf
# apply changes from ~/git/weewx/weewx.conf.part
# but adjust the driver location:
[VueISS]
    # The driver to use:
    #driver = user.drivers.vueiss
    driver = user.vueiss
# and because of https://weewx.com/docs/usersguide.htm#archive_interval
# uncomment:
[StdArchive]
    # If the station hardware supports data logging then the archive interval
    # will be downloaded from the station. Otherwise, specify it (in seconds).
    archive_interval = 60

sudo apt install python3-numpy python-numpy libatlas-base-dev

sudo wee_config --reconfigure
description [Wetterbot2, Balkon, Friedel]:
altitude [42, meter]:
latitude [52.492580]:
longitude [13.428213]:
Include station in the station registry (y/n)? [n]:
units [metric]:
Installed drivers include:
  0) VueISS          (user.vueiss)             
  1) AcuRite         (weewx.drivers.acurite)   
  ...
choose a driver [0]:

sudo service weewx restart

# at this point vueiss.py causes a couple of errors  
# probably due to python2-python3-incompatibilities.

python /usr/share/weewx/user/vueiss.py
  File "/usr/share/weewx/user/vueiss.py", line 532
    print weeutil.weeutil.timestamp_to_string(packet['dateTime']), packet
                ^
SyntaxError: invalid syntax
sudo nano /usr/share/weewx/user/vueiss.py
Alt+c
^_
532
old:         print weeutil.weeutil.timestamp_to_string(packet['dateTime']), packet
new:         print(weeutil.weeutil.timestamp_to_string(packet['dateTime']), packet)

ModuleNotFoundError: No module named 'Queue'
sudo apt-get install libmariadb-dev-compat libmariadb-dev
sudo nano /usr/share/weewx/user/vueiss.py
Alt+c
^_
17
old: import Queue
new: try:
         import queue
     except ImportError:
         import Queue as queue
459
old:        self.packets = Queue.Queue()
new:        self.packets = queue.Queue()


# now driver is loaded successfully and can easily be switched on and off by
# sudo wee_config --reconfigure --driver=user.vueiss --no-prompt
# sudo wee_config --reconfigure --driver=weewx.drivers.simulator --no-prompt
# sudo service weewx restart
# my version of vueiss.py: https://pastebin.com/YHEqnr3u
#
# though noted a massive clock error with vueiss-driver:
Aug 16 00:32:46 wetterbot2 weewx[21924] INFO weewx.engine: Clock error is -1629066766.19 seconds (positive is fast)
# compared to simulater driver:
Aug 16 02:10:46 wetterbot2 weewx[1477] INFO weewx.engine: Clock error is -0.22 seconds (positive is fast)

# Problem: No data is saved in database and/or shown in reports.
# See the logs:
sudo wee_debug --info --verbosity=2
# see https://pastebin.com/0N3m5cJP
tail -f /var/log/syslog
# see https://pastebin.com/67F02BTG

[vince]

I would start by fixing the clock error.  Make sure you install and enable ntp in your os and get the clock to match true time.  You might also check the clock on your Vantage to verify it is correct.

[Loden Hos]

thanks vince, you made me try all options of getting good time: disable all time services and reboot. i even installed a piface shim real time clock and rebooted, allways the same: simulator driver works with correct time. vueiss comes up with a clock error of ~51 years (1.1.1970??). if there is no time service and no rtc at all, weewx is still working properly: if there is no time, it logs "Feb 14 11:12:06 wetterbot2 python2[389]: weewx[389] INFO __main__: Waiting for sane time. Current time is 2019-02-14 11:12:06 CET (1550139126)". and it is really waiting, and if i then start a time demon or the rtc service, it continues starting - unfortunately not with the correct time provided meanwhile but with its 51year-time error.

that all drives me to believe that the time error doesn't result from false rpi or weewx setup but has to do with the vueiss driver.

[gjr80]

Can’t say I’ve ever seen a driver like this before. From a quick look at the code it appears the system is in two parts. First there is a ‘logger’ that runs independently of the driver, the logger intercepts the RF from the ISS and records data (including time stamps) in a separate table (table sensors) in the WeeWX database. Then the driver reads this data and emits loop packets. I would more accurately describe the driver as ‘reading data from a database and emitting loop packets’ rather than reading data from the ISS.

The driver gets its ‘time’ from (initially) the last good record in the WeeWX archive. If there are no records in the archive the ‘time’ is picked up from this intermediate table created and used by the logger. If you have a new install then you likely have no archive records, so there is no last good timestamp from the archive. In this case the driver will go to the intermediate sensors table and get a timestamp. Judging by the log which says ‘starting at 0’ I’m guessing this table is not being populated correctly and hence no ‘time’ is being picked up resulting in the large discrepancy (the driver is effectively being told by the ‘station’ it is the start of Unix epoch time (t = 0))

As Vince said you need to start by fixing the time, without that being fixed WeeWX will never see a loop packet, you will never have archive records created and hence nothing in the archive and no reports; WeeWX will just appear to do nothing. Given the peculiar arrangement within the driver for time I suspect that the logger, whilst maybe getting data from the ISS, is not populating the sensors table. If you can’t figure that out my first point of call would be the author. And if that yields no joy I would seriously consider buying a console :)

Gary

[Loden Hos]

thanks gjr80 for your explanations. until now i did not at all understand what the vueiss driver does. if i get you right, what i should try is: seed some plausible database entries. allow me to add a question: how would i do that "manually"? anyway: in a few days i will be away for a couple weeks. but i will definitely come back to that approach.

best, lh

[gjr80]

No, I never said that at all. In fact looking at the driver again it seems I misread and the driver does not get the time initially from the last record in the WeeWX archive but rather it gets a timestamp from the dateTime field in table last_sensor. The last_sensor table appears to be another table created by the logger. Otherwise it’s as I said previously; the time is then obtained from the last record in the sensors table if it can’t be obtained from the last_sensor table.

So the upshot is that you need to get the last_sensor and sensors tables being populated by the logger. I would be looking closely at the logger part of the system and then worry about the driver,

Gary

[Loden Hos]

good morning,

i solved this with a workaround. since the data is coming in, but i can't get it directly into the weewx database, i now rewrite the incoming date via shell script into a csv file, which i then import cron-based via the weewx import module. it's not 100% live. but at least it's in the db.

i changed the original logger to write to csv: https://pastebin.com/mhAHT2fP

cron does this:

*     *    * * * /home/pi/git/logger/logger.sh >/dev/null 2>&1         # every minute
*/10  *    * * * /home/pi/git/logger/logger-import.sh >/dev/null 2>&1  # every 10 minutes

see

logger.sh: https://pastebin.com/ikMBMsrr

logger-import.sh: https://pastebin.com/zeybbJFB