Datenbank zusammen legen.

893 views
Skip to first unread message

Günther Wrana

unread,
Nov 24, 2019, 4:52:27 AM11/24/19
to weewx-user
Hallo

ich möchte Versuchen zwei Datenbanken von weewx.sdb zusammen zu fügen.

Ich habe einen Rechner mit Ubuntu und kann die Datenbank mit DB Browser for SQLite öffnen.

Die beiden Datenbanken beginnen und enden zu folgenden Zeiten.

1573144800 07.11.2019 17:40:00
1573149300 07.11.2019 18:55:00

1573249500 08.11.2019 22:45:00
1574009700 17.11.2019 17:55:00

Nun habe ich noch nicht herausgefunden wie ich das mit DB Browser for SQLite mache.
Ich weis auch nicht ob das das Richtige Programm für mein Vorhaben ist.

Darum hoffe ich das Ihr mir sagen könnt wie oder mit welchem Programm ich das tun soll.
Das ganze ist einmal ein Versuch danach würde ich dann gerne meine alten Wetterdaten die ich seit November 2014 aufgezeichnet habe auch noch einfügen.

Für alle Tipps bin ich dankbar also dann

Grüße Günther

Xant

unread,
Nov 25, 2019, 10:46:59 PM11/25/19
to weewx-user
Hallo Gunther

I dont know if possible to merge 2 databases.... which it seems can open a "pandora box" and lead to much conflict.

On the other side in a recent posting, Thomas has pointed that there is a whole section in the Customization Guide, on how to address multiple bindings (please, verify):


That said, always verify the User's Guide, do a Forum search prior to new posting, and English is the primary language of WeeWX Forum (not been harsh here, just been clear if you expecting replies).

Best, X
Message has been deleted

Andrew Milner

unread,
Nov 26, 2019, 1:45:39 AM11/26/19
to weewx-user
I would merge only the archive table and then use the database utility to drop the daily summary and rebuild the summaries for the combined database.



On Tuesday, 26 November 2019 06:33:06 UTC+2, J D wrote:
Hi,

Yes you can merge two databases I have done it a number of times. You need to use SQLite commands. At a high level you attached to the two databases and then use the update command. If you google merge sqlite databases you can find a number of ways to do it.
Jerry 

Günther Wrana

unread,
Nov 27, 2019, 8:21:43 AM11/27/19
to weewx-user
Womit bearbeite ich die Archivtabelle und welche Datenbankdienstprogramm verwende ich dafür?

vince

unread,
Nov 27, 2019, 8:48:24 AM11/27/19
to weewx-user
On Wednesday, November 27, 2019 at 5:21:43 AM UTC-8, Günther Wrana wrote:
Womit bearbeite ich die Archivtabelle und welche Datenbankdienstprogramm verwende ich dafür?



Günther Wrana

unread,
Dec 28, 2019, 9:16:15 AM12/28/19
to weewx-user
Mit welchem Programm soll ich das durchführen die Archivtabelle zusammen führen?
Soll ich die Beiden Tabellen als CSV zusamen führen und dann wieder Importieren?

Günther Wrana

unread,
Jan 4, 2020, 10:14:24 AM1/4/20
to weewx-user
Ich habe nun schon die ganze Seite 

WeeWX Utilities Guide

mir durchgelesen.
Aber ich verstehe immer noch nicht wie ich die alten Werte in die neue Datenbank bekomme.

Beide Datenbanken wurden mit weewx erstellt und von der selben Wetterstation mit Werten befüllt.

Der einzige Unterschied ist eine Zeit lang war die Wetterstation an einen PC mit Ubuntu angeschlossen, dann an einen Raspberry.
Es gibt keine Zeit Überschneidung oder sonstiges.

Ich möchte nur alle Werte in einer Datenbank haben das ich die Messwerte mir über längere Zeit zurück ansehen kann.

Beide Datenbanken heißen weewx.sdb nur mir unterschiedlichen Größen und Zeitstempeln.

Wie lege ich das an das ich diese beiden Datenbanken zusammen führe, oder besser gesagt was verstehe ich nicht.

wee_database
wee_import
wee_reports

I apologize but my English is not yellow from the egg so I write German.

gjr80

unread,
Jan 4, 2020, 5:41:23 PM1/4/20
to weewx-user
Hallo Günther,

Ihr Englisch ist wahrscheinlich besser als mein Deutsch, aber lassen Sie uns sehen, wie wir gehen. Ich habe sowohl Englische als auch Google-Deutsche Wörter eingefügt.

Wenn ich das richtig verstehe, haben Sie zwei WeeWX SQLite-Datenbanken, die Sie zusammenführen möchten. Leider gibt es im Utilities Guide nichts, was Ihnen das leicht machen könnte. Dies ist ein paar Mal vorgekommen und wenn Sie das Forum nach Beiträgen mit "merge sqlite datbase" durchsuchen, werden Sie einige Beiträge finden, die Ihnen helfen werden. Ich würde dieses versuchen (https://groups.google.com/d/msg/weewx-user/T2VuiIkcAEM/wZzemNzWknQJ) und das ist das, was ich als Basis für das verwendet habe:

1. Erstellen Sie eine Sicherungskopie Ihrer beiden Datenbanken

2. Installieren Sie das Dienstprogramm sqlite3, falls es noch nicht installiert ist:

$ sudo apt-get install sqlite3

3. Wählen Sie eine Ihrer Datenbanken als endgültige Datenbank aus und benennen Sie die andere in db1.sdb um. Sie sollten jetzt zwei Datenbanken haben, eine mit dem Namen weewx.sdb und eine andere mit dem Namen db1.sdb. Um die Dinge später zu vereinfachen, stellen Sie sicher, dass sich weewx.sdb in dem Verzeichnis befindet, in dem Sie Ihre WeeWX-Datenbank dauerhaft speichern möchten (die Standardverzeichnisse sind /home/weewx/archive für eine setup.py-Installation oder /var/lib/weewx für alle andere installiert). Ich werde /home/weewx/archive in dieser Anleitung verwenden

4. Kopieren Sie die Datenbank db1.sdb in eine Datei:

$ sqlite3 /path/to/db1.sdb
sqlite> .output /var/tmp/db1.dump
sqlite> .dump
sqlite> .quit

5. Importieren Sie nun die gedumpten Daten in die endgültige Datenbank weewx.sdb:

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite> .read /var/tmp/db1.dump

Sie werden wahrscheinlich Beschwerden über 'existing tables' oder 'unique constraint failed' sehen. Diese können ignoriert werden. Es muss jedoch etwas Ernsthafteres betrachtet werden.

sqlite> .quit

6. Der nächste Schritt besteht darin, die täglichen Zusammenfassungen neu zu erstellen. Dies erfolgt mit dem Dienstprogramm wee_database und erfordert, dass Sie eine WeeWX-Datenbindung konfigurieren, um die jetzt konsolidierte Datenbank weewx.sdb zu verwenden. Bearbeiten Sie die Datei weewx.conf und suchen Sie die Zeilengruppe [Databases]. Überprüfen Sie, ob Sie einen Datenbankeintrag für weewx.sdb haben. Dieser sollte ungefähr so ​​aussehen:

[Databases]

    # A SQLite database is simply a single file
    [[archive_sqlite]]
        database_name = weewx.sdb
        database_type = SQLite

Wenn sich weewx.sdb in den Verzeichnissen home/weewx/archive oder /var/lib/weewx befindet, ist das oben Genannte in Ordnung. Beachten Sie den Eintrag in [[ ]], in diesem Fall archive_sqlite.

Überprüfen Sie nun die Zeilengruppe [DataBindings] in der Datei weewx.conf.

[DataBindings]

    [[wx_binding]]
        # The database must match one of the sections in [Databases].
        # This is likely to be the only option you would want to change.
        database = archive_sqlite
        # The name of the table within the database
        table_name = archive
        # The manager handles aggregation of data for historical summaries
        manager = weewx.wxmanager.WXDaySummaryManager
        # The schema defines the structure of the database.
        # It is *only* used when the database is created.
        schema = schemas.wview.schema

Sie müssen überprüfen, ob die Datenbankeinstellung in der Zeilengruppe [[wx_binding]] den Eintrag aus der Zeilengruppe [[ ]] verwendet, den wir zuvor in der Zeilengruppe [Databases] notiert haben, in diesem Fall archive_sqlite.

Sie können nun mit wee_database die täglichen Zusammenfassungen löschen und neu erstellen.

7. Legen Sie die täglichen Zusammenfassungen ab und erstellen Sie sie neu:

$ wee_database --drop-daily
$ wee_database --rebuild-daily

Beantworten Sie die Aufforderungen nach Bedarf

8. Ihre konsolidierte Datenbank sollte nun vollständig und WeeWX gesetzt, es zu benutzen. Der letzte Schritt vor dem Neustart von WeeWX besteht darin, alle Berichte im NOAA-Format zu löschen, damit sie mit den neuen konsolidierten Daten neu generiert werden. Suchen Sie die NOAA-Format Berichte über Ihre WeeWX Maschine und löschen sie alle, Sie werden wahrscheinlich die Berichte im /home/weewx/public_html/NOAA oder /var/www/html/weewx/NOAA-Verzeichnis je nach WeeWX Art installieren finden.

9. Starten Sie WeeWX neu

Überprüfen Sie Ihre Protokolle beim Start von WeeWX auf Fehler. Überprüfen Sie Ihre Ausgabe, um festzustellen, ob die neuen konsolidierten Daten verwendet werden.

Gary

Your English is probably better than my German but let us see how we go. I have included both English and Google-German words.

If I understand correctly you have two WeeWX SQLite databases that you wish to merge. Unfortunately there is nothing in the Utilities Guide that will easily do this for you. This has come up a few times before and if you search the forum for posts with 'merge sqlite database' you will find a few posts that will help. I would try this one (https://groups.google.com/d/msg/weewx-user/T2VuiIkcAEM/wZzemNzWknQJ) and that is the one I have used as the basis for the following steps:

1. Make a backup copy of both your databases

2. Install the sqlite3 utility if it is not already installed:

$ sudo apt-get install sqlite3

3. Select one of your databases as the final database and rename the other to say db1.sdb. You should now have two databases, one named weewx.sdb and another named db1.sdb. To simplify things later make sure that weewx.sdb is located in the directory where you wish to permanently store your WeeWX database (the default locations are /home/weewx/archive for a setup.py install or /var/lib/weewx for all other installs). I will use /home/weewx/archive in these instructions

4. Dump the db1.sdb database to a file:

$ sqlite3 /path/to/db1.sdb
sqlite> .output /var/tmp/db1.dump
sqlite> .dump
sqlite> .quit

5. Now import the dumped data into the final database weewx.sdb:

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite> .read /var/tmp/db1.dump

you will likely see some complaints about 'existing tables' or 'unique constraint failed', these can be ignored. Though anything more serious needs to be looked at.

sqlite> .quit

6. The next step is to rebuild the daily summaries, this is done with the wee_database utilitiy and requires that you configure a WeeWX data binding to use the now consolidated database weewx.sdb. Edit weewx.conf and locate the [Databases] stanza. Check you have a database entry there for weewx.sdb, it should look something like:

[Databases]

    # A SQLite database is simply a single file
    [[archive_sqlite]]
        database_name = weewx.sdb
        database_type = SQLite

if weewx.sdb is in the home/weewx/archive or /var/lib/weewx directories then the above is fine. Take note of the entry in the [[ ]], in this case archive_sqlite.

Now check the [DataBindings] stanza in weewx.conf, it will likely look like this:

[DataBindings]

    [[wx_binding]]
        # The database must match one of the sections in [Databases].
        # This is likely to be the only option you would want to change.
        database = archive_sqlite
        # The name of the table within the database
        table_name = archive
        # The manager handles aggregation of data for historical summaries
        manager = weewx.wxmanager.WXDaySummaryManager
        # The schema defines the structure of the database.
        # It is *only* used when the database is created.
        schema = schemas.wview.schema

You need to check that the database setting in the [[wx_binding]] stanza uses the entry from the [[ ]] stanza that we previously noted in the [Databases] stanza, in this case archive_sqlite.

You can now use wee_database to drop and rebuild the daily summaries.

7. Drop and rebuild the daily summaries:

$ wee_database --drop-daily
$ wee_database --rebuild-daily

answering any prompts as required

8. Your consolidated database should now be complete and WeeWX set to use it. The final step before restarting WeeWX is to delete all the NOAA format reports so that they are re-generated with the new consolidated data. Locate the NOAA format reports on your WeeWX machine and delete them all, you will likely find the reports in the /home/weewx/public_html/NOAA or /var/www/html/weewx/NOAA directory depending on your WeeWX install type.

9. Restart WeeWX

Check your logs from WeeWX startup for errors. Check your output to see that the new consolidated data is being used (checking the NOAA reorts is a good and simple check for this)

Günther Wrana

unread,
Jan 6, 2020, 2:56:41 PM1/6/20
to weewx-user

Hallo wenn ich folgende Befehle im Terminal eingebe wie oben beschrieben.
Bekomme ich folgende Fehlermeldungen.

Es wir irgend etwas in der Datenbank durchgeführt nur wird sie dadurch nicht grösser.
Wo liegt das Problem, oder was mache ich falsch?



daham@daham-p7-1011de:~$ sqlite3 /var/tmp/db1.sdb
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .output/var/tmp/db1.dump
daham@daham-p7-1011de:~$ sqlite3 /media/daham/Neu1601/Eigene\ Dateien/Wetterstation/weewx/271220192133/weewx.sdb
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .read /var/tmp/db1.dump
Error: near line 3: table archive already exists
Error: near line 40840: table archive_day_barometer already exists
Error: near line 40984: table archive_day_pressure already exists
Error: near line 41128: table archive_day_altimeter already exists
Error: near line 41256: table archive_day_inTemp already exists
Error: near line 41400: table archive_day_outTemp already exists
Error: near line 41544: table archive_day_inHumidity already exists
Error: near line 41688: table archive_day_outHumidity already exists
Error: near line 41832: table archive_day_windSpeed already exists
Error: near line 41976: table archive_day_windDir already exists
Error: near line 42120: table archive_day_windGust already exists
Error: near line 42264: table archive_day_windGustDir already exists
Error: near line 42408: table archive_day_rainRate already exists
Error: near line 42552: table archive_day_rain already exists
Error: near line 42696: table archive_day_dewpoint already exists
Error: near line 42840: table archive_day_windchill already exists
Error: near line 42984: table archive_day_heatindex already exists
Error: near line 43128: table archive_day_ET already exists
Error: near line 43272: table archive_day_radiation already exists
Error: near line 43416: table archive_day_UV already exists
Error: near line 43560: table archive_day_extraTemp1 already exists
Error: near line 43704: table archive_day_extraTemp2 already exists
Error: near line 43848: table archive_day_extraTemp3 already exists
Error: near line 43992: table archive_day_soilTemp1 already exists
Error: near line 44136: table archive_day_soilTemp2 already exists
Error: near line 44280: table archive_day_soilTemp3 already exists
Error: near line 44424: table archive_day_soilTemp4 already exists
Error: near line 44568: table archive_day_leafTemp1 already exists
Error: near line 44712: table archive_day_leafTemp2 already exists
Error: near line 44856: table archive_day_extraHumid1 already exists
Error: near line 45000: table archive_day_extraHumid2 already exists
Error: near line 45144: table archive_day_soilMoist1 already exists
Error: near line 45288: table archive_day_soilMoist2 already exists
Error: near line 45432: table archive_day_soilMoist3 already exists
Error: near line 45576: table archive_day_soilMoist4 already exists
Error: near line 45720: table archive_day_leafWet1 already exists
Error: near line 45864: table archive_day_leafWet2 already exists
Error: near line 46008: table archive_day_rxCheckPercent already exists
Error: near line 46152: table archive_day_txBatteryStatus already exists
Error: near line 46296: table archive_day_consBatteryVoltage already exists
Error: near line 46440: table archive_day_hail already exists
Error: near line 46584: table archive_day_hailRate already exists
Error: near line 46728: table archive_day_heatingTemp already exists
Error: near line 46872: table archive_day_heatingVoltage already exists
Error: near line 47016: table archive_day_supplyVoltage already exists
Error: near line 47160: table archive_day_referenceVoltage already exists
Error: near line 47304: table archive_day_windBatteryStatus already exists
Error: near line 47448: table archive_day_rainBatteryStatus already exists
Error: near line 47592: table archive_day_outTempBatteryStatus already exists
Error: near line 47736: table archive_day_inTempBatteryStatus already exists
Error: near line 47880: table archive_day__metadata already exists
Error: near line 47881: UNIQUE constraint failed: archive_day__metadata.name
Error: near line 47882: UNIQUE constraint failed: archive_day__metadata.name
Error: near line 47883: table archive_day_wind already exists
sqlite> 

gjr80

unread,
Jan 6, 2020, 4:43:02 PM1/6/20
to weewx-user
Hallo

Die angezeigten Fehlermeldungen werden erwartet und können ignoriert werden.

Die Dateigröße kann manchmal ein nützlicher Maßstab sein, liefert jedoch nicht immer ein genaues Bild. Der wahre Test besteht darin, die Daten in der neuen zusammengeführten Datenbank zu überprüfen oder zu verwenden. Haben Sie die letzten Schritte befolgt und WeeWX tatsächlich dazu gebracht, die neue zusammengeführte Datenbank zu verwenden? Wenn ja, was zeigen die Berichte jetzt?

Sie können auch die neue Datenbank abfragen und die frühesten und neuesten Zeitstempel finden. Wenn Ihre Datenbank /home/weewx/archive/weewx.sdb lautet, kann die folgende Befehlsfolge verwendet werden:

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite
> SELECT MIN(dateTime),MAX(dateTime),COUNT(dateTime) FROM archive;

Dies zeigt die frühesten und neuesten Zeitstempel, gefolgt von der Anzahl der Archivdatensätze.

1486989000|1578345600|300759

oder um die Zeitstempel in lesbarer Form anzuzeigen:

sqlite> SELECT strftime('%d-%m-%Y %H:%M:%S', datetime(MIN(dateTime), 'unixepoch', 'localtime')),strftime('%d-%m-%Y %H:%M:%S', datetime(MAX(dateTime), 'unixepoch', 'localtime')),COUNT(dateTime) FROM archive;

was so etwas zeigen wird:

13-02-2017 22:30:00|07-01-2020 07:25:00|300760

Sie können die obigen Überprüfungen auch für die beiden Datenbanken vor dem Zusammenführen durchführen. Wenn der früheste und der späteste Zeitstempel und Datensatz übereinstimmen, haben Sie ein gewisses Maß an Sicherheit, dass die Zusammenführung wie erwartet erfolgt ist. Eine umfassendere Überprüfung besteht darin, dass WeeWX die zusammengeführte Datenbank verwendet und anschließend neu generierte NOAA-Berichte überprüft. Dies sollte auf einen Blick zeigen, ob die Zusammenführung funktioniert hat.

Gary

The error messages you see are expected, they can be ignored.

File size can sometimes be a useful yardstick but does not always give an accurate picture. The true test is to check or use the data in the new merged database. Did you follow the last few steps and actually get WeeWX to use the new merged database? If so what do the reports show now?

Another check you can do is to query the new database and find the earliest and latest timestamps. If your database is /home/weewx/archive/weewx.sdb the following sequence of commands can be used:

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite
> SELECT MIN(dateTime),MAX(dateTime),COUNT(dateTime) FROM archive;

this will show the earliest and latest timestamps followed by the archive record count, something like:

1486989000|1578345600|300759

or to view the timstamps in human readable form:

sqlite> SELECT strftime('%d-%m-%Y %H:%M:%S', datetime(MIN(dateTime), 'unixepoch', 'localtime')),strftime('%d-%m-%Y %H:%M:%S', datetime(MAX(dateTime), 'unixepoch', 'localtime')),COUNT(dateTime) FROM archive;

which will show something like:

13-02-2017 22:30:00|07-01-2020 07:25:00|300760

You can also perform the above checks on the two databases before the merge, if the earliest and latest timestamps and record counts tally then you have a degree of confidence that the merge occurred as expected. A more complete check is having WeeWX use the merged database and then checking re-generated NOAA reports. This should show at a glance if the merge worked.

Günther Wrana

unread,
Feb 16, 2020, 3:42:29 AM2/16/20
to weewx-user
Das wird gemeldet wenn ich die alte Datenbank mit den oben angeführten Befehlen abfrage.
Ist da noch etwas zu machen oder ist die alte Datenbank defekt und ich kann sie löschen.

Grüße Günther


daham@hp-pc:/tmp$ sudo sqlite3 /tmp/db1.sdb
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> SELECT MIN(dateTime),MAX(dateTime),COUNT(dateTime) FROM archive;
Error: database disk image is malformed
sqlite> SELECT strftime('%d-%m-%Y %H:%M:%S', datetime(MIN(dateTime), 'unixepoch', 'localtime')),strftime('%d-%m-%Y %H:%M:%S', datetime(MAX(dateTime), 'unixepoch', 'localtime')),COUNT(dateTime) FROM archive;
Error: database disk image is malformed
sqlite> 

Günther Wrana

unread,
Feb 22, 2020, 2:14:26 PM2/22/20
to weewx-user

If i open this merged database on Ubuntu with SQLite there is no problem.
But when i send this datatbase on the raspberry where weewx is runing weewx is out of order.
Where is the problem?

I know that the database on the raspberry is named weewx.sdb

I hope anybody can help me and understand my terrible english.

Thanks

Thomas Keffer

unread,
Feb 22, 2020, 4:54:32 PM2/22/20
to weewx-user
Günther, 

I get "database disk image is malformed" on both my Ubuntu system, and on a Raspberry Pi. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/87c9cfc6-2b06-4924-8479-c996405e1d74%40googlegroups.com.

Günther Wrana

unread,
Feb 23, 2020, 4:37:11 AM2/23/20
to weewx-user
Sorry i have it on Ubuntu with DB Browser for SQLite opend.
And with this sofware i can take a look on the database.
But i dont know were is the problem.
Is this the problem that i opned the database with this software and not in terminal?

Günther
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

Thomas Keffer

unread,
Feb 23, 2020, 7:06:01 AM2/23/20
to weewx-user
I doubt it, but it is easy enough to check. What happens if you use sqlite3?

sqlite3 db3.sdb
sqlite> select datetime(dateTime,'unixepoch','localtime'), outTemp from archive order by dateTime desc limit 10;



To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ff73bff7-949e-4266-9964-62844b78556d%40googlegroups.com.

Günther Wrana

unread,
Feb 23, 2020, 9:56:32 AM2/23/20
to weewx-user
daham@hp-pc:/tmp$ sudo sqlite3 /tmp/db3.sdb
[sudo] Passwort für daham: 
sqlite> select datetime(dateTime,'unixepoch','localtime'), outTemp from archive order by dateTime desc limit 10;
2020-01-16 17:40:00|33.2514285714286
2020-01-16 17:35:00|33.3028571428572
2020-01-16 17:30:00|33.701
2020-01-16 17:25:00|33.7914285714286
2020-01-16 17:20:00|33.9371428571429
2020-01-16 17:15:00|33.9542857142857
2020-01-16 17:10:00|34.0914285714286
2020-01-16 17:05:00|34.34
2020-01-16 17:00:00|34.4685714285714
2020-01-16 16:55:00|34.5628571428571
sqlite> 

This is the answer if i open db3.sdb in terminal on my linux computer.
I want to know, how the syntax looks like, if i want to look for the mistake.

Danke Günther

Thomas Keffer

unread,
Feb 23, 2020, 10:43:41 AM2/23/20
to weewx-user
You can check the integrity of the database using the pragma "integrity_check".

sqlite3 db3.sdb
sqlite> pragma integrity_check;

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c8e26b1c-5cf4-42ec-b1d7-dc40c07703ac%40googlegroups.com.

Günther Wrana

unread,
Feb 23, 2020, 12:34:50 PM2/23/20
to weewx-user
sqlite> pragma integrity_check;
*** in database main ***
On tree page 554 cell 75: invalid page number 1879769860
Multiple uses for byte 1072 of page 554
On tree page 957 cell 28: Rowid 18740937184 out of order
On tree page 957 cell 25: Rowid 18749066592 out of order
On tree page 957 cell 23: Rowid 1561684576 out of order
On tree page 957 cell 19: Rowid 1568679008 out of order
On tree page 957 cell 17: Rowid 1560117600 out of order
On tree page 957 cell 16: Rowid 1593585632 out of order
row 9507 missing from index sqlite_autoindex_archive_1
row 9508 missing from index sqlite_autoindex_archive_1
row 9509 missing from index sqlite_autoindex_archive_1
row 9510 missing from index sqlite_autoindex_archive_1
row 9511 missing from index sqlite_autoindex_archive_1
row 9512 missing from index sqlite_autoindex_archive_1
row 9513 missing from index sqlite_autoindex_archive_1
row 9514 missing from index sqlite_autoindex_archive_1
row 9515 missing from index sqlite_autoindex_archive_1
row 9516 missing from index sqlite_autoindex_archive_1
row 9517 missing from index sqlite_autoindex_archive_1
row 9518 missing from index sqlite_autoindex_archive_1
row 9519 missing from index sqlite_autoindex_archive_1
row 9520 missing from index sqlite_autoindex_archive_1
row 9521 missing from index sqlite_autoindex_archive_1
row 9522 missing from index sqlite_autoindex_archive_1
row 9523 missing from index sqlite_autoindex_archive_1
row 9524 missing from index sqlite_autoindex_archive_1
row 9525 missing from index sqlite_autoindex_archive_1
row 9526 missing from index sqlite_autoindex_archive_1
row 9527 missing from index sqlite_autoindex_archive_1
row 9528 missing from index sqlite_autoindex_archive_1
row 9529 missing from index sqlite_autoindex_archive_1
row 9530 missing from index sqlite_autoindex_archive_1
row 9531 missing from index sqlite_autoindex_archive_1
row 9532 missing from index sqlite_autoindex_archive_1
row 9533 missing from index sqlite_autoindex_archive_1
row 9534 missing from index sqlite_autoindex_archive_1
row 9535 missing from index sqlite_autoindex_archive_1
row 9536 missing from index sqlite_autoindex_archive_1
row 9537 missing from index sqlite_autoindex_archive_1
row 9538 missing from index sqlite_autoindex_archive_1
row 9539 missing from index sqlite_autoindex_archive_1
row 9540 missing from index sqlite_autoindex_archive_1
row 9541 missing from index sqlite_autoindex_archive_1
row 9542 missing from index sqlite_autoindex_archive_1
row 9543 missing from index sqlite_autoindex_archive_1
row 9544 missing from index sqlite_autoindex_archive_1
row 9545 missing from index sqlite_autoindex_archive_1
row 9546 missing from index sqlite_autoindex_archive_1
row 9547 missing from index sqlite_autoindex_archive_1
row 9548 missing from index sqlite_autoindex_archive_1
row 9549 missing from index sqlite_autoindex_archive_1
row 9550 missing from index sqlite_autoindex_archive_1
row 9551 missing from index sqlite_autoindex_archive_1
row 9552 missing from index sqlite_autoindex_archive_1
row 9553 missing from index sqlite_autoindex_archive_1
row 9554 missing from index sqlite_autoindex_archive_1
row 9555 missing from index sqlite_autoindex_archive_1
row 9556 missing from index sqlite_autoindex_archive_1
row 9557 missing from index sqlite_autoindex_archive_1
row 9558 missing from index sqlite_autoindex_archive_1
row 9559 missing from index sqlite_autoindex_archive_1
row 9560 missing from index sqlite_autoindex_archive_1
row 9561 missing from index sqlite_autoindex_archive_1
row 9562 missing from index sqlite_autoindex_archive_1
row 9563 missing from index sqlite_autoindex_archive_1
row 9564 missing from index sqlite_autoindex_archive_1
row 9565 missing from index sqlite_autoindex_archive_1
row 9566 missing from index sqlite_autoindex_archive_1
row 9567 missing from index sqlite_autoindex_archive_1
row 9568 missing from index sqlite_autoindex_archive_1
row 9569 missing from index sqlite_autoindex_archive_1
row 9570 missing from index sqlite_autoindex_archive_1
row 9571 missing from index sqlite_autoindex_archive_1
row 9572 missing from index sqlite_autoindex_archive_1
row 9573 missing from index sqlite_autoindex_archive_1
row 9574 missing from index sqlite_autoindex_archive_1
Error: database or disk is full
sqlite> 

This is the answer and how can i repair the defect datacell?

Thanks

Thomas Keffer

unread,
Feb 23, 2020, 9:11:53 PM2/23/20
to weewx-user
The best solution is to replace with your backup.

If you do not have a backup, well, ... you can try dumping it to a file, then restoring. The steps are (NOT TESTED):

sqlite3 db3.sdb
sqlite> .output db3.sql
sqlite> .dump archive
sqlite> .quit

Check the file db3.sql, then use it to restore to a new database:

sqlite3 db3_restored.sdb < db3.sql

Hopefully the new file, db3_restored.sdb, will be OK.

-tk


To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/555d9cbb-eaac-4be7-a334-cad72f604725%40googlegroups.com.

Günther Wrana

unread,
Feb 26, 2020, 2:36:52 PM2/26/20
to weewx-user
Was habe ich nun kaputt gemacht?


pi@raspberrypi:~/Desktop $ tail -f /var/log/syslog
Feb 26 20:33:39 raspberrypi systemd[1]: Starting LSB: weewx weather system...
Feb 26 20:33:39 raspberrypi weewx[8560]: engine: Initializing weewx version 3.9.2
Feb 26 20:33:39 raspberrypi weewx[8560]: engine: Using Python 2.7.16 (default, Apr  6 2019, 01:42:57) #012[GCC 8.2.0]
Feb 26 20:33:39 raspberrypi weewx[8560]: engine: Platform Linux-4.19.75-v7+-armv7l-with-debian-10.1
Feb 26 20:33:39 raspberrypi weewx[8560]: engine: Locale is 'de_AT.UTF-8'
Feb 26 20:33:39 raspberrypi weewx[8560]: engine: pid file is /var/run/weewx.pid
Feb 26 20:33:39 raspberrypi weewx[8549]: Starting weewx weather system: weewx.
Feb 26 20:33:39 raspberrypi systemd[1]: Started LSB: weewx weather system.
Feb 26 20:33:39 raspberrypi weewx[8564]: engine: Error while parsing configuration file /home/weewx/weewx.conf
Feb 26 20:33:39 raspberrypi weewx[8564]: ****    Reason: 'Parsing failed with several errors.#012First error at line 9.'
Feb 26 20:33:42 raspberrypi dhcpcd[389]: wlan0: Router Advertisement from fe80::9ada:c4ff:feb9:eeda
Feb 26 20:33:42 raspberrypi dhcpcd[389]: wlan0: soliciting a DHCPv6 lease
Feb 26 20:33:42 raspberrypi dhcpcd[389]: wlan0: dropping DHCPv6 due to no valid routers
Feb 26 20:33:52 raspberrypi dhcpcd[389]: wlan0: Router Advertisement from fe80::9ada:c4ff:feb9:eeda
Feb 26 20:33:52 raspberrypi dhcpcd[389]: wlan0: soliciting a DHCPv6 lease
Feb 26 20:33:52 raspberrypi dhcpcd[389]: wlan0: dropping DHCPv6 due to no valid routers

gjr80

unread,
Feb 26, 2020, 4:00:52 PM2/26/20
to weewx-user
Vielleicht Zeile neun in weewx.conf?

Gary

Günther Wrana

unread,
Feb 27, 2020, 12:02:28 AM2/27/20
to weewx-user
Ich habe die weewx.conf aus meiner Datensicherung neu eingespielt nun kommt diese Meldung.
Soll ich das ganze neu installieren oder was ist nun defekt?

Feb 27 05:53:21 raspberrypi weewx[19161]: engine: pid file is /var/run/weewx.pid
Feb 27 05:53:21 raspberrypi weewx[19173]: engine: Using configuration file /home/weewx/weewx.conf
Feb 27 05:53:21 raspberrypi weewx[19173]: engine: Loading station type WH23xx (user.wh23xx)
Feb 27 05:53:21 raspberrypi weewx[19150]: Starting weewx weather system: weewx.
Feb 27 05:53:21 raspberrypi systemd[1]: Started LSB: weewx weather system.
Feb 27 05:53:21 raspberrypi weewx[19173]: engine: Caught unrecoverable exception in engine:
Feb 27 05:53:21 raspberrypi weewx[19173]:     ****  bad marshal data (unknown type code)
Feb 27 05:53:21 raspberrypi weewx[19173]:     ****  Traceback (most recent call last):
Feb 27 05:53:21 raspberrypi weewx[19173]:     ****    File "/home/weewx/bin/weewx/engine.py", line 888, in main
Feb 27 05:53:21 raspberrypi weewx[19173]:     ****    File "/home/weewx/bin/weewx/engine.py", line 72, in __init__
Feb 27 05:53:28 raspberrypi dhcpcd[389]: wlan0: Router Advertisement from fe80::9ada:c4ff:feb9:eeda
Feb 27 05:53:28 raspberrypi dhcpcd[389]: wlan0: soliciting a DHCPv6 lease
Feb 27 05:53:28 raspberrypi dhcpcd[389]: wlan0: dropping DHCPv6 due to no valid routers
Feb 27 05:53:32 raspberrypi dhcpcd[389]: wlan0: Router Advertisement from fe80::9ada:c4ff:feb9:eeda
Feb 27 05:53:32 raspberrypi dhcpcd[389]: wlan0: soliciting a DHCPv6 lease
Feb 27 05:53:32 raspberrypi dhcpcd[389]: wlan0: dropping DHCPv6 due to no valid routers
Feb 27 05:53:36 raspberrypi dhcpcd[389]: wlan0: Router Advertisement from fe80::9ada:c4ff:feb9:eeda
Feb 27 05:53:37 raspberrypi dhcpcd[389]: wlan0: soliciting a DHCPv6 lease
Feb 27 05:53:37 raspberrypi dhcpcd[389]: wlan0: dropping DHCPv6 due to no valid routers

Günther Wrana

unread,
Feb 28, 2020, 6:27:36 AM2/28/20
to weewx-user
It is done i have the database merged but wen i copy the database to /var/lib/weewx on my Ubuntu System come this messages in terminal. 

Feb 28 12:15:32 hp-pc weewx[13721]: engine: retrying...
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Using configuration file /etc/weewx/weewx.conf
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Loading station type WH23xx (user.wh23xx)
Feb 28 12:15:32 hp-pc weewx[13721]: wh23xx: driver version is 0.14
Feb 28 12:15:32 hp-pc weewx[13721]: wh23xx: usb info: pyusb_version=1.0.0
Feb 28 12:15:32 hp-pc weewx[13721]: wh23xx: poll interval is 15
Feb 28 12:15:32 hp-pc weewx[13721]: wh23xx: Found device on USB bus= device=
Feb 28 12:15:32 hp-pc kernel: [65563.123761] usb 2-1.7: reset full-speed USB device number 14 using ehci-pci
Feb 28 12:15:32 hp-pc weewx[13721]: engine: StdConvert target unit is 0x1
Feb 28 12:15:32 hp-pc weewx[13721]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Feb 28 12:15:32 hp-pc weewx[13721]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Archive will use data binding wx_binding
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Record generation will be attempted in 'software'
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Using archive interval of 300 seconds (software record generation)
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Using binding 'wx_binding' to database 'weewx.sdb'
Feb 28 12:15:32 hp-pc weewx[13721]: manager: Starting backfill of daily summaries
Feb 28 12:15:32 hp-pc weewx[13721]: restx: StationRegistry: Station will be registered.
Feb 28 12:15:32 hp-pc weewx[13721]: restx: Wunderground-PWS: Data for station IMATZENR2 will be posted
Feb 28 12:15:32 hp-pc weewx[13721]: restx: PWSweather: Posting not enabled.
Feb 28 12:15:32 hp-pc weewx[13721]: restx: CWOP: Posting not enabled.
Feb 28 12:15:32 hp-pc weewx[13721]: restx: WOW: Posting not enabled.
Feb 28 12:15:32 hp-pc weewx[13721]: restx: AWEKAS: Data will be uploaded for user Gilli
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Starting up weewx version 3.9.2
Feb 28 12:15:32 hp-pc weewx[13721]: engine: Starting main packet loop.
Feb 28 12:15:33 hp-pc weewx[13721]: wh23xx: get_current: failed attempt 1 of 5: [Errno 110] Operation timed out
Feb 28 12:15:44 hp-pc weewx[13721]: wh23xx: get_current: failed attempt 2 of 5: [Errno 110] Operation timed out
Feb 28 12:15:55 hp-pc weewx[13721]: wh23xx: get_current: failed attempt 3 of 5: [Errno 110] Operation timed out
Feb 28 12:16:06 hp-pc weewx[13721]: wh23xx: get_current: failed attempt 4 of 5: [Errno 110] Operation timed out
Feb 28 12:16:17 hp-pc weewx[13721]: wh23xx: get_current: failed attempt 5 of 5: [Errno 110] Operation timed out
Feb 28 12:16:27 hp-pc weewx[13721]: wh23xx: read failed: max retries (5) exceeded
Feb 28 12:16:27 hp-pc weewx[13721]: engine: Main loop exiting. Shutting engine down.
Feb 28 12:16:27 hp-pc weewx[13721]: engine: Caught WeeWxIOError: read failed: max retries (5) exceeded
Feb 28 12:16:27 hp-pc weewx[13721]:     ****  Waiting 60 seconds then retrying...

Where is the problem?

When i delete the database and weewx makes the database new then somtimes it works.

But somtimes the failed messages cames also.

Thanks Günther

Danke Günther

Günther Wrana

unread,
Mar 1, 2020, 8:46:40 AM3/1/20
to weewx-user
Hello and thanks i have succeeded my tow database to merged.

daham@hp-pc:/tmp$ sudo sqlite3 /media/daham/Neu1601/Eigene\ Dateien/Wetterstation/weewx/010320201357/weewx.sdb
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> SELECT strftime('%d-%m-%Y %H:%M:%S', datetime(MIN(dateTime), 'unixepoch', 'localtime')),strftime('%d-%m-%Y %H:%M:%S', datetime(MAX(dateTime), 'unixepoch', 'localtime')),COUNT(dateTime) FROM archive;
24-05-2019 21:10:00|01-03-2020 14:25:00|61866
sqlite> 

This is the begin data 24-05-2019 and the open end 01-03-2020.

Tanks for all the have me supports.

Günther 

Günther Wrana

unread,
Mar 12, 2020, 2:57:37 AM3/12/20
to weewx-user
Hello, since I've merged the databases, weewx has also worked. Only since then has the following been written in the log files.

Mar 12 07:50:27 raspberrypi weewx[29291]: manager: Added record 2020-03-12 07:50:00 CET (1583995800) to database 'weewx.sdb'
Mar 12 07:50:27 raspberrypi weewx[29291]: manager: Added record 2020-03-12 07:50:00 CET (1583995800) to daily summary in 'weewx.sdb'
Mar 12 07:50:29 raspberrypi weewx[29291]: restx: AWEKAS: Failed to publish record 2020-03-12 07:50:00 CET (1583995800): server returned ' - UV Wert administrativ gesperrt  - Transfer OK'
Mar 12 07:50:30 raspberrypi weewx[29291]: restx: Wunderground-PWS: Published record 2020-03-12 07:50:00 CET (1583995800)
Mar 12 07:50:31 raspberrypi weewx[29291]: cheetahgenerator: Generated 9 files for report SeasonsReport in 1.66 seconds
Mar 12 07:50:49 raspberrypi weewx[29291]: reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****  unsupported operand type(s) for -: 'list' and 'float'
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****  Traceback (most recent call last):
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 204, in run
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      obj.start()
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 300, in start
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      self.run()
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/imagegenerator.py", line 33, in run
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      self.genImages(self.gen_ts)
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/imagegenerator.py", line 188, in genImages
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      new_data_vec_t = self.converter.convert(data_vec_t)
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/units.py", line 784, in convert
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      new_val_t = convert(val_t, new_unit_type)
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/units.py", line 1138, in convert
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      new_val = conversion_func(val_t[0]) if val_t[0] is not None else None
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****    File "/home/weewx/bin/weewx/units.py", line 33, in FtoC
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****      return (x - 32.0) * 5.0 / 9.0
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****  TypeError: unsupported operand type(s) for -: 'list' and 'float'
Mar 12 07:50:49 raspberrypi weewx[29291]:         ****  Generator terminated
Mar 12 07:50:49 raspberrypi weewx[29291]: copygenerator: copied 17 files to /var/weewx/reports

Is that a problem or do I have to set something in the database?
I just noticed that some of the annual reports are not displayed.


Günther Wrana

unread,
Mar 20, 2020, 10:48:18 AM3/20/20
to weewx-user

Meteo Oberwallis

unread,
Apr 9, 2020, 5:14:57 AM4/9/20
to weewx-user
Hallo zusammen.

Ich wollte ebenfalls meine beiden .sdb Datenbanken zusammenlegen. Sie stammen beide aus weewx. Nun bekomme ich immer diese Fehlermeldung:

root@raspberrypi:~# sqlite3 weewx.sdb_new_new
SQLite version 3.16.2 2017-01-06 16:32:41

Enter ".help" for usage hints.
sqlite> .read weewx.sdb_new
Error: cannot open "weewx.sdb_new"
sqlite> .read /home/pi/Desktop/weewx.sdb_new
Error: near line 1: near "SQLite": syntax error
Error: near line 403: unrecognized token: " "
Error: near line 5261: near "@n�": syntax error
Error: near line 5337: unrecognized token: " "
Error: near line 6332: unrecognized token: " "
Error: near line 6793: unrecognized token: " "
Error: near line 7575: near "@rH": syntax error
Error: near line 7831: near "@n�": syntax error
sqlite>

Hatte das jemand auch?

Günther Wrana

unread,
Apr 10, 2020, 2:25:51 AM4/10/20
to weewx-user
Hallo

bei mir hat das dann funktioniert nach dem ich die alte Datenbank so wie oben beschrieben in eine .sql Datenbank umgewandelt habe.

sqlite3 db3.sdb
sqlite> .output db3.sql
sqlite> .dump archive
sqlite> .quit

und danach haben sich die beiden Datenbanken zusammmen legen lassen.

Fehlermeldungen sind auch bei mir gekommen nur wenn man am Schluss die Datenbank mit dem checkt.

sqlite3 db3.sdb
sqlite> pragma integrity_check;

und es O.K ausgibt dann passt das.

Grüße Günther 

Meteo Oberwallis

unread,
Apr 10, 2020, 9:25:40 AM4/10/20
to weewx-user
Hallo Günther.

Darf ich dich fragen, ob du vorher auch mal WsWin benutzt hast? Ich habe wetterdaten seit 2005, welche aber natürlich jetzt in weewx fehlen. Es müsste doch möglich sein, die von WsWin erstellten Monatsdateien, also .csv in die weewx Datenbank zu implementieren? Das wäre natürlich genial. anonsten verliere ich die ganzen jahre.

Gruss und danke für eure Hilfe

Günther Wrana

unread,
Apr 10, 2020, 3:03:23 PM4/10/20
to weewx-user
Hallo Meteo

ich habe das Problem auch meine Wetterdaten von 2014 bis zum Mai 2019 als .csv Dateien.
Mit dem weewx_import funktioniert das irgend wie. Ich Versuche das auch gerade aber angeblich soll das dann bei weewx 4.0 dann einfacher oder überhaupt funktionieren.

Ich bin auch erst seit voriges Jahr im Mai mit dem Programm so ein wenig vertraut.
Aber es sind hier alle sehr hilfsbereit bei Fragen.
Nur ist es besser wenn du in Englisch die Fragen stellst dann bekommst du schneller Antworten.

Ich mache das immer mit dem Google Übersetzer denn in Englisch bin ich nicht so versiert.

Aber lies dir die Dokumentation von weewx durch dort findest du oft Hilfen bei manchen Rätseln dieses Programmes.

Sonst funktioniert es sehr gut und ich bin zufrieden den nun kann ich meine Wetterdaten von allen meinen Geräten im Netzwerk zuhause und im Internet ansehen.

Grüße Günther  

gjr80

unread,
Apr 10, 2020, 8:54:36 PM4/10/20
to weewx-user
Hallo,

Ich entschuldige mich für das Google-Deutsche :)

Abhängig vom Format Ihrer WsWin-CSV-Datei sollte es möglich sein, Ihre CSV-Daten mit dem Dienstprogramm wee_import zu importieren. Für einen CSV-Import erfordert wee_import, dass die Datei eine Kopfzeile und dann einen Datensatz pro Zeile enthält. Es muss auch ein einzelnes Feld vorhanden sein, das Datum und Uhrzeit des Datensatzes enthält. Wenn Ihre Daten also separate Felder für Datum und Uhrzeit haben, müssen diese zusammengeführt werden. Außerdem kann wee_import jeweils nur aus einer einzelnen CSV-Datei importieren. Wenn Sie also mehrere CSV-Dateien haben, müssen Sie mehrere wee_import-Sitzungen ausführen oder die Dateien auf andere Weise zu einer zusammenführen.

Wenn Sie eine Ihrer WsWin-CSV-Dateien veröffentlichen möchten, schaue ich sie mir gerne an und prüfe, ob wee_import mit der Datei funktioniert und welche Einstellungen / Änderungen möglicherweise erforderlich sind.

In English:

My apologies for the Google-German :)

Depending on the format of your WsWin .csv file it should be possible to import your csv data using the wee_import utility. For a csv import wee_import requires the file contain a header line and then one record per line. There must also be a single field that contains the date and time of the record, so if your data has separate fields for date and time these would need to be merged. Also, wee_import can only import from a single csv file at a time, so if you have multiple csv files you would need to run multiple wee_import sessions or otherwise merge the files into one.

If you wish to post one of your WsWin csv files I am happy to look at it and see if wee_import will work with the file and what settings/changes may be required.

gruß,

Gary
Message has been deleted

Günther Wrana

unread,
Apr 11, 2020, 2:57:00 AM4/11/20
to weewx-user
Hello,

I would have my .csv file here that I would like to import. How should it look like? Do I have to insert the column names listed in the documentation in the header?

Time,Temp,Dewpoint,Press,WindDir,WindSpeed,WindGust,Hum,dailyrain,SolarRad

Date and time are in one column, do they also have to have this UNIX date format?

Thank you Günther
01052019.csv

gjr80

unread,
Apr 11, 2020, 7:12:08 PM4/11/20
to weewx-user
Hallo Günther,

A few observations on your data:

1. The column names can be whatever you want, the only requirements are they are unique and you can enter them in the [[FieldMap]] portion of the import config file. In theory, your German field names should work but all we need is some sort of unicode failure somewhere in the processing chain and the import will fail. If it were me I would remove the units in brackets and try the import with your fields names via a dry run (using --dry-run). That will show you if it will fail without affecting your WeeWX data.

2. You will need to remove the 'M' from the Intervall field.

3. WeeWX records rain on a per-interval basis whereas your rain data is cumulative. That is fine, wee_import can do the calculations but you need to tell wee_import your rain data is cumulative and you need to select one of the rain fields to use (you have five). I would use the longest period rain field, in your case Gesamter Niederschlag. So in your import config file you will need (untested):

rain = cumulative

and in your field map:

rain = 'Gesamter Niederschlag', mm

4. Your Zeit (date-time) field format is fine, no need for a unix epoch timestamp, wee_import will do that provided you come up with a suitable date-time format string that wee_import can use to recognise the different portions of the date-time. This is covered by the raw_datetime_format setting in the import config file, the wee_import section of the Utilities Guide explains it's use. in your case I suspect that '%d.%m.%Y  %H:%M:%S' will do the job (untested). Again the dry run should be used to check.

Gary

Meteo Oberwallis

unread,
Apr 12, 2020, 10:35:24 AM4/12/20
to weewx-user
Hello Gary.

First of all, thank you for exploring the cvs file. Sounds good that you can import and save them. Now I have to know something else. you talk about adapting the wee_import? How does that work? I also wanted to ask because I didn't quite understand it with rain. Weewx needs the cumulative amount of rain. Wouldn't it be easier if I simply deleted all other rain data from the table? And the last question. You wrote that (--dry-run) can be used to test whether everything is ok. How should I do that? Sorry for the many questions. But is not quite clear ;-)

Many thanks and best regards

William Burton

unread,
Apr 12, 2020, 2:45:34 PM4/12/20
to weewx-user
Hallo Günther,


On Sunday, April 12, 2020 at 10:35:24 AM UTC-4, Meteo Oberwallis wrote:
Hello Gary.

First of all, thank you for exploring the cvs file. Sounds good that you can import and save them. Now I have to know something else. you talk about adapting the wee_import? How does that work? I also wanted to ask because I didn't quite understand it with rain. Weewx needs the cumulative amount of rain. Wouldn't it be easier if I simply deleted all other rain data from the table? And the last question. You wrote that (--dry-run) can be used to test whether everything is ok. How should I do that? Sorry for the many questions. But is not quite clear ;-)

See the instructions at http://www.weewx.com/docs/utilities.htm#wee_import_utility for the wee_import utility.

From your WeeWX install location such as /home/weewx, copy the file utils/import/csv-example.conf and use as a template to make the changes.

Hope this helps,
-Bill

gjr80

unread,
Apr 12, 2020, 7:36:35 PM4/12/20
to weewx-user
Hallo Günther,

Now I have to know something else. you talk about adapting the wee_import? How does that work?

How wee_import processes/interprets your data is controlled by the import config file. As Bill points out there is a template CSV import config file provided with WeeWX, you will find it at /home/weewx/util/import/csv-example.conf or /etc/weewx/import/csv-example.conf depending on your WeeWX install type.

I also wanted to ask because I didn't quite understand it with rain. WeeWX needs the cumulative amount of rain. Wouldn't it be easier if I simply deleted all other rain data from the table?

You can certainly delete the other rain columns in your import data if you want but there is no need, wee_import will only import data in the columns included in the field map ([[FieldMap]] stanza in the import config file) and all other columns are ignored (there is likely a small processing overhead for having the extra unused columns included but I expect this overhead would be negligible).
 
And the last question. You wrote that (--dry-run) can be used to test whether everything is ok. How should I do that?

When you run wee_import you can add the --dry-run option to the command line to run the import but any saves to your archive are skipped, so in effect the import runs as normal (as configured in you import config file) but your database is left untouched. Have a read of Option --dry-run in the Utilities Guide.
 
Sorry for the many questions. But is not quite clear ;-)

No problems, most of what you want to know is covered in the wee_import section of the Utilities Guide. You are interested in a CSV import whereas the wee_import section of the Utilities Guide covers imports from other sources (Weatherunderground, Cumulus etc) as well. You can skip these sections paying attention only to the Actions and options section and the CSV specific sections here and here.
 
If language is a problem or you just don't follow, let me know and I will put together a suitable import config file for you.

Gary

Meteo Oberwallis

unread,
Apr 13, 2020, 7:28:58 AM4/13/20
to weewx-user
Hello.

I have now worked exactly according to the instructions. Unfortunately I always get the same error message when importing the data. I have the relevant data as an attachment here.

wee_import --import-config=/var/tmp/csv.conf --dry-run
Starting wee_import...
A CSV import from source file '/home/pi/Desktop/EXP09_17.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb_new'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
**** Unable to map source data.
**** Field 'timestamp' not found in source data.
**** Nothing done, exiting.

Can someone help me please?
Thx

EXP09_17.csv
csv.conf

Andrew Milner

unread,
Apr 13, 2020, 9:16:59 AM4/13/20
to weewx-user
Your csv file does not contain a field call timestamp, which your .conf file says should be imported to dateTime.  You need to combine Datum and Zeit into a field called timestamp, and then tell wee_import the format of that field.

Meteo Oberwallis

unread,
Apr 13, 2020, 9:20:07 AM4/13/20
to weewx-user
Hello Andrew.

Thank you for your message. What do you mean exactly? Sorry for the question, but this is completely new.
What exactly do I have to do and where do I have to adjust it?

Thank you for your help

rich T

unread,
Apr 13, 2020, 9:37:53 AM4/13/20
to weewx-user
Are you viewing your csv file in excel or openoffice?  If so, add an additional column between Zeit and Temp.Temperatur. In cell C2 add the following formula: =TEXT(A2,"m/dd/yy ")& " "&TEXT(B2,"hh:mm").
excel example.png

Meteo Oberwallis

unread,
Apr 13, 2020, 10:29:35 AM4/13/20
to weewx-user
Hello.
I've done that now. Unfortunately I still get the error message that the "timestamp" is missing. wee_import --import-config=/var/tmp/csv.conf --dry-run

Starting wee_import...
A CSV import from source file '/home/pi/Desktop/EXP09_17.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb_new'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
**** Unable to map source data.
**** Field 'timestamp' not found in source data.
**** Nothing done, exiting.
EXP01_17.csv
Message has been deleted

rich T

unread,
Apr 13, 2020, 11:17:36 AM4/13/20
to weewx-user
Looks like you put all the fields in one column.

Attached is an updated CSV File with timestamp column
Attached is an updated CVS Field map.  Need to verify units are correct.

Make sure to make backup copy of database before proceeding.

Hopefully this helps.


CSV FieldMap.txt
EXPO1_17(1).png
EXP09_17.csv

William Burton

unread,
Apr 13, 2020, 2:06:58 PM4/13/20
to weewx-user
Hello,

The wee_import CSV support appears to only support a comma (,) as a field delimiter or at least it cannot detect a semi-colon (;) field delimiter so you need to take the version of the file with the timestamp fix as suggested by rich T, and from OpenOffice/LibreOffice/Excel, re-save or Save As the standard CSV format so it uses a comma delimiter.

-Bill

Günther Wrana

unread,
Apr 14, 2020, 9:58:19 AM4/14/20
to weewx-user
Hello

I have now adjusted the csv.conf and the datad.csv so that no warnings are given when running dry.
I can't say which units are behind the "," in [[FieldMap]] and whether they are correct.

I only entered them on suspicion and hope that they are correct.
I also entered the inside temperature although they are not shown in these NOAA files but are inside the database.
Can I actually import all the data in my CSV file that is also in the database?

 [[FieldMap]]
        dateTime    = Zeit, unix_epoch
        usUnits     =
        interval    =
        barometer   = Absoluter Luftdruck, inHg
        pressure    =
        altimeter   =
        inTemp      = Innentemperatur, degree_C
        outTemp     = Aussentemperatur, degree_C
        inHumidity  = Innenluftfeuchtigkeit, percent
        outHumidity = Aussenluftfeuchtigkeit, percent
        windSpeed   = Windgeschwindigkeit, km_per_hour
        windDir     = Windrichtung, degree_compass
        windGust    = Windboeen, km_per_hour
        windGustDir = Windrichtung, degree_compass
        rainRate    = 
        rain        = 24-Stunden-Niederschlag, mm
        dewpoint    =
        windchill   =
        heatindex   =
        ET          =
        radiation   =
        UV          =

Thanks Günther
csv.conf
datad.csv

William Burton

unread,
Apr 14, 2020, 2:44:55 PM4/14/20
to weewx-user
Hallo Günther,

Gary and many others have much more experience with this than myself but since I saw a few issues, thought I would reply before it becomes too late for you.

It looks like you changed the CSV file to use the Meteo export file instead of the raw data file which is a good idea. I tried to find where it was documented but was not successful.

On Tuesday, April 14, 2020 at 9:58:19 AM UTC-4, Günther Wrana wrote:
Hello

I have now adjusted the csv.conf and the datad.csv so that no warnings are given when running dry.
I can't say which units are behind the "," in [[FieldMap]] and whether they are correct.

For the Zeit column, if you look closely at the data, there are two spaces between the date and time:
1,01.05.2019  00:02:00,5, 20.9,51, 12.2,74, 985.2, 988.3, 17.3, 20.9,315, 7.7, 7.1, 0.0, 17.1, 0.0, 369.6, 706.5, 0.0,0,0

so:
    raw_datetime_format = %d.%m.%Y %H:%M:%S
might best be changed to:
raw_datetime_format = %d.%m.%Y %H:%M:%S
The field map:
        barometer   = Absoluter Luftdruck, inHg
should be changed to use hectopascals:
barometer = Absoluter Luftdruck, hPa
change:
UV = to:
UV = UV(uW/m2)
For rain, the field specified is supposed to have a value that's reset each midnight when the "rain = cumulative" option is used. So I don't know if the
field "24-Stunden-Niederschlag" (24-hour-precipitation) will work or not.


I only entered them on suspicion and hope that they are correct.
I also entered the inside temperature although they are not shown in these NOAA files but are inside the database.
Can I actually import all the data in my CSV file that is also in the database?

You have to make sure you make a backup of the database first if you already have the station collecting data that you don't want to lose in case something goes wrong. Note that once wee_import loads the file, if something isn't correct and you want to try again, it requires reverting to a backup database or manually deleting the data just imported as wee_import doesn't have an option to overwrite existing data. If wee_import finds existing archive table records that match the ones in the import file, it will skip over them.

Once you import the data, you need to run wee_report and then carefully check in your WeeWX skin reports and graphs to ensure the data has been imported correctly. In the case of rain, make sure to find several days and verify them.

I hesitate to list the exact steps for the import process, etc. as I'm newer to WeeWX and don't want to lead you in the wrong direction.


 [[FieldMap]]
        dateTime    = Zeit, unix_epoch
        usUnits     =
        interval    =
        barometer   = Absoluter Luftdruck, inHg
        pressure    =
        altimeter   =
        inTemp      = Innentemperatur, degree_C
        outTemp     = Aussentemperatur, degree_C
        inHumidity  = Innenluftfeuchtigkeit, percent
        outHumidity = Aussenluftfeuchtigkeit, percent
        windSpeed   = Windgeschwindigkeit, km_per_hour
        windDir     = Windrichtung, degree_compass
        windGust    = Windboeen, km_per_hour
        windGustDir = Windrichtung, degree_compass
        rainRate    = 
        rain        = 24-Stunden-Niederschlag, mm
        dewpoint    =
        windchill   =
        heatindex   =
        ET          =
        radiation   =
        UV          =

Thanks Günther

Hope this helps,
-Bill

Meteo Oberwallis

unread,
Apr 15, 2020, 1:02:07 AM4/15/20
to weewx-user
Hello.

I was now able to retroactively import my data into weewx. It worked wonderfully. Now I have another question. 
After I have written the data into the database, I have to execute another command so that it is also on the homepage, so. e.g. in Belchertown under "Reports"? So far nothing is there, not even after midnight.

Thanks


Günther Wrana

unread,
Apr 15, 2020, 7:18:55 AM4/15/20
to weewx-user
Hallo

hast du die ganze 2MB csv auf einmal in die Datenbank gespielt?
Mit all den Daten und in der Konfiguration?

Grüße Günther

Meteo Oberwallis

unread,
Apr 15, 2020, 2:13:43 PM4/15/20
to weewx-user
Hallo.

Ich mache mittlerweile Jahres Daten und lade das so hoch. Dies klappt, dauert aber rund 3-4 Stunden pro Jahr. Die csv hat dann bei mir rund 24 MB. Leider muss man eine neue Spalte "timestamp" machen, da es sonst nicht klappt.

Gruss Stefan
Unbenannt.PNG

Günther Wrana

unread,
Apr 15, 2020, 2:42:23 PM4/15/20
to weewx-user
Hallo

Bei mir funktioniert das Importieren nur im Trockenlauf. Wenn ich die .csv Datei dann bei gestopptem weewx importiere und weewx wieder starte funktioniert nichts mehr. Es werden folgende Zeilen im log Feil angezeigt. Wenn ich dann die gesicherte Datenbank wieder einspiele und weewx neustarte funktioniert es wieder. Was mache ich falsch oder habe ich übersehen?

Danke Günther

Hello

For me, the import only works in dry running. If I import the .csv file with weewx stopped and weewx start again, nothing works. The following lines are displayed in the log file. If I then import the saved database again and restart weewx, it works again. What am I doing wrong or have I overlooked?

Thanks Günther



csv.conf
datad.csv
syslog

Günther Wrana

unread,
Apr 15, 2020, 2:49:44 PM4/15/20
to weewx-user
Hello

Maybe you can do more with that than with the log file.

Thanks Günther

daham@hp-pc:~$ ssh p...@192.168.1.104
p...@192.168.1.104's password: 
Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Apr 15 17:21:47 2020 from 192.168.1.103
pi@raspberrypi:~ $ cd /home/weewx/archive
pi@raspberrypi:/home/weewx/archive $ dir
weewx.sdb
pi@raspberrypi:/home/weewx/archive $ sudo /etc/init.d/weewx stop
[ ok ] Stopping weewx (via systemctl): weewx.service.
pi@raspberrypi:/home/weewx/archive $ sudo cp weewx.sdb /var/tmp
pi@raspberrypi:/home/weewx/archive $ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf --dry-run
Starting wee_import...
A CSV import from source file '/var/tmp/datad.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
288 records identified for import.
Records processed: 250; Unique records: 250; Last timestamp: 2019-05-01 20:47:00Records processed: 288; Unique records: 288; Last timestamp: 2019-05-01 23:57:00 CEST (1556747820)
Finished dry run import. 288 records were processed and 288 unique records would have been imported.
pi@raspberrypi:/home/weewx/archive $ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf
Starting wee_import...
A CSV import from source file '/var/tmp/datad.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
288 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Records processed: 288; Unique records: 288; Last timestamp: 2019-05-01 23:57:00 CEST (1556747820)
Finished import. 288 raw records resulted in 288 unique records being processed in 10.23 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
pi@raspberrypi:/home/weewx/archive $ sudo /etc/init.d/weewx start
[ ok ] Starting weewx (via systemctl): weewx.service.
pi@raspberrypi:/home/weewx/archive $ sudo tail -f /var/log/syslog
Apr 15 20:14:35 raspberrypi weewx[14619]: wh23xx: Found device on USB bus= device=
Apr 15 20:14:35 raspberrypi kernel: [10925.373718] usb 1-1.5: reset full-speed USB device number 4 using dwc_otg
Apr 15 20:14:35 raspberrypi weewx[14619]: engine: StdConvert target unit is 0x1
Apr 15 20:14:35 raspberrypi weewx[14619]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Apr 15 20:14:35 raspberrypi weewx[14619]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Apr 15 20:14:35 raspberrypi weewx[14619]: engine: Archive will use data binding wx_binding
Apr 15 20:14:35 raspberrypi weewx[14619]: engine: Record generation will be attempted in 'hardware'
Apr 15 20:14:35 raspberrypi weewx[14619]: engine: Using archive interval of 300 seconds (specified in weewx configuration)
Apr 15 20:14:35 raspberrypi weewx[14619]: engine: Using binding 'wx_binding' to database 'weewx.sdb'
Apr 15 20:14:35 raspberrypi weewx[14619]: manager: Starting backfill of daily summaries
Apr 15 20:15:57 raspberrypi weewx[14619]: engine: Caught unrecoverable exception in engine:
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****  accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****  Traceback (most recent call last):
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/engine.py", line 888, in main
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      engine = engine_class(config_dict)
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/engine.py", line 78, in __init__
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      self.loadServices(config_dict)
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/engine.py", line 142, in loadServices
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/engine.py", line 500, in __init__
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      self.setup_database(config_dict)
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/engine.py", line 617, in setup_database
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      _nrecs, _ndays = dbmanager.backfill_day_summary() # @UnusedVariable
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/manager.py", line 1473, in backfill_day_summary
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      day_accum.addRecord(rec, weight=weight)
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/accum.py", line 260, in addRecord
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      func(self, record, obs_type, add_hilo, weight)
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/accum.py", line 318, in add_value
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      self[obs_type].addHiLo(val, record['dateTime'])
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****    File "/home/weewx/bin/weewx/accum.py", line 78, in addHiLo
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****      "got type '%s' ('%s')" % (type(val), val))
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****  ValueError: accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:15:57 raspberrypi weewx[14619]:     ****  Exiting.
^C
pi@raspberrypi:/home/weewx/archive $ sudo /etc/init.d/weewx stop
[ ok ] Stopping weewx (via systemctl): weewx.service.
pi@raspberrypi:/home/weewx/archive $ cd /var/tmp
pi@raspberrypi:/var/tmp $ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf
Starting wee_import...
A CSV import from source file '/var/tmp/datad.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
288 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Records processed: 288; Unique records: 288; Last timestamp: 2019-05-01 23:57:00 CEST (1556747820)
Finished import. 288 raw records resulted in 288 unique records being processed in 4.05 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
pi@raspberrypi:/var/tmp $ sudo /etc/init.d/weewx start
[ ok ] Starting weewx (via systemctl): weewx.service.
pi@raspberrypi:/var/tmp $ sudo tail -f /var/log/syslog
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****    File "/home/weewx/bin/weewx/manager.py", line 1473, in backfill_day_summary
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****      day_accum.addRecord(rec, weight=weight)
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****    File "/home/weewx/bin/weewx/accum.py", line 260, in addRecord
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****      func(self, record, obs_type, add_hilo, weight)
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****    File "/home/weewx/bin/weewx/accum.py", line 318, in add_value
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****      self[obs_type].addHiLo(val, record['dateTime'])
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****    File "/home/weewx/bin/weewx/accum.py", line 78, in addHiLo
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****      "got type '%s' ('%s')" % (type(val), val))
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****  ValueError: accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:17:07 raspberrypi weewx[14938]:     ****  Exiting.
^C
pi@raspberrypi:/var/tmp $ sudo /etc/init.d/weewx reload
[ ok ] Reloading weewx configuration (via systemctl): weewx.service.
pi@raspberrypi:/var/tmp $ sudo /etc/init.d/weewx restart
[ ok ] Restarting weewx (via systemctl): weewx.service.
pi@raspberrypi:/var/tmp $ sudo tail -f /var/log/syslog
Apr 15 20:19:17 raspberrypi weewx[15205]: wh23xx: Found device on USB bus= device=
Apr 15 20:19:17 raspberrypi kernel: [11207.753810] usb 1-1.5: reset full-speed USB device number 4 using dwc_otg
Apr 15 20:19:17 raspberrypi weewx[15205]: engine: StdConvert target unit is 0x1
Apr 15 20:19:17 raspberrypi weewx[15205]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Apr 15 20:19:17 raspberrypi weewx[15205]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Apr 15 20:19:17 raspberrypi weewx[15205]: engine: Archive will use data binding wx_binding
Apr 15 20:19:17 raspberrypi weewx[15205]: engine: Record generation will be attempted in 'hardware'
Apr 15 20:19:17 raspberrypi weewx[15205]: engine: Using archive interval of 300 seconds (specified in weewx configuration)
Apr 15 20:19:17 raspberrypi weewx[15205]: engine: Using binding 'wx_binding' to database 'weewx.sdb'
Apr 15 20:19:17 raspberrypi weewx[15205]: manager: Starting backfill of daily summaries
Apr 15 20:19:22 raspberrypi weewx[15205]: engine: Caught unrecoverable exception in engine:
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  Traceback (most recent call last):
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/engine.py", line 888, in main
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      engine = engine_class(config_dict)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/engine.py", line 78, in __init__
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      self.loadServices(config_dict)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/engine.py", line 142, in loadServices
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/engine.py", line 500, in __init__
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      self.setup_database(config_dict)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/engine.py", line 617, in setup_database
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      _nrecs, _ndays = dbmanager.backfill_day_summary() # @UnusedVariable
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/manager.py", line 1473, in backfill_day_summary
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      day_accum.addRecord(rec, weight=weight)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 260, in addRecord
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      func(self, record, obs_type, add_hilo, weight)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 318, in add_value
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      self[obs_type].addHiLo(val, record['dateTime'])
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 78, in addHiLo
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      "got type '%s' ('%s')" % (type(val), val))
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  ValueError: accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  Exiting.
^C
pi@raspberrypi:/var/tmp $ sudo tail -f /var/log/syslog
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/manager.py", line 1473, in backfill_day_summary
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      day_accum.addRecord(rec, weight=weight)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 260, in addRecord
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      func(self, record, obs_type, add_hilo, weight)
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 318, in add_value
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      self[obs_type].addHiLo(val, record['dateTime'])
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****    File "/home/weewx/bin/weewx/accum.py", line 78, in addHiLo
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****      "got type '%s' ('%s')" % (type(val), val))
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  ValueError: accum: ScalarStats.addHiLo expected float or int, got type '<type 'unicode'>' ('')
Apr 15 20:19:22 raspberrypi weewx[15205]:     ****  Exiting.
^C
pi@raspberrypi:/var/tmp $ sudo /etc/init.d/weewx stop
[ ok ] Stopping weewx (via systemctl): weewx.service.
pi@raspberrypi:/var/tmp $ sudo cp /home/weewx/archive/weewx.sdb /tmp/
pi@raspberrypi:/var/tmp $ sudo cp weewx.sdb /home/weewx/archive
pi@raspberrypi:/var/tmp $ sudo /etc/init.d/weewx start
[ ok ] Starting weewx (via systemctl): weewx.service.
pi@raspberrypi:/var/tmp $ sudo tail -f /var/log/syslog
Apr 15 20:23:19 raspberrypi weewx[15651]: engine: Using binding 'wx_binding' to database 'weewx.sdb'
Apr 15 20:23:19 raspberrypi weewx[15651]: manager: Starting backfill of daily summaries
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: StationRegistry: Station will be registered.
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: Wunderground-PWS: Data for station IMATZENR2 will be posted
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: PWSweather: Posting not enabled.
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: CWOP: Posting not enabled.
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: WOW: Posting not enabled.
Apr 15 20:23:19 raspberrypi weewx[15651]: restx: AWEKAS: Data will be uploaded for user Gilli
Apr 15 20:23:19 raspberrypi weewx[15651]: engine: Starting up weewx version 3.9.2
Apr 15 20:23:19 raspberrypi weewx[15651]: engine: Starting main packet loop.
Apr 15 20:24:13 raspberrypi systemd[1]: Started Session c3 of user pi.
Apr 15 20:24:14 raspberrypi systemd[1]: Started Session c4 of user pi.
Apr 15 20:25:20 raspberrypi weewx[15651]: manager: Added record 2020-04-15 20:25:00 CEST (1586975100) to database 'weewx.sdb'
Apr 15 20:25:20 raspberrypi weewx[15651]: manager: Added record 2020-04-15 20:25:00 CEST (1586975100) to daily summary in 'weewx.sdb'
Apr 15 20:25:22 raspberrypi weewx[15651]: restx: AWEKAS: Failed to publish record 2020-04-15 20:25:00 CEST (1586975100): server returned ' - UV Wert administrativ gesperrt  - Transfer OK'
Apr 15 20:25:23 raspberrypi weewx[15651]: restx: StationRegistry: Published record 2020-04-15 20:25:00 CEST (1586975100)
Apr 15 20:25:24 raspberrypi weewx[15651]: restx: Wunderground-PWS: Published record 2020-04-15 20:25:00 CEST (1586975100)
Apr 15 20:25:28 raspberrypi weewx[15651]: cheetahgenerator: Generated 9 files for report SeasonsReport in 5.58 seconds
Apr 15 20:25:46 raspberrypi weewx[15651]: reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****  unsupported operand type(s) for -: 'list' and 'float'
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****  Traceback (most recent call last):
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 204, in run
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      obj.start()
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 300, in start
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      self.run()
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/imagegenerator.py", line 33, in run
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      self.genImages(self.gen_ts)
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/imagegenerator.py", line 188, in genImages
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      new_data_vec_t = self.converter.convert(data_vec_t)
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/units.py", line 784, in convert
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      new_val_t = convert(val_t, new_unit_type)
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/units.py", line 1138, in convert
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      new_val = conversion_func(val_t[0]) if val_t[0] is not None else None
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****    File "/home/weewx/bin/weewx/units.py", line 33, in FtoC
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****      return (x - 32.0) * 5.0 / 9.0
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****  TypeError: unsupported operand type(s) for -: 'list' and 'float'
Apr 15 20:25:46 raspberrypi weewx[15651]:         ****  Generator terminated
Apr 15 20:25:46 raspberrypi weewx[15651]: copygenerator: copied 34 files to /var/weewx/reports
^C
pi@raspberrypi:/var/tmp $ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf --dry-run
Starting wee_import...
A CSV import from source file '/var/tmp/datad.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
288 records identified for import.
Records processed: 288; Unique records: 288; Last timestamp: 2019-05-01 23:57:00 CEST (1556747820)
Finished dry run import. 288 records were processed and 288 unique records would have been imported.
pi@raspberrypi:/var/tmp $ 

gjr80

unread,
Apr 15, 2020, 8:19:35 PM4/15/20