Data Import and DOM DIR

[shvman]

Hi,

I'm importing about 11 years of data collected using a Vantage Pro2 station and WeatherLink software.  The WeatherLink files are on my computer, so my data are exported from the software as text files.  I bring the data into LibreOffice Calc, make changes and save the data as a CSV file.  When making changes, I change all wind and gust directional data from ordinal/cardinal directions to 'compass degree' numbers (moving around the compass by 22.5 degrees).  N is considered '0' degrees.  One such partial CSV file is attached.  

When I do this, the export "appears" to go well.  After a 'dry run' with no obvious complaints, I import and data and 'generate' reports using the wee_reports utility.  When I look at the data in my browser, the monthly climatological summary looks almost correct EXCEPT for the column named "DOM DIR".  The DOM DIR column is '90' for every day of the month.  

 I feel like I'm close to getting this done, but I must have something set wrong.  I'm also attaching my import conf file.

Any help would be greatly appreciated.  

Thanks, jjs

[Thomas Keffer]

It would be useful to see what actually ended up in the database. Use the tool sqlite3 to do this (you may have to install).

sqlite3 weewx.sdb

sqlite> select dateTime, datetime(dateTime,'unixepoch','localtime'), windSpeed, windDir from archive limit 20;

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/aad806e1-acc2-47f8-a705-cdd52a29fd61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[shvman]

Thanks Tom.

Yes, good point!!

Yup, the data are there....

1554078600|2019-03-31 19:30:00|0.0|

1554080400|2019-03-31 20:00:00|0.0|

1554082200|2019-03-31 20:30:00|0.0|

1554084000|2019-03-31 21:00:00|0.0|

1554085800|2019-03-31 21:30:00|0.0|

1554087600|2019-03-31 22:00:00|0.0|

1554089400|2019-03-31 22:30:00|0.0|

1554091200|2019-03-31 23:00:00|0.0|

1554093000|2019-03-31 23:30:00|0.0|

1554094800|2019-04-01 00:00:00|0.0|

1554096600|2019-04-01 00:30:00|0.0|

1554098400|2019-04-01 01:00:00|0.0|

1554100200|2019-04-01 01:30:00|0.0|

1554102000|2019-04-01 02:00:00|0.0|

1554103800|2019-04-01 02:30:00|0.0|

1554105600|2019-04-01 03:00:00|0.0|

1554107400|2019-04-01 03:30:00|0.0|

1554109200|2019-04-01 04:00:00|0.0|

1554111000|2019-04-01 04:30:00|0.0|

1554112800|2019-04-01 05:00:00|0.0|

1554114600|2019-04-01 05:30:00|0.0|

1554116400|2019-04-01 06:00:00|0.0|

1554118200|2019-04-01 06:30:00|0.0|

1554120000|2019-04-01 07:00:00|0.0|

1554121800|2019-04-01 07:30:00|0.0|

1554123600|2019-04-01 08:00:00|0.0|

1554125400|2019-04-01 08:30:00|0.0|

1554127200|2019-04-01 09:00:00|0.0|

1554129000|2019-04-01 09:30:00|0.0|

1554130800|2019-04-01 10:00:00|1.0|315.0

1554132600|2019-04-01 10:30:00|1.0|337.5

1554134400|2019-04-01 11:00:00|1.0|337.5

1554136200|2019-04-01 11:30:00|1.0|337.5

1554138000|2019-04-01 12:00:00|0.0|

1554139800|2019-04-01 12:30:00|0.0|

I also see windGust and gustDir in the DB too!  Everything looks like it's there. 

jjs

[vince]

On Friday, July 5, 2019 at 12:38:30 PM UTC-7, Thomas Keffer wrote:

It would be useful to see what actually ended up in the database. Use the tool sqlite3 to do this (you may have to install).

sqlite3 weewx.sdb

sqlite> select dateTime, datetime(dateTime,'unixepoch','localtime'), windSpeed, windDir from archive limit 20;

 

one thing I normally do is add a second 'dateTime' to the end of the select statement, so you can see nulls (which aren't always visually obvious), as dateTime is always going to be there

 select dateTime, datetime(dateTime,'unixepoch','localtime'), windSpeed, windDir, dateTime from archive limit 20;

[Thomas Keffer]

OK, the next step is to see if the daily summaries are OK.

select dateTime, datetime(dateTime,'unixepoch','localtime'), xsum, ysum from archive_day_wind limit 20;

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9ae1bc0d-6a3c-4352-a494-5e664a9efaaa%40googlegroups.com.

[shvman]

Maybe it's best to post the whole DB since there isn't much to it. jjs

[shvman]

Here this is:

1554008400|2019-03-31 00:00:00|0.0|0.0

1554094800|2019-04-01 00:00:00|-3339.28274090727|6261.74168169673

1554181200|2019-04-02 00:00:00|-527.207793864214|-1272.79220613579

1554267600|2019-04-03 00:00:00|99686.2672084393|-68132.4427018604

1554354000|2019-04-04 00:00:00|7128.37771018501|516.540171393921

1554440400|2019-04-05 00:00:00|46261.6189147235|-25126.8652699868

1554526800|2019-04-06 00:00:00|63230.7329108605|-51540.8971477026

1554613200|2019-04-07 00:00:00|35059.7143768767|8949.85877881586

1554699600|2019-04-08 00:00:00|-19556.6025679435|41813.8151534698

1554786000|2019-04-09 00:00:00|0.0|0.0

1554872400|2019-04-10 00:00:00|85470.8488865917|-88796.8152036324

1554958800|2019-04-11 00:00:00|7118.40400117596|-58941.4270597173

1555045200|2019-04-12 00:00:00|63155.7979022438|6392.98524749766

1555131600|2019-04-13 00:00:00|-8468.37353223031|17004.9874476823

1555218000|2019-04-14 00:00:00|-52810.2350303602|18778.8694645708

1555304400|2019-04-15 00:00:00|20036.7544231703|-58780.734810682

1555390800|2019-04-16 00:00:00|86931.5664756579|-69271.0851084881

1555477200|2019-04-17 00:00:00|152576.786056679|-111662.360885627

1555563600|2019-04-18 00:00:00|926.66056047536|7459.78712373512

1555650000|2019-04-19 00:00:00|-44647.2253454534|38802.3074638745

1555736400|2019-04-20 00:00:00|-4703.62667355496|4703.62667355496

1555822800|2019-04-21 00:00:00|61680.6367237927|-63058.297080307

1555909200|2019-04-22 00:00:00|55385.7950052227|-37873.6195985791

1555995600|2019-04-23 00:00:00|55410.876221022|-51365.9583394431

1556082000|2019-04-24 00:00:00|16478.8768232805|-22134.0782390054

1556168400|2019-04-25 00:00:00|-17156.417882491|12730.5708627542

1556254800|2019-04-26 00:00:00|-10227.5845234789|34943.5059009812

1556341200|2019-04-27 00:00:00|40220.9750697154|-75931.8550903803

1556427600|2019-04-28 00:00:00|84733.3567021994|-65101.99113641

1556514000|2019-04-29 00:00:00|123488.317569976|-79354.8215370818

1556600400|2019-04-30 00:00:00|151935.996387981|-141220.313605086

1556686800|2019-05-01 00:00:00|123544.376900609|-113802.847097978

1556773200|2019-05-02 00:00:00|42730.1379406706|-51696.5280158417

1556859600|2019-05-03 00:00:00|17316.012962064|-18522.7891199587

1556946000|2019-05-04 00:00:00|-1092.33755450833|5963.1024558241

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Thomas Keffer]

I don't see anything unusual. I was actually able to run wee_reports against it. This is what I got (ignore the station location... that's my station). Looks reasonable to me.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9894df53-d438-4f7b-a1dd-86401f900db5%40googlegroups.com.

[Thomas Keffer]

Forgot to include a NOAA report. Here's April's. Again, looks reasonable:

                   MONTHLY CLIMATOLOGICAL SUMMARY for Apr 2019


NAME: Hood River, Oregon                  
ELEV: 700 feet    LAT: 45-41.16 N    LONG: 121-33.96 W


                   TEMPERATURE (F), RAIN (in), WIND SPEED (mph)

                                         HEAT   COOL         AVG
      MEAN                               DEG    DEG          WIND                   DOM
DAY   TEMP   HIGH   TIME    LOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH   TIME    DIR
---------------------------------------------------------------------------------------
 01   52.9   63.9  13:05   42.8  05:43   12.1    0.0   0.00    0.0   13.0  17:19     84
 02   54.5   61.6  13:51   48.8  06:56   10.5    0.0   0.00    0.0    2.0  17:32     90
 03   52.0   58.1  16:42   47.0  06:52   13.0    0.0   0.00    1.6   18.0  15:28    282
 04   50.2   57.7  16:00   43.8  06:40   14.8    0.0   0.07    0.3    9.0  00:10      1
 05   48.0   52.3  16:34   42.0  23:58   17.0    0.0   0.15    0.9   20.0  13:18    273
 06   44.2   52.0  16:20   37.9  02:25   20.8    0.0   0.40    0.2   10.0  16:17    218
 07   45.8   52.7  12:53   42.5  06:00   19.2    0.0   0.91    0.9   17.0  11:40    224
 08   46.6   52.6  17:16   43.3  04:29   18.4    0.0   1.01    0.2   12.0  23:44    275
 09   48.0   53.1  16:36   44.7  23:59   17.0    0.0   0.09    3.8   27.0  16:07    279
 10   47.5   52.1  16:15   42.9  04:39   17.5    0.0   0.09    4.1   24.0  00:54    281
 11   48.7   53.5  15:47   45.2  08:14   16.3    0.0   0.22    2.0   18.0  02:57    281
 12   51.7   59.1  14:19   46.9  06:37   13.3    0.0   0.01    1.4   20.0  14:21    280
 13   46.8   51.4  11:00   43.9  23:00   18.2    0.0   0.16    4.6   26.0  11:05    284
 14   44.6   50.9  14:35   40.8  23:40   20.4    0.0   0.04    1.3   27.0  02:55    277
 15   44.7   56.0  16:00   32.3  06:10   20.3    0.0   0.02    0.1   11.0  17:10    287
 16   50.5   58.2  14:15   43.4  06:40   14.5    0.0   0.03    0.9   18.0  14:30    280
 17   53.7   61.9  16:35   48.2  04:55   11.3    0.0   0.01    1.5   20.0  15:55    280
 18   59.0   75.8  15:35   41.9  06:15    6.0    0.0   0.00    0.0   13.0  14:35    291
 19   59.7   64.2  00:05   54.9  06:15    5.3    0.0   0.00    0.4   14.0  07:30    275
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
---------------------------------------------------------------------------------------
      49.6   75.8     18   32.3     15  286.0    0.0   3.21    1.3   27.0     09    279
    

[gjr80]

Given the OP had a 'bad' import (separate thread) this sounds like all that is needed is a forced regeneration of the NOAA reports by deleting them all.

Gary

[Thomas Keffer]

Oh. That would have been useful to know.

Thanks, Gary. I'll bet that was the problem.

-tk

On Fri, Jul 5, 2019 at 4:42 PM gjr80 <gjrod...@gmail.com> wrote:

Given the OP had a 'bad' import (separate thread) this sounds like all that is needed is a forced regeneration of the NOAA reports by deleting them all.

Gary

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5d1b04f3-861b-4ebb-9f79-4e53c742f322%40googlegroups.com.

[shvman]

I sure do appreciate all the help!!

OK, well I'll force a regeneration of all NOAA Monthly reports and see what happens.  Gary suggested that might do it.

Many thanks to all.  jjs

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[shvman]

Hello,

Thanks for the suggestion.

I deleted all NOAA text files and regenerated with wee_reports and still get

90's all down the DOM DIR column for the imported month.  All other reports

look OK.  

jjs

[shvman]

Here's what I get:

MONTHLY CLIMATOLOGICAL SUMMARY for Apr 2019 NAME: Shreveport, LA ELEV: 180 feet LAT: 32-24.00 N LONG: 093-42.00 W TEMPERATURE (F), RAIN (in), WIND SPEED (mph) HEAT COOL AVG MEAN DEG DEG WIND DOM DAY TEMP HIGH TIME LOW TIME DAYS DAYS RAIN SPEED HIGH TIME DIR --------------------------------------------------------------------------------------- 01 43.4 53.7 05:30 33.7 23:30 21.6 0.0 0.00 0.1 12.0 10:00 90 02 54.1 71.3 11:30 32.7 02:00 10.9 0.0 0.00 0.0 11.0 09:30 90 03 64.7 76.7 11:00 42.2 01:30 0.3 0.0 0.01 1.4 18.0 11:00 90 04 63.5 74.0 11:30 58.6 00:00 1.5 0.0 0.09 0.3 11.0 12:30 90 05 64.4 76.5 11:00 56.6 03:00 0.6 0.0 0.00 0.6 12.0 14:30 90 06 67.1 72.8 07:30 60.8 01:00 0.0 2.1 0.37 1.0 22.0 20:00 90 07 67.8 79.3 09:00 63.5 18:00 0.0 2.8 0.89 0.6 21.0 11:30 90 08 67.4 79.7 10:30 55.7 00:00 0.0 2.4 0.01 0.5 14.0 10:30 90 09 71.5 89.3 10:30 53.9 02:00 0.0 6.5 0.00 0.0 0.0 00:30 90 10 76.4 90.9 10:00 56.4 00:30 0.0 11.4 0.00 1.5 18.0 12:30 90 11 67.0 81.2 08:30 52.8 00:00 0.0 2.0 0.10 1.0 18.0 06:30 90 12 63.7 76.0 11:30 51.6 02:00 1.3 0.0 0.02 0.9 15.0 00:00 90 13 58.1 62.1 12:00 48.9 00:00 6.9 0.0 0.95 1.2 18.0 18:00 90 14 53.6 67.9 12:00 43.7 00:00 11.4 0.0 0.00 0.7 17.0 00:30 90 15 66.6 81.0 10:30 43.4 00:30 0.0 1.6 0.00 0.8 15.0 18:00 90 16 70.6 83.0 11:30 53.4 02:00 0.0 5.6 0.00 1.3 18.0 12:30 90 17 70.7 76.3 12:00 63.9 01:30 0.0 5.7 0.13 2.2 18.0 09:00 90 18 58.7 66.2 00:30 50.5 21:30 6.3 0.0 1.27 0.6 20.0 13:00 90 19 59.3 67.2 10:30 46.4 00:00 5.7 0.0 0.01 0.7 20.0 12:00 90 20 61.7 75.7 12:00 44.9 01:30 3.3 0.0 0.00 0.1 9.0 08:00 90 21 70.0 84.3 09:30 49.5 02:00 0.0 5.0 0.00 1.0 16.0 09:00 90 22 71.1 83.0 10:30 56.5 01:30 0.0 6.1 0.00 0.8 12.0 05:00 90 23 70.6 81.5 11:30 61.8 00:00 0.0 5.6 0.00 0.9 13.0 08:30 90 24 70.7 77.7 09:30 63.2 00:30 0.0 5.7 0.61 0.4 9.0 06:00 90 25 67.7 76.7 10:30 60.9 22:00 0.0 2.7 0.00 0.3 12.0 09:00 90 26 68.8 82.3 10:00 55.3 22:00 0.0 3.8 0.00 0.4 14.0 04:00 90 27 73.4 85.1 11:00 56.1 00:30 0.0 8.4 0.00 1.1 17.0 06:30 90 28 74.3 86.2 10:30 64.5 02:00 0.0 9.3 0.00 1.3 14.0 09:30 90 29 75.5 82.3 13:00 66.0 00:30 0.0 10.5 0.00 1.8 11.0 12:30 90 30 79.9 87.3 10:30 72.8 00:30 0.0 14.9 0.00 3.1 20.0 12:00 90 --------------------------------------------------------------------------------------- 66.3 90.9 10 32.7 02 69.8 112.2 4.46 0.9 22.0 06 90 jjs

On Friday, July 5, 2019 at 5:51:49 PM UTC-5, Thomas Keffer wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[gjr80]

A few things to check:
- did you delete the NOAA files from the directory on your WeeWX machine where WeeWX originally saves them? If you delete then off the web server then that will have no effect, the files with the incorrect data just get uploaded again next report cycle.
- did you delete every single one, if not the it’s possible only the current month/year may have been regenerated?
- did you check the log to see that regeneration occurred? 11 years of data should have seen some 150 odd files regenerated for the NOAA reports alone, that will take some time and will be very obvious in the log. Suggest you post a log extract from the time you deleted the files covering the next one or two report cycles.

Gary

[shvman]

Yes, I deleted all of the NOAA text files in the /home/weewx/public_html/NOAA subdirectory.  The web server is running on the same machine.  I had a sym link in the /var/www/html directory to the weewx/public_html directory.  

Yes, I deleted ALL NOAA files....all months and the rain data.  

I only imported April 2019 as a test....when I get this working, I'll import the remaining data. 

Here's the interesting thing that I just discovered.....the CORRECT version of the April data are in the /home/weewx/public_html directory (see file attached and compare with my April data shown in the above post)......but it is NOT getting to the server index.html page.  For some reason, the DOM DIR column is NOT getting updated on the webpage.  I have refreshed the page multiple times.

Hope this helps....jjs

[shvman]

The problem appears to be with the browser......Vivaldi (

Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.102 Safari/537.36 Vivaldi/2.6.1566.44

)

does NOT show the DOM DIR column correctly But Chromium does.  I think that's the problem.  Hard to believe.   Could there be any other explanation for this?? 

BTW I'm running Vivaldi on a Raspberry Pi V3B.  

[gjr80]

If the issue is the browser then it has to be a caching issue.

Gary

[shvman]

Yes, that's right.  I flushed the cache and that month is now correct.  

Thanks for everyone's help.  Hope I didn't waste too much of your time.  

jjs