How to Include Modern XTide in Forecasting Extension?

[V. Kelly Bellis]

RE: Forecasting Extension, RPi, Raspbian, weewx v3.9.1, XTide v2.15.2
https://github.com/weewx/weewx/wiki/forecasting

The xtide package (sudo apt-get install xtide) is an old, neglected, and abandoned Debian package. How old? It and its related data: xtide v2.13.2-1+b1 and data circa 20100529. The net result is as you would expect, tide heights and times have drifted enough away from current NOAA predictions to be noticeable or worse.

Building XTide from source files on the RPi (including a path statement in /etc/environment, to the harmonics file; i.e., /usr/local/share/xtide/harmonics-dwf-20181227-free.tcd) has xtide working properly by itself; however, when viewing /var/log/syslog I'm seeing STATION_NOT_FOUND.

I was wondering if there might be any documentation written for non-programmers in how weewx and XTide have been integrated?

[mwall]

On Tuesday, March 12, 2019 at 5:04:16 PM UTC-4, V. Kelly Bellis wrote:

I was wondering if there might be any documentation written for non-programmers in how weewx and XTide have been integrated?

the forecast extension invokes xtide, parses the output, then stores the result in the forecast database.  the results can then be combined with other forecast data when generating reports.

by default, the forecast extension will invoke /usr/bin/tide

if your 'tide' program is located somewhere else, you can tell the extension to find it like this:

[Forecast]

    [[XTide]]

        prog = /path/to/tide

        location = Boston

as for the tide location, you must test that directly with tide.  once you have a string that works, use that string as the 'location' parameter for the forecast extension.  if the location has commas or spaces then put it in quotes.

btw, this spring i'm hoping to have some students put together a guide for how to create your own harmonics from your own data.  we have been using maxbotix ultrasonic sensors on raspberry pi running weewx to collect tide data from a couple of islands in maine.  apparently you can create your own harmonics with 6-9 months of data (the data we have so far tracks the xtide predictions for tenants harbor and camden pretty well, but there are offsets in magnitude and phase).  its been a bit of a struggle - power outages, snow on solar panels, critters chewing through cables, seawater/corrosion.  but we'll get there...

m 

[V. Kelly Bellis]

Matt - Thanks for the reply.

I'm guessing from your response that there is no documentation on xtide integration.

XTide is running perfectly on the RPi, but the syslog is still indicating that there is an issue with weewx: forecast: XTide

even after implementing your suggested path update and location in (presumably) /etc/weewx/weewx.conf

Details

RE: weewx_forecasting-extension_xtide

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Sequence of Events
20190221 Installed forecasting extension, sudo apt-get xtide, and sudo apt-get xtide-data
20190222 Realized deprecated deb package and data, sudo apt-get remove xtide, and sudo apt-get remove xtide-data
20190222 Built XTide v2.15.1 from source files
20190224 XTide v2.15.2 available
20190226 Built XTide v2.15.2 from source files

(time off to work on https://flaterco.com/files/xtide/XTide%20v2.15.2%20Help%20b20190309.chm)

20190312 Examined /var/log/syslog and discovered xtide STATION_NOT_FOUND errors
20190313 Examined /usr/bin/ and discovered tide v2.13.2 was not removed(!) - manually removed - and left wondering what other lingering bits could there be from the original installation on 20190221 of the deprecated stuff, this despite the overt sudo apt-get remove efforts.

9:13 AM   Wednesday, March 13, 2019  (VKB)
pi@raspberrypi:~ $ sudo rm /usr/bin/tide

pi@raspberrypi:~ $ sudo leafpad /etc/weewx/weewx.conf
@ line number 652 [[XTide]], inserted and edited the following lines
     prog = /usr/local/bin/tide

     location = Boston

pi@raspberrypi:~ $ sudo /etc/init.d/weewx stop
[ ok ] Stopping weewx (via systemctl): weewx.service.
pi@raspberrypi:~ $ sudo /etc/init.d/weewx start
[ ok ] Starting weewx (via systemctl): weewx.service.

################ And again from /var/log/syslog
Mar 13 10:30:16 raspberrypi weewx[2614]: manager: Added record 2019-03-13 10:30:00 EDT (1552487400) to database 'weewx.sdb'
Mar 13 10:30:16 raspberrypi weewx[2614]: manager: Added record 2019-03-13 10:30:00 EDT (1552487400) to daily summary in 'weewx.sdb'
Mar 13 10:30:16 raspberrypi weewx[2614]: forecast: XTideThread: XTide: generating tides from 2019-03-13 00:00:00 EDT (1552449600) to 2019-04-10 00:00:00 EDT (1554868800)
Mar 13 10:30:16 raspberrypi weewx[2614]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 13 10:30:17 raspberrypi weewx[2614]: forecast: XTideThread: XTide: got no tidal events
Mar 13 10:30:17 raspberrypi weewx[2614]: forecast: XTideThread: XTide: generate forecast failed: XTide Error:  STATION_NOT_FOUND The specified station was not found in any harmonics file.  Error details: Could not find: Boston

FY(and your students)I & FWIW - https://flaterco.com/xtide/harmonics.html

Kind regards,

Kelly

[mwall]

On Wednesday, March 13, 2019 at 11:03:43 AM UTC-4, V. Kelly Bellis wrote:

Mar 13 10:30:17 raspberrypi weewx[2614]: forecast: XTideThread: XTide: got no tidal events
Mar 13 10:30:17 raspberrypi weewx[2614]: forecast: XTideThread: XTide: generate forecast failed: XTide Error:  STATION_NOT_FOUND The specified station was not found in any harmonics file.  Error details: Could not find: Boston

what happens when you run the tide program that you built and installed?

you should be able to do this:

/usr/local/bin/tide

and get output like this:

Minimal usage:  tide -l "Location name" (or set the environment variable

  XTIDE_DEFAULT_LOCATION to "Location name")

Other switches:

  -b "YYYY-MM-DD HH:MM"

      Specify the begin (start) time for predictions.

  -e "YYYY-MM-DD HH:MM"

      Specify the end (stop) time for predictions.

  -f c|h|i|l|p|t

      Specify the output format as CSV, HTML, iCalendar, LaTeX, PNG, or text.

      The default is text.

  -m a|b|c|C|g|k|l|m|p|r|s

      Specify mode to be about, banner, calendar, alt. calendar, graph, clock,

      list, medium rare, plain, raw, or stats.  The default is plain.

  -o "filename"

      Redirect output to the specified file (appends).

  -s "HH:MM"

      Specify the step interval, in hours and minutes, for raw

      mode predictions.  The default is one hour.

  -v

      Print version string and exit.

These are only the most important switches.  For information on all of

the switches, please read the verbose documentation at:

  http://www.flaterco.com/xtide/

then try specifying a location:

tide -l Boston

or try ellsworth:

tide -l ellsworth

Indexing /usr/share/xtide/harmonics-initial.tcd...

Indexing /usr/share/xtide/harmonics-dwf-20100529-free.tcd...

Ellsworth, Union River, Blue Hill Bay, Maine

44.5350° N, 68.4217° W

2019-03-13  4:34 PM EDT   9.92 feet  High Tide

2019-03-13  6:36 PM EDT   Sunset

2019-03-13 10:40 PM EDT   1.31 feet  Low Tide

2019-03-14  1:53 AM EDT   Moonset

2019-03-14  4:54 AM EDT  10.83 feet  High Tide

2019-03-14  6:26 AM EDT   First Quarter

2019-03-14  6:48 AM EDT   Sunrise

2019-03-14 11:22 AM EDT   Moonrise

2019-03-14 11:22 AM EDT   0.70 feet  Low Tide

2019-03-14  5:34 PM EDT   9.74 feet  High Tide

2019-03-14  6:38 PM EDT   Sunset

2019-03-14 11:40 PM EDT   1.44 feet  Low Tide

2019-03-15  2:56 AM EDT   Moonset

2019-03-15  5:55 AM EDT  10.86 feet  High Tide

2019-03-15  6:46 AM EDT   Sunrise

2019-03-15 12:16 PM EDT   Moonrise

2019-03-15 12:27 PM EDT   0.63 feet  Low Tide

2019-03-15  6:39 PM EDT   Sunset

2019-03-15  6:40 PM EDT   9.76 feet  High Tide

2019-03-16 12:46 AM EDT   1.36 feet  Low Tide

2019-03-16  3:55 AM EDT   Moonset

2019-03-16  6:44 AM EDT   Sunrise

2019-03-16  7:02 AM EDT  11.06 feet  High Tide

2019-03-16  1:19 PM EDT   Moonrise

2019-03-16  1:35 PM EDT   0.34 feet  Low Tide

2019-03-16  6:40 PM EDT   Sunset

2019-03-16  7:48 PM EDT  10.07 feet  High Tide

2019-03-17  1:55 AM EDT   1.00 feet  Low Tide

2019-03-17  4:46 AM EDT   Moonset

2019-03-17  6:42 AM EDT   Sunrise

2019-03-17  8:09 AM EDT  11.49 feet  High Tide

if that does not work, then weewx won't be able to do anything either - you'll have to diagnose the tide program and your harmonics first.

m

 

[V. Kelly Bellis]

Hi Matt,

Thank you for your reply.

Let me know how you make out after you install XTide v2.15.2 and the most recent harmonics file.

In answer to your question, and as I previously said, v2.15.2 works perfectly.

[mwall]

On Wednesday, March 13, 2019 at 1:08:28 PM UTC-4, V. Kelly Bellis wrote:

Hi Matt,

Thank you for your reply.

Let me know how you make out after you install XTide v2.15.2 and the most recent harmonics file.

In answer to your question, and as I previously said, v2.15.2 works perfectly. 

you probably need to tell your tide program where to find the harmonics.  something like this should work:

HFILE_PATH=/path/to/harmonics.tcd tide -l Boston

apparently the environment in which you run weewx is not the same as the environment in which you are directly invoking tide, so you could be safe and wrap your tide program in a shell script, then use the shell script as the 'prog' for forecast.

for example, with weewx installed at /opt/weewx, put in the file /opt/weewx/bin/user/tide.sh something like this:

#!/bin/sh

HFILE_PATH=/path/to/harmonics.tcd tide "$@"

then use this in your forecast configuration:

  prog=/opt/weewx/bin/user/tide.sh

for example, here is a recent harmonics file with the bog-standard tide program from a debian install (xtide version 2.11):

/opt/weewx/bin/user/tide.sh -l "Boston, Boston Harbor"

Indexing /opt/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd...

Boston, Boston Harbor, Massachusetts

42.3539° N, 71.0503° W

2019-03-14  5:59 PM EDT   8.64 feet  High Tide

2019-03-14  6:49 PM EDT   Sunset

2019-03-15 12:05 AM EDT   1.19 feet  Low Tide

2019-03-15  3:00 AM EDT   Moonset

2019-03-15  6:17 AM EDT   9.85 feet  High Tide

2019-03-15  6:56 AM EDT   Sunrise

2019-03-15 12:34 PM EDT   Moonrise

2019-03-15 12:49 PM EDT   0.44 feet  Low Tide

2019-03-15  6:50 PM EDT   Sunset

2019-03-15  7:01 PM EDT   8.69 feet  High Tide

2019-03-16  1:07 AM EDT   1.08 feet  Low Tide

2019-03-16  3:58 AM EDT   Moonset

2019-03-16  6:54 AM EDT   Sunrise

2019-03-16  7:20 AM EDT  10.08 feet  High Tide

2019-03-16  1:37 PM EDT   Moonrise

2019-03-16  1:52 PM EDT   0.14 feet  Low Tide

2019-03-16  6:51 PM EDT   Sunset

2019-03-16  8:05 PM EDT   8.97 feet  High Tide

2019-03-17  2:10 AM EDT   0.73 feet  Low Tide

2019-03-17  4:51 AM EDT   Moonset

2019-03-17  6:53 AM EDT   Sunrise

2019-03-17  8:24 AM EDT  10.46 feet  High Tide

2019-03-17  2:47 PM EDT   Moonrise

2019-03-17  2:53 PM EDT  -0.31 feet  Low Tide

2019-03-17  6:52 PM EDT   Sunset

2019-03-17  9:07 PM EDT   9.46 feet  High Tide

2019-03-18  3:11 AM EDT   0.17 feet  Low Tide

2019-03-18  5:36 AM EDT   Moonset

2019-03-18  6:51 AM EDT   Sunrise

2019-03-18  9:26 AM EDT  10.94 feet  High Tide

[V. Kelly Bellis]

Hello Matt,

Thank you for your reply, though I don't fully understand it.

Have you tried to install XTide v2.15.2 and do you have it working with weewx and the forecast extension?

RE: The environment

/usr/local/bin/tide already knows where to find harmonics as directed by

/etc/environment wherein is contained:

HFILE_PATH=/usr/local/share/xtide
WVS_DIR=/home/pi/XTide/wvs

XTide was installed from source files and into the associated directories following the David Flater's excellent, though sometimes terse, instructions.  Weewx got installed automatically through the apt-get process.

/usr/share/weewx/weewxd

In /etc/weewx/weewx.conf under

[Forecast]

 ...

   [[XTide]]

      prog = /usr/local/bin/tide

IF I understand you,

1) I should create /usr/share/weewx/tide.sh wherein is contained

#!/bin/sh
HFILE_PATH=/usr/local/share/harmonics-dwf-20181227-free.tcd tide "$@"

2) and then, you're saying I should

In /etc/weewx/weewx.conf under

[Forecast]

 ...

   [[XTide]]

      prog =  /usr/share/weewx/tide.sh

 

Is this understanding correct?

FWIW - Found more deprecated droppings from the old deb-package that weren't properly removed by sudo apt-get remove xtide-data; i.e., /usr/share/xtide/harmonics-initial.tcd - manually removed by

pi@raspberrypi:~ $ sudo rm -r /usr/share/xtide

[V. Kelly Bellis]

Hello Matt,

Hope you are well. It's been a few days and I haven't heard back from you yet.

Have you tried to install XTide v2.15.2 and do you have it working with weewx and the forecast extension?

And did I understand you correctly? - see previous post.

Kind regards,

Kelly

[mwall]

On Tuesday, March 19, 2019 at 9:32:20 AM UTC-4, V. Kelly Bellis wrote:

Have you tried to install XTide v2.15.2 and do you have it working with weewx and the forecast extension?

yes.  but there is no need to install the latest xtide - older versions work just fine with the latest harmonics.

 

And did I understand you correctly? - see previous post.

yes

 

 

[V. Kelly Bellis]

Hello Matt,

Thank you for your reply.

I'm still not understanding why in /etc/weewx/weewx.conf under

[Forecast]

 ...

   [[XTide]]

      prog = /usr/local/bin/tide

What's wrong with this?

And why the need for the tide.sh?

[Forecast]

 ...

   [[XTide]]

      prog =  /usr/share/weewx/tide.sh

RE: Updating XTide

The need to update xtide would be predicated on which old version of xtide you are using with the latest harmonics, and you're particular need to have accurate tide information. In your example listed from the forecast extension page, http://sailing.mit.edu/weather/forecast.html, I was unable to reach the site (it keeps timing out) to see it working.

https://flaterco.com/xtide/changelog.html might be worth consulting in making the decision to update xtide as there have been several changes made since 2.13 (deprecated apt-get package linked on your forecast extension page); and of course, comparing your xtide results with https://tidesandcurrents.noaa.gov/stationhome.html is advised.

You might want to consider adding a note on your forecast extension page that harmonics data outside of the U.S. has been discontinued and hasn't been maintained since early 2012.

[mwall]

On Tuesday, March 19, 2019 at 11:52:49 AM UTC-4, V. Kelly Bellis wrote:

I'm still not understanding why in /etc/weewx/weewx.conf under

[Forecast]

 ...

   [[XTide]]

      prog = /usr/local/bin/tide

What's wrong with this?

And why the need for the tide.sh?

apparently the environment in which you run weewx is not the same as the environment in which you run tide directly, so when the forecast extension invokes /usr/local/bin/tide, tide cannot find the harmonics.  the tide.sh script ensures that you use the right harmonics by explicitly telling tide which harmonics file to use.  you do not need tide.sh if you fix your environment issues.

 

[mwall]

On Tuesday, March 19, 2019 at 11:52:49 AM UTC-4, V. Kelly Bellis wrote:

The need to update xtide would be predicated on which old version of xtide you are using with the latest harmonics, and you're particular need to have accurate tide information. In your example listed from the forecast extension page, http://sailing.mit.edu/weather/forecast.html, I was unable to reach the site (it keeps timing out) to see it working.

sorry, mit is mucking about with the network right now (they just sold off a huge block of 18. addresses, and there are still some kinks in the transition to new address space).

 

https://flaterco.com/xtide/changelog.html might be worth consulting in making the decision to update xtide as there have been several changes made since 2.13 (deprecated apt-get package linked on your forecast extension page); and of course, comparing your xtide results with https://tidesandcurrents.noaa.gov/stationhome.html is advised.

yup.  none of the changes have any impact on the output from tide.  you could probably go all the way back to the early 2.x releases circa 1999 and still get the same results.  weewx-forecast uses only tide, not xtide or any of the other stuff include in the xtide packaging.

only the harmonics matter, and how much those matter depends on your location.

 

You might want to consider adding a note on your forecast extension page that harmonics data outside of the U.S. has been discontinued and hasn't been maintained since early 2012.

done

 

[V. Kelly Bellis]

apparently the environment in which you run weewx is not the same as the environment in which you run tide directly, so when the forecast extension invokes /usr/local/bin/tide, tide cannot find the harmonics.  the tide.sh script ensures that you use the right harmonics by explicitly telling tide which harmonics file to use.  you do not need tide.sh if you fix your environment issues.

What does this syslog message suggest:

Mar 20 08:25:31 raspberrypi weewx[16792]: forecast: XTideThread: XTide: got no tidal events

'got no tidal events' is still present after every attempt to remediate forecast not showing tide predictions; e.g.,

[mwall]

this message indicates that the forecast extension is not getting any data from the tide program.

what happens when you use the tide.sh approach?  what output do you get when you invoke tide.sh directly?

 

[V. Kelly Bellis]

pi@raspberrypi:~ $ /usr/local/share/xtide/harmonics-dwf-20181227-free.tcd tide -l Ellsworth
bash: /usr/local/share/xtide/harmonics-dwf-20181227-free.tcd: Permission denied

[V. Kelly Bellis]

The .sh has no luck, but regular tide -l location works.

[mwall]

you might want to purge the debian installation of tide

sudo apt-get purge xtide

otherwise there might be config files lingering around that confound your new xtide installation.  ('apt-get remove' removes the software, but leaves configurations and data behind.  'apt-get purge' is more aggressive, and removes the configurations too.)

or you could do

sudo-apt-get install xtide

then put your new harmonics file somewhere, say

  /opt/tide/harmonics-dwf-20181227-free.tcd

then make the shell script at /opt/tide/tide.sh:

#!/bin/sh

HFILE_PATH=/opt/tide/harmonics-dwf-20181227-free.tcd tide "$@"

then set the permissions:

sudo chmod 755 /opt/tide/tide.sh

then use it in your forecast configuration:

    prog = /opt/tide/tide.sh

that way you can have both - the old xtide and the new xtide, so you can do direct comparisons from the command line.

m

[mwall]

On Wednesday, March 20, 2019 at 9:41:44 AM UTC-4, V. Kelly Bellis wrote:

pi@raspberrypi:~ $ /usr/local/share/xtide/harmonics-dwf-20181227-free.tcd tide -l Ellsworth
bash: /usr/local/share/xtide/harmonics-dwf-20181227-free.tcd: Permission denied

you just tried to execute a harmonics file.  harmonics files are data, not executables.

you want this:

HFILE_PATH=/usr/local/share/xtide/harmonics-dwf-20181227-free.tcd tide -l Ellsworth

or, if you created the tide.sh script, simply:

/path/to/tide.sh -l Ellsworth

but for less grey hair (oops!  too late for that one for many of us!) use the second option in my previous posting.

m

[V. Kelly Bellis]

Will sudo apt-get purge xtide purge only the old deprecated xtide stuff or will frig with all things xtide system-wide?

[mwall]

On Wednesday, March 20, 2019 at 9:52:17 AM UTC-4, V. Kelly Bellis wrote:

Will sudo apt-get purge xtide purge only the old deprecated xtide stuff or will frig with all things xtide system-wide?

purging xtide will probably not touch your installation at /usr/local

packages installed by apt usually go into standard system locations - /etc, /var/lib

packages installed by source typically go in /usr/local or in /opt

so as i mentioned previously, you should be able to have both on your system at the same time (it works for me).  you just have to be explicitly about which harmonics file you want.  you'll have to read the xtide code/docs to find out the configuration precedence for tide.

m

 

[V. Kelly Bellis]

Hello Matt,

Thank you for your reply. My responses are shown below.

On Wednesday, March 20, 2019 at 9:59:07 AM UTC-4, mwall wrote:

On Wednesday, March 20, 2019 at 9:52:17 AM UTC-4, V. Kelly Bellis wrote:

Will sudo apt-get purge xtide purge only the old deprecated xtide stuff or will frig with all things xtide system-wide?

purging xtide will probably not touch your installation at /usr/local

Yes! That's exactly true in my case, thankfully.

Unfortunately simply purging the old deprecated-Debian packaged xtide didn't remedy forecast extension's apparent inability to find tide.

On Wednesday, March 20, 2019 at 9:59:07 AM UTC-4, mwall wrote:

so as i mentioned previously, you should be able to have both on your system at the same time (it works for me).  you just have to be explicitly about which harmonics file you want.  you'll have to read the xtide code/docs to find out the configuration precedence for tide.

Why in the world would I ever want to have anything but a current version of xtide? And moreover, old harmonics data, i.e., circa 20100529 - why would I want that added liability? A better question would be: why isn't forecast able to integrate a modern version of tide? From your remarks, it sounds like you don't have v2.15.2 working yet with forecast and that you've retained v2.13 (or earlier) explicitly for tide's integration into forecast. Or have I misread something?

I'm not a programmer, but it seems that forecast.py (and maybe other related bits of python) is predicated on the placement of xtide in a particular environment instilled from a Debian-style package installation without any provision for building xtide from source files. All of this is still pretty alien to me, but in so far as I'm able, I'm interested in helping find a solution to this problem as I love that local tide predictions can be presented alongside local weather forecasts.

In answer to an earlier question that you asked (and yes, all remaining hairs are gray!), going the tty and tide.sh HFILE_PATH= approach both worked fine:

Mar 21 07:30:25 raspberrypi weewx[566]: manager: Added record 2019-03-21 07:30:00 EDT (1553167800) to database 'weewx.sdb'
Mar 21 07:30:25 raspberrypi weewx[566]: manager: Added record 2019-03-21 07:30:00 EDT (1553167800) to daily summary in 'weewx.sdb'
Mar 21 07:30:26 raspberrypi weewx[566]: forecast: XTideThread: XTide: generating tides from 2019-03-21 00:00:00 EDT (1553140800) to 2019-04-18 00:00:00 EDT (1555560000)
Mar 21 07:30:26 raspberrypi weewx[566]: engine: Garbage collected 6168 objects
Mar 21 07:30:26 raspberrypi weewx[566]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 21 07:30:26 raspberrypi weewx[566]: forecast: XTideThread: XTide: got no tidal events
Mar 21 07:30:28 raspberrypi weewx[566]: cheetahgenerator: Generated 8 files for report SeasonsReport in 1.88 seconds
Mar 21 07:30:29 raspberrypi weewx[566]: imagegenerator: Generated 14 images for SeasonsReport in 1.02 seconds
Mar 21 07:30:29 raspberrypi weewx[566]: copygenerator: copied 0 files to /var/www/html/weewx
Mar 21 07:30:30 raspberrypi weewx[566]: cheetahgenerator: Generated 12 files for report forecast in 1.02 seconds
Mar 21 07:30:30 raspberrypi weewx[566]: copygenerator: copied 0 files to /var/www/html/weewx/forecast
Mar 21 07:35:01 raspberrypi CRON[13115]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 21 07:35:29 raspberrypi weewx[566]: manager: Added record 2019-03-21 07:35:00 EDT (1553168100) to database 'weewx.sdb'
Mar 21 07:35:29 raspberrypi weewx[566]: manager: Added record 2019-03-21 07:35:00 EDT (1553168100) to daily summary in 'weewx.sdb'
Mar 21 07:35:29 raspberrypi weewx[566]: forecast: XTideThread: XTide: generating tides from 2019-03-21 00:00:00 EDT (1553140800) to 2019-04-18 00:00:00 EDT (1555560000)
Mar 21 07:35:29 raspberrypi weewx[566]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 21 07:35:29 raspberrypi weewx[566]: forecast: XTideThread: XTide: got no tidal events
Mar 21 07:35:31 raspberrypi weewx[566]: cheetahgenerator: Generated 8 files for report SeasonsReport in 1.88 seconds
Mar 21 07:35:32 raspberrypi weewx[566]: imagegenerator: Generated 14 images for SeasonsReport in 1.03 seconds
Mar 21 07:35:32 raspberrypi weewx[566]: copygenerator: copied 0 files to /var/www/html/weewx
Mar 21 07:35:33 raspberrypi weewx[566]: cheetahgenerator: Generated 12 files for report forecast in 1.01 seconds
Mar 21 07:35:33 raspberrypi weewx[566]: copygenerator: copied 0 files to /var/www/html/weewx/forecast

Thank you very much for all of your help.

Kind regards,

Kelly

 

[mwall]

On Thursday, March 21, 2019 at 7:39:23 AM UTC-4, V. Kelly Bellis wrote:

Why in the world would I ever want to have anything but a current version of xtide? And moreover, old harmonics data, i.e., circa 20100529 - why would I want that added liability? A better question would be: why isn't forecast able to integrate a modern version of tide?

ah, but the forecast extension DOES work with ANY version of xtide.

why would you want to install the latest xtide, when the one from 2010 produces exactly the same results?

only the harmonics need to be updated, and for some locations those don't even matter that much

when someone asks "does forecast work with the latest version of xtide?" it is rather handy to have 2 (or 5 or 10) different versions of xtide or its harmonics installed at once so that i can answer accurately.

 

From your remarks, it sounds like you don't have v2.15.2 working yet with forecast and that you've retained v2.13 (or earlier) explicitly for tide's integration into forecast. Or have I misread something?

apparently you misread

 

I'm not a programmer, but it seems that forecast.py (and maybe other related bits of python) is predicated on the placement of xtide in a particular environment instilled from a Debian-style package installation without any provision for building xtide from source files. All of this is still pretty alien to me, but in so far as I'm able, I'm interested in helping find a solution to this problem as I love that local tide predictions can be presented alongside local weather forecasts. 

the default installation assumes that tide is located at /usr/bin/tide

this can be changed with the `prog` option.

as long as the thing referred to by `prog` produces output in the format produced by the 'tide' program (which is unchanged since 1999 or so), then everything works.

so maybe some examples will help:

here is output from tide version 2.11 for boston harbor using two different harmonics files.  the first harmonics file is from 2010, the second one is from 2018.  the differences in tide magnitude and timing are minimal.  if you do the same test for boston light, there is more variation, but still pretty small.

tide -l'Boston, Boston Harbor' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

Indexing /usr/share/xtide/harmonics-initial.tcd...

Indexing /usr/share/xtide/harmonics-dwf-20100529-free.tcd...

Boston, Boston Harbor, Massachusetts

42.3550° N, 71.0533° W

2019-03-21  5:58 AM EDT  -1.53 feet  Low Tide

2019-03-21  6:46 AM EDT   Sunrise

2019-03-21  7:26 AM EDT   Moonset

2019-03-21 12:13 PM EDT  11.82 feet  High Tide

2019-03-21  6:26 PM EDT  -1.72 feet  Low Tide

2019-03-21  6:57 PM EDT   Sunset

2019-03-21  7:50 PM EDT   Moonrise

HFILE_PATH=/opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd tide -l'Boston, Boston Harbor' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

Indexing /opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd...

Boston, Boston Harbor, Massachusetts

42.3539° N, 71.0503° W

2019-03-21  5:58 AM EDT  -1.52 feet  Low Tide

2019-03-21  6:46 AM EDT   Sunrise

2019-03-21  7:25 AM EDT   Moonset

2019-03-21 12:13 PM EDT  11.83 feet  High Tide

2019-03-21  6:26 PM EDT  -1.71 feet  Low Tide

2019-03-21  6:57 PM EDT   Sunset

2019-03-21  7:50 PM EDT   Moonrise

now lets look at ellsworth maine:

tide -l'Ellsworth' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

Indexing /usr/share/xtide/harmonics-initial.tcd...

Indexing /usr/share/xtide/harmonics-dwf-20100529-free.tcd...

Ellsworth, Union River, Blue Hill Bay, Maine

44.5350° N, 68.4217° W

2019-03-21  5:49 AM EDT  -1.50 feet  Low Tide

2019-03-21  6:35 AM EDT   Sunrise

2019-03-21  7:15 AM EDT   Moonset

2019-03-21 11:59 AM EDT  13.23 feet  High Tide

2019-03-21  6:15 PM EDT  -1.72 feet  Low Tide

2019-03-21  6:47 PM EDT   Sunset

2019-03-21  7:39 PM EDT   Moonrise

HFILE_PATH=/opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd tide -l'Ellsworth' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

Indexing /opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd...

Ellsworth, Union River, Blue Hill Bay, Maine

44.5350° N, 68.4217° W

2019-03-21  5:48 AM EDT  -1.29 feet  Low Tide

2019-03-21  6:35 AM EDT   Sunrise

2019-03-21  7:15 AM EDT   Moonset

2019-03-21 12:00 PM EDT  13.01 feet  High Tide

2019-03-21  6:16 PM EDT  -1.63 feet  Low Tide

2019-03-21  6:47 PM EDT   Sunset

2019-03-21  7:39 PM EDT   Moonrise

interesting!  there is more variation here, probably because of the geography - the ellsworth tidal area is considerably different from boston harbor.  but the differences are still pretty small (unless you're on a boat and you really need that additional 1.-2.5" of draft or that 1 minute time shift).

of course, if you run your own tide station you could figure out whether the xtide harmonics are actually accurate for your location - they might be inter-station approximations.

so it might not be worth sweating the 1.-2.5" difference unless/until you get some data that show the 1.-2.5" is accurate.

[mwall]

On Thursday, March 21, 2019 at 7:39:23 AM UTC-4, V. Kelly Bellis wrote:

Mar 21 07:30:26 raspberrypi weewx[566]: forecast: XTideThread: XTide: generating tides from 2019-03-21 00:00:00 EDT (1553140800) to 2019-04-18 00:00:00 EDT (1555560000)
Mar 21 07:30:26 raspberrypi weewx[566]: engine: Garbage collected 6168 objects
Mar 21 07:30:26 raspberrypi weewx[566]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 21 07:30:26 raspberrypi weewx[566]: forecast: XTideThread: XTide: got no tidal events


so you are still not getting output from the tide program that the weewx-forecast extension can understand.

please set debug=1 then restart weewx.  that will tell us exactly what command weewx is running when it tries to generate the tide data.

m

[mwall]

and just for completeness, here are the two boston harbor cases using tide 2.15.2.  as you can see, the output is identical to that of tide 2.11

/opt/xtide-2.15.2/bin/tide -l'Boston, Boston Harbor' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

Indexing /usr/share/xtide/harmonics-initial.tcd...

Indexing /usr/share/xtide/harmonics-dwf-20100529-free.tcd...

Boston, Boston Harbor, Massachusetts

42.3550° N, 71.0533° W

2019-03-21  5:58 AM EDT  -1.53 feet  Low Tide

2019-03-21  6:46 AM EDT   Sunrise

2019-03-21  7:26 AM EDT   Moonset

2019-03-21 12:13 PM EDT  11.82 feet  High Tide

2019-03-21  6:26 PM EDT  -1.72 feet  Low Tide

2019-03-21  6:57 PM EDT   Sunset

2019-03-21  7:50 PM EDT   Moonrise

HFILE_PATH=/opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd /opt/xtide-2.15.2/bin/tide -l'Boston, Boston Harbor' -b'2019-03-21 00:00' -e'2019-03-22 00:00'

[V. Kelly Bellis]

Unsure where and how many possible places in the weewx realm there are where debugging might be available, I took a guess that that /etc/weewx/weewx.conf was what you meant:

Just out of curiosity, was this where you meant? and are there any other places where to check for verbose bug reports related to weewx+forecast+tide?

Mar 22 07:40:21 raspberrypi weewx[16352]: manager: Added record 2019-03-22 07:40:00 EDT (1553254800) to database 'weewx.sdb'
Mar 22 07:40:21 raspberrypi weewx[16352]: manager: Added record 2019-03-22 07:40:00 EDT (1553254800) to daily summary in 'weewx.sdb'
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: MainThread: Zambretti: starting thread
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: MainThread: NWS: not yet time to do the forecast
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: MainThread: OWM: not yet time to do the forecast
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: MainThread: XTide: starting thread
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: ZambrettiThread: Zambretti: forecast was already calculated for 2019-03-21 09:00:00 EDT (1553173200)
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: ZambrettiThread: Zambretti: terminating thread
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: generating tides from 2019-03-22 00:00:00 EDT (1553227200) to 2019-04-19 00:00:00 EDT (1555646400)
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: running command '/usr/share/weewx/tide.sh -fc -df'%Y.%m.%d' -tf'%H:%M' -l'Ellsworth' -b'2019-03-22 00:00' -e'2019-04-19 00:00''
Mar 22 07:40:21 raspberrypi weewx[16352]: restx: StationRegistry: wait interval (71400 < 604800) has not passed for record 2019-03-22 07:40:00 EDT (1553254800)
Mar 22 07:40:21 raspberrypi weewx[16352]: reportengine: Running reports for latest time in the database.
Mar 22 07:40:21 raspberrypi weewx[16352]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 22 07:40:21 raspberrypi weewx[16352]: reportengine: Running report 'SeasonsReport'
Mar 22 07:40:21 raspberrypi weewx[16352]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: got no tidal events
Mar 22 07:40:21 raspberrypi weewx[16352]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.forecast.ForecastVariables']
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: terminating thread
Mar 22 07:40:21 raspberrypi weewx[16352]: manager: Daily summary version is 2.0
Mar 22 07:40:23 raspberrypi weewx[16352]: cheetahgenerator: Generated 8 files for report SeasonsReport in 2.12 seconds
Mar 22 07:40:23 raspberrypi weewx[16352]: manager: Daily summary version is 2.0
Mar 22 07:40:24 raspberrypi weewx[16352]: imagegenerator: Generated 14 images for SeasonsReport in 1.09 seconds
Mar 22 07:40:24 raspberrypi weewx[16352]: copygenerator: copied 0 files to /var/www/html/weewx
Mar 22 07:40:24 raspberrypi weewx[16352]: reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Mar 22 07:40:24 raspberrypi weewx[16352]: reportengine: Report 'MobileReport' not enabled. Skipping.
Mar 22 07:40:24 raspberrypi weewx[16352]: reportengine: Report 'StandardReport' not enabled. Skipping.
Mar 22 07:40:24 raspberrypi weewx[16352]: reportengine: Running report 'forecast'
Mar 22 07:40:24 raspberrypi weewx[16352]: reportengine: Found configuration file /etc/weewx/skins/forecast/skin.conf for report 'forecast'
Mar 22 07:40:24 raspberrypi weewx[16352]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.forecast.ForecastVariables']
Mar 22 07:40:25 raspberrypi weewx[16352]: manager: Daily summary version is 2.0
Mar 22 07:40:26 raspberrypi weewx[16352]: cheetahgenerator: Generated 12 files for report forecast in 1.22 seconds
Mar 22 07:40:26 raspberrypi weewx[16352]: copygenerator: copied 0 files to /var/www/html/weewx/forecast
Mar 22 07:40:26 raspberrypi weewx[16352]: reportengine: Report 'FTP' not enabled. Skipping.
Mar 22 07:40:26 raspberrypi weewx[16352]: reportengine: Report 'RSYNC' not enabled. Skipping.

[V. Kelly Bellis]

If the forecast interval has been set to n days, should the tide interval; i.e., -b -e, do likewise?

What does restx accomplish and is it a forecast term, linux, xtide, etc., and why is it set to be (71400 < 604800) - whatever that means??

[V. Kelly Bellis]

So far I haven't gotten forecast to run the latest version of tide, but I understand that you as a developer might want to support tide users of legacy versions, and I find that refreshing - thank you for helping them! - and for helping me in trying to get tide to work in forecast.

[V. Kelly Bellis]

There's a great deal more to xtide than what is used in forecast. In 2014 all so-called common files where moved into a separate library and namespace: libxtide, and a number of new settings were introduced relating to making tide calendars. Additionally, the developer, Dave Flater, has actively and routinely updated his program and continues its development - see the change log for detail.  Besides being able to produce widely ranging formatted tide calendars, there are tide clocks, tide graphs, and statistical extractions that no other program I've found handles as adeptly.

In addition to all of that which tide and xtide can accomplish, xttpd is another executable component for interactive website and application developers. I know nothing about how to use xttpd, but do see it's reflections all over the place, be they in various tide-related Android apps or websites. Flater's XTide and his steadfast development of it over the past 25 years is to be commended. Unfortunately, Debian has not exercised good governance over its ecosystem and the dpkg maintainer appears to have abdicated responsibility during the Bush administration.

On Thursday, March 21, 2019 at 9:56:56 AM UTC-4, mwall wrote:

[V. Kelly Bellis]

Does the weewx.conf call of:

prog = /usr/share/weewx/tide.sh

take precedence over the call at line 3906 in forecast.py?

On Thursday, March 21, 2019 at 9:56:56 AM UTC-4, mwall wrote:

[mwall]

On Friday, March 22, 2019 at 7:55:12 AM UTC-4, V. Kelly Bellis wrote:

Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: generating tides from 2019-03-22 00:00:00 EDT (1553227200) to 2019-04-19 00:00:00 EDT (1555646400)
Mar 22 07:40:21 raspberrypi weewx[16352]: forecast: XTideThread: XTide: running command '/usr/share/weewx/tide.sh -fc -df'%Y.%m.%d' -tf'%H:%M' -l'Ellsworth' -b'2019-03-22 00:00' -e'2019-04-19 00:00''

and what happens when you execute the command directly?  try it:

/usr/share/weewx/tide.sh -fc -df'%Y.%m.%d' -tf'%H:%M' -l'Ellsworth' -b'2019-03-22 00:00' -e'2019-04-19 00:00'

 

On Friday, March 22, 2019 at 9:00:34 AM UTC-4, V. Kelly Bellis wrote:

There's a great deal more to xtide than what is used in forecast.

yup.  all good stuff, and wonderful that Dave Flater can still find the time to keep things updated.  

but if you don't need that other stuff, tide still works the same as it did 20 years ago, and the results are still reliable.

yay standards!  yay for good design!  yay for not letting marketing steamroll your APIs and naming and versioning!  yay for respecting your users!  yay open source!

yay Dave Flater!

 

[Kelly Bellis]

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/CEs-qlhj69U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[V. Kelly Bellis]

It works and produces 4-weeks worth of tide tables  - why 4 weeks and not the same number of forecast days?

[Andrew Milner]

in the command you issued you requested tide information for the 4 weeks from 22 march thru 19 april - so that is why 4 weeks of data was returned.

[V. Kelly Bellis]

Exactly! And was why I questioned that the period isn't matching the forecast number of days; e.g., 7 days;

-b'2019-03-22 00:00' -e'2019-03-30 00:00'

[mwall]

On Friday, March 22, 2019 at 11:36:09 AM UTC-4, V. Kelly Bellis wrote:

It works and produces 4-weeks worth of tide tables 

yay!  i see exactly the same output when i run tide with those options, whether i use tide 2.11 or tide 2.15.2

but your weewx-forecast still gets nothing.  on the other hand, my weewx-forecast *does* see the output and happily parses it.

so the obvious difference is still not so obvious.

please try the attached forecast code to see if that will shed any more light.  it has some additional generate/parse debug output, as well as python3-readiness (not that that will help you, but since i'm in the code, we may as well take care of everything and test tom's python3 changes.  it has been running fine for me.)

after you download forecast-3.4.0rc1.py to the Downloads folder on the pi, the process looks something like this:

sudo mv /usr/share/weewx/user/forecast.py /usr/share/weewx/user/forecast-3.2.2.py

sudo cp ~/Downloads/forecast-3.4.0rc1.py /usr/share/weewx/user/forecast.py

sudo /etc/init.d/weewx stop

sudo /etc/init.d/weewx start

m

[V. Kelly Bellis]

Thank you Matt for the help, but still no cigar, unfortunately.

Mar 22 13:40:22 raspberrypi weewx[3297]: manager: Added record 2019-03-22 13:40:00 EDT (1553276400) to database 'weewx.sdb'
Mar 22 13:40:22 raspberrypi weewx[3297]: manager: Added record 2019-03-22 13:40:00 EDT (1553276400) to daily summary in 'weewx.sdb'
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: MainThread: Zambretti: starting thread
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: MainThread: NWS: not yet time to do the forecast
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: MainThread: OWM: not yet time to do the forecast
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: MainThread: XTide: starting thread
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: generating forecast for 2019-03-22 09:00:00 EDT (1553259600)
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: XTideThread: XTide: generating tides from 2019-03-22 00:00:00 EDT (1553227200) to 2019-04-19 00:00:00 EDT (1555646400)
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: using winddir from 2019-03-22 08:30:00 EDT (1553257800) to 2019-03-22 09:00:00 EDT (1553259600)
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: XTideThread: XTide: running command '/usr/share/weewx/tide.sh -fc -df'%Y.%m.%d' -tf'%H:%M' -l'Ellsworth' -b'2019-03-22 00:00' -e'2019-04-19 00:00''
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: using pressure from 2019-03-22 06:00:00 EDT (1553248800) to 2019-03-22 09:00:00 EDT (1553259600)
Mar 22 13:40:22 raspberrypi weewx[3297]: atmocomudp: MainThread: Listening for UDP broadcasts to IP address <broadcast> on port 12000, with timeout 90 and share_socket True...
Mar 22 13:40:22 raspberrypi weewx[3297]: reportengine: Running reports for latest time in the database.
Mar 22 13:40:22 raspberrypi weewx[3297]: reportengine: Running report 'SeasonsReport'
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: XTideThread: XTide: got no tidal events
Mar 22 13:40:22 raspberrypi weewx[3297]: manager: Daily summary version is 2.0
Mar 22 13:40:22 raspberrypi weewx[3297]: forecast: XTideThread: XTide: terminating thread
Mar 22 13:40:22 raspberrypi weewx[3297]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Mar 22 13:40:23 raspberrypi weewx[3297]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.forecast.ForecastVariables']
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: units=1 winddir=22.8282497722 pressure=29.669207315 first_p=29.7504201375 last_p=29.547317715
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: pressure=1004.71452333 month=2 winddir=1 trend=-2.29261212944 north=True
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: code is O
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: generated 1 forecast record
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: saving 1 forecast records
Mar 22 13:40:23 raspberrypi weewx[3297]: manager: Added record 2019-03-22 13:40:00 EDT (1553276400) to database 'forecast.sdb'
Mar 22 13:40:23 raspberrypi weewx[3297]: manager: Daily summary version is 2.0
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: saved 1 forecast records
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: deleting forecasts prior to 1552671623
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: deleted forecasts prior to 1552671623
Mar 22 13:40:23 raspberrypi weewx[3297]: forecast: ZambrettiThread: Zambretti: terminating thread
Mar 22 13:40:28 raspberrypi weewx[3297]: cheetahgenerator: Generated 8 files for report SeasonsReport in 5.59 seconds
Mar 22 13:40:28 raspberrypi weewx[3297]: manager: Daily summary version is 2.0
Mar 22 13:40:29 raspberrypi weewx[3297]: imagegenerator: Generated 14 images for SeasonsReport in 1.05 seconds
Mar 22 13:40:29 raspberrypi weewx[3297]: copygenerator: copied 87 files to /var/www/html/weewx
Mar 22 13:40:29 raspberrypi weewx[3297]: reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Mar 22 13:40:29 raspberrypi weewx[3297]: reportengine: Report 'MobileReport' not enabled. Skipping.
Mar 22 13:40:29 raspberrypi weewx[3297]: reportengine: Report 'StandardReport' not enabled. Skipping.
Mar 22 13:40:29 raspberrypi weewx[3297]: reportengine: Running report 'forecast'
Mar 22 13:40:29 raspberrypi weewx[3297]: reportengine: Found configuration file /etc/weewx/skins/forecast/skin.conf for report 'forecast'
Mar 22 13:40:29 raspberrypi weewx[3297]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.forecast.ForecastVariables']
Mar 22 13:40:29 raspberrypi weewx[3297]: manager: Daily summary version is 2.0
Mar 22 13:40:33 raspberrypi weewx[3297]: cheetahgenerator: Generated 12 files for report forecast in 3.61 seconds
Mar 22 13:40:33 raspberrypi weewx[3297]: copygenerator: copied 87 files to /var/www/html/weewx/forecast
Mar 22 13:40:33 raspberrypi weewx[3297]: reportengine: Report 'FTP' not enabled. Skipping.
Mar 22 13:40:33 raspberrypi weewx[3297]: reportengine: Report 'RSYNC' not enabled. Skipping.

[V. Kelly Bellis]

Matt - How's it going?

One thought that keeps coming up, are any of your tests being done on a system where the only xtide installation is v2.15.2?

[mwall]

On Monday, March 25, 2019 at 11:22:28 AM UTC-4, V. Kelly Bellis wrote:

One thought that keeps coming up, are any of your tests being done on a system where the only xtide installation is v2.15.2?

i have been testing on a raspberry pi and an x64 box, both running some version of debian.  both have old and new xtide on them.

i'll try removing the debian xtide from the pi so see if that gives me results like yours (but it won't be for a few days - boats going in the water now, so lots of bottom painting and logistics to take care of)

 

[V. Kelly Bellis]

Wow - the ice is totally gone out of the Charles? Bet the water is still pretty chilly... though I suppose it wouldn't matter all that much on the right size boat :)

Whenever you're able to get to it will be just fine; no rush. Maybe me and my blind man's cane will wander into the reeds.

[V. Kelly Bellis]

Hi Matt,

Within the entirety of the forecast extension, is forecast.py the only file that contains the path to tide? Are there compiled versions of forecast files that may still hold vestiges to tide?

Thank you for any reply.

Kind regards,

Kelly

On Friday, March 22, 2019 at 1:08:39 PM UTC-4, mwall wrote:

[V. Kelly Bellis]

/usr/share/weewx/user/forecast.py

@ line 25 "A single table stores all forecast information."

Not sure if I've found the correct database @

/var/lib/weewx/forecast.sdb

If this is the correct file, there aren't any tide forecasts; only NWS, OWM, and Zambretti

Is this why the syslog includes:

"Mar 31 16:00:28 raspberrypi weewx[20798]: forecast: XTideThread: XTide: got no tidal events"

?

[mwall]

please try the attached forecast 3.4.0rc2.py.  i did an install of tide 2.15.2 on a raspberry pi, with no other tide installation, and it works with forecast 3.4.0rc2.py

this is how i compiled libtcd:

./configure

make

sudo make install

this is how i compiled tide:

./configure

make

sudo make install

then i installed the harmonics file:

sudo mkdir /opt/harmonics

cd /opt/harmonics

tar xvf ~/Downloads/harmonics-dwf-20181227-free.tar.bz2

this is my tide wrapper at /opt/weewx/bin/user/tide.sh (there are only 2 lines in the file):

#!/bin/sh

HFILE_PATH=/opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/tide "$@"

i generated tide output for testing like this:

/opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt

then i verified the tide parsing like this:

PYTHONPATH=bin python bin/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

and that resulted in parsed records:

[{'event_ts': 1554502500, 'dateTime': 1554493866, 'location': None, 'hilo': 'L', 'offset': '0.67', 'issued_ts': 1554493866, 'method': 'XTide', 'usUnits': 1}, ...]

finally, when i run weewx with a configuration that contains tide forecasting, this is what it looks like in the weewx log (with debug enabled):

Apr  5 16:35:52 7up weewx[16424]: engine: Loading service user.forecast.XTideForecast

Apr  5 16:35:52 7up weewx[16424]: forecast: MainThread: XTide: forecast version 3.4.0rc2

Apr  5 16:35:52 7up weewx[16424]: manager: Created and initialized table 'archive' in database 'forecast.sdb'

Apr  5 16:35:52 7up weewx[16424]: forecast: MainThread: XTide: interval=1209600 max_age=2419200 location='Ellsworth' duration=2419200

Apr  5 16:35:52 7up weewx[16424]: engine: Finished loading service user.forecast.XTideForecast

...

Apr  5 16:35:52 7up weewx[16424]: engine: Starting main packet loop.

Apr  5 16:40:14 7up weewx[16424]: forecast: MainThread: XTide: starting thread

Apr  5 16:40:14 7up weewx[16424]: forecast: XTideThread: XTide: generating tides from 2019-04-05 00:00:00 EDT (1554436800) to 2019-05-03 00:00:00 EDT (1556856000)

Apr  5 16:40:14 7up weewx[16424]: forecast: XTideThread: XTide: running command '/opt/weewx/bin/user/tide.sh -fc -df'%Y.%m.%d' -tf'%H:%M' -l'Ellsworth' -b'2019-04-05 00:00' -e'2019-05-03 00:00''

...

Apr  5 16:40:14 7up weewx[16424]: manager: Added record 2019-04-05 16:40:00 EDT (1554496800) to database 'weewx.sdb'

Apr  5 16:40:15 7up weewx[16424]: manager: Added record 2019-04-05 16:40:00 EDT (1554496800) to daily summary in 'weewx.sdb'

Apr  5 16:40:15 7up weewx[16424]: reportengine: Running reports for latest time in the database.

Apr  5 16:40:15 7up weewx[16424]: reportengine: Running report 'SeasonsReport'

Apr  5 16:40:15 7up weewx[16424]: reportengine: Found configuration file /opt/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'

Apr  5 16:40:15 7up weewx[16424]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.forecast.ForecastVariables']

Apr  5 16:40:15 7up weewx[16424]: forecast: XTideThread: XTide: got 222 lines of output

Apr  5 16:40:15 7up weewx[16424]: forecast: XTideThread: XTide: location mismatch: 'Ellsworth' != 'Ellsworth, Union River, Blue Hill Bay, Maine'

Apr  5 16:40:15 7up weewx[16424]: forecast: XTideThread: XTide: tide matrix: [{'event_ts': 1554458400, 'dateTime': 1554496815, 'location': 'Ellsworth', 'hilo': 'L', 'offset': '0.64', 'issued_ts': 1554496815, 'method': 'XTide', 'usUnits': 1}, ...]

Apr  5 16:40:15 7up weewx[16424]: forecast: XTideThread: XTide: saving 108 forecast records

Apr  5 16:40:15 7up weewx[16424]: manager: Added record 2019-04-05 16:40:15 EDT (1554496815) to database 'forecast.sdb'

...

Apr  5 16:40:16 7up weewx[16424]: manager: Added record 2019-04-05 16:40:15 EDT (1554496815) to database 'forecast.sdb'

Apr  5 16:40:16 7up weewx[16424]: forecast: XTideThread: XTide: saved 108 forecast records

Apr  5 16:40:16 7up weewx[16424]: forecast: XTideThread: XTide: deleting forecasts prior to 1552077616

Apr  5 16:40:16 7up weewx[16424]: forecast: XTideThread: XTide: deleted forecasts prior to 1552077616

Apr  5 16:40:16 7up weewx[16424]: forecast: XTideThread: XTide: terminating thread

the attachment shows what it looks like in the Seasons report.

m

[V. Kelly Bellis]

This is wonderful Matt, thank you so very much!! I will study this and try implementing this weekend.

FWIW, I followed the procedure as outlined in the section entitled "Installation Guide For New Users of XTide" that I wrote when I created XTide v2.15.2 Help (b20190309). Although it was written with Ubuntu in play, installation on the RPi was nearly identical.

Kind regards,

Kelly

[V. Kelly Bellis]

Notes on my way to nowhere

11:41 AM   Saturday, April 6, 2019  (VKB)

Matt says:
then i installed the harmonics file:

sudo mkdir /opt/harmonics
cd /opt/harmonics
tar xvf ~/Downloads/harmonics-dwf-20181227-free.tar.bz2

=========================
1)
Just checking, should permissions be addressed?, i.e.:
sudo mkdir /opt/harmonics
sudo chmod 755 /opt/harmonics

or are they okay by default?

2)

cd /opt/harmonics
tar xvf ~/Downloads/harmonics-dwf-20181227-free.tar.bz2

This results in failure.
pi@raspberrypi:~ $ sudo mkdir /opt/harmonics
pi@raspberrypi:~ $ cd /opt/harmonics
pi@raspberrypi:/opt/harmonics $ tar xvf ~/Downloads/harmonics-dwf-20181227-free.tar.bz2
harmonics-dwf-20181227/
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd: Cannot open: No such file or directory
harmonics-dwf-20181227/README
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/README: Cannot create symlink to ‘COPYING’: No such file or directory
harmonics-dwf-20181227/Disclaimers
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/Disclaimers: Cannot create symlink to ‘COPYING’: No such file or directory
harmonics-dwf-20181227/ChangeLog
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/ChangeLog: Cannot open: No such file or directory
harmonics-dwf-20181227/COPYING
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/COPYING: Cannot open: No such file or directory
harmonics-dwf-20181227/AUTHORS
tar: harmonics-dwf-20181227: Cannot mkdir: Permission denied
tar: harmonics-dwf-20181227/AUTHORS: Cannot create symlink to ‘COPYING’: No such file or directory
tar: Exiting with failure status due to previous errors

3)
pi@raspberrypi:/opt/harmonics $ sudo tar xvf ~/Downloads/harmonics-dwf-20181227-free.tar.bz2

sudo is required because we're making a sub-directory during extraction; i.e.,
/opt/harmonics/harmonics-dwf-20181227

4)
Just checking, should permissions be addressed?, i.e.:
pi@raspberrypi:$ sudo chmod 644 /opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd

or are they okay by default?

5)
Why bury/ nest the harmonics file below /opt/harmonics/
Why not have /opt/harmonics/harmonics-dwf-20181227-free.tcd

Matt says:
this is my tide wrapper at /opt/weewx/bin/user/tide.sh (there are only 2 lines in the file):

#!/bin/sh
HFILE_PATH=/opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/tide "$@"

=========================

6)
No such folders exist.

pi@raspberrypi:/ $ sudo mkdir /opt/weewx
pi@raspberrypi:/ $ sudo mkdir /opt/weewx/bin
pi@raspberrypi:/ $ sudo mkdir /opt/weewx/bin/user

Created tide.sh, then
sudo cp /home/pi/weewx/extensions/forecast/tide.sh /opt/weewx/bin/user/tide.sh

Here again, just checking, should permissions be addressed?

Matt says:

i generated tide output for testing like this:

/opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt

============================

7)
Permission denied.

pi@raspberrypi:/opt/weewx/bin/user $ /opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt
bash: /opt/weewx/bin/user/tide.sh: Permission denied

pi@raspberrypi:/opt/weewx/bin/user $ sudo /opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt
sudo: /opt/weewx/bin/user/tide.sh: command not found

pi@raspberrypi:/ $ cd  /opt/weewx/bin/user/
pi@raspberrypi:/opt/weewx/bin/user $ ls
tide.sh
pi@raspberrypi:/opt/weewx/bin/user $


8)
Has Matt directed to wrong location or is it only an issue of permissions to be considered in any of what he outlined?

First, let's try it with permissions set:
sudo chmod 755 /opt/weewx/bin/user/tide.sh

Then once again:
pi@raspberrypi:/ $ /opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt
bash: /opt/weewx/bin/user/tide.sh: /bin/sh^M: bad interpreter: No such file or directory
pi@raspberrypi:/ $

Issue created by initial creating tide.sh in TextPad on PC, specifically, new line in PC mode. Edited again in TextPad and saved as UNIX.

And again:
pi@raspberrypi:~/weewx/extensions/forecast $ /opt/weewx/bin/user/tide.sh -l ellsworth -fc -df'%Y.%m.%d' -tf'%H:%M' > /var/tmp/ellsworth.txt
Indexing /opt/harmonics/harmonics-dwf-20181227/harmonics-dwf-20181227-free.tcd...

Success! /var/tmp/ellsworth.txt was created YAY!

Matt says:

then i verified the tide parsing like this:

PYTHONPATH=bin python bin/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

==================================

9)
pi@raspberrypi:~/weewx/extensions/forecast $ PYTHONPATH=bin python bin/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug
python: can't open file 'bin/user/forecast.py': [Errno 2] No such file or directory
pi@raspberrypi:~/weewx/extensions/forecast $



I'm thinking that Matt says the wrong location for forecast.py

Trying next:
PYTHONPATH=bin python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

Which then reults:
pi@raspberrypi:/ $ PYTHONPATH=bin python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug
Traceback (most recent call last):
  File "/usr/share/weewx/user/forecast.py", line 566, in <module>
    import weewx
ImportError: No module named weewx


10)
At this point I cry uncle. 2:18 PM   Saturday, April 6, 2019  (VKB)

[mwall]

On Saturday, April 6, 2019 at 2:21:39 PM UTC-4, V. Kelly Bellis wrote:

Trying next:

PYTHONPATH=bin python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

Which then reults:
pi@raspberrypi:/ $ PYTHONPATH=bin python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug
Traceback (most recent call last):
  File "/usr/share/weewx/user/forecast.py", line 566, in <module>
    import weewx
ImportError: No module named weewx

for weewx installed via apt, you want:

PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

you should probably put tide.sh in /usr/share/weewx/user/tide.sh

or maybe create a directory /opt/tide and fill it with:

/opt/tide/tide.sh

/opt/tide/harmonics-dwf-20181227-free.tcd

then adjust paths in tide.sh and weewx.conf appropriately

i put it in /opt/weewx/bin/user/tide.sh because my weewx is installed at /opt/weewx

m

 

[V. Kelly Bellis]

YAY MATT!! Success at long last :) THANK YOU MATT!!!

My notes along the way to success:

10:07 AM   Sunday, April 7, 2019  (VKB)

Matt says:

for weewx installed via apt, you want:

PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug

Results:
pi@raspberrypi:~ $ PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/forecast.py --action=parse --method=xtide --filename=/var/tmp/ellsworth.txt --debug
[{'event_ts': 1554590640, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '0.64', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554612480, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '11.10', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554635100, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '0.14', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554657000, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '10.76', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554679020, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '0.64', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554700740, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '11.33', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554723720, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '-0.03', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554745500, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '10.73', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554767640, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '0.70', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554789300, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '11.48', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554812640, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '-0.10', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554834360, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '10.62', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554856560, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '0.83', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554878280, 'dateTime': 1554646140, 'location': None, 'hilo': 'H', 'offset': '11.52', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}, {'event_ts': 1554901860, 'dateTime': 1554646140, 'location': None, 'hilo': 'L', 'offset': '-0.08', 'issued_ts': 1554646140, 'method': 'XTide', 'usUnits': 1}]
pi@raspberrypi:~ $

==============================

Matt says:

you should probably put tide.sh in /usr/share/weewx/user/tide.sh

Then:
pi@raspberrypi:/opt/weewx/bin/user $ sudo mv tide.sh /usr/share/weewx/user/tide.sh
=========

Matt says:

then adjust paths in weewx.conf appropriately

Then in /etc/weewx/weewx.conf

Looking @ line 639, et als #####################################################

# Options for extension 'forecast'
[Forecast]
    data_binding = forecast_binding
    [[NWS]]
        lid = MEZ029
        foid = CAR
    ...
    [[XTide]]
        prog = /usr/share/weewx/user/tide.sh
        location = Ellsworth
       

Success at long last! YAY Matt!
10:32 AM   Sunday, April 7, 2019  (VKB)