Packaging the weather34 Template (WX-HWS)

[steeple ian]

Gary,
Thank your offer to guide me through this process.

First of all, WX-HWS is a port of weather34 template designed by Brian Underdown. Brian has given his permission for his original work to maintained in three versions, Meteobridge maintained by Lightmaster, Cumulus maintained by Ken True and WeeWX by myself. The WeeWX version is closely aligned visually, functionally and in performance to the Meteobridge version as I recognised that WeeWX can do what MB can do (and much more). All three versions are under active development.

My repository is at https://github.com/steepleian/WX-HWS . As you will see, there are a lot of folders and files. Simplistically the CRT extension is used to feed the live data and report generator to feed archive data for the weather almanacs and weather charts. Forecast data is optionally captured either Dark Sky or WU 5 day forecast.

What do I want to achieve? Although I have managed to streamline the manual install somewhat, it still causes problems for some users. My ideal is a robust install which includes all required extensions etc, insert the correct mods to weewx.conf and run a settings process where users can add their api keys, webcam and other hardware.

I have read the documentation and looked at several install.py files from the very simple to the more complex. I think it will be wise to take baby steps initially by creating an install just to place files. Once that is working, the more complex functions can be added.

One more thing to add, I am not a Python coder although I can follow what a piece of code doing and adapt it.

Many thanks,
Ian

[steeple ian]

Forgot to add, my public facing site is https://claydonsweather.org.uk/pws

[mwall]

ian,

you mentioned that your extension (lets call it weewx-wx34) depends on at least two other extensions (weewx-crt and weewx-forecast).

when one extension depends on others, you should make the dependent extension work as much as possible even if the independent extension(s) is not installed.  do not assume that a user has or ever will install all of the dependencies.  and definitely do not make one extension forcibly install another extension.  for example, if you made it so that installing weewx-wx34 also installs weewx-crt, you could very well break an installation that is already using weewx-crt.

in this example, the dependency on weewx-crt is required - weewx-wx34 will not be very useful without weewx-crt.  the dependency on weewx-forecast is optional - weewx-wx34 could be very useful, but it will not have forecast information.

so if a user installs weewx-wx34 but not weewx-crt, the installation of weewx-wx34 should complete successfully, but when the user opens the web page for weewx-wx34 s/he should see a message something like "No data are available.  Is the weewx-crt extension installed and configured?".

similarly, if weewx-forecast is not installed, consider making weewx-wx34 work completely - only the forecast parts would not display.

if you do this right, then the dependent extension will 'degrade gracefully' - it will work as much as possible without the dependencies, but it will offer information about those dependencies discretely and in a way that leads the user to discover how to install those dependencies.

m

[mwall]

another thing to consider is when to write something yourself in your extension versus use what has already been written in another extension.

for example, forecasting.

you could easily make queries to wu or other forecast service in the weewx-wx34 extension.  or you could make the weewx-wx34 extension use the forecast variables in the weewx-forecast extension.

which do you do?

i depends how much you want to duplicate effort, and how how much of the functionality from another extension that you need.

for example, the weewx-forecast extension works with many different forecasting services, and it provides a unified way to query and display forecast information.  that could make your work easier in weewx-wx34, and it could make life easier for users of weewx-wx34 (e.g., if a user cannot access the forecast service you hard-coded into weewx-wx34, or if a user wants to extend weewx-wx34 forecasting in a 'standard' way)

many of the lower-level extensions do not have to worry about this kind of tradeoff, but almost any skin/template extension will.

another example: the seasons skin.  in that skin we did not provide explicit support for air quality or forecasting or many other useful extensions.  but we did provide a structure that makes it trivially simple to add sections to the seasons skin, without breaking the original implementation.  if you want to make your system modular and easily customized, spend some time on that aspect of the design.  (you may not be able to do that much with wx34 - it strikes me as a rather monolithic design and implementation.  modularity might not be possible unless you do a complete fork, or get the upstream to accept significant changes.)

m

[steeple ian]

Matthew and Gary,

A quick question, if I wanted the packaged skin to install some files directly into the web root rather than skin or bin/user, (a) is this possible? (b) what is the syntax? I have tried all the various path options that I can think of and install.py keeps throwing syntax errors.

Thanks,

Ian

[Vince Skahan]

On Wednesday, July 3, 2019 at 2:39:02 PM UTC-7, steeple ian wrote:

Matthew and Gary,

A quick question, if I wanted the packaged skin to install some files directly into the web root rather than skin or bin/user, (a) is this possible? (b) what is the syntax? I have tried all the various path options that I can think of and install.py keeps throwing syntax errors.

My understanding of the correct way is to install your extension to the skin root and then use the 'copy_once' feature in skin.conf to get those files to get into the docroot when weewx starts up. 

For an example of one of my extensions, see  https://github.com/vinceskahan/vds-weewx-local-skin/blob/master/skins/vds-local/skin.conf (line 40 or so) and the matching installer script that's at https://github.com/vinceskahan/vds-weewx-local-skin/blob/master/install.py as well.   It's worked well for me for several years that way FWIW. 

[steeple ian]

Thanks Vince,

That works perfectly. If I want to copy_once an entire folder containing mixed type files and sub folders, is there a syntax for that or do I have to list each wild-card file type and sub-folder?

[mwall]

On Wednesday, July 3, 2019 at 7:21:53 PM UTC-4, steeple ian wrote:

That works perfectly. If I want to copy_once an entire folder containing mixed type files and sub folders, is there a syntax for that or do I have to list each wild-card file type and sub-folder?

just to be explicit - this is a distinction between when your skin is installed versus when a report is generated from that skin.

at installation time:  the installer places the files into the user-writable areas of a weewx configuration - in the case of a skin, that means the skins directory and the user directory.  those are the files that are used to generate a report.

at report generation time:  the copy_once directive is one mechanism for telling the report generator what to do when it generates reports. 

if a directory is specified as an element in the list for copy_once, then copying should recurse into that directory.  so be sure to clean out any dot files you might have lingering about.

m

[mwall]

On Wednesday, July 3, 2019 at 7:21:53 PM UTC-4, steeple ian wrote:

If I want to copy_once an entire folder containing mixed type files and sub folders, is there a syntax for that or do I have to list each wild-card file type and sub-folder?

did you try it?

no magic syntax, just what you would expect:

given a file 'a', directory 'b', and directory 'c', the directive:

copy_once = a, b, c/*.txt

will copy the file 'a', the directory 'b' and all of its contents, and only files that end in '*.txt' from directory 'c'

if you experience other behavior, please report it.

if some of the files/directories that feed your skin change, for example, by some mechanism outside of weewx, then use the `copy_always` directive for those assets.

m

[mwall]

On 04 Jul 2019, at 08:56, steeple ian wrote:

Matthew,

Yes I did,

This is from the syslog: -

reportengine: Caught unrecoverable exception in generator 'weewx.reportengine.CopyGenerator'

Jul  4 13:40:44 DietPi weewx[7627]:         ****  [Errno 21] Is a directory: 'w34pages'

Jul  4 13:40:44 DietPi weewx[7627]:         ****  Traceback (most recent call last):

Jul  4 13:40:44 DietPi weewx[7627]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 204, in run

Jul  4 13:40:44 DietPi weewx[7627]:         ****      obj.start()

Jul  4 13:40:44 DietPi weewx[7627]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 300, in start

Jul  4 13:40:44 DietPi weewx[7627]:         ****      self.run()

Jul  4 13:40:44 DietPi weewx[7627]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 460, in run

Jul  4 13:40:44 DietPi weewx[7627]:         ****      shutil.copy(_file, dest_dir)

Jul  4 13:40:44 DietPi weewx[7627]:         ****    File "/usr/lib/python2.7/shutil.py", line 119, in copy

Jul  4 13:40:44 DietPi weewx[7627]:         ****      copyfile(src, dst)

Jul  4 13:40:44 DietPi weewx[7627]:         ****    File "/usr/lib/python2.7/shutil.py", line 82, in copyfile

Jul  4 13:40:44 DietPi weewx[7627]:         ****      with open(src, 'rb') as fsrc:

Jul  4 13:40:44 DietPi weewx[7627]:         ****  IOError: [Errno 21] Is a directory: 'w34pages'

Jul  4 13:40:44 DietPi weewx[7627]:         ****  Generator terminated

It seems to have choked on the fact that it was a directory!

i thought i remembered something about this.  there was a fix in 3.9.2 (not yet released) for something similar - see #379:

https://github.com/weewx/weewx/issues/379

otherwise, just explicitly specify the patterns for files that you need to copy.

m

[steeple ian]

Matthew,

Getting back to this task after the Summer, I now have the Weather34 skin working with a conventional wee_extension --install process. It works like a dream thanks to your changes in 3.9.2 to allow nested copying. There is just one little niggle which I trying to exclude. Once everything is setup, if the user restarts WeeWX for any reason, the copy_once process starts again and wipes out some settings. At the moment, I ask users to edit the skin.conf file after the first archive cycle, to comment out the copy_once line with a #.

I would like to find a more elegant solution to this to minimise the amount of editing for the user. Do you have any suggestions of what I could try?

You can see the current build at https://github.com/steepleian/WX-HWS

Many thanks,

Ian