Calling custom posts

[Mike Revitt]

I have written a report to copy files to an AWS S3 bucket, separate thread on that, but now an trying to incorporate it into my deployment.

The easy way, and the way I have done it so far is to modify reportengine.py, but there must be a better way

# =============================================================================
#                    Class AWS_Sync

# =============================================================================

class AWS_Sync(ReportGenerator):

    

    """Class for managing the "AWS generator".

    This will copy everything in the public_html subdirectory to a webserver."""

    def run(self):

        import weeutil.s3upload

        # determine how much logging is desired

        log_success = to_bool(search_up(self.skin_dict, 'log_success', True))

        t1 = time.time()

        try:

            local_root = os.path.join(self.config_dict['WEEWX_ROOT'],

                                      self.skin_dict.get('HTML_ROOT', self.config_dict['StdReport']['HTML_ROOT']))

            s3_data = weeutil.s3upload.S3Upload(

                bucket=self.skin_dict['bucket'],

                profile=self.skin_dict['profile'],

                local_root=local_root,

                remote_root=self.skin_dict['S3_ROOT'],

                name=self.skin_dict['REPORT_NAME'],

                debug=int(self.skin_dict.get('debug', 0)),

                secure_data=to_bool(self.skin_dict.get('secure_data', True)))

        except KeyError:

            syslog.syslog(syslog.LOG_DEBUG,

                          "AWS_Sync: AWS upload not requested. Skipped.")

            return

        try:

            n = s3_data.run()

        except (socket.timeout, socket.gaierror, ftplib.all_errors, IOError) as e:

            (cl, unused_ob, unused_tr) = sys.exc_info()

            syslog.syslog(syslog.LOG_ERR, "AWS_Sync: "

                                          "Caught exception %s: %s" % (cl, e))

            weeutil.weeutil.log_traceback("        ****  ")

            return

        if log_success:

            t2 = time.time()

            syslog.syslog(syslog.LOG_INFO,

                          "AWS_Sync: awscli S3'd %d files in %0.2f seconds" %

                          (n, (t2 - t1)))

Thoughts

[gjr80]

A few things. Rather than put your generator in reportengine.py put it in its own file in /home/weewx/bin/user or /usr/share/weewx/user, that way it will survive WeeWX.

To invoke your generator create a skin ala FTP and in your skin.conf you will invoke your generator rather than the FTP generator. Then in weewx.conf under [StdReport] you will include a report just like the [[FTP]] but with the details/settings for your generator.

That’s a fairly broad brush high level view or have I missed your question completely?

Gary

[Mike Revitt]

That makes sense, I have created a skin with its own weewx.conf already, was struggling to work out how to call the generator in its own file

Do I then call it from the S3 conf file like this

[Generators]

    generator_list = weeutil.s3upload.S3Upload

------

where

weeutil is the directory

s3upload is the py file

S3Upload is the class within the py file

---

Already have a section in the main conf file that I have just tested via my other thread on this subject

[Paul Anderson]

Hi Mike,
Reading your post jogged my memory that I had read of someone doing this previously.
weewx-S3upload
https://github.com/wmadill/weewx-S3upload

Maybe you can try it, or use it for ideas. Have fun!
Paul

[gjr80]

On Thursday, 13 February 2020 08:39:26 UTC+10, Mike Revitt wrote:

That makes sense, I have created a skin with its own weewx.conf already, was struggling to work out how to call the generator in its own file

I suspect you mean its own skin.conf, your only connection to weewx.conf will be that your skin will have it's own sub-section under [StdReport] in weewx.conf.

 

Do I then call it from the S3 conf file like this

[Generators]

    generator_list = weeutil.s3upload.S3Upload

The structure is right but I think the detail is wrong. Let's take a step back. You have created class S3Upload in s3upload.py in the weeutil directory, this is the class that does the actual upload work. To perform the S3 upload as a generator in a skin (al the FTP generator) you then need to call a class based on class ReportGenerator, this appears to be your class AWS_Sync in your first post. As I outlined above any code you add is best located in the user directory (/home/weewx/bin/user or /usr/share/weewx/user depending on your WeeWX install type) in order to survive WeeWX upgrades. In your case there is no reason why you cannot place both classes in the same file, let's say you place both in the file user/s3upload.py. In your skin.conf you would call your generator with:

[Generators]
    generator_list = user.s3upload.AWS_Sync

In your class AWS_Sync instead of using

   def run(self):
        import weeutil.s3upload

        # determine how much logging is desired
        log_success = to_bool(search_up(self.skin_dict, 'log_success', True))

        t1 = time.time()
        try:
            local_root = os.path.join(self.config_dict['WEEWX_ROOT'],
                                      self.skin_dict.get('HTML_ROOT', self.config_dict['StdReport']['HTML_ROOT']))

            s3_data = weeutil.s3upload.S3Upload(

you would use

   def run(self):
        import user.s3upload

        # determine how much logging is desired
        log_success = to_bool(search_up(self.skin_dict, 'log_success', True))

        t1 = time.time()
        try:
            local_root = os.path.join(self.config_dict['WEEWX_ROOT'],
                                      self.skin_dict.get('HTML_ROOT', self.config_dict['StdReport']['HTML_ROOT']))

            s3_data = user.s3upload.S3Upload(

You then have everything contained in a single file that is survivable across WeeWX upgrades. One other thing you might consider is renaming class AWS_Sync, WeeWX uses the naming structure XxxxxxGenrator for its generator classes, so something like class S3UploadGenerator might make it a bit more obvious as to the purpose of the class (which aids others when they look at your code later). Of course if you chnage the name of class AWS_Sync you need to change the same in the examples I have used above.

Gary

[Mike Revitt]

You have been super helpful so far Gary, so bear with me, I am almost there.

My program is now called S3UploadGenerator and is in the bin/user directory. I have also successfully installed weewx 4.0 on my raspberry pi using python3, which is also now the default

Calling it from the command line now works perfectly and I also successfully call it when running weewxd 

But: 1 error remains when s3upload.py is called:

Traceback (most recent call last):

  File "/home/weewx/bin/weewx/reportengine.py", line 202, in run

    obj.start()

AttributeError: 'S3Upload' object has no attribute 'start'

Exception in thread ReportThread:

Traceback (most recent call last):

  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner

    self.run()

  File "/home/weewx/bin/weewx/reportengine.py", line 215, in run

    obj.finalize()

AttributeError: 'S3Upload' object has no attribute 'finalize'

I have searched the manuals and looked at all of the existing code and can't work out what to do to resolve this.

my S3/skin.conf contains the following:

[Generators]

    generator_list = user.s3upload.S3Upload

This is the relevant part of s3upload.py

import logging

import os

import sys

import time

import boto3

from    weewx.reportengine  import ReportGenerator

from    six.moves           import cPickle

log = logging.getLogger(__name__)

# =============================================================================

#                    Class S3UploadGenerator

# =============================================================================

class S3UploadGenerator(ReportGenerator):

    

    """Class for managing the "AWS generator".

    This will copy everything in the public_html subdirectory to a webserver."""

    

    def run(self):

        import user.s3upload

        # determine how much logging is desired

        log_success = to_bool(search_up(self.skin_dict, 'log_success', True))

        t1 = time.time()

        try:

            S3_upload = user.s3upload.S3Upload(bucket=self.skin_dict['S3_BUCKET'],

                                            profile=self.skin_dict['AWS_Profile'],

                                            html_root=self.config_dict['StdReport']['HTML_ROOT'],

                                            remote_root=self.skin_dict['S3_ROOT'],

                                            name=self.skin_dict['REPORT_NAME'])

        except KeyError:

            syslog.syslog(syslog.LOG_DEBUG,

                          "S3UploadGenerator: AWS upload not requested. Skipped.")

            return

        try:

            n = S3_upload.run()

        except (socket.timeout, socket.gaierror, ftplib.all_errors, IOError) as e:

            (cl, unused_ob, unused_tr) = sys.exc_info()

            syslog.syslog(syslog.LOG_ERR, "S3UploadGenerator: "

                                          "Caught exception %s: %s" % (cl, e))

            weeutil.weeutil.log_traceback("        ****  ")

            return

        if log_success:

            t2 = time.time()

            syslog.syslog(syslog.LOG_INFO,

                          "S3UploadGenerator: AWS-S3 S3'd %d files in %0.2f seconds" %

                          (n, (t2 - t1)))

And this is the main, which works from the command line

# =============================================================================

#                    Main

# =============================================================================

if __name__ == '__main__':

    import configobj

    import weewx

    import weeutil.logger

    weewx.debug = 1

    weeutil.logger.setup('S3upload', {})

    

    if len(sys.argv) < 2:

        print("""Usage: s3upload.py path-to-configuration-file [path-to-be-ftp'd]""")

        sys.exit(weewx.CMD_ERROR)

    try:

        config_dict = configobj.ConfigObj(sys.argv[1], file_error=True, encoding='utf-8')

    except IOError:

        print("Unable to open configuration file %s" % sys.argv[1])

        raise

    S3_upload = S3Upload(config_dict['StdReport']['AWS-S3']['S3_BUCKET'],

                         config_dict['StdReport']['AWS-S3']['AWS_Profile'],

                         config_dict['StdReport']['HTML_ROOT'],

                         config_dict['StdReport']['AWS-S3']['S3_ROOT'],

                         'S3')

                         

    print(config_dict['StdReport']['AWS-S3']['S3_BUCKET'],

    config_dict['StdReport']['AWS-S3']['AWS_Profile'],

    config_dict['WEEWX_ROOT'],

    config_dict['StdReport']['AWS-S3']['S3_ROOT'],

    'S3')

    S3_upload.run()

And this is the relevant section of weewx.conf

    [[AWS-S3]]

        # Using AWSCLI to copy the results to a webserver is treated as just

        # another report, albeit one with an unusual report generator!

        skin = S3
        enable = true

        # You must configure AWS at the command line and create some logon credentials

        # in the credentials file. This is the name of the profile defined in that

        # file that you wish to use for the copy

        AWS_Profile = mike

        # This is the name of the S3 bucket where the files will be copied

        S3_BUCKET = s3-archive.revittmk.aws.co.uk

        # This is the folder into which the files will be copied within the S3 bucket,

        S3_ROOT = /MountWeather

[Mike Revitt]

This is the s3upload.py file, might be easier than posing extracts

[gjr80]

Mike,

The generator_list setting should contain one or more references to generator classes to be used, ie classes that are based on class ReportGenerator. Given your code that would be S3UploadGenerator, so something like:

[Generators]

    generator_list = user.s3upload.S3UploadGenerator

should do the trick.

Gary

[Mike Revitt]

Thanks for all your help Gary,

That did the trick, Just fixed all off the error handling today, now that I have it running, and am ready to package it up and put my new website live.

What is the process for adding a plugin to weewx?

Mike

[gjr80]

That is good news.

As for adding to WeeWX, typically only services/drivers etc that are/will be widely used are added to the code base. Anything else is maintained externally by the author and made available as they see fit, that can be anything from a few scratchie bits of code and some instructions on a web page somewhere through to a WeeWX extension that can be easily installed/managed/removed by the user using the wee_extension utility. Extensions etc are commonly listed in the WeeWX wiki (anyone can edit/add to the wiki) with a link to any source files/extension packages (the WeeWX wiki is great for documenting instructions etc but does not store files as such). Many of us host our code on GitHub (just search for weewx in GitHub) and many of the extensions you will find in the wiki have links to GitHub repos.

Gary