Delaying report generation to include solar power data

[Risto Virtanen]

I've successfully used weewx for my home weather station (a FineOffset WH1080) for over a year now practically without any problems.

Recently I had a solar power system installed and in order to monitor the production I soon installed the SBFspot software that gathers the data from the SMA inverter and stores it into a mysql database. Quite soon I decided that it would be nice to have the actual production power visible also on the weather station page (http://www.myrskyt.org/rouhunsaa/index.html)

For this I thought I could use the solar radiation field that is already in the database and in the skins, so I wrote a small shell script that copies the power value from the SBFspot database into the weewx.archive.radiation field, run every minute by crond. The values seem to enter into the database but for some reason they are not shown in the web page created (a N/A is displayed)

I suspect this must be due to some timing issue: the value for radiation is not yet available in the database when the report generation starts.

So my first question is: is it possible to delay the report generation a bit (10-20 seconds max) so that the radiation figures have time to get into the database? If yes, how? I took a look at the file /usr/share/weewx/engine.py and also added a line with  time.sleep(15) into the section  def launch_report_thread(self, event): but it doesn't seem to help. Please, understand, that I'm not a python programmer:(

My 2nd question is related to the procedure: is it possible/reasonable to add a 'sensor' value like that at all or should I have to try to do it the long and possibly hard way (i.e. properly:) as described in the documentation, adding a service/observation type with classes etc?

//rkv

[Thomas Keffer]

You did not include your configuration file (nor a log!), but I suspect the problem is as you say: the reports are getting generated before the solar data are available. 

You can delay the report generation with option archive_delay, but that will delay the saving of the archive record as well. You may end up with a situation where weewx overwrites your changes.

But, why not add the second database explicitly? It's not so hard. 

Here's how. Say your MySQL database is on a machine solar_computer, in a database called solar_database, in a table called solar_table. Then your configuration file would look something like (additions in yellow and NOT TESTED):

[DataBindings]

    

    [[wx_binding]]

        # The database must match one of the sections in [Databases].

        # This is likely to be the only option you would want to change.

        database = archive_sqlite

        # The name of the table within the database

        table_name = archive

        # The manager handles aggregation of data for historical summaries

        manager = weewx.wxmanager.WXDaySummaryManager

        # The schema defines the structure of the database.

        # It is *only* used when the database is created.

        schema = schemas.wview.schema

    [[solar_binding]]

        database = solar_mysql

        table_name = solar_table

        manager = weewx.manager.Manager

        # schema = dummy_schema        # Schema not needed

##############################################################################

#   This section defines various databases.

[Databases]

    

    # A SQLite database is simply a single file

    [[archive_sqlite]]

        database_name = weewx.sdb

        database_type = SQLite

    

    # MySQL

    [[archive_mysql]]

        database_name = weewx

        database_type = MySQL

    [[solar_mysql]]

        database_name = solar_database

        database_type = MySQL2

##############################################################################

#   This section defines defaults for the different types of databases.

[DatabaseTypes]

    # Defaults for SQLite databases

    [[SQLite]]

        driver = weedb.sqlite

        # Directory in which the database files are located

        SQLITE_ROOT = %(WEEWX_ROOT)s/archive

    

    # Defaults for MySQL databases

    [[MySQL]]

        driver = weedb.mysql

        # The host where the database is located

        host = localhost

        # The user name for logging in to the host

        user = weewx

        # The password for the user name

        password = weewx

    [[MySQL2]]

        driver = weedb.mysql

        host = solar_computer

        user = your_user_name

        password = your_password

Substitute a real user and password for your_user_name and your_password. I would strongly suggest a user name with only 'select' privileges in case something goes wrong! 

As I've outlined it, the solar database will not have daily summaries, so you will be limited in what statistics you can ask of it. To add the daily summaries would require write privileges (specifically, "update", "create", "insert") to the database, which may not be a good idea until you understand what you're doing.

Having done this, and assuming you have a SQL  type solarPower in the schema on the other database, you could get the current value as

$latest($data_binding='solar_binding').solarPower

You could plot it with

[[[daySolarPower]]]

   [[[[solarPower]]]]

       label = Power from solar

       data_binding = solar_binding

This would create a daySolarPower.png file.

No Python programming involved. 

Details on all this is in the section Using multiple bindings in the Customizing Guide.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Risto Virtanen]

perjantai 26. helmikuuta 2016 14.48.02 UTC+2 Tom Keffer kirjoitti:

You did not include your configuration file (nor a log!), but I suspect the problem is as you say: the reports are getting generated before the solar data are available. 

No, I didn't as I was a bit lost. I removed my edits from the engine.py and restarted weewx. And it seems it is the reason (syslog):

Feb 26 15:11:01 hanuri /USR/SBIN/CRON[26460]: (hassula) CMD (/home/hassula/bin/sma-tila2sql.sh > /dev/null)

Feb 26 15:11:11 hanuri weewx[8533]: manager: added record 2016-02-26 15:10:04 HMT (1456492204) to database 'weewxdb'

Feb 26 15:11:11 hanuri weewx[8533]: manager: added record 2016-02-26 15:10:04 HMT (1456492204) to daily summary in 'weewxdb'

Feb 26 15:11:12 hanuri weewx[8533]: restx: AWEKAS: Published record 2016-02-26 15:10:04 HMT (1456492204)

Feb 26 15:11:15 hanuri weewx[8533]: cheetahgenerator: Generated 23 files for report StandardReport in 3.34 seconds

Feb 26 15:11:16 hanuri weewx[8533]: genimages: Generated 12 images for StandardReport in 1.50 seconds

Feb 26 15:11:18 hanuri weewx[8533]: rsyncupload: rsync'd 36 files (113805 bytes) in 1.73 seconds

The script sma-tila2sql.sh makes the database update and is run 10 seconds *before* the record  gets added. It seems there is a few seconds gap between AWEKAS publish and cheetahgeneration, but I doubt it can be trusted enough.

You can delay the report generation with option archive_delay, but that will delay the saving of the archive record as well. You may end up with a situation where weewx overwrites your changes.

This is what I understood from the documentation.

But, why not add the second database explicitly? It's not so hard. 

That was one thing that also popped into my mind (related to Q#2), and actually I tried/started work on that also, but then I got stuck in the process. But it seems that it is the proper way to proceed. Thanks for the clarified instructions.

The DataBinding and Database sections are otherwise quite clear but I couldn't figure out the correct entries for the lines

       manager = weewx.manager.Manager

        # schema = dummy_schema        # Schema not needed

The "Schema not needed" is the key here. And a manager does not have be created/edited? I'm only interested in the current values into the html pages, the solar power history can be fetched from the SBFspot database (or from SunnyPortal.com)

Below you write also:

"Having done this, and assuming you have a SQL  type solarPower in the schema on the other database, you could get the current value as

$latest($data_binding='solar_binding').solarPower"

"Schema of the other database"? Do you mean there is a field with name 'solarPower' in the SBFspot database table?

I noticed that in my weewx.conf the [Databases] section contains all the items for the database access (including username and password), there is no separate [DatabaseTypes] section. Seems to work, anyway.

I'll re-read the instructions once more and edit the files. Thanks!

//rkv

[Thomas Keffer]

On Fri, Feb 26, 2016 at 5:55 AM, Risto Virtanen <ris...@gmail.com> wrote:

perjantai 26. helmikuuta 2016 14.48.02 UTC+2 Tom Keffer kirjoitti:

You did not include your configuration file (nor a log!), but I suspect the problem is as you say: the reports are getting generated before the solar data are available. 

No, I didn't as I was a bit lost. I removed my edits from the engine.py and restarted weewx. And it seems it is the reason (syslog):

Feb 26 15:11:01 hanuri /USR/SBIN/CRON[26460]: (hassula) CMD (/home/hassula/bin/sma-tila2sql.sh > /dev/null)

Feb 26 15:11:11 hanuri weewx[8533]: manager: added record 2016-02-26 15:10:04 HMT (1456492204) to database 'weewxdb'

Feb 26 15:11:11 hanuri weewx[8533]: manager: added record 2016-02-26 15:10:04 HMT (1456492204) to daily summary in 'weewxdb'

Feb 26 15:11:12 hanuri weewx[8533]: restx: AWEKAS: Published record 2016-02-26 15:10:04 HMT (1456492204)

Feb 26 15:11:15 hanuri weewx[8533]: cheetahgenerator: Generated 23 files for report StandardReport in 3.34 seconds

Feb 26 15:11:16 hanuri weewx[8533]: genimages: Generated 12 images for StandardReport in 1.50 seconds

Feb 26 15:11:18 hanuri weewx[8533]: rsyncupload: rsync'd 36 files (113805 bytes) in 1.73 seconds

The script sma-tila2sql.sh makes the database update and is run 10 seconds *before* the record  gets added. It seems there is a few seconds gap between AWEKAS publish and cheetahgeneration, but I doubt it can be trusted enough.

W

​eewx adds a record using a SQL INSERT statement, so this would overwrite whatever you already had in the database.

As an experiment, you could try changing it to an UPDATE statement, but this would require some Python programming.

​

The DataBinding and Database sections are otherwise quite clear but I couldn't figure out the correct entries for the lines

       manager = weewx.manager.Manager

        # schema = dummy_schema        # Schema not needed

The "Schema not needed" is the key here. And a manager does not have be created/edited? I'm only interested in the current values into the html pages, the solar power history can be fetched from the SBFspot database (or from SunnyPortal.com)

​The schema is not needed because weewx is not creating the database --- it already exists, presumably created by the SBFspot software.​

 

Below you write also:

"Having done this, and assuming you have a SQL  type solarPower in the schema on the other database, you could get the current value as

$latest($data_binding='solar_binding').solarPower"

"Schema of the other database"? Do you mean there is a field with name 'solarPower' in the SBFspot database table?

​That's correct. I don't know what the actual field name is, so I used the placeholder 'solarPower'. Substitute as necessary.

​

 

​-tk​

[Risto Virtanen]

perjantai 26. helmikuuta 2016 16.14.29 UTC+2 Tom Keffer kirjoitti:

W

​eewx adds a record using a SQL INSERT statement, so this would overwrite whatever you already had in the database.

Ok.

As an experiment, you could try changing it to an UPDATE statement, but this would require some Python programming.

I'm not afraid of Python programming (actually, I took an Edx course on subject last year, but as there was no further experiment, the skills are missing:) And basically it is the order of the INSERT and UPDATE that matters. My shell script does an update.

​That's correct. I don't know what the actual field name is, so I used the placeholder 'solarPower'. Substitute as necessary.

Ok.

Now I have it as follows:

/etc/weewx/weewx.conf:

[DataBindings]

    [[sma_binding]]

        database = SBFspot_mysql

        table_name = vwSpotData

        manager = weewx.wxmanager.smaManager  # not used?

        schema = dummy_schema 

[Databases]

    [[SBFspot_mysql]]

        host = localhost

        user = USERNAME # SELECT privilege only

        password = PASSWORD

        database_name = SBFspot

        driver = weedb.mysql

/etc/weewx/skins/Rouhunsaa/index.html.tmpl

              <tr>

                <td class="stats_label">Solar Power</td>

                <td class="stats_data">$latest.($data_binding='sma_binding').PacTot</td>

              </tr>

And in the sql database SBFspot:

mysql> describe vwSpotData;

+---------------+---------------+------+-----+---------+-------+

| Field         | Type          | Null | Key | Default | Extra |

+---------------+---------------+------+-----+---------+-------+

| TimeStamp     | datetime      | YES  |     | NULL    |       |

| Nearest5min   | datetime      | YES  |     | NULL    |       |

| Name          | varchar(32)   | YES  |     | NULL    |       |

| Type          | varchar(32)   | YES  |     | NULL    |       |

| Serial        | int(4)        | NO   |     | NULL    |       |

| Pdc1          | int(4)        | YES  |     | NULL    |       |

| Pdc2          | int(4)        | YES  |     | NULL    |       |

| Idc1          | float         | YES  |     | NULL    |       |

| Idc2          | float         | YES  |     | NULL    |       |

| Udc1          | float         | YES  |     | NULL    |       |

| Udc2          | float         | YES  |     | NULL    |       |

| Pac1          | int(4)        | YES  |     | NULL    |       |

| Pac2          | int(4)        | YES  |     | NULL    |       |

| Pac3          | int(4)        | YES  |     | NULL    |       |

| Iac1          | float         | YES  |     | NULL    |       |

| Iac2          | float         | YES  |     | NULL    |       |

| Iac3          | float         | YES  |     | NULL    |       |

| Uac1          | float         | YES  |     | NULL    |       |

| Uac2          | float         | YES  |     | NULL    |       |

| Uac3          | float         | YES  |     | NULL    |       |

| PdcTot        | bigint(12)    | YES  |     | NULL    |       |

| PacTot        | bigint(13)    | YES  |     | NULL    |       |

| Efficiency    | decimal(17,1) | YES  |     | NULL    |       |

| EToday        | int(8)        | YES  |     | NULL    |       |

| ETotal        | int(8)        | YES  |     | NULL    |       |

| Frequency     | float         | YES  |     | NULL    |       |

| OperatingTime | double        | YES  |     | NULL    |       |

| FeedInTime    | double        | YES  |     | NULL    |       |

| BT_Signal     | double(18,1)  | YES  |     | NULL    |       |

| Status        | varchar(10)   | YES  |     | NULL    |       |

| GridRelay     | varchar(10)   | YES  |     | NULL    |       |

| Temperature   | double(18,1)  | YES  |     | NULL    |       |

+---------------+---------------+------+-----+---------+-------+

32 rows in set (0.01 sec)

If you take a look at the page  http://www.myrskyt.org/rouhunsaa/index.html you can see there is field at the bottom labeled 'Solar Power' but it is not showing '18' but the string $latest.($data_binding='sma_binding').PacTot instead.

I also tried to set Debug value to 1 but no additional relevant info was entered into syslog (also the html update stopped working for some reason, after AWEKAS upload no RSYNC for some time, I fiddled with the skin.conf a bit)

Most probably I've missed something. Any ideas?

//rkv

[Thomas Keffer]

On Fri, Feb 26, 2016 at 7:45 AM, Risto Virtanen <ris...@gmail.com> wrote:

/etc/weewx/weewx.conf:

[DataBindings]

    [[sma_binding]]

        database = SBFspot_mysql

        table_name = vwSpotData

        manager = weewx.wxmanager.smaManager  # not used?

        schema = dummy_schema 

​Should be 

manager = weewx.manager.Manager

​

 

[Databases]

    [[SBFspot_mysql]]

        host = localhost

        user = USERNAME # SELECT privilege only

        password = PASSWORD

        database_name = SBFspot

        driver = weedb.mysql

/etc/weewx/skins/Rouhunsaa/index.html.tmpl

              <tr>

                <td class="stats_label">Solar Power</td>

                <td class="stats_data">$latest.($data_binding='sma_binding').PacTot</td>

              </tr>

​Should be:

$latest($data_binding='sma_binding').PacTot​

​-tk​

[Risto Virtanen]

perjantai 26. helmikuuta 2016 17.56.43 UTC+2 Tom Keffer kirjoitti:

​Should be 

manager = weewx.manager.Manager

​

Corrected.

​Should be:

$latest($data_binding='sma_binding').PacTot​

My bad. Did not (this time) do a copy-paste, and left that extra dot. Corrected.

After restart the index.html is not generated anymore. Lines in syslog:

Feb 26 20:06:24 hanuri weewx[10062]: manager: added record 2016-02-26 20:05:01 HMT (1456509901) to database 'weewxdb'

Feb 26 20:06:24 hanuri weewx[10062]: manager: added record 2016-02-26 20:05:01 HMT (1456509901) to daily summary in 'weewxdb'

Feb 26 20:06:24 hanuri weewx[10062]: restx: AWEKAS: Published record 2016-02-26 20:05:01 HMT (1456509901)

Feb 26 20:06:27 hanuri weewx[10062]: cheetahgenerator: Generate failed with exception '<type 'exceptions.ValueError'>'

Feb 26 20:06:27 hanuri weewx[10062]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Rouhunsaa/index.html.tmpl

Feb 26 20:06:27 hanuri weewx[10062]: cheetahgenerator: **** Reason: Empty module name

Feb 26 20:06:27 hanuri weewx[10062]: ****  Traceback (most recent call last):

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 296, in generate

Feb 26 20:06:27 hanuri weewx[10062]: ****      print >> _file, text

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/lib/pymodules/python2.6/Cheetah/Template.py", line 1005, in __str__

Feb 26 20:06:27 hanuri weewx[10062]: ****      rc = getattr(self, mainMethName)()

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 428, in respond

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 189, in __errorCatcher18

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "<string>", line 1, in <module>

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weewx/tags.py", line 65, in latest

Feb 26 20:06:27 hanuri weewx[10062]: ****      manager = self.db_lookup(data_binding)

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weewx/manager.py", line 846, in db_lookup

Feb 26 20:06:27 hanuri weewx[10062]: ****      return self.get_manager(data_binding)

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weewx/manager.py", line 832, in get_manager

Feb 26 20:06:27 hanuri weewx[10062]: ****      default_binding_dict=self.default_binding_dict)

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weewx/manager.py", line 901, in get_manager_dict

Feb 26 20:06:27 hanuri weewx[10062]: ****      manager_dict['schema'] = weeutil.weeutil._get_object(schema_name)

Feb 26 20:06:27 hanuri weewx[10062]: ****    File "/usr/share/weewx/weeutil/weeutil.py", line 894, in _get_object

Feb 26 20:06:27 hanuri weewx[10062]: ****      mod = __import__(module)

Feb 26 20:06:27 hanuri weewx[10062]: ****  ValueError: Empty module name

Feb 26 20:06:27 hanuri weewx[10062]: cheetahgenerator: Generated 22 files for report StandardReport in 3.34 seconds

Feb 26 20:06:29 hanuri weewx[10062]: genimages: Generated 12 images for StandardReport in 1.44 seconds

Feb 26 20:06:31 hanuri weewx[10062]: rsyncupload: rsync'd 34 files (108263 bytes) in 2.55 seconds

Should I comment the schema with a # altogether?

//rkv

 

[Thomas Keffer]

Yes, comment it out as per my original post.

-tk

--

[Risto Virtanen]

Done. Now it complains about usUnits:

Feb 26 20:41:39 hanuri weewx[12156]: ****  OperationalError: (1054, "Unknown column 'usUnits' in 'field list'")

syslog:

Feb 26 20:41:39 hanuri weewx[12156]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Rouhunsaa/index.html.tmpl

Feb 26 20:41:39 hanuri weewx[12156]: cheetahgenerator: **** Reason: (1054, "Unknown column 'usUnits' in 'field list'")

Feb 26 20:41:39 hanuri weewx[12156]: ****  Traceback (most recent call last):

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 296, in generate

Feb 26 20:41:39 hanuri weewx[12156]: ****      print >> _file, text

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/lib/pymodules/python2.6/Cheetah/Template.py", line 1005, in __str__

Feb 26 20:41:39 hanuri weewx[12156]: ****      rc = getattr(self, mainMethName)()

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 428, in respond

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 189, in __errorCatcher18

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "<string>", line 1, in <module>

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/tags.py", line 65, in latest

Feb 26 20:41:39 hanuri weewx[12156]: ****      manager = self.db_lookup(data_binding)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 846, in db_lookup

Feb 26 20:41:39 hanuri weewx[12156]: ****      return self.get_manager(data_binding)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 833, in get_manager

Feb 26 20:41:39 hanuri weewx[12156]: ****      self.manager_cache[data_binding] = open_manager(manager_dict, initialize)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 918, in open_manager

Feb 26 20:41:39 hanuri weewx[12156]: ****      manager_dict['table_name'])

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 103, in open

Feb 26 20:41:39 hanuri weewx[12156]: ****      dbmanager = cls(connection, table_name)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 85, in __init__

Feb 26 20:41:39 hanuri weewx[12156]: ****      self._sync()

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 191, in _sync

Feb 26 20:41:39 hanuri weewx[12156]: ****      _row = self.getSql("SELECT usUnits FROM %s LIMIT 1;" % self.table_name)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weewx/manager.py", line 380, in getSql

Feb 26 20:41:39 hanuri weewx[12156]: ****      _cursor.execute(sql, sqlargs)

Feb 26 20:41:39 hanuri weewx[12156]: ****    File "/usr/share/weewx/weedb/mysql.py", line 30, in guarded_fn

Feb 26 20:41:39 hanuri weewx[12156]: ****      raise weedb.OperationalError(e)

Feb 26 20:41:39 hanuri weewx[12156]: ****  OperationalError: (1054, "Unknown column 'usUnits' in 'field list'")

Where is that field_list and should it contain all the fields on the SNBFspot table?

//rkv

[Thomas Keffer]

Rats. Didn't think about that failure mode. 

Come to think of it, this is not going to be as easy as I thought, and would require a custom Manager class.

OK, let's try another way: a weewx service that reads the database then inserts what you need in the archive record. It would look something like this:

File user/solar.py:

import weewx.manager

import weedb

from weewx.engine import StdService

class GetSolar(StdService):

    """Custom service that reads the solar database"""

    

    def __init__(self, engine, config_dict):

        # Pass the initialization information on to my superclass:

        super(GetSolar, self).__init__(engine, config_dict)

        

        self.solar_dict = weewx.manager.get_database_dict_from_config(config_dict, 'SBFspot_mysql')

        

        self.bind(weewx.NEW_ARCHIVE_RECORD, self.newArchiveRecord)

        

    def newArchiveRecord(self, event):

        

        connect = weedb.connect(self.solar_dict)

        try:

            cursor = connect.cursor()

            cursor.execute("SELECT PacTot FROM vwSpotData WHERE TimeStamp = (SELECT MAX(TimeStamp) FROM vwSpotData)")

            result = cursor.fetchone()

            event.record['radiation'] = result[0] if result is not None else None

            cursor.close()

        finally:

            connect.close()

In your weewx.conf, keep the changes you made, but add the service you just wrote (highlighted in yellow) to the list of services to be run

    [[Services]]

        # This section specifies the services that should be run. They are

        # grouped by type, and the order of services within each group

        # determines the order in which the services will be run.

        prep_services = weewx.engine.StdTimeSynch

        data_services = user.solar.GetSolar

        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, ... etc.

The new service GetSolar will put the SQL type vwSpotData into every weewx archive record, using key radiation. In turn, this will cause it to be put in the weewx database under the SQL type radiation, which, I think, was your original intent. Now you can use it like any other weewx type, although it may not have the right units. 

-tk

[Risto Virtanen]

Hi,

was away the whole day, now finally had time to try the solution you wrote. Copy-pasted it into user/solar.py, added the Service and restarted. Sadly no success:( 

syslog:

Feb 27 19:39:31 hanuri weewx[22337]: engine: Initializing weewx version 3.1.0

Feb 27 19:39:31 hanuri weewx[22337]: engine: Using Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40) #012[GCC 4.4.5]

Feb 27 19:39:31 hanuri weewx[22337]: engine: pid file is /var/run/weewx.pid

Feb 27 19:39:32 hanuri weewx[22339]: engine: Using configuration file /etc/weewx/weewx.conf

Feb 27 19:39:32 hanuri weewx[22339]: engine: Loading station type FineOffsetUSB (weewx.drivers.fousb)

Feb 27 19:39:32 hanuri weewx[22339]: fousb: driver version is 1.7

Feb 27 19:39:32 hanuri weewx[22339]: fousb: polling mode is PERIODIC

Feb 27 19:39:32 hanuri weewx[22339]: fousb: polling interval is 60

Feb 27 19:39:32 hanuri weewx[22339]: fousb: found station on USB bus=001 device=054

Feb 27 19:39:32 hanuri weewx[22339]: engine: Caught unrecoverable exception in engine:

Feb 27 19:39:32 hanuri weewx[22339]:     ****  'module' object has no attribute 'get_database_dict_from_config'

Feb 27 19:39:32 hanuri weewx[22339]:     ****  Traceback (most recent call last):

Feb 27 19:39:32 hanuri weewx[22339]:     ****    File "/usr/share/weewx/weewx/engine.py", line 831, in main

Feb 27 19:39:32 hanuri weewx[22339]:     ****      engine = EngineClass(config_dict)

Feb 27 19:39:32 hanuri weewx[22339]:     ****    File "/usr/share/weewx/weewx/engine.py", line 77, in __init__

Feb 27 19:39:32 hanuri weewx[22339]:     ****      self.loadServices(config_dict)

Feb 27 19:39:32 hanuri weewx[22339]:     ****    File "/usr/share/weewx/weewx/engine.py", line 141, in loadServices

Feb 27 19:39:32 hanuri weewx[22339]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

Feb 27 19:39:32 hanuri weewx[22339]:     ****    File "/usr/share/weewx/user/solar.py", line 12, in __init__

Feb 27 19:39:32 hanuri weewx[22339]:     ****      self.solar_dict = weewx.manager.get_database_dict_from_config(config_dict, 'SBFspot_mysql')

Feb 27 19:39:32 hanuri weewx[22339]:     ****  AttributeError: 'module' object has no attribute 'get_database_dict_from_config'

Feb 27 19:39:32 hanuri weewx[22339]:     ****  Exiting.

I browsed through the file manager.py and did not find the string 'get_database_dict_from_config' anywhere. Maybe I have too old a version (this is 3.10)? Could the same module be phrased some other way?

I also read once more the configuration guide but the examples there were a bit different

//rkv

[Thomas Keffer]

The function was introduced last summer, in v3.2.

You could just hardwire in the database dictionary. Replace

    self.solar_dict = weewx.manager.get_database_dict_from_config(config_dict, 'SBFspot_mysql')

with

    self.solar_dict = {'database_name' : 'SBFspot', 'user' : yourusername, 'password' : yourpassword, 'driver': 'weedb.mysql'}

-tk

--

[Risto Virtanen]

lauantai 27. helmikuuta 2016 20.44.03 UTC+2 Tom Keffer kirjoitti:

The function was introduced last summer, in v3.2.

Ok. Haven't upgraded as I've tried to follow the rule "If it works, don't touch"

You could just hardwire in the database dictionary. Replace

with

    self.solar_dict = {'database_name' : 'SBFspot', 'user' : yourusername, 'password' : yourpassword, 'driver': 'weedb.mysql'}

Tried that, but the error about usUnits re-appeared:

Feb 28 11:16:09 hanuri weewx[14836]: cheetahgenerator: Generate failed with exception '<class 'weedb.OperationalError'>'

Feb 28 11:16:09 hanuri weewx[14836]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Rouhunsaa/index.html.tmpl

Feb 28 11:16:09 hanuri weewx[14836]: cheetahgenerator: **** Reason: (1054, "Unknown column 'usUnits' in 'field list'")

Feb 28 11:16:09 hanuri weewx[14836]: ****  Traceback (most recent call last):

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 296, in generate

Feb 28 11:16:09 hanuri weewx[14836]: ****      print >> _file, text

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/lib/pymodules/python2.6/Cheetah/Template.py", line 1005, in __str__

Feb 28 11:16:09 hanuri weewx[14836]: ****      rc = getattr(self, mainMethName)()

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 427, in respond

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "_etc_weewx_skins_Rouhunsaa_index_html_tmpl.py", line 189, in __errorCatcher18

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "<string>", line 1, in <module>

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/tags.py", line 65, in latest

Feb 28 11:16:09 hanuri weewx[14836]: ****      manager = self.db_lookup(data_binding)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 846, in db_lookup

Feb 28 11:16:09 hanuri weewx[14836]: ****      return self.get_manager(data_binding)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 833, in get_manager

Feb 28 11:16:09 hanuri weewx[14836]: ****      self.manager_cache[data_binding] = open_manager(manager_dict, initialize)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 918, in open_manager

Feb 28 11:16:09 hanuri weewx[14836]: ****      manager_dict['table_name'])

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 103, in open

Feb 28 11:16:09 hanuri weewx[14836]: ****      dbmanager = cls(connection, table_name)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 85, in __init__

Feb 28 11:16:09 hanuri weewx[14836]: ****      self._sync()

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 191, in _sync

Feb 28 11:16:09 hanuri weewx[14836]: ****      _row = self.getSql("SELECT usUnits FROM %s LIMIT 1;" % self.table_name)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weewx/manager.py", line 380, in getSql

Feb 28 11:16:09 hanuri weewx[14836]: ****      _cursor.execute(sql, sqlargs)

Feb 28 11:16:09 hanuri weewx[14836]: ****    File "/usr/share/weewx/weedb/mysql.py", line 30, in guarded_fn

Feb 28 11:16:09 hanuri weewx[14836]: ****      raise weedb.OperationalError(e)

Feb 28 11:16:09 hanuri weewx[14836]: ****  OperationalError: (1054, "Unknown column 'usUnits' in 'field list'")

Then I temporarily modified the database name to the binding name in weewx.conf (SBFspot -> SBFspot_mysql), and it surely tries to connect to the db but of course fails:

Feb 28 11:20:03 hanuri weewx[16020]: engine: Caught database OperationalError: (1044, "Access denied for user 'kerttu'@'localhost' to database 'SBFspot_mysql'") while opening database 'SBFspot_mysql'

Would it suffice if I add a column 'usUnits' into the SBFspot db table? Might of course break something on SBFspot side...

//rkv

[Risto Virtanen]

Hi,

did some tweaking and now it  seems to work, see: http://www.myrskyt.org/rouhunsaa/index.html

I added a table ('weesolar') into the database SBFspot with columns dateTime, usUnits and PacTot and let my shell script copy the values from table vwSpotData (could not alter it as it is a view table). Edited the user/solar.py to use this new table instead and now it does not complain anymore. Interestingly, the summary page seem to have worked already before.

Anyway, the index.html.tmpl now contains a block:

              #if $day.radiation.has_data

              <tr>

                <td class="stats_label">Radiation</td>

                <td class="stats_data">$current.radiation</td>

              </tr>

              <tr>

                <td class="stats_label">Solar Power</td>

                <td class="stats_data">$latest($data_binding='sma_binding').PacTot</td>

              </tr>

              #end if

and the solar power data is now appearing twice on the page. I'll clean up that soon, but it shows that now BOTH ways of querying the database work.

Anyway the problem is solved. Like others on this group have already pointed out: fantastic software, awesome support.

Thanks!

//rkv

[Liz]

On Fri, 26 Feb 2016 02:18:48 -0800 (PST)
Risto Virtanen <ris...@gmail.com> wrote:

> Recently I had a solar power system installed and in order to monitor
> the production I soon installed the SBFspot software that gathers the
> data from the SMA inverter and stores it into a mysql database. Quite
> soon I decided that it would be nice to have the actual production
> power visible also on the weather station page
> (http://www.myrskyt.org/rouhunsaa/index.html)

I see you have Low Radiation on your dailystats page
http://www.myrskyt.org/rouhunsaa/dailystats.html
I removed Low Radiation because here it is always zero.

Liz

[Risto Virtanen]

maanantai 29. helmikuuta 2016 11.47.55 UTC+2 Liz kirjoitti:

I see you have Low Radiation on your dailystats page
http://www.myrskyt.org/rouhunsaa/dailystats.html
I removed Low Radiation because here it is always zero.

So far I've not touched the summaries page so much, but it is true, that could be removed. OTOH, during midsummer there just might be a nonzero value (I'm on 60 degrees N),  which remains to be seen:)

//rkv

 

[Liz]

When I removed mine, I wasn't able to see a use for that line at all.
When I looked at your location, I realised that it might have a use in
certain parts of the world.

Liz