weewx on WRT router running TomatoUSB

[Nigel Brownjohn]

OK,  I've tried to glue this together without asking questions, but I'm falling at the last hurdle!

Have a Netgear WNR3500L/U/v2 running Tomato Firmware v1.28.7505 MIPSR2Toastman-RT K26 USB Ext.

Installed python via optware to a USB key, after trying various I have Python25, py25-usb, py25-cheetah and py25-configobj with relevant dependencies.

Could not find a python-imaging for this rev, but saw it was likely not needed as all I want to do is push data from Acurite 5-in-1 to Wunderground.

(Thanks to those who recently produced the Acurite.py driver).

Finally got everything to build and apparently happy.

Start with: weewxd weewx.conf

Get:

Error: Traceback (most recent call last):

  File "./bin/weewxd", line 61, in <module>

    weewx.engine.main(options, args)

  File "/opt/apps/weewx/bin/weewx/engine.py", line 834, in main

    engine = EngineClass(config_dict)

  File "/opt/apps/weewx/bin/weewx/engine.py", line 74, in __init__

    self.loadServices(config_dict)

  File "/opt/apps/weewx/bin/weewx/engine.py", line 140, in loadServices

    self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))

  File "/opt/apps/weewx/bin/weewx/engine.py", line 503, in __init__

    self.setup_database(config_dict)

  File "/opt/apps/weewx/bin/weewx/engine.py", line 601, in setup_database

    dbmanager = self.engine.db_binder.get_manager(self.data_binding, initialize=True)

  File "/opt/apps/weewx/bin/weewx/manager.py", line 833, in get_manager

    self.manager_cache[data_binding] = open_manager(manager_dict, initialize)

  File "/opt/apps/weewx/bin/weewx/manager.py", line 915, in open_manager

    manager_dict['schema'])

  File "/opt/apps/weewx/bin/weewx/manager.py", line 126, in open_with_create

    connection = weedb.connect(database_dict)

  File "/opt/apps/weewx/bin/weedb/__init__.py", line 49, in connect

    __import__(db_dict['driver'])

  File "/opt/apps/weewx/bin/weedb/sqlite.py", line 19, in <module>

    from pysqlite2 import dbapi2 as sqlite3  #@Reimport @UnresolvedImport

ImportError: No module named pysqlite2

So it's falling back to attempt to import from pysqlite2 - which is not installed.  Question is why?

Do I need to get pysqlite2 installed also, or is this a configuration issue with sqlite3 ?

Tried stopping it from dropping sqlite3, and I then get this error in the syslog:

weewx[23092]: engine: Caught database OperationalError: cannot start a transaction within a transaction

So close, can anyone who has actually got this working let me know what I'm missing - or give me any pointers to how they did this differently?

Thanks to everyone contributing to the project - looking forward to getting it running.

Nigel

[Thomas Keffer]

Looks like your version of Python did not come with module sqlite3 (pysqlite2 was its name before it became part of the Python standard library). You'll have to install it.

You may have to install sqlite itself as well.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nigel Brownjohn]

Tom, thanks for the thoughts. sqlite3 is present, hence the second half of my message - if I force weedb to use sqlite3 I get the other error.

Not keen to install yet another package in a resource tight environment, if I can avoid it.

[Thomas Keffer]

Sorry. I jumped to conclusions.

But, let's unpack this a bit.

The fallback occurred because your version of sqlite3 does not support the "with" statement, used extensively in weewx. So, weewx went looking for an alternative, and tried pysqlite2, which wasn't present.

Let's test whether your version of sqlite3 supports the "with statement" or not. Try running this:

from __future__ import with_statement

import sqlite3

with sqlite3.connect(":memory:") as conn:

    conn.execute("CREATE TABLE foo(i)")

See if that executes without error. If not, you'll have to find a more modern version of sqlite3, one that supports "with" statements.

-tk

[mwall]

On Saturday, January 31, 2015 at 9:41:26 AM UTC-5, Tom Keffer wrote:

Looks like your version of Python did not come with module sqlite3 (pysqlite2 was its name before it became part of the Python standard library). You'll have to install it.

tom,

i think there is a bigger issue here that we should address.

i think we should make it possible to run weewx without the StdArchive service.  or perhaps i should say with a StdArchive service that has no database dependencies.

there is a use case where you run weewx with no local storage - use it just to collect data then send the data on to twitter or wunderground or whatever.

then you could have a minimal weewx configuration that has no image or database dependencies.

what do you think?

m

[Thomas Keffer]

In general it could be done, but some uploaders require specialized values, such as rain-since-midnight, that would be tough to do without a database.

What's so bad about running sqlite? Lots of O/S applications use it.

-tk

--

[mwall]

On Saturday, January 31, 2015 at 10:50:01 AM UTC-5, Tom Keffer wrote:

In general it could be done, but some uploaders require specialized values, such as rain-since-midnight, that would be tough to do without a database.

What's so bad about running sqlite? Lots of O/S applications use it.

nothing wrong with sqlite, and if you have the resources it is wonderful for keeping a local copy of the data.

there is a use case where you want a completely stripped-down weewx installation - no database, no images, no cheetah.  in that case all weewx does is get the data from the device then send it on.  in that case restful services that do rain-since-midnight and other db queries would send all of the other data except for those.  (btw, services that *require* the client to do rain-since-midnight and other aggregation are broken, imho.  of course we'll work with them, but by requiring those calculations they are introducing a vector for data inconsistency).

sometimes weewx acts is a data collector, sometimes does store-and-forward, sometimes it is a complete reporting tool.

isolating those functions might help us keep the codebase cleaner.

m

[Nigel Brownjohn]

Tom,

I assume this confirms your theory on 'with', even though the error message isn't specifically about 'with'?

with sqlite3.connect(":memory:") as conn:

AttributeError: 'sqlite3.Connection' object has no attribute '__exit__'

Version of sqlite is:   3.8.1

Will investigate further ... there's a 2.6 version of Python, which might have later sqlite, but I had other issues with it.

Thanks for the replies.

Nige ...

[Nigel Brownjohn]

Personally this would be very useful to me for the current use case, which is attaching a cheap sensor to Wunderground without having to run a PC.  

Router is running anyway, and has *some* spare capacity, but I am not looking to run the fully blown weewx on it, if I did want that I'd be writing to a network device and running the rest of weewx on a higher spec. machine.

Would be good to be modular.

Love the product design so far, clean, nice install, as you say this would be even cleaner - but right now I am just looking to get it working as I know others have.  Once it's working I can determine whether I can afford the resources on the router on an on-going basis.

Thanks for all the hard work on this project.

Nigel

[Thomas Keffer]

Yes it does. Sorry, buddy, but you're going to have to find yourself a more recent version of sqlite3!

Be careful you don't confuse the version of sqlite with the version of its Python interface. The former is given by attribute sqlite3.sqlite_version, the latter by sqlite3.version.

Just for reference, some version info from copies of sqlite I have lying around the house:

Python version | sqlite version | sqlite3 version

    V2.5.6           3.8.2           2.3.2

    V2.6.5           3.6.22          2.4.1

    V2.7.6           3.8.2           2.6.0

-tk

[Nigel Brownjohn]

Tom,

Thanks for the pointer.  For others hitting this thread, CTE and WITH were added to SQLITE in version 3.8.3.  I'll go see if I can find that for mips, or whether I'm now going to have to cross-compile for WRT.

Nigel

[Thomas Keffer]

Are you sure? Again, you're confusing the version of sqlite with the version of its Python interface. The latter is frequently called "PySqlite," and is only up to version 2.6.3. This is what will have "with" statement. 

-tk

[Nigel Brownjohn]

Tom you've been at this longer than me, I am sure you are right.  I took this from the sqlite changelog, which listed version 3.8.3 as the point they introduced WITH.  Correlates to sqlite3.sqlite_version rather than sqlite3.version I guess.

I don't know.  Am looking at the Shibby version of Tomato which was built later in 2014, vs. the 2013 version of Toasterman which I am using ... it claims sqlite V3.8.6 (which you are saying is the version of Python - I hear what you are saying).

We'll see.  Again, thanks for the pointers and clarifications.  If/when I get it working, I'll post the firmware version in use, and the various packages (I am ipkg loading sqlite independent of Python).

Nige ...