Python 3 and the future of pywws
449 views
Skip to first unread message
Jim Easterbrook
Apr 4, 2018, 5:32:16 AM4/4/18
to py...@googlegroups.com
Friends,
As you probably know Python 2 is on the way out and Python 3 is the way
of the future. Up to now pywws has supported both versions by using the
"2to3" tool when installing with Python 3. This is not very satisfactory
as it slows down installation and stops pywws taking advantage of new
language features.
I've created a new branch on GitHub (called "python3") where I've made
the minimum changes to make pywws work directly with Python 3. It's
still compatible with Python 2.7, and possibly 2.6, but almost certainly
not 2.5. I'm running this now and it seems to work OK, but more testers
are welcome. You can easily switch back to the "master" branch if you
run into problems.
I'll make a few more sweeping changes later, such as rationalising the
module names a bit, before making the "python3" branch the "master"
branch and renaming the old "master" branch to "legacy" or similar. At
that time anyone importing pywws into other Python programs will
probably need to make some changes.
--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>
As you probably know Python 2 is on the way out and Python 3 is the way
of the future. Up to now pywws has supported both versions by using the
"2to3" tool when installing with Python 3. This is not very satisfactory
as it slows down installation and stops pywws taking advantage of new
language features.
I've created a new branch on GitHub (called "python3") where I've made
the minimum changes to make pywws work directly with Python 3. It's
still compatible with Python 2.7, and possibly 2.6, but almost certainly
not 2.5. I'm running this now and it seems to work OK, but more testers
are welcome. You can easily switch back to the "master" branch if you
run into problems.
I'll make a few more sweeping changes later, such as rationalising the
module names a bit, before making the "python3" branch the "master"
branch and renaming the old "master" branch to "legacy" or similar. At
that time anyone importing pywws into other Python programs will
probably need to make some changes.
--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>
Jim Easterbrook
Apr 5, 2018, 4:34:13 AM4/5/18
to py...@googlegroups.com
On 04/04/18 10:32, Jim Easterbrook wrote:
>
> I'll make a few more sweeping changes later, such as rationalising the
> module names a bit, before making the "python3" branch the "master"
> branch and renaming the old "master" branch to "legacy" or similar. At
> that time anyone importing pywws into other Python programs will
> probably need to make some changes.
I think "rationalising the module names" should be a more thorough
>
> I'll make a few more sweeping changes later, such as rationalising the
> module names a bit, before making the "python3" branch the "master"
> branch and renaming the old "master" branch to "legacy" or similar. At
> that time anyone importing pywws into other Python programs will
> probably need to make some changes.
exercise. The current pywws code is very inconsistent in its use of
lower case, upper case, camel case and so on. I'm going to create a new
branch called "pep8" and try and make everything in it consistent with
the Python style guide: https://www.python.org/dev/peps/pep-0008/
Some of the pywws code is ten years old now. Looking at it now I'm
amazed how little Python I knew ten years ago. I really didn't
appreciate the value of classes and inheritance. Look out for some major
rewrites in the near future!
Jim Easterbrook
Apr 6, 2018, 10:57:28 AM4/6/18
to py...@googlegroups.com
On 05/04/18 09:34, Jim Easterbrook wrote:
> On 04/04/18 10:32, Jim Easterbrook wrote:
>>
>> I'll make a few more sweeping changes later, such as rationalising the
>> module names a bit, before making the "python3" branch the "master"
>> branch and renaming the old "master" branch to "legacy" or similar. At
>> that time anyone importing pywws into other Python programs will
>> probably need to make some changes.
>
> I think "rationalising the module names" should be a more thorough
> exercise. The current pywws code is very inconsistent in its use of
> lower case, upper case, camel case and so on. I'm going to create a new
> branch called "pep8" and try and make everything in it consistent with
> the Python style guide: https://www.python.org/dev/peps/pep-0008/
OK, the pep8 branch has been created, heavily modified, and now merged
> On 04/04/18 10:32, Jim Easterbrook wrote:
>>
>> I'll make a few more sweeping changes later, such as rationalising the
>> module names a bit, before making the "python3" branch the "master"
>> branch and renaming the old "master" branch to "legacy" or similar. At
>> that time anyone importing pywws into other Python programs will
>> probably need to make some changes.
>
> I think "rationalising the module names" should be a more thorough
> exercise. The current pywws code is very inconsistent in its use of
> lower case, upper case, camel case and so on. I'm going to create a new
> branch called "pep8" and try and make everything in it consistent with
> the Python style guide: https://www.python.org/dev/peps/pep-0008/
back in to the python3 branch. This branch now has many renamed modules,
classes and functions, but appears to be working OK in my (not
comprehensive) tests.
Jim Easterbrook
Apr 10, 2018, 2:44:02 AM4/10/18
to py...@googlegroups.com
"master" branch. This branch is no longer compatible with Python 2.6 or
earlier.
The next step is to rewrite the "service" uploaders. They're currently a
complete mess and I've got some ideas for making them easier to maintain
and still able to support a wide range of upload protocols. This work
will be done on the "newservices" branch.
Jim Easterbrook
Apr 10, 2018, 6:31:38 AM4/10/18
to py...@googlegroups.com
On 10/04/18 07:44, Jim Easterbrook wrote:
>
> The next step is to rewrite the "service" uploaders. They're currently a
> complete mess and I've got some ideas for making them easier to maintain
> and still able to support a wide range of upload protocols. This work
> will be done on the "newservices" branch.
I've checked in three of the new "service" uploaders - underground,
>
> The next step is to rewrite the "service" uploaders. They're currently a
> complete mess and I've got some ideas for making them easier to maintain
> and still able to support a wide range of upload protocols. This work
> will be done on the "newservices" branch.
metoffice and cwop. Using Python modules (one for each service) makes it
a lot easier to cope with oddities such as using Weather Underground's
"rapid fire" server for live data. You'll no longer need to use
'underground_rf' in your weather.ini. Similarly the cwop uploader
detects if you have a "ham" ID or not. Each uploader runs in its own
thread, so one slow server won't delay other services' uploads.
The disadvantage of this new system is that you can't supply your own
template to control what data is uploaded. As pywws's reputation might
be affected by malformed data this may be no bad thing, but I know some
of you are using custom templates. Is anyone doing this for any service
other than MQTT? I believe that's the only one where you really need a
high degree of customisation. (Unless you MQTT users can put your heads
together and agree a common specification...)
Tony Gilbert
Apr 10, 2018, 3:57:45 PM4/10/18
to py...@googlegroups.com
I can't easily check at the moment but I seem to recall that I am using a custom template to include Inside Temp on wunderground.
Cheers
Tony
--
You received this message because you are subscribed to the Google Groups "pywws" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pywws+un...@googlegroups.com.
To post to this group, send an email to py...@googlegroups.com.
Visit this group at https://groups.google.com/group/pywws.
For more options, visit https://groups.google.com/d/optout.
Jim Easterbrook
Apr 10, 2018, 5:28:14 PM4/10/18
to py...@googlegroups.com
On 10/04/18 20:57, Tony Gilbert wrote:
> I can't easily check at the moment but I seem to recall that I am using
> a custom template to include Inside Temp on wunderground.
That's one of the features I've added - a simple True/False setting in
> I can't easily check at the moment but I seem to recall that I am using
> a custom template to include Inside Temp on wunderground.
weather.ini.
Jim Easterbrook
Apr 11, 2018, 8:34:31 AM4/11/18
to py...@googlegroups.com
On 10/04/18 11:31, Jim Easterbrook wrote:
>
> The disadvantage of this new system is that you can't supply your own
> template to control what data is uploaded. As pywws's reputation might
> be affected by malformed data this may be no bad thing, but I know some
> of you are using custom templates. Is anyone doing this for any service
> other than MQTT? I believe that's the only one where you really need a
> high degree of customisation. (Unless you MQTT users can put your heads
> together and agree a common specification...)
I've created a new mqtt service uploader that gets its template from
>
> The disadvantage of this new system is that you can't supply your own
> template to control what data is uploaded. As pywws's reputation might
> be affected by malformed data this may be no bad thing, but I know some
> of you are using custom templates. Is anyone doing this for any service
> other than MQTT? I believe that's the only one where you really need a
> high degree of customisation. (Unless you MQTT users can put your heads
> together and agree a common specification...)
weather.ini. This allows the user to adjust it to suit their needs, but
needs to be done with care as there are a lot of escaped quote marks and
so on.
I'd really like some MQTT users to test this (and anyone to test my
other new uploaders). You'll need to have cloned pywws from GitHub
though. Here's how you switch to the test branch:
git fetch origin
git checkout --track origin/newservices
python setup.py build
sudo python setup.py install
To switch back to the "master" branch you just 'git checkout master' and
then compile and install again. If you want to go back to the last
release branch, do 'git checkout --track origin/legacy'.
Jim Easterbrook
Apr 14, 2018, 5:05:40 AM4/14/18
to py...@googlegroups.com
I'm now having a thorough rummage through pywws's attempts to use local
time zones. It's a complete mess and I'm making some interesting
discoveries. "rain since midnight" gets interesting when the clocks
change to and from daylight saving time.
I also plan to allow the "day end hour" to be DST aware (optionally).
time zones. It's a complete mess and I'm making some interesting
discoveries. "rain since midnight" gets interesting when the clocks
change to and from daylight saving time.
I also plan to allow the "day end hour" to be DST aware (optionally).
Richard Truran
Apr 15, 2018, 4:25:09 AM4/15/18
to pywws
Dear Jim,
I have a spare Raspberry Pi, so I’ll be setting it up with a fresh install of Raspian and can switch to the github development branch following your instructions.
When I do this do any settings in the weather.ini file need to change or can I simply use the same file that is currently live?
As my weather station records solar measurements I have switched to running pywws.hourly every 10 minutes by a cron job rather than running as a daemon. Do you think there’s any chance your rationalisation of code could help with the lock ups?
It might take a week or two for to get this up and running, I’ll provide feedback when it’s up and running.
R
I have a spare Raspberry Pi, so I’ll be setting it up with a fresh install of Raspian and can switch to the github development branch following your instructions.
When I do this do any settings in the weather.ini file need to change or can I simply use the same file that is currently live?
As my weather station records solar measurements I have switched to running pywws.hourly every 10 minutes by a cron job rather than running as a daemon. Do you think there’s any chance your rationalisation of code could help with the lock ups?
It might take a week or two for to get this up and running, I’ll provide feedback when it’s up and running.
R
Jim Easterbrook
Apr 15, 2018, 5:08:08 AM4/15/18
to py...@googlegroups.com
On 15/04/18 09:25, Richard Truran wrote:
>
> I have a spare Raspberry Pi, so I’ll be setting it up with a fresh
> install of Raspian and can switch to the github development branch
> following your instructions.
>
> When I do this do any settings in the weather.ini file need to change
> or can I simply use the same file that is currently live?
You can use a copy of the weather.ini file. Forwards compatibility is
>
> I have a spare Raspberry Pi, so I’ll be setting it up with a fresh
> install of Raspian and can switch to the github development branch
> following your instructions.
>
> When I do this do any settings in the weather.ini file need to change
> or can I simply use the same file that is currently live?
guaranteed, backwards compatibility may not be.
> As my weather station records solar measurements I have switched to
> running pywws.hourly every 10 minutes by a cron job rather than
> running as a daemon. Do you think there’s any chance your
> rationalisation of code could help with the lock ups?
you get lockups I can't be sure. If it's a similar problem to the
"normal" lockups then I guess a third avoidance strategy is needed, to
synchronise to and then avoid times when the solar data arrives. As I
don't have a solar station (and given their problems I'm not sure I want
one!) this would be hard to develop.
I don't know if running 'pywws-testweatherstation -l -vv' for ten
minutes would show when the solar data changes happen relative to the
live and logged data. It would probably crash your station though.
Richard Truran
Apr 15, 2018, 10:41:40 AM4/15/18
to pywws
Dear Jim,
I managed to setup the spare Raspberry Pi quicker than I thought it would be possible, unfortunately I've run into a problem:
Here is the log from pywws-hourly
15:38:09:pywws.logger:pywws version 18.4.2, build 1488 (a8cf20c)
15:38:09:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:38:10:pywws.weatherstation:using pywws.device_ctypes_hidapi
15:38:11:pywws.logdata:Synchronising to weather station
15:38:12:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:38:12:pywws.weatherstation:live_data log extended
15:38:53:pywws.logdata:Fetching data
15:39:05:pywws.logdata:1 catchup records
15:39:05:pywws.process:Generating summary data
15:39:05:pywws.calib:Using user calibration
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 66, in h
ourly
pywws.process.process_data(context)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/process.py", line 747, in
process_data
start = calibrate_data(context.params, context.raw_data, context.calib_data)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/process.py", line 530, in
calibrate_data
calibrator = Calib(params, raw_data)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/calib.py", line 133, in _
_init__
module, globals(), locals(), ['Calib'])
File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 22, in <module>
from pywws.Process import SECOND
ImportError: No module named Process
15:38:09:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:38:10:pywws.weatherstation:using pywws.device_ctypes_hidapi
15:38:11:pywws.logdata:Synchronising to weather station
15:38:12:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:38:12:pywws.weatherstation:live_data log extended
15:38:53:pywws.logdata:Fetching data
15:39:05:pywws.logdata:1 catchup records
15:39:05:pywws.process:Generating summary data
15:39:05:pywws.calib:Using user calibration
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 66, in h
ourly
pywws.process.process_data(context)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/process.py", line 747, in
process_data
start = calibrate_data(context.params, context.raw_data, context.calib_data)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/process.py", line 530, in
calibrate_data
calibrator = Calib(params, raw_data)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/calib.py", line 133, in _
_init__
module, globals(), locals(), ['Calib'])
File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 22, in <module>
from pywws.Process import SECOND
ImportError: No module named Process
Would you be able to help?
Thanks
Richard
Richard Truran
Apr 15, 2018, 10:50:45 AM4/15/18
to pywws
Dear Jim,
I fixed the previous pywws.Process error by updating my user_calib file to update the reference to pywws.process, however I now have a new error.
5:46:54:pywws.logger:pywws version 18.4.2, build 1488 (a8cf20c)
15:46:54:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:46:54:pywws.weatherstation:using pywws.device_libusb1
15:47:05:pywws.logdata:Synchronising to weather station
15:47:41:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:47:41:pywws.logdata:Fetching data
15:47:42:pywws.process:Generating summary data
15:47:42:pywws.calib:Using user calibration
15:46:54:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:46:54:pywws.weatherstation:using pywws.device_libusb1
15:47:05:pywws.logdata:Synchronising to weather station
15:47:41:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:47:41:pywws.logdata:Fetching data
15:47:42:pywws.process:Generating summary data
15:47:42:pywws.calib:Using user calibration
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 68, in h
ourly
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py", line 79
, in __init__
import croniter
ImportError: No module named croniter
ourly
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py", line 79
, in __init__
import croniter
ImportError: No module named croniter
I've confirmed I have croniter installed from pip.
Thanks for your help.
R
Jim Easterbrook
Apr 15, 2018, 10:52:07 AM4/15/18
to py...@googlegroups.com
On 15/04/18 15:41, Richard Truran wrote:
>
> I managed to setup the spare Raspberry Pi quicker than I thought it
> would be possible, unfortunately I've run into a problem:
>
> Here is the log from pywws-hourly
[snip]
>
> I managed to setup the spare Raspberry Pi quicker than I thought it
> would be possible, unfortunately I've run into a problem:
>
> Here is the log from pywws-hourly
> File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 22, in
> <module>
> from pywws.Process import SECOND
> ImportError: No module named Process
renamed modules. pywws.Process is now called pywws.process, as Python
modules should have lower case names.
Jim Easterbrook
Apr 15, 2018, 10:54:16 AM4/15/18
to py...@googlegroups.com
On 15/04/18 15:50, Richard Truran wrote:
>
> I fixed the previous pywws.Process error by updating my user_calib file
> to update the reference to pywws.process, however I now have a new error.
>
>
> I fixed the previous pywws.Process error by updating my user_calib file
> to update the reference to pywws.process, however I now have a new error.
>
> File
> "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py",
> line 79
> , in __init__
> import croniter
> ImportError: No module named croniter
>
> I've confirmed I have croniter installed from pip.
Have you run a Python shell and tried "import croniter"?
> "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py",
> line 79
> , in __init__
> import croniter
> ImportError: No module named croniter
>
> I've confirmed I have croniter installed from pip.
Richard Truran
Apr 15, 2018, 11:00:19 AM4/15/18
to pywws
Dear Jim,
OK, so that fails, so ran sudo pip install croniter, which has solved the error. I think we are making progress, I now get the following error:
15:57:48:pywws.logger:pywws version 18.4.2, build 1488 (a8cf20c)
15:57:48:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:57:48:pywws.weatherstation:using pywws.device_libusb1
15:57:50:pywws.logdata:Synchronising to weather station
15:58:05:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:58:05:pywws.logdata:Fetching data
15:58:06:pywws.logdata:2 catchup records
15:58:06:pywws.process:Generating summary data
15:58:06:pywws.calib:Using user calibration
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:13:59 0:20:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:17:59 0:04:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:23:59 0:01:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:32:59 0:04:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:38:59 0:06:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:47:59 0:04:00
15:57:48:pywws.logger:Python version 2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
15:57:48:pywws.weatherstation:using pywws.device_libusb1
15:57:50:pywws.logdata:Synchronising to weather station
15:58:05:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
15:58:05:pywws.logdata:Fetching data
15:58:06:pywws.logdata:2 catchup records
15:58:06:pywws.process:Generating summary data
15:58:06:pywws.calib:Using user calibration
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:13:59 0:20:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:17:59 0:04:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:23:59 0:01:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:32:59 0:04:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:38:59 0:06:00
15:58:06:pywws.process:unexpected data interval 2018-04-15 14:47:59 0:04:00
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 68, in h
ourly
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/hourly.py", line 68, in h
ourly
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py", line 96
, in __init__
mod = importlib.import_module('pywws.service.' + name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/service/openweathermap.py
", line 32, in <module>
logger = logging.getLogger(__package__ + '.' + service_name)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
, in __init__
mod = importlib.import_module('pywws.service.' + name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/service/openweathermap.py
", line 32, in <module>
logger = logging.getLogger(__package__ + '.' + service_name)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
Something to do with the openweathermap uploader?
Thanks
R
Jim Easterbrook
Apr 15, 2018, 11:07:02 AM4/15/18
to py...@googlegroups.com
On 15/04/18 16:00, Richard Truran wrote:
>
> OK, so that fails, so ran sudo pip install croniter, which has solved
> the error. I think we are making progress, I now get the following error:
>
>
> OK, so that fails, so ran sudo pip install croniter, which has solved
> the error. I think we are making progress, I now get the following error:
>
> Traceback (most recent call last):
> File
> File
> "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/regulartasks.py",
> line 96
> , in __init__
> mod = importlib.import_module('pywws.service.' + name)
> File "/usr/lib/python2.7/importlib/__init__.py", line 37, in
> import_module
> __import__(name)
> File
> "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/service/openweathermap.py
> ", line 32, in <module>
> logger = logging.getLogger(__package__ + '.' + service_name)
> TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
>
> Something to do with the openweathermap uploader?
Looks like a Python 2 compatibility problem - I've not been testing
> line 96
> , in __init__
> mod = importlib.import_module('pywws.service.' + name)
> File "/usr/lib/python2.7/importlib/__init__.py", line 37, in
> import_module
> __import__(name)
> File
> "/usr/local/lib/python2.7/dist-packages/pywws-18.4.2-py2.7.egg/pywws/service/openweathermap.py
> ", line 32, in <module>
> logger = logging.getLogger(__package__ + '.' + service_name)
> TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
>
> Something to do with the openweathermap uploader?
thoroughly with Python 2 during this development phase. It should work
with Python 3 if you fancy trying that. I'll investigate further later
this week.
Richard Truran
Apr 15, 2018, 11:15:24 AM4/15/18
to pywws
Dear Jim,
I was just coming to the same conclusion and was compiling again using python3...
16:10:34:pywws.logger:pywws version 18.4.2, build 1488 (a8cf20c)
16:10:34:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
16:10:34:pywws.weatherstation:using pywws.device_libusb1
16:10:35:pywws.logdata:Synchronising to weather station
16:10:40:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
16:11:05:pywws.logdata:Fetching data
16:11:06:pywws.logdata:1 minutes gap in data detected
16:11:06:pywws.logdata:3 catchup records
16:11:06:pywws.process:Generating summary data
16:11:06:pywws.calib:Using user calibration
16:10:34:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
16:10:34:pywws.weatherstation:using pywws.device_libusb1
16:10:35:pywws.logdata:Synchronising to weather station
16:10:40:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
16:11:05:pywws.logdata:Fetching data
16:11:06:pywws.logdata:1 minutes gap in data detected
16:11:06:pywws.logdata:3 catchup records
16:11:06:pywws.process:Generating summary data
16:11:06:pywws.calib:Using user calibration
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 97, in m
ain
return hourly(args[0])
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 66, in h
ourly
pywws.process.process_data(context)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/process.py", line 747, in
ourly
pywws.process.process_data(context)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/process.py", line 747, in
process_data
start = calibrate_data(context.params, context.raw_data, context.calib_data)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/process.py", line 544, in
calibrate_data
calib_data[idx] = calibrator.calib(data)
File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 49, in calib
median = history[(len(history) - 1) / 2]
TypeError: list indices must be integers or slices, not float
calibrate_data
calib_data[idx] = calibrator.calib(data)
File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 49, in calib
median = history[(len(history) - 1) / 2]
TypeError: list indices must be integers or slices, not float
Which looks to be an issue with my user_calib file, any thoughts?
R
Jim Easterbrook
Apr 15, 2018, 11:21:12 AM4/15/18
to py...@googlegroups.com
On 15/04/18 16:15, Richard Truran wrote:
>
> I was just coming to the same conclusion and was compiling again using
> python3...
>
>
> I was just coming to the same conclusion and was compiling again using
> python3...
>
> File
> "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/process.py",
> line 544, in
> calibrate_data
> calib_data[idx] = calibrator.calib(data)
> File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 49, in
> calib
> median = history[(len(history) - 1) / 2]
> TypeError: list indices must be integers or slices, not float
>
> Which looks to be an issue with my user_calib file, any thoughts?
In Python 3 '/' is floating point division. If you want an integer
> "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/process.py",
> line 544, in
> calibrate_data
> calib_data[idx] = calibrator.calib(data)
> File "/home/pi/NAS-drive/weather/modules/user_calib.py", line 49, in
> calib
> median = history[(len(history) - 1) / 2]
> TypeError: list indices must be integers or slices, not float
>
> Which looks to be an issue with my user_calib file, any thoughts?
result use '//'.
Richard Truran
Apr 15, 2018, 11:40:06 AM4/15/18
to pywws
Dear Jim,
Thanks, I would have never figured that out. Working through a couple of dependancies which needed python3 versions installing. I now get this:
16:34:51:pywws.logger:pywws version 18.4.2, build 1488 (a8cf20c)
16:34:51:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
16:34:51:pywws.weatherstation:using pywws.device_libusb1
16:35:05:pywws.logdata:Synchronising to weather station
16:35:41:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
16:35:41:pywws.logdata:Fetching data
16:35:42:pywws.process:Generating summary data
16:35:42:pywws.calib:Using user calibration
16:35:43:pywws.regulartasks:doing task sections ['logged', 'hourly']
16:35:44:pywws.service.openweathermap:server response "{'message': '', 'id': 1451413658, 'cod': '200'}"
16:35:44:pywws.service.openweathermap:3 records sent
16:35:46:pywws.service.underground:server response "success"
16:35:46:pywws.regulartasks:Graphing today.png.xml
16:35:48:pywws.service.underground:9 records sent
16:35:56:pywws.service.metoffice:OK
16:35:56:pywws.service.metoffice:1 record sent
16:34:51:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
16:34:51:pywws.weatherstation:using pywws.device_libusb1
16:35:05:pywws.logdata:Synchronising to weather station
16:35:41:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
16:35:41:pywws.logdata:Fetching data
16:35:42:pywws.process:Generating summary data
16:35:42:pywws.calib:Using user calibration
16:35:43:pywws.regulartasks:doing task sections ['logged', 'hourly']
16:35:44:pywws.service.openweathermap:server response "{'message': '', 'id': 1451413658, 'cod': '200'}"
16:35:44:pywws.service.openweathermap:3 records sent
16:35:46:pywws.service.underground:server response "success"
16:35:46:pywws.regulartasks:Graphing today.png.xml
16:35:48:pywws.service.underground:9 records sent
16:35:56:pywws.service.metoffice:OK
16:35:56:pywws.service.metoffice:1 record sent
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 97, in main
return hourly(args[0])
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 97, in main
return hourly(args[0])
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 68, in hourly
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 251, in do_tasks
self._do_common(now, sections)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 165, in _do_common
plot_file = self.do_plot(template, local=local)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 273, in do_plot
self.plotter.do_plot(input_xml, output_file) == 0):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/plot.py", line 734, in do_plot
of.write(self.plot_data(plot_no, plot, source))
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/plot.py", line 911, in plot_data
value = eval(subplot.ycalc)
File "<string>", line 1, in <module>
AttributeError: 'GraphPlotter' object has no attribute 'raw_data'
pywws.regulartasks.RegularTasks(context).do_tasks()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 251, in do_tasks
self._do_common(now, sections)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 165, in _do_common
plot_file = self.do_plot(template, local=local)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/regulartasks.py", line 273, in do_plot
self.plotter.do_plot(input_xml, output_file) == 0):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/plot.py", line 734, in do_plot
of.write(self.plot_data(plot_no, plot, source))
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/plot.py", line 911, in plot_data
value = eval(subplot.ycalc)
File "<string>", line 1, in <module>
AttributeError: 'GraphPlotter' object has no attribute 'raw_data'
Thanks
R
Jim Easterbrook
Apr 15, 2018, 11:57:43 AM4/15/18
to py...@googlegroups.com
On 15/04/18 16:40, Richard Truran wrote:
>
> Thanks, I would have never figured that out. Working through a couple of
> dependancies which needed python3 versions installing. I now get this:
>
>
> Thanks, I would have never figured that out. Working through a couple of
> dependancies which needed python3 versions installing. I now get this:
>
> Traceback (most recent call last):
> File
> File
> "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/plot.py",
> line 911, in plot_data
> value = eval(subplot.ycalc)
> File "<string>", line 1, in <module>
> AttributeError: 'GraphPlotter' object has no attribute 'raw_data'
I guess one of your plot templates is doing something (possibly
> line 911, in plot_data
> value = eval(subplot.ycalc)
> File "<string>", line 1, in <module>
> AttributeError: 'GraphPlotter' object has no attribute 'raw_data'
undocumented) with raw_data.
Message has been deleted
Richard Truran
Apr 15, 2018, 12:51:06 PM4/15/18
to pywws
Dear Jim,
Yes, some of my templates use an undocumented feature you pointed out to me to calculate the amount of rain in the last hour:
Any chance this could become documented?
I also had issues using the weathercloud service as discussed here:
but I got round that by deleting the reference to it :)
R
Thanks
R
Jim Easterbrook
Apr 15, 2018, 1:17:09 PM4/15/18
to py...@googlegroups.com
On 15/04/18 17:51, Richard Truran wrote:
>
> Yes, some of my templates use an undocumented feature you pointed out to
> me to calculate the amount of rain in the last hour:
>
> https://groups.google.com/forum/#!topic/pywws/EKLeMnnRsDA
>
> Any chance this could become documented?
No, but if you change from raw_data to calib_data it should still work.
>
> Yes, some of my templates use an undocumented feature you pointed out to
> me to calculate the amount of rain in the last hour:
>
> https://groups.google.com/forum/#!topic/pywws/EKLeMnnRsDA
>
> Any chance this could become documented?
The better solution is to make the rain_hour function available in plots
- another thing for me to look at in the current rewriting spree.
> I also had issues using the weathercloud service as discussed here:
>
> https://groups.google.com/forum/#!searchin/pywws/weathercloud|sort:date/pywws/02vc-X0KwOQ/3GW_7vyUAwAJ
>
> but I got round that by deleting the reference to it :)
src/pywws/service
I'm wondering if I should add the user's "user module" directory to the
search path for service modules - it would make it easier for
individuals to write one-off specials for their own use.
Richard Truran
Apr 15, 2018, 2:00:13 PM4/15/18
to pywws
Dear Jim,
Thanks for the advice, I'll convert the raw_data references to calib_data and see how I get on.
I'll have a look at the services code and see if I can figure out how to adapt to fit with weather cloud.
R
Richard Truran
Apr 15, 2018, 2:38:33 PM4/15/18
to pywws
Dear Jim,
I think my graph templates are going to cause a bit of trouble, can you review the attached and let me know if there are any other undocumented functions which are removed or need adapting.
My suspicion is that the rain graphs are the ones causing the trouble.
Thanks
R
Jim Easterbrook
Apr 16, 2018, 5:58:45 AM4/16/18
to py...@googlegroups.com
On 15/04/18 19:51, Richard Truran wrote:
> Now with template and not output file...
>
> On Sunday, 15 April 2018 19:38:33 UTC+1, Richard Truran wrote:
>
> I think my graph templates are going to cause a bit of trouble, can
> you review the attached and let me know if there are any other
> undocumented functions which are removed or need adapting.
>
> My suspicion is that the rain graphs are the ones causing the trouble.
Once you change the self.raw_data it should work. But it's easier for
> Now with template and not output file...
>
> On Sunday, 15 April 2018 19:38:33 UTC+1, Richard Truran wrote:
>
> I think my graph templates are going to cause a bit of trouble, can
> you review the attached and let me know if there are any other
> undocumented functions which are removed or need adapting.
>
> My suspicion is that the rain graphs are the ones causing the trouble.
you just to try it than for me to process it in my head. As you're
already using "source" you could use that instead of "self.calib_data".
It's shorter and would be more consistent with your other calcs.
Jim Easterbrook
Apr 16, 2018, 8:35:46 AM4/16/18
to py...@googlegroups.com
On 15/04/18 18:17, Jim Easterbrook wrote:
> On 15/04/18 17:51, Richard Truran wrote:
>>
>> Yes, some of my templates use an undocumented feature you pointed out
>> to me to calculate the amount of rain in the last hour:
>>
>> https://groups.google.com/forum/#!topic/pywws/EKLeMnnRsDA
>>
>> Any chance this could become documented?
>
> No, but if you change from raw_data to calib_data it should still work.
> The better solution is to make the rain_hour function available in plots
> - another thing for me to look at in the current rewriting spree.
I've added the rain computation functions from pywws.template to
> On 15/04/18 17:51, Richard Truran wrote:
>>
>> Yes, some of my templates use an undocumented feature you pointed out
>> to me to calculate the amount of rain in the last hour:
>>
>> https://groups.google.com/forum/#!topic/pywws/EKLeMnnRsDA
>>
>> Any chance this could become documented?
>
> No, but if you change from raw_data to calib_data it should still work.
> The better solution is to make the rain_hour function available in plots
> - another thing for me to look at in the current rewriting spree.
pywws.plot now so you should be able to use rain_hour in your graph
calculations. (Do a 'git pull' to fetch the latest changes, then build
and install again.)
Richard Truran
Apr 16, 2018, 9:00:13 AM4/16/18
to pywws
Dear Jim,
Thanks I've harmonised on using source references as I had issues with calib_data. I'm currently back using the release version on me production Raspberry Pi, and all is working well with this syntax. I'll test with the python3 compiled development branch tonight.
R
Richard Truran
Apr 16, 2018, 3:07:56 PM4/16/18
to pywws
Der Jim,
I've updated the pywws3 development version and with the new templates using source references, it runs correctly and appears to succeed (that is it doesn't crash out):
19:55:03:pywws.logger:pywws version 18.4.2, build 1493 (2593383)
19:55:03:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
19:55:04:pywws.weatherstation:using pywws.device_libusb1
19:55:05:pywws.logdata:Synchronising to weather station
19:55:29:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
19:55:41:pywws.logdata:Fetching data
19:55:42:pywws.logdata:2 catchup records
19:55:42:pywws.process:Generating summary data
19:55:42:pywws.calib:Using user calibration
19:55:43:pywws.regulartasks:doing task sections ['logged']
19:55:44:pywws.regulartasks:Graphing today.png.xml
19:55:44:pywws.service.openweathermap:server response "{'message': '', 'cod': '200', 'id': 1451413658}"
19:55:44:pywws.service.metoffice:OK
19:55:44:pywws.service.openweathermap:2 records sent
19:55:44:pywws.service.metoffice:3 records sent
19:55:44:pywws.service.underground:server response "success"
19:55:45:pywws.service.underground:2 records sent
Warning: empty y range [0:0], adjusting to [0:1]
Warning: empty y2 range [0:0], adjusting to [0:1]
19:55:48:pywws.regulartasks:Graphing today_rose.png.xml
19:55:48:pywws.regulartasks:Templating 1hrs_lux.txt
19:55:48:pywws.regulartasks:Templating 6hrs_lux_freq.txt
19:55:49:pywws.regulartasks:Templating en_current.txt
19:55:49:pywws.regulartasks:Templating live_json.txt
19:55:49:pywws.regulartasks:Templating 1hr_json.txt
19:55:49:pywws.towebsite:Uploading to web site with FTP
19:55:50:pywws.towebsite:local variable 'OK' referenced before assignment
19:55:03:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124]
19:55:04:pywws.weatherstation:using pywws.device_libusb1
19:55:05:pywws.logdata:Synchronising to weather station
19:55:29:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
19:55:41:pywws.logdata:Fetching data
19:55:42:pywws.logdata:2 catchup records
19:55:42:pywws.process:Generating summary data
19:55:42:pywws.calib:Using user calibration
19:55:43:pywws.regulartasks:doing task sections ['logged']
19:55:44:pywws.regulartasks:Graphing today.png.xml
19:55:44:pywws.service.openweathermap:server response "{'message': '', 'cod': '200', 'id': 1451413658}"
19:55:44:pywws.service.metoffice:OK
19:55:44:pywws.service.openweathermap:2 records sent
19:55:44:pywws.service.metoffice:3 records sent
19:55:44:pywws.service.underground:server response "success"
19:55:45:pywws.service.underground:2 records sent
Warning: empty y range [0:0], adjusting to [0:1]
Warning: empty y2 range [0:0], adjusting to [0:1]
19:55:48:pywws.regulartasks:Graphing today_rose.png.xml
19:55:48:pywws.regulartasks:Templating 1hrs_lux.txt
19:55:48:pywws.regulartasks:Templating 6hrs_lux_freq.txt
19:55:49:pywws.regulartasks:Templating en_current.txt
19:55:49:pywws.regulartasks:Templating live_json.txt
19:55:49:pywws.regulartasks:Templating 1hr_json.txt
19:55:49:pywws.towebsite:Uploading to web site with FTP
19:55:50:pywws.towebsite:local variable 'OK' referenced before assignment
I'm not sure if the last line is good or bad.
However it appears the files for ftp uploading are not being sent, the files for upload are in the work directory as indicated by the weather.ini, however there is also a directory called upload in there which is empty. Should the files for ftp upload go in the 'upload' directory.
I have used the ftp command to log into the remote server and it works correctly.
Thanks
Richard
Jim Easterbrook
Apr 16, 2018, 4:01:19 PM4/16/18
to py...@googlegroups.com
On 16/04/18 20:07, Richard Truran wrote:
>
> I've updated the pywws3 development version and with the new templates
> using source references, it runs correctly and appears to succeed (that
> is it doesn't crash out):
>
> 19:55:03:pywws.logger:pywws version 18.4.2, build 1493 (2593383)
Good, that's the latest.
>
> I've updated the pywws3 development version and with the new templates
> using source references, it runs correctly and appears to succeed (that
> is it doesn't crash out):
>
> 19:55:03:pywws.logger:pywws version 18.4.2, build 1493 (2593383)
> 19:55:49:pywws.towebsite:Uploading to web site with FTP
> 19:55:50:pywws.towebsite:local variable 'OK' referenced before assignment
>
> I'm not sure if the last line is good or bad.
currently using FTP OK. One vague possibility is a leftover old file.
Try deleting your build directory then build and install again.
Richard Truran
Apr 16, 2018, 4:07:43 PM4/16/18
to pywws
Dear Jim,
I’ll delete my weather/pywws directory and clone in the current github respository tomorrow and let you know how I get on. Do I need to do anything else?
Is there anything else I should check?
R
I’ll delete my weather/pywws directory and clone in the current github respository tomorrow and let you know how I get on. Do I need to do anything else?
Is there anything else I should check?
R
Jim Easterbrook
Apr 17, 2018, 1:55:13 PM4/17/18
to py...@googlegroups.com
On 14/04/18 10:05, Jim Easterbrook wrote:
> I'm now having a thorough rummage through pywws's attempts to use local
> time zones. It's a complete mess and I'm making some interesting
> discoveries. "rain since midnight" gets interesting when the clocks
> change to and from daylight saving time.
>
> I also plan to allow the "day end hour" to be DST aware (optionally).
This is done now, so all of you who want a day end hour of midnight
> I'm now having a thorough rummage through pywws's attempts to use local
> time zones. It's a complete mess and I'm making some interesting
> discoveries. "rain since midnight" gets interesting when the clocks
> change to and from daylight saving time.
>
> I also plan to allow the "day end hour" to be DST aware (optionally).
local time, summer or winter, will be able to do so. And you should also
get the correct 23 hour and 25 hour days when the clocks change.
(Testing that was a bit of a challenge.)
Today I've updated the OpenWeatherMap uploader to their latest API,
which is rather different to what's gone before. Existing weather.ini
files will need a fair bit of editing.
http://pywws.readthedocs.io/en/latest/api/pywws.service.openweathermap.html
Richard Truran
Apr 17, 2018, 2:29:02 PM4/17/18
to pywws
Dear Jim,
I've now ran into a problem with twitter. I have used the pip command to instal both python-twitter and tweepy, but am unable to get either to be recognised in python 3 (ie import python and import tweepy both fail) however both are present in python 2. Can you give advise on how you got twitter uploads working for python3 pywws.
I will remove the twitter requirement of my weather.ini file so I can test the FTP issue I experienced yesterday.
R
Jim Easterbrook
Apr 17, 2018, 2:35:13 PM4/17/18
to py...@googlegroups.com
On 17/04/18 19:29, Richard Truran wrote:
>
> I've now ran into a problem with twitter. I have used the pip command to
> instal both python-twitter and tweepy, but am unable to get either to be
> recognised in python 3 (ie import python and import tweepy both fail)
> however both are present in python 2. Can you give advise on how you got
> twitter uploads working for python3 pywws.
Are you saying the command 'python3 -c "import twitter"' fails? I've
>
> I've now ran into a problem with twitter. I have used the pip command to
> instal both python-twitter and tweepy, but am unable to get either to be
> recognised in python 3 (ie import python and import tweepy both fail)
> however both are present in python 2. Can you give advise on how you got
> twitter uploads working for python3 pywws.
installed python-twitter on a Pi running Python3 with no difficulty.
Note that the package is called "python-twitter" on PyPI, even though
it's just "twitter" that you import in Python.
Richard Truran
Apr 17, 2018, 2:39:14 PM4/17/18
to pywws
Dear Jim
Yes unfortunately the command you mention fails, I have install the python-twitter package from pip, as documented in the pywws documentation.
Also I have just ran pywws compiled in python 3 and I am still getting the same pywws.towebsite:local variable 'OK' referenced before assignment error.
I've never really used python 3 before, hence the number of issues.
R
Richard Truran
Apr 17, 2018, 2:50:15 PM4/17/18
to pywws
Dear Jim,
I've solved the twitter issue. Just realised to install python3 libraries I need to use pip3 not pip...
As I say, not used python3 too much..
R
Richard Truran
Apr 17, 2018, 3:11:51 PM4/17/18
to pywws
Dear Jim,
Good news, I’ve managed to get the FTP transfer to complete.
I install ftp and pysftp with both pip and pip3, I’m beginning to suspect I had a python 2 only version of ftp installed.
I’ll run the system overnight to ensure everything is stable and then update to your latest version and begin testing the new openweathermap service.
R
Good news, I’ve managed to get the FTP transfer to complete.
I install ftp and pysftp with both pip and pip3, I’m beginning to suspect I had a python 2 only version of ftp installed.
I’ll run the system overnight to ensure everything is stable and then update to your latest version and begin testing the new openweathermap service.
R
Jim Easterbrook
Apr 17, 2018, 3:11:53 PM4/17/18
to py...@googlegroups.com
On 17/04/18 19:39, Richard Truran wrote:
>
> Also I have just ran pywws compiled in python 3 and I am still getting
> the same pywws.towebsite:local variable 'OK' referenced before
> assignment error.
I've just found a potential cause of that and have committed a fix to
>
> Also I have just ran pywws compiled in python 3 and I am still getting
> the same pywws.towebsite:local variable 'OK' referenced before
> assignment error.
GitHub. Do a 'git pull' and rebuild / install to test it.
PS you can go back to Python 2 if you like, but many of the problems
you're running in to are not Python 3 specific.
Richard Truran
Apr 17, 2018, 3:27:01 PM4/17/18
to pywws
Dear Jim,
It's not a problem to help, its a way to say thank you for developing the software I enjoy using to make my weatherstation much more useful than the standard software.
I spoke too soon on the FTP front. I have updated by performing a git pull and rebuilding and installing. The log shows:
20:23:03:pywws.regulartasks:Graphing today_rose.png.xml
20:23:04:pywws.regulartasks:Templating 1hrs_lux.txt
20:23:04:pywws.regulartasks:Templating 6hrs_lux_freq.txt
20:23:05:pywws.regulartasks:Templating en_current.txt
20:23:05:pywws.regulartasks:Templating live_json.txt
20:23:05:pywws.regulartasks:Templating 1hr_json.txt
20:23:05:pywws.towebsite:Uploading to web site with FTP
20:23:05:pywws.towebsite:220 DiskStation FTP server ready.
20:23:04:pywws.regulartasks:Templating 1hrs_lux.txt
20:23:04:pywws.regulartasks:Templating 6hrs_lux_freq.txt
20:23:05:pywws.regulartasks:Templating en_current.txt
20:23:05:pywws.regulartasks:Templating live_json.txt
20:23:05:pywws.regulartasks:Templating 1hr_json.txt
20:23:05:pywws.towebsite:Uploading to web site with FTP
20:23:05:pywws.towebsite:220 DiskStation FTP server ready.
Which I think suggests the ftp transfer appears successful, however no files are transferred and the work folder is full of the generated templates.
Any other thoughts?
R
Jim Easterbrook
Apr 17, 2018, 5:09:15 PM4/17/18
to py...@googlegroups.com
On 17/04/18 20:27, Richard Truran wrote:
>
> I spoke too soon on the FTP front. I have updated by performing a git
> pull and rebuilding and installing. The log shows:
>
> 20:23:03:pywws.regulartasks:Graphing
> today_rose.png.xml
> 20:23:04:pywws.regulartasks:Templating
> 1hrs_lux.txt
> 20:23:04:pywws.regulartasks:Templating
> 6hrs_lux_freq.txt
> 20:23:05:pywws.regulartasks:Templating
> en_current.txt
> 20:23:05:pywws.regulartasks:Templating
> live_json.txt
> 20:23:05:pywws.regulartasks:Templating
> 1hr_json.txt
> 20:23:05:pywws.towebsite:Uploading to web site with
> FTP
> 20:23:05:pywws.towebsite:220 DiskStation FTP server ready.
>
> Which I think suggests the ftp transfer appears successful, however no
> files are transferred and the work folder is full of the generated
> templates.
I'd expect an 'OK' message if it's successful the first time. (Repeated
>
> I spoke too soon on the FTP front. I have updated by performing a git
> pull and rebuilding and installing. The log shows:
>
> 20:23:03:pywws.regulartasks:Graphing
> today_rose.png.xml
> 20:23:04:pywws.regulartasks:Templating
> 1hrs_lux.txt
> 20:23:04:pywws.regulartasks:Templating
> 6hrs_lux_freq.txt
> 20:23:05:pywws.regulartasks:Templating
> en_current.txt
> 20:23:05:pywws.regulartasks:Templating
> live_json.txt
> 20:23:05:pywws.regulartasks:Templating
> 1hr_json.txt
> 20:23:05:pywws.towebsite:Uploading to web site with
> FTP
> 20:23:05:pywws.towebsite:220 DiskStation FTP server ready.
>
> Which I think suggests the ftp transfer appears successful, however no
> files are transferred and the work folder is full of the generated
> templates.
messages are logged at a lower level, so don't repeat.) You might get
more detail if you run at a higher logging level.
Looking at this list suggests 220 means "Service ready for new user"
whatever that means. It's not something I've seen before, but I use a
different server.
https://en.wikipedia.org/wiki/List_of_FTP_server_return_code.
Jim Easterbrook
Apr 18, 2018, 2:05:22 AM4/18/18
to py...@googlegroups.com
On 16/04/18 20:07, Richard Truran wrote:
>
>
> I've updated the pywws3 development version and with the new templates
> using source references, it runs correctly and appears to succeed (that
> is it doesn't crash out):
>
> 19:55:03:pywws.logger:pywws version 18.4.2, build 1493 (2593383)
> 19:55:03:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
> [GCC 6.3.0 20170124]
> 19:55:04:pywws.weatherstation:using pywws.device_libusb1
> 19:55:05:pywws.logdata:Synchronising to weather station
I've just realised that you're using 'hourly' logging rather than the
> using source references, it runs correctly and appears to succeed (that
> is it doesn't crash out):
>
> 19:55:03:pywws.logger:pywws version 18.4.2, build 1493 (2593383)
> 19:55:03:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04)
> [GCC 6.3.0 20170124]
> 19:55:04:pywws.weatherstation:using pywws.device_libusb1
> 19:55:05:pywws.logdata:Synchronising to weather station
'live' logging I use. I've just tried 'hourly' and have reproduced your
problems. Which is good.
the recent changes, but I'm going to go back to using it. The ftp
uploader needs a lot more work...
Jim Easterbrook
Apr 18, 2018, 2:06:50 AM4/18/18
to py...@googlegroups.com
On 17/04/18 22:09, Jim Easterbrook wrote:
>
> Looking at this list suggests 220 means "Service ready for new user"
> whatever that means.
> https://en.wikipedia.org/wiki/List_of_FTP_server_return_code.
>
> Looking at this list suggests 220 means "Service ready for new user"
> whatever that means.
It's a typical greeting from an FTP server. The one I use has several
lines of welcome message and warnings about misuse.
Richard Truran
Apr 18, 2018, 2:16:03 AM4/18/18
to pywws
Dear Jim,
I'm glad you've managed to replicate the issue, hopefully that will make developing the fix a lot easier.
Let me know when you need a second tester for the FTP functionality again.
I'll look into using the 'local' setting in the 'ftp' as my webserver is on my local network, when ready I can switch back uploading over FTP.
I'll also give the new OWM settings in 'weather.ini' a go.
If necessary I can also test running as a Daemon for short periods (several hours) to help ensure all usage scenarios are independently tested, however I would prefer to do this at the weekend as it does increase the likelihood of lock-ups.
Thanks
R
Jim Easterbrook
Apr 18, 2018, 2:52:59 AM4/18/18
to py...@googlegroups.com
On 18/04/18 07:05, Jim Easterbrook wrote:
>
> I've just realised that you're using 'hourly' logging rather than the
> 'live' logging I use. I've just tried 'hourly' and have reproduced your
> problems. Which is good.
And now I've found the problem. The upload thread is being terminated
>
> I've just realised that you're using 'hourly' logging rather than the
> 'live' logging I use. I've just tried 'hourly' and have reproduced your
> problems. Which is good.
before it's done anything useful. (This ought to have affected other
uploaders as well, but they're probably a bit quicker.)
Jim Easterbrook
Apr 18, 2018, 5:42:22 AM4/18/18
to py...@googlegroups.com
On 18/04/18 07:52, Jim Easterbrook wrote:
> On 18/04/18 07:05, Jim Easterbrook wrote:
>>
>> I've just realised that you're using 'hourly' logging rather than the
>> 'live' logging I use. I've just tried 'hourly' and have reproduced your
>> problems. Which is good.
>
> And now I've found the problem. The upload thread is being terminated
> before it's done anything useful. (This ought to have affected other
> uploaders as well, but they're probably a bit quicker.)
All sorted (I think) in the latest commit.
> On 18/04/18 07:05, Jim Easterbrook wrote:
>>
>> I've just realised that you're using 'hourly' logging rather than the
>> 'live' logging I use. I've just tried 'hourly' and have reproduced your
>> problems. Which is good.
>
> And now I've found the problem. The upload thread is being terminated
> before it's done anything useful. (This ought to have affected other
> uploaders as well, but they're probably a bit quicker.)
Richard Truran
Apr 18, 2018, 6:15:19 AM4/18/18
to pywws
Dear Jim,
That latest commit seems to have fixed the file transfer, however when I ran the pywws-hourly to test the process didn't terminate and I had to ctrl-c to end it.
11:10:27:pywws.regulartasks:Templating 1hrs_lux.txt
11:10:27:pywws.regulartasks:Templating 6hrs_lux_freq.txt
11:10:28:pywws.regulartasks:Templating en_current.txt
11:10:28:pywws.regulartasks:Templating live_json.txt
11:10:28:pywws.regulartasks:Templating 1hr_json.txt
11:10:29:pywws.totwitter:Using python-twitter library
11:10:29:pywws.regulartasks:Templating tweet.txt
11:10:29:pywws.regulartasks:Tweeting
11:10:29:pywws.regulartasks:Templating 6hrs_lux.txt
11:10:29:pywws.regulartasks:Templating 24hrs.txt
11:10:29:pywws.regulartasks:Templating 24hrs_table_lux.txt
11:10:29:pywws.regulartasks:Templating daily_table.txt
11:10:29:pywws.totwitter:OK
11:10:29:pywws.totwitter:1 record sent
11:10:29:pywws.towebsite:Uploading to web site with FTP
11:10:31:pywws.towebsite:OK
11:10:31:pywws.towebsite:1 record sent
^CTraceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 99, in mai
n
return hourly(args[0])
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 70, in hou
rly
context.stop()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/storage.py", line 789, in s
top
thread.stop()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/service/__init__.py", line
72, in stop
self.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
Thanks
R
Jim Easterbrook
Apr 18, 2018, 6:39:43 AM4/18/18
to py...@googlegroups.com
On 18/04/18 11:15, Richard Truran wrote:
>
> That latest commit seems to have fixed the file transfer, however when I
> ran the pywws-hourly to test the process didn't terminate and I had to
> ctrl-c to end it.
How long did you wait? It might take 30 seconds or more. Next time can
>
> That latest commit seems to have fixed the file transfer, however when I
> ran the pywws-hourly to test the process didn't terminate and I had to
> ctrl-c to end it.
you run it at higher verbosity so we can see which thread was the problem.
Richard Truran
Apr 18, 2018, 7:24:27 AM4/18/18
to pywws
Dear Jim,
Sure, I left the process running for several minutes, here is the output with high verbosity (I've replace sensitive text with XXX)
12:16:38:pywws.regulartasks:Templating live_json.txt
12:16:38:pywws.regulartasks:Templating 1hr_json.txt
12:16:38:pywws.towebsite:thread started Thread-4
12:16:38:pywws.service.openweathermap:waiting for upload thread
12:16:38:pywws.towebsite:Uploading to web site with FTP
12:16:38:pywws.towebsite:welcome message
220 DiskStation FTP server ready.
12:16:39:pywws.towebsite:OK
12:16:39:pywws.towebsite:1 record sent
12:17:12:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): api.openweathermap.org
12:17:12:requests.packages.urllib3.connectionpool:http://api.openweathermap.org:80 "POST /data/3.0/measurements?appid=XXXXXXXXXX HTTP/1.1
" 400 48
12:17:12:pywws.service.openweathermap:http status: 400 {"code":400001,"message":"Got empty station id"}
12:17:52:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): api.openweathermap.org
12:17:52:requests.packages.urllib3.connectionpool:http://api.openweathermap.org:80 "POST /data/3.0/measurements?appid=XXXXXXXXXX HTTP/1.1
" 400 48
12:17:52:pywws.service.openweathermap:http status: 400 {"code":400001,"message":"Got empty station id"}
12:18:32:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): api.openweathermap.org
12:18:32:requests.packages.urllib3.connectionpool:http://api.openweathermap.org:80 "POST /data/3.0/measurements?appid=XXXXXXXXXX HTTP/1.1
" 400 48
12:18:32:pywws.service.openweathermap:http status: 400 {"code":400001,"message":"Got empty station id"}
^X
^C12:19:08:pywws.storage:waiting for thread Thread-1
12:19:08:pywws.storage:waiting for thread Thread-4
12:19:08:pywws.storage:waiting for thread Thread-3
12:19:08:pywws.storage:waiting for thread Thread-2
12:19:08:pywws.storage:flushing
Traceback (most recent call last):
File "/usr/local/bin/pywws-hourly", line 11, in <module>
load_entry_point('pywws==18.4.2', 'console_scripts', 'pywws-hourly')()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 99, in main
return hourly(args[0])
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/hourly.py", line 70, in hourly
context.stop()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/storage.py", line 789, in stop
thread.stop()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/service/__init__.py", line 72, in stop
self.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
I've checked my weather.ini file and I definitely have something in the id and external id lines for openweathermap
R
Richard Truran
Apr 18, 2018, 7:39:27 AM4/18/18
to pywws
OK, so the issue is that I haven't registered my station with the new API. I'm having trouble understanding how to submit the required station information to the api to get the station id variable.
The process ends cleanly without the openweathermap service in weather.ini
12:38:13:pywws.regulartasks:Graphing today_rose.png.xml
12:38:14:pywws.regulartasks:Templating 1hrs_lux.txt
12:38:14:pywws.regulartasks:Templating 6hrs_lux_freq.txt
12:38:14:pywws.regulartasks:Templating en_current.txt
12:38:14:pywws.regulartasks:Templating live_json.txt
12:38:14:pywws.regulartasks:Templating 1hr_json.txt
12:38:15:pywws.towebsite:thread started Thread-3
12:38:15:pywws.towebsite:waiting for upload thread
12:38:15:pywws.towebsite:Uploading to web site with FTP
12:38:15:pywws.towebsite:welcome message
220 DiskStation FTP server ready.
12:38:15:pywws.towebsite:OK
12:38:15:pywws.towebsite:1 record sent
12:38:15:pywws.storage:waiting for thread Thread-1
12:38:15:pywws.storage:waiting for thread Thread-2
12:38:15:pywws.storage:flushing
R
Jim Easterbrook
Apr 18, 2018, 7:40:23 AM4/18/18
to py...@googlegroups.com
mean it's no longer waiting for a thread, yet when you stopped it it was
in a thread's "join" method. It had finished waiting for threads 1, 2, 3
& 4 but you didn't include the bits of the log file where the threads
started so I don't know which they are or what other threads might have
been running.
> I've checked my weather.ini file and I definitely have something in the
> id and external id lines for openweathermap
by registering your site. This is why I included a link to the new
documentation yesterday.
http://pywws.readthedocs.io/en/latest/api/pywws.service.openweathermap.html
Jim Easterbrook
Apr 18, 2018, 7:44:35 AM4/18/18
to py...@googlegroups.com
On 18/04/18 12:39, Richard Truran wrote:
> OK, so the issue is that I haven't registered my station with the new
> API. I'm having trouble understanding how to submit the required station
> information to the api to get the station id variable.
>
> The process ends cleanly without the openweathermap service in weather.ini
Thanks, that points to any sort of "broken" service being a problem.
> OK, so the issue is that I haven't registered my station with the new
> API. I'm having trouble understanding how to submit the required station
> information to the api to get the station id variable.
>
> The process ends cleanly without the openweathermap service in weather.ini
Something that should enable me to reproduce the bug.
I must thank you for being so patient with all this. It's a lesson in
how something that "works for me" is not always fit for wider use.
Richard Truran
Apr 18, 2018, 7:56:32 AM4/18/18
to pywws
Dear Jim,
Reading through your notes in the github repository, I've realised you have created a script for this, thanks. Unfortunately the -r flag is not recognised:
pi@raspberrypi:~/weather/pywws $ python -m pywws.service.openweathermap -r -v /home/pi/weather/data
usage: openweathermap.py [-h] [-c] [-v] data_dir
openweathermap.py: error: unrecognized arguments: -r
pi@raspberrypi:~/weather/pywws $ python -m pywws.service.openweathermap -h
usage: openweathermap.py [-h] [-c] [-v] data_dir
Upload data to OpenWeatherMap
positional arguments:
data_dir root directory of the weather data
optional arguments:
-h, --help show this help message and exit
-c, --catchup upload all data since last upload
-v, --verbose increase amount of reassuring messages
pi@raspberrypi:~/weather/pywws $ python -m pywws.service.openweathermap -r -v /home/pi/weather/data
usage: openweathermap.py [-h] [-c] [-v] data_dir
openweathermap.py: error: unrecognized arguments: -r
Thanks
R
Jim Easterbrook
Apr 18, 2018, 8:03:16 AM4/18/18
to py...@googlegroups.com
On 18/04/18 12:56, Richard Truran wrote:
>
> Reading through your notes in the github repository, I've realised you
> have created a script for this, thanks. Unfortunately the -r flag is not
> recognised:
>
> pi@raspberrypi:~/weather/pywws $ python -m pywws.service.openweathermap
> -r -v /home/pi/weather/data
> usage: openweathermap.py [-h] [-c] [-v] data_dir
> openweathermap.py: error: unrecognized arguments: -r
Shouldn't that be python3? You're probably using your old Python2 version.
>
> Reading through your notes in the github repository, I've realised you
> have created a script for this, thanks. Unfortunately the -r flag is not
> recognised:
>
> pi@raspberrypi:~/weather/pywws $ python -m pywws.service.openweathermap
> -r -v /home/pi/weather/data
> usage: openweathermap.py [-h] [-c] [-v] data_dir
> openweathermap.py: error: unrecognized arguments: -r
Jim Easterbrook
Apr 18, 2018, 8:44:50 AM4/18/18
to py...@googlegroups.com
On 18/04/18 12:44, Jim Easterbrook wrote:
> Thanks, that points to any sort of "broken" service being a problem.
> Something that should enable me to reproduce the bug.
OK, I've worked out what's going on. The initial design premise is that
> Thanks, that points to any sort of "broken" service being a problem.
> Something that should enable me to reproduce the bug.
in 'live' logging items are queued for uploading (to a service) in a
separate thread. If the upload is successful the item is taken off the
queue, otherwise it is left on the queue so it can be tried again later.
This allows for temporary loss of connection errors and ensures nothing
goes missing when these occur.
With 'hourly' logging any upload that fails will be done again from
scratch next time so we don't want this behaviour. But we do want to be
able to shut down the thread cleanly. At present the thread won't quit
if there's still something on the queue, as all the expected uploads
haven't been done yet. (See the earlier problem with ftp uploads not
happening.) It's still nice to have threads so you can upload to several
services at the same time.
I'm going to have to think about this.
Richard Truran
Apr 18, 2018, 9:08:41 AM4/18/18
to pywws
Dear Jim,
Yes, of course. That fixed it, however to get the output to see the station id I had to turn the verbosity up to -vvv. As as low priority perhaps you'll want to reflect that in your notes at the top of the code.
I'm now up and running on the python3 development version, the only outstanding thing is for me to look at the service coding to see if I can make one for weathercloud (I'm not hopeful, as I'm well past my comfort zone)
Looking at the data generated I think there is an issue with the illuminance data as it appears to be very low and not matching what is on the console.
If you make any changes and need me to update the install please let me know.
R
Jim Easterbrook
Apr 18, 2018, 9:13:31 AM4/18/18
to py...@googlegroups.com
On 18/04/18 14:08, Richard Truran wrote:
>
> Yes, of course. That fixed it, however to get the output to see the
> station id I had to turn the verbosity up to -vvv. As as low priority
> perhaps you'll want to reflect that in your notes at the top of the code.
Why did you need to see the station id? It's stored in your weather.ini
>
> Yes, of course. That fixed it, however to get the output to see the
> station id I had to turn the verbosity up to -vvv. As as low priority
> perhaps you'll want to reflect that in your notes at the top of the code.
for you.
Richard Truran
Apr 18, 2018, 9:20:35 AM4/18/18
to pywws
Dear Jim,
My weather.ini file didn't get automatically updated with the station id allocation.
R
Jim Easterbrook
Apr 18, 2018, 10:02:15 AM4/18/18
to py...@googlegroups.com
On 18/04/18 13:44, Jim Easterbrook wrote:
>
> OK, I've worked out what's going on. The initial design premise is that
> in 'live' logging items are queued for uploading (to a service) in a
> separate thread. If the upload is successful the item is taken off the
> queue, otherwise it is left on the queue so it can be tried again later.
> This allows for temporary loss of connection errors and ensures nothing
> goes missing when these occur.
>
> With 'hourly' logging any upload that fails will be done again from
> scratch next time so we don't want this behaviour. But we do want to be
> able to shut down the thread cleanly. At present the thread won't quit
> if there's still something on the queue, as all the expected uploads
> haven't been done yet. (See the earlier problem with ftp uploads not
> happening.) It's still nice to have threads so you can upload to several
> services at the same time.
>
> I'm going to have to think about this.
I have thought. The latest commit has much nicer loop control and
>
> OK, I've worked out what's going on. The initial design premise is that
> in 'live' logging items are queued for uploading (to a service) in a
> separate thread. If the upload is successful the item is taken off the
> queue, otherwise it is left on the queue so it can be tried again later.
> This allows for temporary loss of connection errors and ensures nothing
> goes missing when these occur.
>
> With 'hourly' logging any upload that fails will be done again from
> scratch next time so we don't want this behaviour. But we do want to be
> able to shut down the thread cleanly. At present the thread won't quit
> if there's still something on the queue, as all the expected uploads
> haven't been done yet. (See the earlier problem with ftp uploads not
> happening.) It's still nice to have threads so you can upload to several
> services at the same time.
>
> I'm going to have to think about this.
termination, and removes the 'stop' methods I introduced this morning.
Realising that 'live logging' is the special case and doesn't need clean
shutdown was the key.
Jim Easterbrook
Apr 18, 2018, 10:05:02 AM4/18/18
to py...@googlegroups.com
On 18/04/18 14:20, Richard Truran wrote:
>
> My weather.ini file didn't get automatically updated with the station id
> allocation.
That's odd. I want to revisit the registration process anyway, as it
>
> My weather.ini file didn't get automatically updated with the station id
> allocation.
would be nice to be able to reuse an id if you lose it (e.g. your
weather.ini gets lost or corrupted). You've probably generated a bunch
of ids in your experiments - I want to list them and give you the option
of using or deleting or ignoring each one.
But I need to go and finish mowing the grass first.
Jim Easterbrook
Apr 18, 2018, 10:08:07 AM4/18/18
to py...@googlegroups.com
On 18/04/18 14:49, Richard Truran wrote:
>
> Please see the attached file showing the illuminance data issue.
>
>
> Please see the attached file showing the illuminance data issue.
>
> On Wednesday, 18 April 2018 14:08:41 UTC+1, Richard Truran wrote:
>
> Looking at the data generated I think there is an issue with the
> illuminance data as it appears to be very low and not matching what
> is on the console.
That looks very odd. I assume pywws used to cope OK with bright sunshine
>
> Looking at the data generated I think there is an issue with the
> illuminance data as it appears to be very low and not matching what
> is on the console.
like we're having today. Are the low values present in your raw data?
(I.e. not caused by subsequent processing.)
Richard Truran
Apr 18, 2018, 10:18:03 AM4/18/18
to pywws
Dear Jim,
Yes pywws has always handled sunshine well, the low values are in the raw data files as well as the processed data files. The incorrect values recorded don't match the values on the console either.
R
Jim Easterbrook
Apr 18, 2018, 10:20:50 AM4/18/18
to py...@googlegroups.com
On 18/04/18 15:08, Jim Easterbrook wrote:
>
> That looks very odd. I assume pywws used to cope OK with bright sunshine
> like we're having today. Are the low values present in your raw data?
> (I.e. not caused by subsequent processing.)
Found it. I was stupidly only processing two of the three bytes. Commit
>
> That looks very odd. I assume pywws used to cope OK with bright sunshine
> like we're having today. Are the low values present in your raw data?
> (I.e. not caused by subsequent processing.)
940c082 fixes it.
Message has been deleted
Richard Truran
Apr 18, 2018, 11:00:49 AM4/18/18
to pywws
Dear Jim,
Thanks for the rapid fix. I've now switched back to using the python 3 compiled development branch and all appears to be going well. Illuminance data is as expected and uploads to services and my website are working well.
I have noticed one thing, from switching between the current production version and development version I have sometimes forgotten to remove the reference to weathercloud when using the development version. You appear to have coded in a check to see if a service uploader file exists and it is correctly identified as being a service without and uploader. However later on pywws then tries to upload and crashes as there is no uploader. Would it be possible to link the initial check to a variable which would skip attempting to upload data to services without an uploader?
I hope that is clear.
R
Jim Easterbrook
Apr 18, 2018, 11:10:31 AM4/18/18
to py...@googlegroups.com
On 18/04/18 16:00, Richard Truran wrote:
>
> Thanks for the rapid fix. I've now switched back to using the python 3
> compiled development branch and all appears to be going well.
> Illuminance data is as expected and uploads to services and my website
> are working well.
Hooray for some good news! (-:
>
> Thanks for the rapid fix. I've now switched back to using the python 3
> compiled development branch and all appears to be going well.
> Illuminance data is as expected and uploads to services and my website
> are working well.
> I have noticed one thing, from switching between the current production
> version and development version I have sometimes forgotten to remove the
> reference to weathercloud when using the development version. You appear
> to have coded in a check to see if a service uploader file exists and it
> is correctly identified as being a service without and uploader. However
> later on pywws then tries to upload and crashes as there is no uploader.
> Would it be possible to link the initial check to a variable which would
> skip attempting to upload data to services without an uploader?
Jim Easterbrook
Apr 18, 2018, 11:13:35 AM4/18/18
to py...@googlegroups.com
On 18/04/18 15:37, Richard Truran wrote:
>
> This last update seems to have broken something:
>
> File
> "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/storage.py",
> line 774, in __init__
> self.params = ParamStore(data_dir, 'weather.ini')
> File "/usr/lib/python3.5/codecs.py", line 321, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0:
> invalid start byte
I would guess that while editing your weather.ini you've managed to
insert a non-utf_8 character, possibly right at the start (maybe a byte
order mark?).
>
> This last update seems to have broken something:
>
> File
> "/usr/local/lib/python3.5/dist-packages/pywws-18.4.2-py3.5.egg/pywws/storage.py",
> line 774, in __init__
> self.params = ParamStore(data_dir, 'weather.ini')
> File "/usr/lib/python3.5/codecs.py", line 321, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0:
> invalid start byte
I would guess that while editing your weather.ini you've managed to
insert a non-utf_8 character, possibly right at the start (maybe a byte
order mark?).
Richard Truran
Apr 18, 2018, 11:30:41 AM4/18/18
to pywws
Dear Jim,
Yes the problem was firmly at my end, apologies. I tried to delete the post when I realised.
All seems to be working well on the latest commit.
R
Jim Easterbrook
Apr 18, 2018, 12:00:24 PM4/18/18
to py...@googlegroups.com
On 18/04/18 15:05, Jim Easterbrook wrote:
> On 18/04/18 14:20, Richard Truran wrote:
>>
>> My weather.ini file didn't get automatically updated with the station id
>> allocation.
>
> That's odd. I want to revisit the registration process anyway, as it
> would be nice to be able to reuse an id if you lose it (e.g. your
> weather.ini gets lost or corrupted). You've probably generated a bunch
> of ids in your experiments - I want to list them and give you the option
> of using or deleting or ignoring each one.
The latest commit (90d53ac) adds this. It displays a list of all your
> On 18/04/18 14:20, Richard Truran wrote:
>>
>> My weather.ini file didn't get automatically updated with the station id
>> allocation.
>
> That's odd. I want to revisit the registration process anyway, as it
> would be nice to be able to reuse an id if you lose it (e.g. your
> weather.ini gets lost or corrupted). You've probably generated a bunch
> of ids in your experiments - I want to list them and give you the option
> of using or deleting or ignoring each one.
stations, then lets you choose which one to use (or register a new one),
then lets you delete any unwanted ones (e.g. created in error).
Message has been deleted
Richard Truran
Apr 19, 2018, 6:38:10 AM4/19/18
to pywws
Dear Jim,
My installation of the development version has been working well overnight, however I have now ran into a problem with uploading to twitter.
My weather.ini file has a twitter upload in both the 'logged' and '12 hourly' sections.
pywws crashes when it initiates the second twitter upload with the following error:
raise RuntimeError ("threads can only be started once").
Thanks
Richard
Jim Easterbrook
Apr 19, 2018, 6:43:00 AM4/19/18
to py...@googlegroups.com
On 19/04/18 11:38, Richard Truran wrote:
>
> My installation of the development version has been working well
> overnight, however I have now ran into a problem with uploading to twitter.
>
> My weather.ini file has a twitter upload in both the 'logged' and '12
> hourly' sections.
>
> pywws crashes when it initiates the second twitter upload with the
> following error:
>
> raise RuntimeError ("threads can only be started once").
I think what's happening is that you have two twitter jobs and the first
>
> My installation of the development version has been working well
> overnight, however I have now ran into a problem with uploading to twitter.
>
> My weather.ini file has a twitter upload in both the 'logged' and '12
> hourly' sections.
>
> pywws crashes when it initiates the second twitter upload with the
> following error:
>
> raise RuntimeError ("threads can only be started once").
is completing before the second gets put on the queue. Because of the
changes I made yesterday to do with thread termination.
Another bit of thinking required.
Jim Easterbrook
Apr 19, 2018, 7:52:26 AM4/19/18
to py...@googlegroups.com
Richard Truran
Apr 19, 2018, 2:15:19 PM4/19/18
to pywws
Dear Jim,
Thanks, I’ve updated and will monitor the logs to ensure it is behaving correctly when the ‘hourly’ and ‘12 hourly’ groups run together.
R
Thanks, I’ve updated and will monitor the logs to ensure it is behaving correctly when the ‘hourly’ and ‘12 hourly’ groups run together.
R
Igg
Apr 20, 2018, 7:45:02 AM4/20/18
to pywws
Interesting to see Jim, but it doesn't appear to me that python 2 is going anywhere particularly quickly. I''m quite happy to "upgrade" to python 3 if there are advantages, and of course if future pywws makes specific use of python 3 differences. I'm not a luddite, just an "if it's not broken...." type of person. It might well be that this change will force you to restructure the current set up, and make improvements/rethinks to the modules. If so, that will be enough for me to jump on board. I should have some time over the next couple of weeks to trial a set up on a spare RPi.
Thanks again for your time and efforts in producing this piece of software; it's been a great pleasure to use over the last 10 years :)
Ian
Igg
Apr 20, 2018, 7:53:13 AM4/20/18
to pywws
P.S. I'm going to add this manually, but I wondered if you might be interested in the python library for calculating sunrise/set times https://github.com/jebeaudet/SunriseSunsetCalculator
Jim Easterbrook
Apr 20, 2018, 8:29:19 AM4/20/18
to py...@googlegroups.com
On 20/04/18 12:45, Igg wrote:
> Interesting to see Jim, but it doesn't appear to me that python 2 is
> going anywhere particularly quickly. I''m quite happy to "upgrade" to
> python 3 if there are advantages, and of course if future pywws makes
> specific use of python 3 differences. I'm not a luddite, just an "if
> it's not broken...." type of person. It might well be that this change
> will force you to restructure the current set up, and make
> improvements/rethinks to the modules. If so, that will be enough for me
> to jump on board. I should have some time over the next couple of weeks
> to trial a set up on a spare RPi.
Python 3 is not compulsory, but pywws was due for a major overhaul and
> Interesting to see Jim, but it doesn't appear to me that python 2 is
> going anywhere particularly quickly. I''m quite happy to "upgrade" to
> python 3 if there are advantages, and of course if future pywws makes
> specific use of python 3 differences. I'm not a luddite, just an "if
> it's not broken...." type of person. It might well be that this change
> will force you to restructure the current set up, and make
> improvements/rethinks to the modules. If so, that will be enough for me
> to jump on board. I should have some time over the next couple of weeks
> to trial a set up on a spare RPi.
making it compatible with Python 2 & 3 (without any conversion required)
was just one of the things I wanted to do.
Python 2 has had its life extended to 2020, but that's less than two
years away.
https://www.python.org/dev/peps/pep-0373/
Richard Truran
Apr 27, 2018, 11:33:21 AM4/27/18
to pywws
Dear Jim,
I've been successfully using the development branch for about a week now with no issues.
As predicted the weathercloud service uploader script was deleted from the standard module directory when I installed a new build, would it be possible to implement a custom upload script directory in a future build?
R
Jim Easterbrook
Apr 28, 2018, 4:20:32 AM4/28/18
to py...@googlegroups.com
search path.
If you added your module to your pywws sources it shouldn't have been
affected if you update with a 'git pull'.
MyForest
Nov 3, 2018, 4:52:54 PM11/3/18
to pywws
Hi Jim,
I know Docker certainly isn't your target environment but I thought I'd mention that I was able to migrate my container to Python 3 and things seem to be working absolutely fine with release 18.10.1.
Keep up the good work.
David Bowen
Mark Jarvis
Feb 27, 2020, 8:41:18 AM2/27/20
to pywws
Hi Jim,
Now that Python2.7 is now officially EOL, would you say future contributions to the pywws project can exclusively be in Python3 without needing to consider Python2.7 compatibility?
If so, it would be good to draw a line in the sand and start seeing the abundant use of f-strings!
Mark
Jim Easterbrook
Feb 27, 2020, 9:05:54 AM2/27/20
to py...@googlegroups.com
On 27/02/2020 13:41, Mark Jarvis wrote:
>
> Now that Python2.7 is now officially EOL, would you say future
> contributions to the pywws project can exclusively be in Python3 without
> needing to consider Python2.7 compatibility?
A bit too soon I think. (There were complaints when I withdrew Python
>
> Now that Python2.7 is now officially EOL, would you say future
> contributions to the pywws project can exclusively be in Python3 without
> needing to consider Python2.7 compatibility?
2.6 compatibility.) Maybe at the end of the year.
> If so, it would be good to draw a line in the sand and start seeing the
> abundant use of f-strings!
Reply all
Reply to author
Forward
0 new messages