wee_import: Novice user, and Python newbie

[KnC Mc]

I was using wee_import with WU to generate a local webpage, and gave up since May when it stopped functioning.  I am not a developer at all, but have been playing with HTML since v1, and hacked and thrashed my way thru coding since those days for front end projects.  I recently have started an adventure learning python, and have figured out how to use the WU API key to get some results I think I can parse.  So the question...

Is anyone still working on wee_import?  Can I beta test on CentOS 7, and if I can contribute I am willing to try.  Weeding thru the code has been an experience, but it is helping me learn.  Still a total newbie with weather stations and python, but this seems like a worthwhile project I am willing to contribute.  I am currently running a standard packaged version of weewx... 3.8.0.  I am not all that familiar with git, but there too, am working on it.  If anyone is working on wee_import, I am willing to beta test and test imperial data.  Haven't figured out the database piece, but I have php/mysql experience that might be useful, and I am going to try and apply that to my python lessons next.  

If nothing else I can stress test code (which I seem to be good at), and we might be able to get wee_import in a functional state again.  I don't have any illusions of doing it on my own, but I am willing to help. ;)

[gjr80]

Hi,

wee_import is certainly still being worked on, this thread refers (and there are probably others). The main chnages being the WeeWX v4 wee_import once again supports importing from WU and it also now supports Weather Display imports.

I had planned on having the wee_import code finished some time before WeeWX v4 was released, and fully intended posting some instructions for using the WeeWX v4 wee_import code under a WeeWX 3.x install. Though as the wee_import code took much longer than I thought (it has some significant changes to how it works in WeeWX v4) and since WeeWX v4 will be released RSN, I decided not to post these instructions but rather wait for v4 to be released. On reflection, I realise a number of folks may not upgrade to v4 immediately, due in part to the number of non-core WeeWX drivers and extensions that are yet WeeWX v4 compatible. So I have put together the following instructions that will allow you to use the WeeWX v4 wee_import code under a WeeWX 3.x install (note this will have no affect on your WeeWX 3.x install):

1. install git if not already installed:

$ sudo apt-get install git

2. create a directory for our clone and chnage to that directory:

$ mkdir /var/tmp/weewx4_src
$ cd /var/tmp/weewx4_src

3. clone the WeeWX GitHub development:

$ git clone -b development https://github.com/weewx/weewx

4. check wee_import works:

$ sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --help
$ sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --version


the version should be 0.4

5. an example import might look like:

$ sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --config=/home/weewx/weewx.conf --import-config=/var/tmp/wu.conf --date=2019-11-30

Adjust paths to suit (you could probably leave the --config option out and wee_import will find the correct weewx.conf but better safe than sorry)

You should be able to use wee_import with a suitable import config file and options using your existing WeeWX 3.x install setup (ie databases) but with the WeeWX v4 wee_import code. Note that you will need to refer to the v4 wee_import documentation which you will find in /var/tmp/weewx4_scr/weewx/docs/utilities.htm and the example import config files will be in /var/tmp/weewx4_src/weewx/util/import. Importantly, the usual caveats apply when using wee_import, ie back your database, do a dry run first etc but you should find importing from WU works provided you are using a new style long API key.

Finally, you can remove the WeeWX v4 code using:

$ rm -R /var/tmp/weewx4_src

Gary

[gjr80]

One thing I forgot to mention in the above post is that once WeeWX v4 is released the -b development option should be omitted from the git clone command, ie step 3 would become:

3. clone the WeeWX GitHub master branch:

$ git clone https://github.com/weewx/weewx

Gary

[KnC Mc]

git went fine...

git clone -b development https://github.com/weewx/weewx

Cloning into 'weewx'...

remote: Enumerating objects: 45, done.

remote: Counting objects: 100% (45/45), done.

remote: Compressing objects: 100% (37/37), done.

remote: Total 35534 (delta 17), reused 24 (delta 8), pack-reused 35489

Receiving objects: 100% (35534/35534), 15.58 MiB | 306.00 KiB/s, done.

Resolving deltas: 100% (27219/27219), done.

However, on my Centos7 machine running both python 2.7 and python 3.6 (python is symlinked to 3.6) I get the following...

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --help

Traceback (most recent call last):

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 723, in <module>

    import weecfg

  File "/var/tmp/weewx4_src/weewx/bin/weecfg/__init__.py", line 22, in <module>

    import configobj

ModuleNotFoundError: No module named 'configobj'

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --version

Traceback (most recent call last):

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 723, in <module>

    import weecfg

  File "/var/tmp/weewx4_src/weewx/bin/weecfg/__init__.py", line 22, in <module>

    import configobj

ModuleNotFoundError: No module named 'configobj'

I haven't had the time to dig, but I assume I am missing some modules in an expected path?  I still have the default weewx 3.8 installed by yum running.  Does 3.9.2 install some extra magic that I need for the 4.0b to work?

[gjr80]

I suspect your issue is that your symlink to python3 is causing wee_import to run under python 3 and hence the python 3 versions of the WeeWX python pre-requisites are required. Since you are running WeeWX under python these pre-requisites are not installed. You could try installing the following (untested):

$ sudo yum install python3-configobj

There may be other flow on dependency issues.

Alternatively, you could temporarily remove your python 3 symlink and run wee_import, this would run wee_import under python2 and use your (presumably) already installed pre-requisites.

Gary

[KnC Mc]

My bad... I thought it required Python3.  I screwed up removing the symlink (always get the target-link reversed and -f habit bites me.), so I am in the process of rolling back to 2.7.  I only have 3 installed for learning.  ugh!  Hopefully, I can test within the hour.  :-|

[gjr80]

Ah, a costly (in time) exercise. WeeWX 4 will run under either python 2.7 or python 3.5 or later. Am sure you will not be the only one to be caught out.

Gary

[vince]

You need the prerequisite modules installed for whatever version of python you are using.   Best thing is to specify either 'python2' or 'python3' explicitly when you run setup.py and then the installation will do the right thing.  You can switch back and forth if needed by simply rerunning setup.py using whatever python you want.

I'd recommend not using the 'python' path even if you symlinked it, just to be safe.

Same goes for installing modules.  If you want python3, specify 'pip3' when installing the modules.

[KnC Mc]

Yep. Lesson learned. Been up way to long, and made a mess of my python installs. Had sense enough to do a clone before mucking around so I am back to functioning. Project for another day I think.

Pleasantly surprised this forum is so active. Been lurking for awhile but never turned on the email notifies. Wow?!

Thanks for all the tips and advice. Going to give it another go tomorrow.

[gjr80]

True, but in this case setup.py may never have been run. Running wee_ import in this manner could be used with a packaged 3.x install. The link to python 3 was a complicating factor I did not anticipate.

Just goes to show there will likely be a number users experiencing WeeWX/python version issues when 4.0 is released.

Gary

[gjr80]

Darn iPad, that should be ‘number of users’

Gary

[KnC Mc]

OK... Not going to start modifying things until I understand what is going on... Here is the situation:

CentOS 7 running on a quad core NUC.  Python 2.7.5 & Python 3 installed.  Standard python installs under yum with the following symlinks created under the  Python 2.7.5 install:

/usr/bin/python -> python2

/usr/bin/python2 -> python2.7

/usr/bin/python2.7

I ran the following...

mkdir /var/tmp/weewx4_src

cd /var/tmp/weewx4_src

git clone -b development https://github.com/weewx/weewx

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --help

I am actually getting more errors than before (that is my luck)...

Traceback (most recent call last):

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 726, in <module>

    import weeimport.weeimport

  File "/var/tmp/weewx4_src/weewx/bin/weeimport/weeimport.py", line 30, in <module>

    import weecfg.database

  File "/var/tmp/weewx4_src/weewx/bin/weecfg/database.py", line 22, in <module>

    import weewx.manager

  File "/var/tmp/weewx4_src/weewx/bin/weewx/manager.py", line 21, in <module>

    import weewx.accum

  File "/var/tmp/weewx4_src/weewx/bin/weewx/accum.py", line 95, in <module>

    defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8')

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1242, in __init__

    self._load(infile, configspec)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1302, in _load

    infile = self._handle_bom(infile)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1442, in _handle_bom

    if not line.startswith(BOM):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

 I have verified all the files exist, now I am going to look at the lines in the files.  I am not a programmer, but I sometimes can figure things out with the limited knowledge of programming languages I have, so we will see.  If there is a ENV variable that is broke because of the multiple versions of python, hopefully I can figure that out... just to reiterate all the file paths in the Traceback exist, including /usr/lib/python2.7/site-packages/configobj.py.

Any thoughts?

[Thomas Keffer]

Thanks for giving the beta a try!

You are not the first to be having this problem, but, unfortunately, I haven't figured out how to replicate the error yet. Could you do me a favor, and tell me the results of running these 4 commands. They will help isolate the versions that are causing the problem.

python2 -c "import configobj; print(configobj.__version__)"

python3 -c "import configobj; print(configobj.__version__)"

head -n 1 /var/tmp/weewx4_src/weewx/bin/wee_import

/usr/bin/env python -V

Hopefully, I will be able to replicate the error, but so far I have not

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b9bafcb0-5496-4f72-bb04-7db24b157670%40googlegroups.com.

[KnC Mc]

Here ya go...

/var/tmp/weewx4_src # python2 -c "import configobj; print(configobj.__version__)"

4.7.2

/var/tmp/weewx4_src # python3 -c "import configobj; print(configobj.__version__)"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

ModuleNotFoundError: No module named 'configobj'

/var/tmp/weewx4_src # head -n 1 /var/tmp/weewx4_src/weewx/bin/wee_import

#!/usr/bin/env python

/var/tmp/weewx4_src # /usr/bin/env python -V

Python 2.7.5

Looks like I do not have the python3 configobj installed... let me go back and reread that post, and install it, and try again.

[KnC Mc]

hmmm...

No package python3-configobj available.

Error: Nothing to do

[KnC Mc]

Doesn't look like it is provided under python3...

yum provides configobj.py

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

 * atomic: www7.atomicorp.com

 * base: mirror.sfo12.us.leaseweb.net

 * epel: mirror.sfo12.us.leaseweb.net

 * extras: mirror.sfo12.us.leaseweb.net

 * updates: mirror.sfo12.us.leaseweb.net

python-configobj-4.7.2-7.el7.noarch : Config file reading, writing, and

                                    : validation

Repo        : base

Matched from:

Filename    : /usr/lib/python2.7/site-packages/configobj.py

python-configobj-4.7.2-7.el7.noarch : Config file reading, writing, and

                                    : validation

Repo        : @anaconda

Matched from:

Filename    : /usr/lib/python2.7/site-packages/configobj.py

[Thomas Keffer]

Slow down there, knc! Let's try and isolate this problem first.

You're using Python 2, which should work fine. I suspect the problem is an older version of configobj. But, before you try to upgrade configobj, can you try one other thing?

Go into the file bin/weewx/accum.py, Line 95, and change this

defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8')

to this

defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI))

Then try running wee_import again.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/486ae325-251b-42aa-9163-ac48d6e1c81b%40googlegroups.com.

[KnC Mc]

OK... I found this:

https://stackoverflow.com/questions/32208421/ascii-codec-error-when-writing-configobj

Thinking this might have something to do with it because of:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

in my Traceback?

Not sure how to force python2.7.  yum and other python dependent software are using 2.7 (As I found out last night when I broke my system playing with the symlinks).  I only use python 3 for one piece of software, and I am using it to learn the language.  I know there are differences, but I am definitely no where near knowledgeable enough to know what they are.  I have only been working thru some online tutorials and barely know how to write simple programs to calculate shipping cost, and play with turtle (very distracting since I remember turtle from Vic-20 days. :P).  I am still exploring the code, but not great at following all the modules, and figuring out what is being called where.  [sigh]

let me know if I can share any other output.

[KnC Mc]

---

vim +95 /var/tmp/weewx4_src/weewx/bin/weewx/accum.py 

---

#defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8')

defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI)) # Modification

accum_dict = None

---

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --help

Traceback (most recent call last):

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 726, in <module>

    import weeimport.weeimport

  File "/var/tmp/weewx4_src/weewx/bin/weeimport/weeimport.py", line 30, in <module>

    import weecfg.database

  File "/var/tmp/weewx4_src/weewx/bin/weecfg/database.py", line 22, in <module>

    import weewx.manager

  File "/var/tmp/weewx4_src/weewx/bin/weewx/manager.py", line 21, in <module>

    import weewx.accum

  File "/var/tmp/weewx4_src/weewx/bin/weewx/accum.py", line 96, in <module>

    defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI))

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1242, in __init__

    self._load(infile, configspec)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1302, in _load

    infile = self._handle_bom(infile)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1457, in _handle_bom

    if not line.startswith(BOM):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

[KnC Mc]

Not to make things crazy... but, I am eager, and am looking...

https://stackoverflow.com/questions/3423601/python-2-7-exec-what-is-wrong/3423935#3423935

Specifically at "#2"

io.StringIO wants to work with unicode. You might think you can fix it by putting a u in front of the string you want to print like this

I don't know if that applies, but I see the "u" when you are declaring DEFAULTS_INI.

This is all greek to me right now, but I am reading.  ;)

[KnC Mc]

What version should I be running?  yum indicates that I am running the recent version.  Have to go to work, but I will keep looking in my spare time.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[vince]

On Sunday, January 5, 2020 at 1:22:19 PM UTC-8, KnC Mc wrote:

What version should I be running?  yum indicates that I am running the recent version.  Have to go to work, but I will keep looking in my spare time.

I got the beta running in Simulator mode under python3 on centos7 after a bit of fiddling.   Let me let it run for a couple hours and I'll check it at halftime of the football game and post details if it stays up ok.

 

[Thomas Keffer]

OK, the problem is your older version of configobj. However, I found a workaround that will work for all versions.

Look for v4.0.0b7 in the usual place.

Please try this version of WeeWX. Do not upgrade your configobj. We want to make sure it will work with all versions!

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/23d6cea0-8cb2-40f2-9464-93707911fc2e%40googlegroups.com.

[gjr80]

Just remembering the OP 'installed' the WeeWX v4 by cloning the development branch. I think the easiest way for him to update to b7 will be to pull down the updated development branch. The following is all that should be required:

$ cd /var/tmp/weewx4_src/weewx
$ git pull

Then you should be able to run wee_import as you have been.

Gary

On Monday, 6 January 2020 07:49:16 UTC+10, Thomas Keffer wrote:

OK, the problem is your older version of configobj. However, I found a workaround that will work for all versions.

Look for v4.0.0b7 in the usual place.

Please try this version of WeeWX. Do not upgrade your configobj. We want to make sure it will work with all versions!

-tk

On Sun, Jan 5, 2020 at 2:44 PM vince <vince...@gmail.com> wrote:

On Sunday, January 5, 2020 at 1:22:19 PM UTC-8, KnC Mc wrote:

What version should I be running?  yum indicates that I am running the recent version.  Have to go to work, but I will keep looking in my spare time.

I got the beta running in Simulator mode under python3 on centos7 after a bit of fiddling.   Let me let it run for a couple hours and I'll check it at halftime of the football game and post details if it stays up ok.

 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[vince]

Just to document it, for centos7 running b6 in Simulator mode under python3, you want to:

• install python3 with yum
• install all the required python modules with pip3
• also be careful to not let selinux break you, which it will do if you do things like  symlinking /home/weewx/public_html to under your webserver document root.  You can either fix selinux by relabeling things, or disabling selinux, depending on your preferences.
  

nginx is not available for centos7 unless you do:

• rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
  
• yum install nginx
  
• systemctl enable nginx
  
• systemctl start nginx

My Simulator VM has the following modules installed (some were autoselected):

• Cheetah3 (3.2.4)
  
• configobj (5.0.6)
  
• Pillow (7.0.0)
  
• pip (9.0.3)
  
• pyephem (3.7.7.0)
  
• pyserial (3.4)
  
• pyusb (1.0.2)
  
• setuptools (39.2.0)
  
• six (1.13.0)
  

The system has the following python3 rpms installed, again many were autoinstalled when I installed python3:

• python3.x86_64                              3.6.8-10.el7               @base
  
• python3-libs.x86_64                         3.6.8-10.el7               @base
  
• python3-pip.noarch                          9.0.3-5.el7                @base
  
• python3-setuptools.noarch                   39.2.0-10.el7              @base
  
• python3.i686                                3.6.8-10.el7               base
  
• python3-debug.i686                          3.6.8-10.el7               base
  
• python3-debug.x86_64                        3.6.8-10.el7               base
  
• python3-devel.i686                          3.6.8-10.el7               base
  
• python3-devel.x86_64                        3.6.8-10.el7               base
  
• python3-idle.i686                           3.6.8-10.el7               base
  
• python3-idle.x86_64                         3.6.8-10.el7               base
  
• python3-libs.i686                           3.6.8-10.el7               base
  
• python3-rpm-generators.noarch               6-2.el7                    base
  
• python3-rpm-macros.noarch                   3-32.el7                   base
  
• python3-test.i686                           3.6.8-10.el7               base
  
• python3-test.x86_64                         3.6.8-10.el7               base
  
• python3-tkinter.i686                        3.6.8-10.el7               base
  
• python3-tkinter.x86_64                      3.6.8-10.el7               base
  
• python3-wheel.noarch                        0.31.1-4.el7               base
  

Bottom line, there's nothing special needed to install v4 using python3 on centos7 - just use pip3 to install the modules.

[KnC Mc]

had to: 

cd /var/tmp/weewx4_src/weewx

rm /var/tmp/weewx4_src/bin/weewx/accum.py

(I had modified this file, and it was the only way I could figure out how to get git to pull)

then

git pull 

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --help

Traceback (most recent call last):

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 900, in <module>

    main()

  File "/var/tmp/weewx4_src/weewx/bin/wee_import", line 760, in main

    weeutil.logger.setup('wee_import', {})

  File "/var/tmp/weewx4_src/weewx/bin/weeutil/logger.py", line 83, in setup

    log_config = configobj.ConfigObj(StringIO(LOGGING_STR), interpolation=False)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1242, in __init__

    self._load(infile, configspec)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1302, in _load

    infile = self._handle_bom(infile)

  File "/usr/lib/python2.7/site-packages/configobj.py", line 1457, in _handle_bom

    if not line.startswith(BOM):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

I am willing to keep trying things if that is beneficial, or just try vince's solution.  Not in any hurry (I have been down since May).  Just trying to learn, and contribute.  I have a couple Raspberry Pi's that I might try to get this running on next weekend, but I have to get some SD Cards, and try the new buster install, then figure out how to get nginx running on it.    

[KnC Mc]

Don't run NGINX and I am not familiar with pip, so I have to look into that.  

I run NGINX on some Raspberry Pis, so I might try to adapt your instructions to one of those.

[KnC Mc]

Sounds contradictory?  I don't run NGINX on my CentOS machine (to clarify).  I have been running apache for 10 years or better, but installed NGINX a couple years ago just to see how hard it was.  Haven't really looked at it since.  Like git, I have issues 'pip' commands to support open source projects, but really have no idea how things work.  I am really more interested in learning python at the moment, so if pip is a repository (similar to cpan for perl?), I should probably dive into that sooner than later? 

[KnC Mc]

I checked out pip on one of my Pis and see it is a package manager for python.  Very cool!  I won't do anything on my CentOS machine with this experiment until I hear back.  I have a couple other projects at the moment, so I will work thru the weewx issue on my CentOS machine with your guidance.  I hope to have another instance up on a Pi next weekend, so I can explore the code, and tweak things on a machine that isn't something I worry about breaking.  ;)

[Thomas Keffer]

Apparently, I did not catch all the instances where a ConfigObj is initialized against a unicode string.

Try it now. Use "git pull" as Gary suggested to refresh your repository.

-tk

On Sun, Jan 5, 2020 at 7:40 PM KnC Mc <knc.geo...@gmail.com> wrote:

I checked out pip on one of my Pis and see it is a package manager for python.  Very cool!  I won't do anything on my CentOS machine with this experiment until I hear back.  I have a couple other projects at the moment, so I will work thru the weewx issue on my CentOS machine with your guidance.  I hope to have another instance up on a Pi next weekend, so I can explore the code, and tweak things on a machine that isn't something I worry about breaking.  ;)

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/00c4437b-8486-4ab4-90de-5c343bb6e993%40googlegroups.com.

[KnC Mc]

Look at that... Great Job! 

sudo PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --version

Using WeeWX configuration file /var/tmp/weewx4_src/weewx/weewx.conf

wee_import version: 0.4

I have to look at the differences in the conf file, and make the necessary changes to give it a test.   

[KnC Mc]

I added my 'api_key' and it acts like it is working, but the web pages are not getting updated...

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf

Using WeeWX configuration file /var/tmp/weewx4_src/weewx/weewx.conf

Starting wee_import...

Observation history for Weather Underground station 'KCAANAHE97' will be imported.

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

Starting import ...

Records covering multiple periods have been identified for import.

256 records identified for import.

Proceeding will save all imported records in the WeeWX archive.

Are you sure you want to proceed (y/n)? y

Unique records processed: 250; Last timestamp: 2020-01-06 20:49:52 PST (15783725Unique records processed: 256; Last timestamp: 2020-01-06 21:19:48 PST (1578374388)

Calculating missing derived observations ...

Processing record: 256; Last record: 2020-01-07 00:00:00 PST (1578384000)

Recalculating daily summaries...

Finished recalculating daily summaries

Finished calculating missing derived observations

Finished import

256 records were processed and 256 unique records imported in 30.47 seconds.

Those records with a timestamp already in the archive will not have been

imported. Confirm successful import in the WeeWX log file.

/var/tmp/weewx4_src/weewx/bin # ./wee_database --check

Using configuration file /var/tmp/weewx4_src/weewx/weewx.conf

Using database binding 'wx_binding', which is bound to database 'archive_sqlite'

Checking daily summary tables version...

Daily summary tables are at version 2.0

Interval Weighting Fix is not required.

Daily summary tables version check completed in 0.01 seconds.

Preparing Null String Fix, this may take a while...

 Checking record: 1408; Timestamp: 2020-01-06 21:19:48 PST (1578374388)

No null strings found.

Completed Null String Check in 0.09 seconds.

/var/tmp/weewx4_src/weewx/bin # ./wee_reports 

Using configuration file /var/tmp/weewx4_src/weewx/weewx.conf

Generating for all time

/var/log/messages...

Jan  6 21:37:14 www2 journal: wee_import[32763] INFO weeimport.wuimport: Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Jan  6 21:37:15 www2 journal: wee_import[32763] INFO weewx.manager: Added record 2020-01-05 00:04:50 PST (1578211490) to database 'weewx.sdb'

Jan  6 21:37:15 www2 journal: wee_import[32763] INFO weewx.manager: Added record 2020-01-05 00:04:50 PST (1578211490) to daily summary in 'weewx.sdb'

*** Multiple lines nipped for conciseness ***

Jan  6 21:37:18 www2 journal: wee_import[32763] INFO weewx.manager: Added record 2020-01-05 23:59:48 PST (1578297588) to database 'weewx.sdb'

Jan  6 21:37:18 www2 journal: wee_import[32763] INFO weewx.manager: Added record 2020-01-05 23:59:48 PST (1578297588) to daily summary in 'weewx.sdb'

Jan  6 21:37:18 www2 journal: wee_import[32763] INFO weewx.wxservices: The following values will be calculated: altimeter=prefer_hardware, appTemp=prefer_hardware, barometer=prefer_hardware, beaufort=prefer_hardware, cloudbase=prefer_hardware, dewpoint=prefer_hardware, ET=prefer_hardware, heatindex=prefer_hardware, humidex=prefer_hardware, inDewpoint=prefer_hardware, maxSolarRad=prefer_hardware, pressure=prefer_hardware, rainRate=prefer_hardware, windchill=prefer_hardware, windrun=prefer_hardware

Jan  6 21:37:18 www2 journal: wee_import[32763] INFO weewx.wxservices: The following algorithms will be used for calculations: altimeter=aaASOS, maxSolarRad=RS

Jan  6 21:37:20 www2 journal: wee_import[32763] INFO weewx.manager: Starting backfill of daily summaries

Jan  6 21:37:20 www2 journal: wee_import[32763] INFO weewx.manager: Processed 288 records to backfill 1 day summaries in 0.51 seconds

Jan  6 21:37:30 www2 wee_reports[300]: copygenerator: copied 9 files to /www/html/weewx

I verified the .htm files were updated in my weewx directory with the import date, but the data is still from 5/20/2019.  

I am not familiar with getting a Select out of the sql database for weewx.  Is there a simple command I can use to see if the data from 1/5/2020 or 1/6/2020 actually exists in the database?  Thanks. 

[gjr80]

Hang on, I'm a little confused with what you have posted about. You performed a WU import on 6 January with no --date option:

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf

so wee_import will have imported data for the current day and that is what is confirmed in the wee_import console output:

Unique records processed: 250; Last timestamp: 2020-01-06 20:49:52 PST (15783725Unique records processed: 256; Last timestamp: 2020-01-06 21:19:48 PST (1578374388)

yet the log extract you presented shows records being imported from 5 January. Maybe there is an error in wee_import or maybe we are looking at the log from another import? Also, wee_import reports it took 30 odd seconds to execute yet the log indicates somewhere around 15 seconds. Noit trying to be smart but want to make sure that we are actually looking at log output from the same wee_import sessison, the answers are in the log but you need to be looking at the answer to your question not some other question.

No need to check the database to confirm what was added, the line

Jan  6 21:37:18 www2 journal: wee_import[32763] INFO weewx.manager: Added record 2020-01-05 23:59:48 PST (1578297588) to database 'weewx.sdb'

tells you that a record stamped 2020-01-05 23:59:48 PST was added to your database, if you imported 6 January and the only such entries are for 5 January then something is wrong.

Gary

[gjr80]

Actually, thinking some more, WU has shown some peculiar behaviour when querying their API for the current day. Why not try importing from an earlier day.

Gary

[KnC Mc]

Good eye!!! You are quite correct!  My bad again.  I have actually issued a few import commands:

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --from=2019-05-20T07:00 --to=2019-05-20T23:59

--- just a couple log entries ---

Jan  6 21:22:55 www2 journal: wee_import[31892] INFO weewx.manager: Added record 2019-05-20 23:54:48 PDT (1558421688) to database 'weewx.sdb'

Jan  6 21:22:55 www2 journal: wee_import[31892] INFO weewx.manager: Added record 2019-05-20 23:54:48 PDT (1558421688) to daily summary in 'weewx.sdb'

A ton of successful hits, but the html did not update.

---

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf 

Success again, but when I ran it a second time I see this:

Jan  6 21:43:48 www2 journal: wee_import[759] ERROR weewx.manager: Unable to add record 2020-01-06 21:19:48 PST (1578374388) to database 'weewx.sdb': PRIMARY KEY must be unique

which I believe tells me that the records were imported on the first run, but because they exist it can not import them due to the primary key exists, which is probably set to the date stamp?

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-05

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-04

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-03

All of these have similar entries telling me Added records and the same fail when running a second time.

I ran 'wee_database --check' and 'wee_reports' just because I ran those in my cron job before everything broke.  I was thinking one of those updated the html, and png files.

I thought about it before I ran the commands, and I could have a permission issue on the directory because the cron ran as root, so I logged in as root before issuing the commands.  Might have to go back in and fix some permissions after, but running as root should have resolved any permission issues.  However when I cd to the weewx directory on my web server and issue:

lynx index.html

-----

05/20/2019 07:04:57 AM

   Current Conditions

   Outside Temperature       57.6°F

   Wind Chill                57.6°F

-----

the previous date is shown in the html.  :(

[gjr80]

A few comments embedded below.

Gary

On Tuesday, 7 January 2020 17:56:32 UTC+10, KnC Mc wrote:

Good eye!!! You are quite correct!  My bad again.  I have actually issued a few import commands:

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --from=2019-05-20T07:00 --to=2019-05-20T23:59

--- just a couple log entries ---

Jan  6 21:22:55 www2 journal: wee_import[31892] INFO weewx.manager: Added record 2019-05-20 23:54:48 PDT (1558421688) to database 'weewx.sdb'

Jan  6 21:22:55 www2 journal: wee_import[31892] INFO weewx.manager: Added record 2019-05-20 23:54:48 PDT (1558421688) to daily summary in 'weewx.sdb'

A ton of successful hits, but the html did not update.

---

OK, those log entries are consistent with the command, no issue here.

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf 

Success again, but when I ran it a second time I see this:

Jan  6 21:43:48 www2 journal: wee_import[759] ERROR weewx.manager: Unable to add record 2020-01-06 21:19:48 PST (1578374388) to database 'weewx.sdb': PRIMARY KEY must be unique

which I believe tells me that the records were imported on the first run, but because they exist it can not import them due to the primary key exists, which is probably set to the date stamp?

Correct, wee_import will attempt to write an imported record to archive whether it exists or not. If a record with that timestamp already exists the WeeWX API logs the failure and moves on. Unfortunately wee_import has no way of knowing whether the record was successfully added or not hence the seemingly vague messages on the console about records processed (not saved) and the need to check the log to see what was actually saved. Again nothing wrong here.

 

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-05

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-04

PYTHONPATH=/var/tmp/weewx4_src/weewx/bin /var/tmp/weewx4_src/weewx/bin/wee_import --import-config=/var/tmp/wu.conf --date=2020-01-03

All of these have similar entries telling me Added records and the same fail when running a second time.

As explained above. 

I ran 'wee_database --check' and 'wee_reports' just because I ran those in my cron job before everything broke.  I was thinking one of those updated the html, and png files.

I would not bother with wee_database,  it will add nothing here, the checks performed check for an old algorithm that was used in an earlier version as well as null strings; you will have none of those from this version of wee_import.

I thought about it before I ran the commands, and I could have a permission issue on the directory because the cron ran as root, so I logged in as root before issuing the commands.  Might have to go back in and fix some permissions after, but running as root should have resolved any permission issues.  However when I cd to the weewx directory on my web server and issue:

lynx index.html

Not sure which directory you refer to re your permissions problem but if it is to do with wee_reports I would expect you would see some complaints in the log if there was a problem. Could be wrong on that though. Moving on, what is index.html? A report generated by Cheetah via a skin? What is in the report template used to generate index.html?

[KnC Mc]

Thank you for making me think... I found the weewx.conf file I stupidly forgot to modify, making the assumption that because I didn't specify a --config on the command like that it was pulling it from my previous weewx.conf... but now that doesn't make sense since I am passing the PYTHONPATH to tell it where all the files are?  SO I have replaced the sample config data in /var/tmp/weewx4_src/weewx/weewx.conf to better match what I previously had.  I still have to look at the paths more when I get time, because there are more options than I had in the 3.8.0 weewx.conf file, and I still can not seem to get the html files to update.  I believe I was using the standard template before (slightly modified).  Nothing Fancy, just added some extra text and a footer via css that had a drop down menu for day/week/month/year.  

I will look at this again tomorrow.  Thanks Again!

[KnC Mc]

I have the v4b wee_import working thru cron now.  I was able to use it to back fill my database to May when things stopped working.  

I am still trying to look at the new version of wee_reports to try and get the cheetahgenerator to create the graphs, but unsuccessful so far.  I need to do more reading to understand the settings again?  I found that my v3.8.0 wee_reports generated the graphics on the first run, so I added that to my cron for the time being, but it seems to have stopped on the last run.  I am going to give it a couple more tries as I made some changes to my script to specify the config files.

I am not sure if WU will throttle my usage, so I set my cron to twice a day, and I will see how it goes.  I will nose around the docs folder and see if their are other utilities I might be able to beta test.