How to find a line in index.html.tmpl from the error log?

[WindnFog]

Hi guys. I'm a little late converting from python 2.7 to python 3.7, but true to my word last autumn, I'm now up to my elbows into it. It's going well, and I'm using Matthews sftp extension to move it to my VPS web server in Toronto, etc. This project will keep me busy for a while.

My question is sort of simple. I have my index.html.tmpl heavily customized, and I want to keep the features I've added over the past 4-5 years. Some of them are causing it to choke because what I'm doing has to be coded differently in Python 3.7. I'll figure all that out. The question I have is: How do I find the line causing the error in index.html.tmpl? Here's an example:

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] INFO weewx.imagegenerator: Generated 15 images for report SeasonsReport in 0.54 seconds

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: Generate failed with exception '<class 'TypeError'>'

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/Standard/index.html.tmpl

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** Reason: unsupported operand type(s) for *: 'NoneType' and 'float'

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** Traceback (most recent call last):

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** File "/usr/share/weewx/weewx/cheetahgenerator.py", line 323, in generate

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** unicode_string = compiled_template.respond()

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** File "_etc_weewx_skins_Standard_index_html_tmpl.py", line 1135, in respond

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] ERROR weewx.cheetahgenerator: **** TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

Dec 29 16:15:18 carrot python3[9457]: weewx[9457] INFO weewx.cheetahgenerator: Generated 13 files for report StandardReport in 0.67 seconds

Dec 29 16:15:19 carrot python3[9457]: weewx[9457] INFO weewx.imagegenerator: Generated 12 images for report StandardReport in 0.43 seconds

Dec 29 16:15:19 carrot python3[9457]: weewx[9457] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx

This specific error doesn't matter because I can fix it if I can find the offending like. Because index.html.tmpl goes through cheetah, I don't think line 323 is the one causing the problem. I guessed correctly at a few, but I know I'm going to get quite a few more. I need a way to narrow it down. My version of index.html.tmpl is 867 lines, compared to the stock one of 527 lines.

- Paul VE1DX

[vince]

Cheetah is notoriously hard to debug.  It seeeeems like you are trying to do math in your template on a value that is a NoneType (ie, on a reading that has no value) but that's just a guess.  

Best I can suggest is the old sysadmin credo of 'when in doubt, comment it out' and try to build back your modified version tweak by tweak.  Or perhaps post your .tmpl as an attachment and somebody might be able to take a look for likely areas it might go boom if it gets unexpected input (or lack of input).

[gjr80]

Hi,

Short answer is there is no direct way of obtaining the offending line number in your template file. It would be helpful if we had access to the temporary .py file that Cheetah creates (for which the line number is provided) but unfortunately we don’t have access to that file. The best you can do is to infer from the error message. In the case above you are looking for a multiplication operation for which the left hand operand is (could be) None and the right hand operand is a float. Usually not too hard to spot possible problem locations (unless you have lots and lots of multiplication operators). Also, NoneType issues typically come from WeeWX fields/obs that are unexpectedly None, so something like $current.outTemp.raw * 1.2 would be a prime candidate (outTemp could be None for some reason) but 3.23 * 1.2 (no chance of None) is not nor would $current.outHumidity.raw * 2 (does not use a float).

If you are unfortunate enough to get an error message that is not very descriptive, or you have just too many possible error sites in your template, your only real option is to start breaking down your template into smaller chunks to find the location. I favour splitting the template in half, find the offending half template and then breaking that into halves and repeating the process, tedious but about the best you can do.

This is a good incentive to make sure your code can tolerate variable/fields being None.

Gary

[WindnFog]

Thanks, guys.  That's what I figured.  While I'm no Python guru, I was in the IT field for 39 years.  This isn't the first time I ran into things like this.  I will do as you both suggest.  I can add a "feature" to the stock file one at a time, and fix the errors as I encounter them.  The good news is I have two parallel systems . . . one is my live site:

https://www.ve1dx.net

And the other is sending the simulator data to a subdirectory under it.  I even have two Davis receivers,  each with a data logger and USB interface.  I do so because I have this obsession not to let my site go offline (failing power outages.)  I always try to have a backup plan and duplicate the hardware if possible.  I used to manage Unix servers for up to 2000 clients, and downtime was a no-no!

So . . . it looks like I have my work cut out for me.  It's something to do during this long, cold, dark winter.  One bite (byte?) at a time is the way to go when facing an elephant.

- Paul VE1DX

[p q]

I just went through this same process. Either comment stuff out or start with a clean system and add back in each piece one at a time.

You might check the if statements for None. I think this is one of the things that tripped me up. Like:

#if $day.rain.sum.raw is not None and $day.rain.sum.raw > 0.0

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fd7aac2f-2b0f-41ce-a379-6f3898c37b12n%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[vince]

Running the simulator and looping through "make a change, manually run wee_reports, check the log, check the results" works pretty well as one way of doing it.   I don't think you even need weewx running to do that.   I typically just spin up a Simulator VM with vagrant+VirtualBox and battle through it.

But cheetah can certainly make it unpleasant to say the least.

[vince]

FWIW, I've had some luck with the approach in one of my skins (here) where I check for $current.whatever.has_data to show things in my skin only if there's data to show. I don't know if that's a preferred failsafe construct though.  Can something.has_data be true and still have the value be None ?  I dunno.

[WindnFog]

Vince, Peter, Gary, and of course Tom,

It's done! I used suggestions from all of you, and it wasn't as painful as I had thought. There was a lot of math in index.html.tmpl, but I can't say exactly which line(s) were causing the errors. My Pi's Raspiban O/S was sort of messed up because of a year of hacks, upgrades, and other unknowns. I rebuilt the O/S from scratch and added the latest versions of the extensions I needed. Then I performed a clean install of weewx 4.2.0. 

I went through index.html.tmpl using more or less a "binary search" method by commenting out half, then a quarter, then an 8th, etc. Each iteration resolved something.

This also gave me a chance to fix up weewx.conf that had been modified with parts commented out over the years, etc. As of today, my system is running flawlessly on Python 3.7.3. It's reassuring to know everything is newly installed and up to date. This had been one of those things on my "to do" list for 6-8 months. My takeaway: As with most software you've upgraded for some time, there comes the point when you need to bite the bullet and do a fresh install.

A big thank YOU to everyone who puts up with my questions. Sometimes they are apparent and in the documentation, and other times they are a bit obscure. Either way, I can say that weewx has been running my weather station for over 5 years, and it's been a joy to use.

https://ve1dx.net/

- Paul VE1DX