formatting issue in cheetah

284 views
Skip to first unread message

vince

unread,
Aug 24, 2023, 3:05:29 PM8/24/23
to weewx-user
A couple cheetah questions...

I'm displaying AQI data which uses group_concentration and it displays oddly for me.  Is there a simple fix to remove the funky 'A' characters before/after the value ? 

(see attached screenshot)

Second question - the values are coming from the following:

AQI: $latest('purpleair_binding').aqi2_5.format(format_string="%.0f")    ($latest('purpleair_binding').pm2_5_cf_1.format(format_string="%.1f"))

I have not found a combination of formatting directives to show the second value without the units being displayed 'and' with it formatted down to one decimal place. 

If I have a value of 15.6123 micrograms/cubic-meter how would I display just 15.6 ?


Screenshot 2023-08-24 at 11.59.23 AM.png

Karen K

unread,
Aug 24, 2023, 3:13:31 PM8/24/23
to weewx-user
I guess that there is an issue with the character set ISO8859-1 vs. UTF-8.

May be, #encoding is missing at the beginning of the Cheetah template file or does not match the real encoding of the file.

Or the template is saved in ISO8859 character set instead of UTF-8.

vince

unread,
Aug 24, 2023, 3:29:46 PM8/24/23
to weewx-user
I have absolutely no idea what you just said.  Sorry - US ASCII person here :-)

View source shows 
AQI: 56    (14.6 µg/m³)

and https://www.w3schools.com/charsets/ref_html_8859.asp matches up (circumflex accent and micro symbol) but I have no idea what to do about it.


Karen K

unread,
Aug 24, 2023, 3:54:21 PM8/24/23
to weewx-user
Unfortunately the greek my character and the cubic symbol are not US-ASCII. So they have the 8th bit set. There are different methods to do that. The older one is ISO 8859, where one byte is always one character. The newer one is UTF-8 where one character can be 1, 2 or even more bytes. So if you have an UTF-8 character, containing of 2 bytes interpreted as ISO 8859, you see 2 characters instead of the one you want to see. 

So the question is: What does the browser think about the character set? And what character set is used for Cheetah?

See the first line of the templates *.html.tmpl. There is a line beginning with #encoding followed by the name of a character set. That name must match the character set you used to save the file.

Then there may be a line <meta charset="UTF-8" /> (or ISO-8859-1 instead of UTF-8).

Then, in skin.conf there may be lines "encoding = ...". What do they say?

Make sure, all the encoding directives match your file's encoding.

vince

unread,
Aug 24, 2023, 5:19:12 PM8/24/23
to weewx-user
Well I can't explain it at all.  I appended the same .tmpl line at the bottom of index.html.tmpl for the Seasons skin (trusting Tom gets this stuff correctly) and I still see funny encoding in Safari + Chrome (mac) and in Chrome (iphone).   I can only guess it has something to do with the locale or i18n settings on the pi4 that weewx is running on, but I'm just guessing completely at this point.

gjr80

unread,
Aug 24, 2023, 6:16:01 PM8/24/23
to weewx-user
I notice the Seasons index.html.tmpl does not have a #encoding directive (though the other Seasons .html.tmpl files do). Have you tried adding a #encoding directive à la:

#errorCatcher Echo #encoding UTF-8 <!DOCTYPE html>

Gary

vince

unread,
Aug 24, 2023, 6:30:43 PM8/24/23
to weewx-user
Thanks.  No joy.

Added to top of the file under the echo line ala...

## Copyright 2009-2018 Tom Keffer, Matthew Wall
## Distributed under terms of GPLv3.  See LICENSE.txt for your rights.
#errorCatcher Echo
#encoding UTF-8

#set $periods = $to_list($DisplayOptions.get('periods', ['day', 'week', 'month', 'year']))
#set $plot_groups = $to_list($DisplayOptions.get('plot_groups', ['tempdew', 'wind', 'rain']))

## use this span to determine whether there are any data to consider.
#set $recent=$span($day_delta=30, boundary='midnight')

<!DOCTYPE html>
<html lang="$lang">
  <head>
    <meta charset="UTF-8">
    <title>$station.location</title>
    <link rel="icon" type="image/png" href="favicon.ico" />
    <link rel="stylesheet" type="text/css" href="seasons.css"/>
    #if $station.station_url
    <link rel="canonical" href="$station.station_url/$filename" />
    #end if
    <script src="seasons.js"></script>
  </head>

  <body onload="setup();">
    #include "titlebar.inc"

And appended the AQI line to the footer to make it easier for me to find during testing...

    <p class="footnote">
      $gettext("This station is controlled by <a href='https://weewx.com'>WeeWX</a>, an experimental weather software system written in Python.")

      AQI: $latest('purpleair_binding').aqi2_5.format(format_string="%.0f")    ($latest('purpleair_binding').pm2_5_cf_1.format(format_string="%.1f"))


Result looks like:
      This station is controlled by WeeWX, an experimental weather software system written in Python. AQI: 64 (17.9 µg/m³)

Output in the .html is:
   <p class="footnote">
      This station is controlled by <a href='https://weewx.com'>WeeWX</a>, an experimental weather software system written in Python.
AQI: 64    (17.9 &#194;&#181;g/m&#194;&#179;)

Tom Keffer

unread,
Aug 24, 2023, 6:58:44 PM8/24/23
to weewx...@googlegroups.com
Internal to WeeWX, the label microgram_per_meter_cubed is encoded in Unicode. To be included in a template, it has to be converted to an appropriate byte string using an encoding. By default, the Seasons templates use html entities to do this encoding. The results will be in simple ASCII.

Note: this is separate from the encoding used by the templates themselves, which is generally UTF-8. This is what you're setting with the #encoding directive. This is useful if one were to include UTF-8 characters directly in the templates themselves. But, that's not what's happening here.

The html entity encoding for the cube symbol is either "&sup3;", or "&#179;". For whatever reason, your Python installation is choosing the latter, which your browser does not seem to be capable of interpreting as the cube symbol. I've seen this happen when the file suffix is ".txt", but yours is ".html" (check!).

It may be a browser specific thing. What browser are you using? Perhaps you can point us to the page?

One thing you can try is another encoding besides html entities. For example, utf-8.

[CheetahGenerator]
  encoding=utf_8 



--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d39d52c7-3a0d-4a2a-93a8-97fb2f329b7fn%40googlegroups.com.

vince

unread,
Aug 24, 2023, 8:15:22 PM8/24/23
to weewx-user
URL that rsync uploads to is https://www.skahan.net/weewx/purpleair.html - this will not be available unless you're in US/CA/AU probably due to some geoip blocking that I have set up on the ISP side to try to limit how many bots bang on the site.

Same thing happens on chrome on mac os and iphone as well as Safari on mac os.   The weewx host is a current RaspiOS pi4.   Adding the encoding=utf_8 line didn't help, unfortunately.  The encoding was previously set to html_entities in skin.conf

I've attached the .tmpl file and skin.conf for the skin in case there are obvious errors therein.  FWIW the skin is kinda ancient so it's possible I missed adding something along the way.
purpleair.html.tmpl
skin.conf

Tom Keffer

unread,
Aug 25, 2023, 7:10:00 PM8/25/23
to weewx...@googlegroups.com
Both the template and the skin configuration look fine.

Despite being just down the road from you, I am unable to connect to your webserver.



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/34810d7b-e939-4f26-9868-41da36ffc992n%40googlegroups.com.

Greg from Oz

unread,
Aug 25, 2023, 7:29:00 PM8/25/23
to weewx-user
I am in Australia and https://www.skahan.net/weewx/purpleair.html works. So the Australian geo block is working OK.
Screenshot from 2023-08-26 09-25-36.png

Tom Keffer

unread,
Aug 25, 2023, 8:32:28 PM8/25/23
to weewx...@googlegroups.com
The problem is the 'xmlns' declaration in your <html> tag. It is causing metadata to be pulled in, which says that the character set is "windows-1252".

Change this

<html xmlns="http://www.w3.org/1999/xhtml">
  <head profile="http://www.w3.org/2005/10/profile">

to this

<html lang="en">
  <head>

and I think your troubles will go away.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/36b2cb84-ceb4-4898-9f5b-dd351ebb0461n%40googlegroups.com.

vince

unread,
Aug 25, 2023, 9:08:56 PM8/25/23
to weewx-user
No joy.

Try "test.html" now for a far simpler variant.
It'll take a few minutes for the rsync to update the public version.

Here's the page source.

<html lang="en">
  <head>
  <title> test page </title>
  </head>

  <body>
          <table>
            <tbody>
              <tr>
                <td class="stats_label">AQI (current)</td>
                <td class="stats_data">80.5</td>
              </tr>
              <tr>
                <td class="stats_label">AQI (last hour)</td>
                <td class="stats_data">72.25</td>
              </tr>
              <tr>
                <td class="stats_label">AQI (last day)</td>
                <td class="stats_data">76.80139372822299</td>
              </tr>
              <tr>
                <td class="stats_label">pm2_5_cf_1</td>
                <td class="stats_data">25.960  &#194;&#181;g/m&#194;&#179;</td>
              </tr>
            </tbody>
          </table>
   </body>
</html>

Tom Keffer

unread,
Aug 25, 2023, 9:47:54 PM8/25/23
to weewx...@googlegroups.com
Your little test page is still being interpreted by Chrome as being encoded in "windows-1252". Try including a meta tag for the encoding:

<html lang="en">
<head>
    <title> test page </title>
    <meta charset="UTF-8">
</head>
<body>

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c46b679f-4c2a-407e-83f9-57836f539714n%40googlegroups.com.

vince

unread,
Aug 25, 2023, 11:04:46 PM8/25/23
to weewx-user
Did not help.  Is it possible there is something in the nginx configuration 'or' the combination of python3/cheetah in the venv that might be at fault ?   I'm running a v5 beta pip install FWIW.

Here's the simplest example I can cook up.
Simply try to show the units for something in group_concentration....

#errorCatcher Echo

<html lang="en">
<head>
  <title> test page </title>
  <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
      This should look ok ==> $unit.label.pm2_5_cf_1<
   </body>
</html>

vince

unread,
Aug 25, 2023, 11:05:49 PM8/25/23
to weewx-user
Doh - delete that last < in the line above before the body of course...

Tom Keffer

unread,
Aug 26, 2023, 8:37:45 AM8/26/23
to weewx...@googlegroups.com
After I put the following in user.extensions, your little test template worked for me, using both html entities, and utf-8 encoding.

import weewx.units
weewx.units.obs_group_dict['pm2_5_cf_1'] = 'group_concentration'

I assume you did the same?

The label for the unit microgram_per_meter_cubed is normally defined in weewx/defaults.py. Did you change this?

Are labels for other observation types, such as pm2_5, getting garbled? Or, just pm2_f_cf_1?

One thing I'm noticing: your test template is emitting 

&#194;&#181;g/m&#194;&#179;

One thing I'm noticing: the character sequence &#194;&#181; is hex C2 B5, which is the UTF-8 encoding for the micro character. Except it's being emitted using HTML entities for C2 and for B5, instead of the actual bytes. Same with the cubed symbol.

Not sure what to make of that.


To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8792f25d-84c0-4e4b-8fd9-9526a13a4a77n%40googlegroups.com.

vince

unread,
Aug 26, 2023, 2:40:22 PM8/26/23
to weewx-user
I had not changed defaults.py but I 'had' neglected to add to extensions.py, which didn't help unfortunately.

All the labels in my skin for concentrations that refer to elements from the secondary purpleair.sdb are garbled even when added to extensions.py but if I check some single character labels in weewx.sdb (outTemp, radiation) those look ok.  Checking the concentration label (pm2_5) used in weewx.sdb looks garbled too.  Seems to be something with labels with more than one character to interpret perhaps.  I dunno.  So odd.

I tried adding my test page to Seasons, thinking skin.conf perhaps, but that didn't help.  Garbled there too.

Switching to encoding=utf-8 breaks the degree symbol everywhere in 'my' skin, garbling it similar to the concentration label.  Interestingly when I go utf-8 in skin.conf for Seasons 'that' skin's output still looks fine (hmmm) other than my test page therein which still looks garbled (sigh....)

I'm going to try my modified test page on v4 .deb in a x86 VM and take the pi os and v5 pip/venv differences out of the equation and see if there's any difference.

#errorCatcher Echo
<html lang="en">
<head>
  <meta charset="utf-8">

  <title> test page </title>
  </head>

  <body>
These are in purpleair.sdb
<br>  pm1_0_cf_1           $unit.label.pm1_0_cf_1
<br>  pm1_0_atm            $unit.label.pm1_0_atm
<br>  pm2_5_cf_1           $unit.label.pm2_5_cf_1
<br>  pm2_5_atm            $unit.label.pm2_5_atm
<br>  pm10_0_cf_1          $unit.label.pm10_0_cf_1
<br>  pm10_0_atm           $unit.label.pm10_0_atm
<p>
These are in weewx.sdb
<br>  pm1_0                $unit.label.pm1_0
<br>  pm2_5                $unit.label.pm2_5
<br>  pm10_0               $unit.label.pm10_0
<br>  outTemp              $unit.label.outTemp
<br>  radiation            $unit.label.radiation
   </body>
</html>

vince

unread,
Aug 26, 2023, 3:10:52 PM8/26/23
to weewx-user
In deb12 dpkg of weewx 4 in a x86_64 vm all the pm labels are still garbled using my test file even when run out of the Seasons skin.

What is interesting is that 'without' adding the elements to extensions.py it somehow knows that the label is micrograms per cubic meter, it just doesn't know how to format that label.   Adding the items to extensions.py had no effect either way. Still garbled.

I think it wins.

Tom Keffer

unread,
Aug 27, 2023, 7:48:09 AM8/27/23
to weewx...@googlegroups.com
I must admit to being a little lost.

I wrote an article Character encodings to explain how encodings work within WeeWX. Maybe you'll recognize something?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b435f3c6-c25b-474c-8d61-d5595b6686cfn%40googlegroups.com.

jterr...@gmail.com

unread,
Aug 27, 2023, 10:04:17 AM8/27/23
to weewx-user
hex C2 B5 is the UTF-8 binary encoding of "micro" in a "text" file such as a python script.
But as far as HTML is concerned, the html  UTF-8 encoding of "micro" is either &#181; or &#xB5; or &micro;  - see https://www.w3schools.com/charsets/tryit.asp?deci=181&ent=micro

May be  cheetah converts the two binary UTF-8 codes found in defaults.py , but in that case it is not working for html.

vince

unread,
Aug 27, 2023, 11:23:40 AM8/27/23
to weewx-user
I guess if anybody else could try the test.index.html file I previously posted in https://groups.google.com/g/weewx-user/c/nW3U8DVMZyU/m/qSosva-hAAAJ on their systems it might help, if you have time to do so of course.  If you get it working somehow, let us know and if there's a URL to let others access it from their browser(s) that would be helpful too.

For me, I cannot find a combination where the group_concentration label looks correct when the .html is generated.

jterr...@gmail.com

unread,
Aug 27, 2023, 11:43:33 AM8/27/23
to weewx-user
here is the result of your test.index.html in my system  :  http://meteo-sciez.fr/weewx/tagssciez/test.html

the pmxxx units are correctly shown, but not the outTemp and radiation units !

vince

unread,
Aug 27, 2023, 12:48:08 PM8/27/23
to weewx-user
Fascinating.  I came up with a modified test template that prints out the units for everything in the extended schema, but my little one is sufficient to show the issue.

For me if encoding=html_entities then labels using group_concentration are garbled, but temperatures and radiation units and wind direction are ok.   If I switch the skin to encoding=utf-8 then concentration, radiation, and anything with a degree symbol in it is garbled.

I've attached a modified .tmpl that displays the labels for all elements in the extended schema.  Maybe this helps.
test.html.tmpl

Tom Keffer

unread,
Aug 27, 2023, 3:45:19 PM8/27/23
to weewx...@googlegroups.com
I tried your test template (on my Mac) and it worked perfectly using both HTML entities and UTF-8 encodings. The results are attached.

I'm grasping here, but it must have something to do with your environment. What do you get for

echo $LANG

python3 -c "import locale;print(locale.setlocale(locale.LC_ALL))"

python3 -c "import locale;locale.setlocale(locale.LC_ALL,'');print(locale.setlocale(locale.LC_ALL))"


-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5de3914e-24e4-4592-adf8-6bcbf9625400n%40googlegroups.com.
test-utf8.html
test-entities.html

Jacques Terrettaz

unread,
Aug 27, 2023, 3:57:09 PM8/27/23
to weewx...@googlegroups.com
Tom, here are my settings :

root@meteopi:~# echo $LANG

fr_CH.UTF-8

root@meteopi:~# python3 -c "import locale;print(locale.setlocale(locale.LC_ALL))"

LC_CTYPE=fr_CH.UTF-8;LC_NUMERIC=C;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C

root@meteopi:~# python3 -c "import locale;locale.setlocale(locale.LC_ALL,'');print(locale.setlocale(locale.LC_ALL))"

fr_CH.UTF-8




You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nW3U8DVMZyU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECeEPkcBP7ChEV77OXaTxVkG6NuquCk%3Dg3o_4OAjCNRzw%40mail.gmail.com.
<test-utf8.html><test-entities.html>

vince

unread,
Aug 27, 2023, 5:37:05 PM8/27/23
to weewx-user
My m1 mini running latest MacOS worked fine with the test.html.tmpl template.
Here are the python diagnostics.

$ echo $LANG

python3 -c "import locale;print(locale.setlocale(locale.LC_ALL))"
python3 -c "import locale;locale.setlocale(locale.LC_ALL,'');print(locale.setlocale(locale.LC_ALL))"

en_US.UTF-8
C/en_US.UTF-8/C/C/C/C
en_US.UTF-8


My pi4 has the issue still.  Here are the diagnostics for the pi.
I got the same command results below without activating the venv too.

pi@pi4:~$ source ~/weewx-venv/bin/activate
(weewx-venv) pi@pi4:~$ echo $LANG

python3 -c "import locale;print(locale.setlocale(locale.LC_ALL))"
python3 -c "import locale;locale.setlocale(locale.LC_ALL,'');print(locale.setlocale(locale.LC_ALL))"
en_US.UTF-8
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C
en_US.UTF-8


vince

unread,
Aug 27, 2023, 5:50:38 PM8/27/23
to weewx-user
Tom - my lightsail VM works fine.  Ubuntu 22.04.3 LTS all patched up.  Note the diagnostic results differ from the (garbled) pi.

$ echo $LANG
python3 -c "import locale;print(locale.setlocale(locale.LC_ALL))"
python3 -c "import locale;locale.setlocale(locale.LC_ALL,'');print(locale.setlocale(locale.LC_ALL))"

C.UTF-8
LC_CTYPE=C.UTF-8;LC_NUMERIC=C;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C
C.UTF-8

Tom Keffer

unread,
Aug 27, 2023, 5:59:27 PM8/27/23
to weewx...@googlegroups.com
I guess the next question is what version of Python and Cheetah?

python3 -V
python3 -c "import Cheetah;print(Cheetah.Version)"

Mine is Python 3.7.16 and Cheetah V3.3.2

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9752dfa4-6be1-4509-8049-5fe348653496n%40googlegroups.com.

vince

unread,
Aug 27, 2023, 6:18:00 PM8/27/23
to weewx-user
Garbled pi4:
(weewx-venv) pi@pi4:~$ python3 -V

python3 -c "import Cheetah;print(Cheetah.Version)"
Python 3.9.2
3.3.1

Working mac mini:
python3 -c "import Cheetah;print(Cheetah.Version)"
Python 3.10.9
3.3.2

Working AWS lightsail ubuntu:
$ python3 -V

python3 -c "import Cheetah;print(Cheetah.Version)"
Python 3.10.12
3.3.2

Tom Keffer

unread,
Aug 27, 2023, 6:47:08 PM8/27/23
to weewx...@googlegroups.com
I have tried Python 3.7.10, 3.10.5, and 3.11.4, with Cheetah 3.2.6, 3.3.1, and 3.3.2. I even tried Python 3.9.2 with Cheetah 3.3.1 (your combination). All gave the expected results.

Unfortunately, I am unable to reproduce what you're seeing.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b04e8c8f-99f7-442a-be91-63d0518c241cn%40googlegroups.com.

vince

unread,
Aug 27, 2023, 7:17:25 PM8/27/23
to weewx-user
That's ok.  Thanks for trying.  There is definitely something going on with locale/cheetah since Jacques had odd results too, but this one might just not be readily solvable.

Jim Munro

unread,
Aug 27, 2023, 11:47:22 PM8/27/23
to weewx-user
just a guess.  entry missing in units section of weewx.conf.  Check locale setting in raspi-config on RPI. ???
Reply all
Reply to author
Forward
0 new messages