PM.5 units bargled a bit in history PNG files...

[richard maliszewski]

This is a minor annoyance, but I am failing to figure out where this is being done, or how to fix it.  Units for PM2.5 are micrograms per meter cubed.  I've got a "current" where the units look as expected: µg/m³.

But in the generated history images, the units in the upper show as shown in this image: 

Station data is coming from the WX1000 extension and in this case, some of it from the PurpleAir extension.  Looking through both extensions, I am not seeing who/what is choosing the unit string for these values, nor where it's getting sucked into the image generator.  Hints?  I am a week into using WeeWX, and am quite impressed, but finding where the bodies are buried for this sort of magic.

--Richard

[gjr80]

Hi,

Nothing to do with drivers or extensions, it’s a font issue. Have a read of the section Funky symbols in plots in the User’s Guide.

Gary

[richard maliszewski]

Dug through that.  The font referenced for unit_label_font_path in skin.conf for Seasons is "font/OpenSans-Bold.ttf", which does exist.  Thinking it might not have full support for the Unicode math symbols, I changed that to "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", which supposedly covers all 256 symbols.  And I still have the weird unit label...nothing there changed.  I see no error messages WRT rendering in the log as described in the User's Guide.

What's quite odd is that the rendered text contains both the "micro" and the "cubed" symbols...it just has the extra garbage as well.

And I found where this unit value gets defined:  in /usr/share/weewx/unit.py: "microgram_per_meter_cubed": u"µg/m³".  In the HTML for the "current" line, this looks reasonable:  "µg/m³".   But I have yet to make the image generator happy.

--Richard

[p q]

You might check to see what font is actually used by looking at the developer tools browser. Most browsers use F12 to activate the developer tools or it's in the menu. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/31dd2730-21d9-4e86-85a2-8a42d3e339b3n%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[richard maliszewski]

So the problematic text rendering is within a generated PNG, right?  So the debugging has to be on the generation end...there's no font in the PNG, just pixels.

--Richard

[p q]

Ah missed that. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/020e85a4-d40f-4498-b231-7a0d9f16a699n%40googlegroups.com.

[gjr80]

So when you say that '"font/OpenSans-Bold.ttf", which does exist', where exactly does it exist? /etc/weewx/skins/Seasons/font/OpenSans-Bold.ttf? OpenSans-Bold.ttf is more than adequate, my install is using it and rendering µg/m³ just fine. The symptoms sounds very much like PIL is giving you a default font every time. What version of WeeWx are you using and under which version of python is it running(will be in the log on WeeWX startup)?

Gary

[richard maliszewski]

Yes, /etc/weewx/skins/Seasons/font/OpenSans-Bold.ttf.  Weewx version is 4.40, Python is 2.7.17.  This is on Ubuntu 18.04.5 LTS.

--Richard

[gjr80]

Attached are a couple of instrumented files that should tell us what is happening. Could you rename:

/usr/share/weewx/weeplot/genplot.py to /usr/share/weewx/weeplot/genplot_orig.py

/usr/share/weewx/weeplot/utilities.py to /usr/share/weewx/weeplot/utilities_orig.py

and then download and save the attached genplot.py and utilities.py to directory usr/share/weewx/weeplot

Then restart WeeWX and let it complete at least one full report cycle. Take a copy of the log from WeeWX startup through until the first report cycle finishes and post the log here.

Gary

[Tom Keffer]

One other thing: what version of PIL are you using? You can find out by running:

python -c "import PIL;print(PIL.__version__)"

Let's also double check the version of Python:

python -V

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d9e7b7d1-bbf1-470b-97fe-d3a6ffbfd651n%40googlegroups.com.

[richard maliszewski]

Python reports 2.7.17.  PIL reports 5.1.0.  Log output pretty long-winded, attached.

--Richard

[gjr80]

Well it’s clear from the log that right font is being loaded, and more importantly, a default font is never being used by PIL. My PIL is 5.4.1 but it is being run under python3. I am about out of ideas, maybe Tom has more?

You can revert you WeeWX install by deleting utilities.py and genplot.py and renaming the two _orig files back to their former names. You will need to restart WeeWX.

Gary

[richard maliszewski]

I had enabled the Smartphone skin as well as the default.  On the off chance that it was using some default font that was getting sucked into the Seasons image gen, I disabled it.  Made no differece, but was a low-effort thing to try.

--Richard

[gjr80]

Well I am at a complete loss. I downloaded Ubuntu 18.04.5 LTS and installed it on a VM this morning. Installed WeeWX v4.4.0 via dpkg and reassigned outHumidity to group_concentration:

Apr  7 04:04:58 ubuntu weewx[3902]:  * Starting weewx weather system weewx
Apr  7 04:04:58 ubuntu weewx[3924] INFO __main__: Initializing weewx version 4.4.0
Apr  7 04:04:58 ubuntu weewx[3924] INFO __main__: Using Python 2.7.17 (default, Feb 27 2021, 15:10:58) #012[GCC 7.5.0]
Apr  7 04:04:58 ubuntu weewx[3924] INFO __main__: Platform Linux-4.15.0-140-generic-x86_64-with-Ubuntu-18.04-bionic
Apr  7 04:04:58 ubuntu weewx[3924] INFO __main__: Locale is 'C.UTF-8'

outHumidity plot came out just file with no funky characters:

PIL is 5.1.0:

gary@ubuntu:~$ python -c "import PIL;print(PIL.__version__)"
5.1.0

Perhaps a locale issue, I don't know, getting out of my depth.

Gary

[Tom Keffer]

I'm at loss, too. My Ubuntu 20.04, with Python 2.7, renders it perfectly.

It's a long shot, but you can try upgrading Pillow. Pillow 6.2.2 was the last version to support Python 2.7. 

sudo python2 -m pip install Pillow==6.2.2

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fa487133-f08f-4f8c-bec2-84d71537d51an%40googlegroups.com.

[richard maliszewski]

Have tried several long shots:

1) the examples above of this working fine only had single values being graphed, mine has two.  Commented each one out in turn, made no difference.  Worth a poke.

2) tried Tom's suggestion for upgrading Pillow, only to find that nothing else to date had precipitated a need for pip on this server.  So installed pip, and saw a surprising amount of stuff get pulled in along the way.  Restarted WeeWX in case any of those pip-dependencies made a difference.  Nope.

3) upgraded Pillow, restarted WeeWX.  Same, same.

Appreciate the help, regardless.

--Richard

[richard maliszewski]

And BTW:

rlmalisz@minix:~$ locale

LANG=en_US.UTF-8

LANGUAGE=

LC_CTYPE="en_US.UTF-8"

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_PAPER="en_US.UTF-8"

LC_NAME="en_US.UTF-8"

LC_ADDRESS="en_US.UTF-8"

LC_TELEPHONE="en_US.UTF-8"

LC_MEASUREMENT="en_US.UTF-8"

LC_IDENTIFICATION="en_US.UTF-8"

LC_ALL=

[Tom Keffer]

That all looks normal, but the default encoding could be making a difference. What do you get with

python -c "import sys;print(sys.getdefaultencoding())"

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7443c87b-292f-4fb6-8a03-5305ff7bc48dn%40googlegroups.com.

[richard maliszewski]

That comes back "ascii".

--Richard

[Tom Keffer]

I was actually able to reproduce your error by writing a very simple program that uses PIL. The problem seems to come from passing in a byte string that encode "µg/m³", instead of Unicode. WeeWX is not supposed to do that, but obviously it is.

Let me work on this a bit. 

I've created issue #662 to track.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fffe253e-a7d9-4218-af39-37210a3aa287n%40googlegroups.com.

[Tom Keffer]

Richard, before I get too deep into this, we should check that the operating thesis is correct. Could you please replace your copy of weeutil/genplot.py with the attached? It will tell us what types the labels are.

Then restart weewx, run through the first reporting cycle. Post the log through that first cycle.

-tk

[richard maliszewski]

I suspect as you suspected:

Apr  7 16:45:21 minix weewx[29999] INFO weeplot.genplot: Unit label set to '%', which is type '<type 'unicode'>'

Apr  7 16:45:21 minix weewx[29999] INFO weeplot.genplot: Unit label set to '%', which is type '<type 'unicode'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to 'in', which is type '<type 'unicode'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to 'mph', which is type '<type 'unicode'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to 'mph', which is type '<type 'unicode'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to 'µg/m³', which is type '<type 'str'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to 'µg/m³', which is type '<type 'str'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to '', which is type '<type 'str'>'

Apr  7 16:45:22 minix weewx[29999] INFO weeplot.genplot: Unit label set to '', which is type '<type 'unicode'>'

--Richard

[Tom Keffer]

Alright! Now we're getting somewhere! For some reason 'µg/m³' is showing up as a string, despite it being marked as unicode in units.py.

And, you are sure that the only place µg/m³ is used is in units.py? You can search by using the following:

find /usr/share/weewx /etc/weewx -type f|xargs grep 'µg/m³'

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/67cc02d7-5159-4f0a-8848-e614ffaa4d64n%40googlegroups.com.

[richard maliszewski]

It looks like it gets defined identically in /usr/share/weewx/gw1000.py.  But also gets defined in /usr/share/weewx/user/purpleair.py line 77 via:

weewx.units.default_unit_label_dict['microgram_per_meter_cubed']  = '  \xc2\xb5g/m\xc2\xb3'

Python are not close friends, but that smelled like a string rather than Unicode, so I commented it out.  Log now says:

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to '%', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to '%', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to 'in', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to 'mph', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to 'mph', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to 'µg/m³', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to 'µg/m³', which is type '<type 'unicode'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to '', which is type '<type 'str'>'

Apr  7 18:40:26 minix weewx[30594] INFO weeplot.genplot: Unit label set to '', which is type '<type 'unicode'>'

And graph label looks fine.  I'll let the Purple Air extension author know to have a look at this.  Thanks for the help!

--Richard

[gjr80]

Just to clarify the gw1000 definition, it is solely used when the driver is run directly (ie without WeeWX) so that gw1000 data can be displayed in either US customary or Metric units. It is not (and cannot) be used when used as a WeeWX driver or service.

Gary

[Tom Keffer]

So, it seems the problem is 3rd parties defining labels as byte strings, rather than as unicode strings. We can guard against that by converting to unicode in the plotting modules. 

Thanks, Richard.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4753a452-710a-48a5-86e6-0500e95484a1n%40googlegroups.com.

[Tom Keffer]

One more thing to add: it's OK to define the label using UTF-8, but the purple air author should have indicated that s/he was using a byte string. 

weewx.units.default_unit_label_dict['microgram_per_meter_cubed']  = b'  \xc2\xb5g/m\xc2\xb3'

[Tom Keffer]

Fixed in commit e51b2a6.