Utf8 error

[Thomas Hackler]

Hello,

I have the following utf-8 error and I don't understand why. Maybe you can help me?

0xe4is a "ä" in German but I try to avoid these special letters.

How can I find the position which causes the problem?

Thank you

Regards

Thomas

11:08:32 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: Generate failed with exception '<type 'exceptions.UnicodeDecodeError'>'
Mar 19 11:08:32 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/Seasons/NOAA/NOAA-%Y-%m.txt.tmpl
Mar 19 11:08:32 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Reason: 'utf8' codec can't decode byte 0xe4 in position 1: unexpected end of data
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 323, in generate
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_Seasons_NOAA_NOAA__Y__m_txt_tmpl.py", line 379, in respond
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 641, in filter
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      filtered = val.decode('utf-8')
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      return codecs.utf_8_decode(input, errors, True)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 1: unexpected end of data
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: Generate failed with exception '<type 'exceptions.UnicodeDecodeError'>'
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/Seasons/NOAA/NOAA-%Y.txt.tmpl
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Reason: 'utf8' codec can't decode byte 0xe4 in position 2: unexpected end of data
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 323, in generate
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_Seasons_NOAA_NOAA__Y_txt_tmpl.py", line 892, in respond
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_Seasons_NOAA_NOAA__Y_txt_tmpl.py", line 568, in __errorCatcher64
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "<string>", line 1, in <module>
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 1143, in format
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      addLabel=add_label, localize=localize)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 1128, in toString
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      localize=localize)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 859, in toString
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      val_ustr = six.ensure_text(val_str)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/six.py", line 904, in ensure_text
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      return s.decode(encoding, errors)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      return codecs.utf_8_decode(input, errors, True)
Mar 19 11:08:33 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 2: unexpected end of data
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: Generate failed with exception '<type 'exceptions.UnicodeDecodeError'>'
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/Seasons/rss.xml.tmpl
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: **** Reason: 'utf8' codec can't decode byte 0xe4 in position 8: invalid continuation byte
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 323, in generate
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_Seasons_rss_xml_tmpl.py", line 102, in respond
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 1143, in format
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      addLabel=add_label, localize=localize)
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 1128, in toString
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      localize=localize)
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/units.py", line 859, in toString
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      val_ustr = six.ensure_text(val_str)
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/six.py", line 904, in ensure_text
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      return s.decode(encoding, errors)
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****      return codecs.utf_8_decode(input, errors, True)
Mar 19 11:08:36 raspberrypi weewx[4685] ERROR weewx.cheetahgenerator: ****  UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 8: invalid continuation byte
Mar 19 11:08:36 raspberrypi weewx[4685] INFO weewx.cheetahgenerator: Generated 5 files for report SeasonsReport in 4.10 seconds
Mar 19 11:08:37 raspberrypi weewx[4685] DEBUG weewx.manager: Daily summary version is 2
Mar 19 11:08:38 raspberrypi weewx[4685] INFO weewx.imagegenerator: Generated 16 images for report SeasonsReport in 1.64 seconds
Mar 19 11:08:38 raspberrypi weewx[4685] INFO weewx.reportengine: Copied 5 files to /var/www/html/weewx

[Tom Keffer]

Hello,

Did you edit any of these files in a text editor? 

Seasons/NOAA/NOAA-%Y.txt.tmpl

Seasons/NOAA/NOAA-%Y.txt.tmpl, or 

Seasons/rss.xml.tmpl

My suspicion is that whatever text editor you used assumed the file was encoded in cp1252 (aka windows-1252), rather than UTF-8. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d58160e4-7eaf-4773-b472-f53849ffe02an%40googlegroups.com.

[Thomas Hackler]

thank you for this advice

how can I encode the file again in UTF-8 ?

maybe I opened the files with the standard editor from the raspberry pi but I did not open them in windows

to edit the config files of weewx I use nano

[Tom Keffer]

Unfortunately, character encoding can be a complicated topic. Let's take this in two parts: the encoding used by the template, and the final encoding that will appear in the NOAA .txt reports. These are two separate topics.

1, It appears that you edited the template file and inserted a character, "ä", encoded in cp1252, into the template. Cheetah must be informed of this. You do this by including a "#encoding" directive in the file. So, the top of the file should look like:

#errorCatcher Echo
#encoding cp1252
#set $YM="%Y %m"

...

where you should add the highlighted line.

2. Second, now you must tell Cheetah to emit the file using the same encoding. You do that by going into skin.conf and setting the encoding to cp1252. When you're done, you end up with something like this:

    [[SummaryByMonth]]
        # Reports that summarize "by month"
        [[[NOAA_month]]]
            encoding = cp1252
            template = NOAA/NOAA-%Y-%m.txt.tmpl

    [[SummaryByYear]]
        # Reports that summarize "by year"
        [[[NOAA_year]]]
            encoding = cp1252
            template = NOAA/NOAA-%Y.txt.tmpl

Then delete all your NOAA files. At the next reporting cycle, WeeWX will regenerate the reports hopefully, this time, with the correct encodings.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a31c853b-2864-4868-9ab7-397aec2cd291n%40googlegroups.com.

[Thomas Hackler]

unfortunately I deleted the two file in the NOAA directory before adding your yellow marks :/

I thought that they are created after every running but the NOAA directory is empty now and I get a lot other errors because of the missing files

how can I create them again ?

[Tom Keffer]

You can download them from the repository. For example, here is NOAA-%Y.txt.tmpl and NOAA-%Y-%m.txt.tmpl for Version 4.4.0.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/50b3a076-2c04-4c2b-aab7-6f39dc715044n%40googlegroups.com.

[Thomas Hackler]

before I make something wrong again .....

I have now the files of the NOAA diretory again

in which file should I copy the  #encoding cp1252 ?

In file NOAA-%Y.txt.tmpl or NOAA-%Y-%m.txt.tmpl or both ? I guess both?

Which files in which directory do you mean with " Then delete all your NOAA files " ?

[Tom Keffer]

Yes, put the #encoding directive in both templates.

WeeWX uses the templates to generate files, which then get uploaded to your web server. It's these generated files that can be deleted. They will automatically be regenerated.

The location is system dependent, but most likely in /var/www/html/weewx/NOAA

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f62b62a6-27b6-49df-b4fa-c0745e16f633n%40googlegroups.com.

[Thomas Hackler]

I made the changes but I still get the same errors utf8 blabla :/

I don't understand this because I never used a "ä" ?

in the skin.conf is written

[CheetahGenerator]

encoding = html_entities

should I change this too to cd1252 ?

[gert.a...@gmail.com]

Hi

I use Danish characters, ÆØÅ,  and I just used utf8 for the NOAA reports, encoding= utf8 and that is working. CP1252 did not work for me. Delete the NOAA txt files and let weewx generate them again.

Gert