UnicodeDecodeError when extracting image metadata using the Image Service

[Nickolas Daskalou]

Looks like this has been happening for at least one week, not sure beyond that.

In the Python runtime, when calling images.execute_transforms() with parse_source_metadata=True, for a UnicodeDecodeError is always raised for certain images. The images are valid images.

Last part of Traceback:

File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 810, in execute_transforms return rpc.get_result() File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self) File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 912, in execute_transforms_hook self._original_metadata = json.loads(response.source_metadata()) File "/base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/json/decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) UnicodeDecodeError: 'utf8' codec can't decode byte 0xbc in position 12: invalid start byte

App Engine team - is this intended behaviour?

I assume this is a bug in the App Engine image service (it looks like a unicode problem when the image service module tries to use JSON to decode a string).

I can send image details in a private message if required.

Thanks,

Nick

[George (Cloud Platform Support)]

Hello Nickolas, 

A public URL of your application suffering from this issue would enable reproduction. 

If you attach some of the images that trigger the error, you'll facilitate debugging further. 

Any other details you deem relevant may prove of great help as well.  

[Nickolas Daskalou]

Hi George,

I'm pretty sure I replied to you in a private message, but in case you didn't receive it...

For privacy reason I can send you a URL of an image in a private message, would that suffice?

Thanks,

Nick

[George Suceveanu]

Hi Nickolas, 

You are most welcome to send private messages any time. 

If the URL you mention is publicly accessible, it sounds a bit contradictory: you wanting to send it in a private message. Please go ahead anyhow, if you think all this OK. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/vsaXkhU-QfM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8d4beec0-22df-49b7-a014-3297e55fba61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

In the EXIF metadata of the provided file one can see: Software ... �Ʀ즨�� . What happens if you modify this entry (replacing the characters with some ASCII)? 

If you check for all other images causing failure, do you see similar non-ASCII characters? Otherwise, do images that do not cause error contain only ASCII characters? 

[George (Cloud Platform Support)]

The API expects EXIF tags to conform to specifications, usually UTF. Your program's flow assumes that images contain EXIF text hat does conform to specifications. This might not be always the case. In particular, some image editors apply transform to the image that may jumble the embedded text. You may consider a safety step in your app, that deals with these special EXIF cases, and solves related problems.