Property is corrupt (Unicode Decode Error)

[Ravindra B]

My GAE app sometimes gives the following error. I uploaded all my data using bulk uploader, but never faced any problem in uploading data to datastore.

Property keywords is corrupt in the datastore:
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 958, in _FromPb
    value = datastore_types.FromPropertyPb(prop)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1753, in FromPropertyPb
    value = unicode(value.decode('utf-8'))
  File "/base/python_runtime/python_dist/lib/python2.5/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 22-27: unsupported Unicode code range
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 700, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/e-shiksha/1.351017675093647993/common.py", line 36, in wrapper
    func(self, *args, **kw)
  File "/base/data/home/apps/e-shiksha/1.351017675093647993/search.py", line 95, in get
    query = q.fetch(1000)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2025, in fetch
    raw = raw_query.Get(limit, offset, config=config)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1862, in Get
    val = iterator.next()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2051, in IterateResults
    results_to_push.append(top_result.GetNext())
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1908, in GetNext
    self.__orderings)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1889, in __init__
    self.__entity = entity_iterator.next()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2313, in next
    next_batch = self.__batcher.next()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2183, in next
    return self.next_batch(self.AT_LEAST_ONE)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2220, in next_batch
    batch = self.__next_batch.get_result()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
    return self.__get_result_hook(self)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 1998, in __query_result_hook
    self.__results = self._process_results(query_result.result_list())
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2031, in _process_results
    for result in results]
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 127, in pb_to_query_result
    return self.pb_to_entity(pb)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 242, in pb_to_entity
    return Entity._FromPb(pb)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 962, in _FromPb
    (prop.name(), traceback.format_exc()))
Error: Property keywords is corrupt in the datastore:
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 958, in _FromPb
    value = datastore_types.FromPropertyPb(prop)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1753, in FromPropertyPb
    value = unicode(value.decode('utf-8'))
  File "/base/python_runtime/python_dist/lib/python2.5/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 22-27: unsupported Unicode code range

[Jc Lanoë]

Hi,

I'm having the same issue.

Did you find a solution to fix it, or at least access the other properties?

Thanks.

[Vinny P]

On Fri, Feb 7, 2014 at 10:07 PM, Jc Lanoë <jc.l...@starmakerinteractive.com> wrote:

I'm having the same issue. Did you find a solution to fix it, or at least access the other properties?

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 22-27: unsupported Unicode code range

Most of the time, this error happens because (as the error message implies) there are non UTF-8 encoded characters in the property.

What you can do is try to catch entities that are storing improper information and edit them, or verify that the data is entirely UTF-8 before sending it to the bulkloader.

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com