GAE: Error when downloading data, if ndb.KeyProperty(repeated=True)

[Kave via StackOverflow]

I am creating the bulkloader.yaml automatically from my existing schema and have trouble downloading my data due the repeated=True of my KeyProperty.

class User(ndb.Model):
    firstname = ndb.StringProperty()
    friends = ndb.KeyProperty(kind='User', repeated=True) 

The automatic created bulkloader looks like this:

- kind: User
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: firstname
      external_name: firstname
      # Type: String Stats: 2 properties of this type in this kind.

    - property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: transform.create_foreign_key('User')
      export_transform: transform.key_id_or_name_as_string

This is the error message I am getting:

Code: transform.key_id_or_name_as_string Details: 'list' object has no attribute 'to_path'

What can I do please?

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/17485250/gae-error-when-downloading-data-if-ndb-keypropertyrepeated-true

[Kave via StackOverflow]

I am creating the bulkloader.yaml automatically from my existing schema and have trouble downloading my data due the repeated=True of my KeyProperty.

class User(ndb.Model):
    firstname = ndb.StringProperty()
    friends = ndb.KeyProperty(kind='User', repeated=True) 

The automatic created bulkloader looks like this:

- kind: User
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: firstname
      external_name: firstname
      # Type: String Stats: 2 properties of this type in this kind.

    - property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: transform.create_foreign_key('User')
      export_transform: transform.key_id_or_name_as_string

This is the error message I am getting:

google.appengine.ext.bulkload.bulkloader_errors.ErrorOnTransform: Error on transform. Property: friends External Name: friends. Code: transform.key_id_or_name_as_string Details: 'list' object has no attribute 'to_path'

[tony via StackOverflow]

When you are using repeated properties and exporting to a CSV, you should be doing some formatting to concatenate the list into a CSV understood format. Please check the example here on import/export of list of dates and hope it can help you.

Please DO NOT REPLY directly to this email but go to StackOverflow:

http://stackoverflow.com/questions/17485250/gae-error-when-downloading-data-if-ndb-keypropertyrepeated-true/17487622#17487622

[Kave via StackOverflow]

I am creating the bulkloader.yaml automatically from my existing schema and have trouble downloading my data due the repeated=True of my KeyProperty.

class User(ndb.Model):
    firstname = ndb.StringProperty()
    friends = ndb.KeyProperty(kind='User', repeated=True) 

The automatic created bulkloader looks like this:

- kind: User
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: firstname
      external_name: firstname
      # Type: String Stats: 2 properties of this type in this kind.

    - property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: transform.create_foreign_key('User')
      export_transform: transform.key_id_or_name_as_string

This is the error message I am getting:

google.appengine.ext.bulkload.bulkloader_errors.ErrorOnTransform: Error on transform. Property: friends External Name: friends. Code: transform.key_id_or_name_as_string Details: 'list' object has no attribute 'to_path'

What can I do please?

Edit:

After Tony's tip I came up with this:

- property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      #import_transform: transform.create_foreign_key('User')
      #export_transform: myfriends.transform
      import_transform: "lambda x: x.split(';')"
      export_transform: myfriends.valueToString(';')

myfriends.py

def valueToString(delimiter):
    def key_list_to_string(value):
        keyStringList = []
        for val in value:
            keyStringList.append(transform.key_id_or_name_as_string(val))        
        return delimiter.join(keyStringList)
    return key_list_to_string

The result in the csv file becomes Chinese though:

慹潨⹯敤␬␵潲湵獤㠽〰〰樤

Any idea please?

[Kave via StackOverflow]

- property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: "lambda x: x.split(';')"
      export_transform: myfriends.valueToString(';')

[Kave via StackOverflow]

edit 2: The encoding is in Unicode: UTF-8. Somehow the default encoding of LibreOffice (Unicode) can't read it anymore.

Tony. Please allow me to test also the upload and I mark it as an answer.

[Kave via StackOverflow]

And it works! The encoding is in Unicode though: UTF-8. Make sure to open the file in LibreOffice as such or you would see garbled content.

Import: ??

[Kave via StackOverflow]

I am creating the bulkloader.yaml automatically from my existing schema and have trouble downloading my data due the repeated=True of my KeyProperty.

class User(ndb.Model):
    firstname = ndb.StringProperty()
    friends = ndb.KeyProperty(kind='User', repeated=True) 

The automatic created bulkloader looks like this:

- kind: User
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: firstname
      external_name: firstname
      # Type: String Stats: 2 properties of this type in this kind.

    - property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: transform.create_foreign_key('User')
      export_transform: transform.key_id_or_name_as_string

This is the error message I am getting:

google.appengine.ext.bulkload.bulkloader_errors.ErrorOnTransform: Error on transform. Property: friends External Name: friends. Code: transform.key_id_or_name_as_string Details: 'list' object has no attribute 'to_path'

What can I do please?

Possible Solution:

After Tony's tip I came up with this:

- property: friends
      external_name: friends
      # Type: Key Stats: 2 properties of this type in this kind.
      import_transform: myfriends.stringToValue(';') 
      export_transform: myfriends.valueToString(';')

myfriends.py

def valueToString(delimiter):
    def key_list_to_string(value):
        keyStringList = []
        if value == '' or value is None or value == []:
            return None 
        for val in value:                
            keyStringList.append(transform.key_id_or_name_as_string(val))        
        return delimiter.join(keyStringList)
    return key_list_to_string

And this works! The encoding is in Unicode though: UTF-8. Make sure to open the file in LibreOffice as such or you would see garbled content.

The biggest challenge is import. This is what I came up with without any luck:

def stringToValue(delimiter):    
    def string_to_key_list(value):
        keyvalueList = []
        if value == '' or value is None or value == []:
            return None        
        for val in value.split(';'):            
            keyvalueList.append(transform.create_foreign_key('User'))        
        return keyvalueList
    return string_to_key_list

I get the error message:

BadValueError: Unsupported type for property friends: <type 'function'>

According to Datastore viewer, I need to create something like this:

[datastore_types.Key.from_path(u'User', u'ka...@gmail.com', _app=u's~myapp1')]