Quoting in export_to_csv_file() / import_from_csv_file()

[Wes Hall]

After using import_from_csv_file(), additional quotes are showing up in the db.

Should I be using some different quoting in export_to_csv_file() or is this an inherent problem with specifying filter_out in in the model, or am I completely off the mark?

Exporting db = 2.4.6

Importing db = 2.5.1

Table definition

db.define_table('evaluation',
    Field('form', type='text',
          label=T('Evaluation form')),
    Field('results', type='text',
          label=T('JSON results')),
    auth.signature,
    format='%(form)s',
    migrate=settings.migrate)

db.evaluation.results.filter_in = lambda obj, dumps=dumps: dumps(obj)
db.evaluation.results.filter_out = lambda txt, loads=loads: loads(txt)

Export

db.export_to_csv_file(open('test.csv', 'w'))

(Excerpt from test.csv)

89,se2,"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u""This answer should've had an apostrophe.""}, u'email': u'u...@mail.com'}",<auth fields>

Import

db.import_from_csv_file(open('test.csv', 'r'))

MySQL row.results from within current application on original and new db

{"name": "Good User", "answers": {"q1": "5", "comment1": "This answer should've had an apostrophe."}, "email": "u...@mail.com"}

MySQL row.results from import on new db (note the new enclosing double quotes)

"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u\"This answer should've had an apostrophe.\"}, u'email': u'u...@mail.com'}"

This all leads to the following error:

evaluation.results['name']),
TypeError: string indices must be integers

[dlypka]

I was able to import rows which have minimal quotes, using the default import settings.

Here is a sample row which worked for me:

Atlanta,4214,GANGV19175,GANGV1917,S2C1,437,RBS6601W,"15-B Gainesville_3C",F3,NRCSGAJTCRBR12,Yes,Yes,Yes,Yes,No,Yes,Yes,Yes,CXP9018350/1_R10A/1,IP,Yes,Yes,Yes,Yes,Yes,FALSE,Yes,454,325,454,Yes,CXP9014711/7_R10CA24,Yes,"2013-06-06 10:06:46"

On Wednesday, July 10, 2013 4:40:59 PM UTC-5, Wes Hall wrote:

After using import_from_csv_file(), additional quotes are showing up in the db.

Should I be using some different quoting in export_to_csv_file() or is this an inherent problem with specifying filter_out in in the model, or am I completely off the mark?

Exporting db = 2.4.6

Importing db = 2.5.1

Table definition

db.define_table('evaluation',
    Field('form', type='text',
          label=T('Evaluation form')),
    Field('results', type='text',
          label=T('JSON results')),
    auth.signature,
    format='%(form)s',
    migrate=settings.migrate)

db.evaluation.results.filter_in = lambda obj, dumps=dumps: dumps(obj)
db.evaluation.results.filter_out = lambda txt, loads=loads: loads(txt)

Export

db.export_to_csv_file(open('test.csv', 'w'))

(Excerpt from test.csv)

89,se2,"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u""This answer should've had an apostrophe.""}, u'email': u'...@mail.com'}",<auth fields>

Import

db.import_from_csv_file(open('test.csv', 'r'))

MySQL row.results from within current application on original and new db

{"name": "Good User", "answers": {"q1": "5", "comment1": "This answer should've had an apostrophe."}, "email": "u...@mail.com"}

MySQL row.results from import on new db (note the new enclosing double quotes)

"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u\"This answer should've had an apostrophe.\"}, u'email': u'...@mail.com'}"

[Massimo Di Pierro]

I opened an issue about this until I can verify the problem:

https://code.google.com/p/web2py/issues/detail?id=1585&thanks=1585&ts=1373793319

On Wednesday, 10 July 2013 16:40:59 UTC-5, Wes Hall wrote:

After using import_from_csv_file(), additional quotes are showing up in the db.

Should I be using some different quoting in export_to_csv_file() or is this an inherent problem with specifying filter_out in in the model, or am I completely off the mark?

Exporting db = 2.4.6

Importing db = 2.5.1

Table definition

db.define_table('evaluation',
    Field('form', type='text',
          label=T('Evaluation form')),
    Field('results', type='text',
          label=T('JSON results')),
    auth.signature,
    format='%(form)s',
    migrate=settings.migrate)

db.evaluation.results.filter_in = lambda obj, dumps=dumps: dumps(obj)
db.evaluation.results.filter_out = lambda txt, loads=loads: loads(txt)

Export

db.export_to_csv_file(open('test.csv', 'w'))

(Excerpt from test.csv)

89,se2,"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u""This answer should've had an apostrophe.""}, u'email': u'...@mail.com'}",<auth fields>

Import

db.import_from_csv_file(open('test.csv', 'r'))

MySQL row.results from within current application on original and new db

{"name": "Good User", "answers": {"q1": "5", "comment1": "This answer should've had an apostrophe."}, "email": "u...@mail.com"}

MySQL row.results from import on new db (note the new enclosing double quotes)

"{u'name': u'Good User', u'answers': {u'q1': u'5', u'comment1': u\"This answer should've had an apostrophe.\"}, u'email': u'...@mail.com'}"