unable to parse csv file

[fredg]

Hi,

I'm using the last web2py release on localhost with python-2.5.4 on a
LFS based system.
I'm trying to import a CSV file through the web2py interface, I have
tried several CVS output,but everytime,I've got the same error which
is "unable to parse csv file"

Here is the sqlite database created with web2py :
db.py :
<code>
#!/usr/bin/python
# -*- coding: utf-8 -*-

db = SQLDB('sqlite://bddpackages.db')

db.define_table("packages",
SQLField("nom","string",length=32,notnull=True,default=""),
SQLField
("description","string",length=128,notnull=True,default=""),
SQLField("url","string",length=64,notnull=True,default=""),
SQLField("packager","string",length=64,default=""),
SQLField("categorie","string",length=32,default=""))
</code>

Here is the CSV file that I have made (a package database), it's the
output of (prt-get is a package manager) :
<code>
$ prt-get printf "\"\",\"%n\",\"%d\",\"%u\",\"%P\",\"cat\"\n" >> /tmp/
db_packages.csv
</code>
Yes, the first line of this file is written by myself (I saw that line
when I have exported a little database made by hand)

<code>
packages.id,packages.nom,packages.description,packages.url,packages.packager,packages.categorie
"","AfterStep","Gestionnaire de fenêtres","http://
www.afterstep.org/","geanbrun","cat"
"","ICU","International Components for Unicode for C","http://www.icu-
project.org/","","cat"
"","LibVNCServer","seveur VNC","http://
libvncserver.sourceforge.net/","floreal.c at orange dot fr","cat"
"","MesaLib","OpenGL compatible 3-D graphics library","http://
mesa3d.sourceforge.net","Thierry Nuttens, thierryn1 at hispeed dot
ch","cat"
"","Mowitz","Librairie pour siag","http://siag.nu/","","cat"
...
...
...
</code>
> 1100 lines

Maybe because the first ""(packages.id) is empty instead of being
"1","2","3","4"... but don't see how to do this automatically.

I will be very pleased if someone have a clue.

Thanks.

++ fredg

[mdipierro]

You should be able ti just delete the "id" column and if there are no
dates it should work.

> "","AfterStep","Gestionnaire de fenêtres","http://www.afterstep.org/","geanbrun","cat"

[fredg]

> You should be able ti just delete the "id" column and if there are no
> dates it should work.

Trying with that CSV file (with and without the 1st line) :

packages.nom,packages.description,packages.url,packages.packager,packages.categorie

"AfterStep","Gestionnaire de fenêtres","http://
www.afterstep.org/","geanbrun","cat"

"ICU","International Components for Unicode for C","http://www.icu-
project.org/","","cat"

"LibVNCServer","seveur VNC","http://
libvncserver.sourceforge.net/","floreal.c at orange dot fr","cat"

..
..

No success :(

++ fredg

[mdipierro]

What error do you get?

Here are some other possible causes of errors:
- There are column names that do have a corresponding field in the
table
- field types do not match
- you have dates NOT in ISO format
- you have special characters in the strings (I see you have a "ê").
Strings must be UTF8 to be imported.

Massimo

On Apr 7, 12:06 pm, fredg <fred.galu...@gmail.com> wrote:
> > You should be able ti just delete the "id" column and if there are no
> > dates it should work.
>
> Trying with that CSV file (with and without the 1st line) :
>
> packages.nom,packages.description,packages.url,packages.packager,packages.categorie

> "AfterStep","Gestionnaire de fenêtres","http://www.afterstep.org/","geanbrun","cat"

[fredg]

> What error do you get?

I don' t have any ticket just the red warning.
Is there a log file where I can find something useful for you ?

> Here are some other possible causes of errors:
> - There are column names that do have a corresponding field in the
> table
> - field types do not match
> - you have dates NOT in ISO format
> - you have special characters in the strings (I see you have a "ê").
> Strings must be UTF8 to be imported.

I have changed string to text, remove special characters from desc.
(stup** frenchies :P), remove the not null condition, no dates.
Here is the new db .py

<code>
#!/usr/bin/python
# -*- coding: utf-8 -*-

db = SQLDB('sqlite://bddpackages.db')

## Table des Packages
db.define_table("packages",
SQLField("nom","text",length=32,notnull=True,default=""),
SQLField("description","text",length=128,default=""),
SQLField("url","text",length=64,default=""),
SQLField("packager","text",length=64,default=""),
SQLField("categorie","text",length=32,default=""))
</code>

The new CSV file looks like that :
packages.nom,packages.description,packages.url,packages.packager,packages.categorie
"AfterStep","desc","http://www.afterstep.org/","geanbrun","cat"
"ICU","desc","http://www.icu-project.org/","","cat"
"LibVNCServer","desc","http://

libvncserver.sourceforge.net/","floreal.c at orange dot fr","cat"

"MesaLib","desc","http://mesa3d.sourceforge.net","Thierry Nuttens,

thierryn1 at hispeed dot ch","cat"

..
..

I 've got the same warning "unable to parse CSV file", but this time
only 51 packages were added to the db.
I will check line by line if there is a special character
somewhere ...

Best regards.

++ fredg

[mdipierro]

Because if appadmin were to give a ticket people would interpret this
as a web2py (appamdin) bug.

If you call

db.youtable.import_from_csv_file(open(....,'rb'))

and it fails, this will give you a ticket.

Massimo

[fredg]

On 7 avr, 21:34, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Because if appadmin were to give a ticket people would interpret this
> as a web2py (appamdin) bug.

Yes, I understand.

>    db.youtable.import_from_csv_file(open(....,'rb'))
<code>
fredg [ ~/Desktop/web2py ]$ python web2py.py -S nutyx
WARNING:root:unable to import dbhash
default applications appear to be installed already
web2py Enterprise Web Framework
Created by Massimo Di Pierro, Copyright 2007-2009
Version 1.59 (2009-03-16 13:10:16)
Database drivers available: SQLite3
WARNING:root:import IPython error, use default python shell
Python 2.5.4 (r254:67916, Dec 30 2008, 15:35:29)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> db.packages.import_from_csv_file(open('/tmp/db_packages.csv','rb'))
Traceback (most recent call last):
File "<console>", line 1, in <module>
NameError: name 'db' is not defined
</code>

I have try to add packages by hand through your web2py admin interface
in my database created by :

<code>
#!/usr/bin/python
# -*- coding: utf-8 -*-

db = SQLDB('sqlite://packages.db')

## Table des Packages
db.define_table("packages",

SQLField("nom","text",length=32,default=""),

SQLField("description","text",length=128,default=""),
SQLField("url","text",length=64,default=""),
SQLField("packager","text",length=64,default=""),
SQLField("categorie","text",length=32,default=""))
</code>

And it works like a charm, here is the export CSV file (with special
characters ;) )
<code>
packages.id,packages.nom,packages.description,packages.url,packages.packager,packages.categorie

1,b43-cutter,plop plip (po),http://www.plop.org/~down4/plop,thierryn1
at plop dot com,cat

2,tpokoij-iuhuhh32,kjihiy y_y_yy-tèuggôih,http://www.plop.pop,,cat

3,blah-blah-2,"ihyszui, yh_y_yi-zu34ih jôîuh hyuhy",http://
www.plop.plop/~ljij/123,fredg at web2py dot org,cat
</code>

Hope it helps.

++ fredg

[fredg]

Is that CSV import impossible because web2py can't set a primary key
that is not id ?
Not even possible with your sqldesigner :
http://mdp.cti.depaul.edu/sqldesigner/default/sqldesigner

I don't have any trouble importing my CSV into an sqlite database made
with the firefox extension (SQLite Manager) and with my first field
"nom" set as text and primary key.
If I try with a db where the primary key is id, it's impossible to do
the import.
Trying with an db made with web2py an open with SQLite Manager too,
import impossible.

++ fredg

> 3,blah-blah-2,"ihyszui, yh_y_yi-zu34ih jôîuh hyuhy",http://www.plop.plop/~ljij/123,fredgat web2py dot org,cat