How to populate the database

[Zeynel]

Can anyone point me in the right place in documentation for populating
my sqlite3 tables with the data in the .csv file? Thanks.

[Shawn Milochik]

There are different ways to do it, depending on how much data you have and how often you plan to do it.

The fastest way for large files is to use sqlite3's .import command to directly import a file. However, this will bypass any validation done by your models.

The way I do it is to read the csv with csv.DictReader, then create instances of my model with the values from the resulting dictionary, then calling a .save() on the new instance. This is fairly slow, but thorough; you won't realize belatedly that your database is missing required fields or has invalid values, because the script will just blow up if you try.

Shawn

[Zeynel]

Thanks. I couldn't make the sqlite3 shell work on windows command
prompt, so I cannot use .import.

> read the csv with csv.DictReader, then create instances of my model
> with the values from the resulting dictionary, then calling a .save() on
> the new instance.

Do you have more detailed instructions on this method?

[Shawn Milochik]

Sure, here's a quick & dirty sample I put up on pastebin:

http://pastebin.com/f651cf8de

[John M]

You could also use OpenOffice with the SQLIte connector (I think)

I use Access in Windows, works great!

But you might try getting the CSV into a JSON format that the
manage.py loaddata command could use, that set's you up for the future
too.

J

[Zeynel]

Thanks, I'll try it now. What does

os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'

do? I read the documentation here http://docs.python.org/library/os.html
but I didn't understand.

My app is in

C:.../Documents/PROJECTS/Django/sw2/wkw2.

Do I enter?

os.environ['DJANGO_SETTINGS_MODULE'] = 'sw2.settings'?
from sw2.wkw2.models import School, Lawyer

By INPUT_FILE you mean the file where data I want to upload to sqlite3
is, correct? That file is sw2/csvtest1.csv

What's 'Internal ID'?

....


This is my models.py:

from django.db import models

class School(models.Model):
school = models.CharField(max_length=300)
def __unicode__(self):
return self.school


class Lawyer(models.Model):
firm_url = models.URLField('Bio', max_length=200)
firm_name = models.CharField('Firm', max_length=100)
first = models.CharField('First Name', max_length=50)
last = models.CharField('Last Name', max_length=50)
year_graduated = models.IntegerField('Year graduated')
school = models.CharField(max_length=300)
school = models.ForeignKey(School)
class Meta:
ordering = ('?',)
def __unicode__(self):
return self.first


Thanks!

[Zeynel]

I tried to convert my csv file with csv2json.py and it worked before
(see this thread
http://groups.google.com/group/django-users/browse_frm/thread/a00b529ba2147d91/efb82ba2893cc0a7?lnk=gst&q=csv#efb82ba2893cc0a7)
but now I am trying the same exact thing in a different directory and
I get this error:

C:\...\Django\sw2\wkw2>csv2json.py csvtest1.csv wkw2.Lawyer
Converting C:...\Django\sw2\wkw2csvtest1.csv from CSV to JSON as
C:\...\Django\sw2\wkw2csvtest1.csv.json

Traceback (most recent call last):
File "C:\...\Django\sw2\wkw2\csv2json.py", line 37, in <module>
f = open(in_file, 'r' )
IOError: [Errno 2] No such file or directory:
'C:\\...\\Django\\sw2\\wkw2csvtest1.csv'

Below is the link to csv2jason.py and line 37:

http://www.djangosnippets.org/snippets/1680/
...
in_file = dirname(__file__) + input_file_name
out_file = dirname(__file__) + input_file_name + ".json"

print "Converting %s from CSV to JSON as %s" % (in_file, out_file)

[line 37] --> f = open(in_file, 'r' )
fo = open(out_file, 'w')

Do you know an easier way to convert csv to json? (Or how to fix
this?)