data import

[sector119]

Hi ALL!

I need to import data in CSV format into my models everyday ~ 500.000
rows. But my CSV files are uncomplite, I have to append some data to
every row.

I use something like this to import data into my model:
for d in csv.DictReader(...):
d.update(my_data)
MyModel.objects.create(**d)

But I don't want to invent a bisycle and want to do this in django
way, may be it'll be better to create intermediate XML fixtures and
then load them using manage.py loaddata ? But if I do so, I do my work
twise. And it takes much more time... load data from XML is very slow,
it takes 14 minutes to import 85.000 rows.

What you can recomend in my case?

Thanks!

[Ben Ford]

How about setting a default value on all of the database tables the you are specifying in the dict that you're doing the update with?

[sector119]

On 4 Бер, 13:33, "Ben Ford" <ben.for...@gmail.com> wrote:
> How about setting a default value on all of the database tables the you are
> specifying in the dict that you're doing the update with?

Hi, Ben, those values can't be set as default value for the table
column. Each model may have not only one import source, and values
from my dict - are foreign key values, and may differ from one source
to another.