importing csv data into database

[sum abiut]

Hi,

Could someone please advise this i am trying to import data from csv file or write to database but i am confuse on how to start. i am using sqllite database.

Cheers,

[James Schneider]

Is it a one-time import or are you looking to have a form that accepts CSV files?

Either way, I would start with the Python csv module. Works really well and should be able to convert your CSV file to a standard Python list of lists, which you should be able to coerce into models, etc.

https://docs.python.org/2/library/csv.html

Write a small script outside of Django that reads a file representing the typical CSV format you are expecting. Once that is working, you can incorporate the code from that script into a model or form. If you only need to import a CSV file one time, just modify that script to bootstrap the Django libraries and modules, then create your model instances from there.

You can also write directly to SQLite directly at that point if needed.

-James

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y5Nk_ATFgYk-hduiLsrxBasS%3Dzu4Qj8Gp5F9vo%3DNRwF%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Farrell]

Hello Sum,

There are two approaches you could take.

One is to install a plugin like django-import-export and then use it's import functionality. Its been a while since I've done this, but when I did under django-1.4 it Just Worked. This is the best approach if you know your data is clean and doesn't require any sort of processing before saving it.

The other is to write a custom manage.py command. This would let you load the data using a csv.DictReader, do some logic on it, and then bulk-create objects in the database. This approach has the disadvantage that if you have a bug in your processing script, you will end up adding messy data to your database and having to clean it up. To avoid this, you could write a unit test case which calls your import function in its setUp() method. This testcase would create a separate test database, import the data into it, run your test functions, and then clear the test database. This approach would also let you, for example, deliberately insert malformed info into the data and verify that your import function would catch those errors.

[sum abiut]

Thanks guys,

appreciate your help. Yes i am looking to have a form that accepts CSV files. I will have a read on the link that you guys have provided and see how it goes.

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLbaBcs8%2BrJsNmXFknRGNTC5%2BOr3URs53eiFBM8GLekmcA%40mail.gmail.com.

[sum abiut]

Hi,

could you please help me figure out why i am getting this error i am trying to import data from csv file to model buty i am getting the error: list index out of range and if refering to line 20 on view which is: data.FirstName=row[1]

view.py
def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)
        for row in readata:
            data=test()
            data.PersonID=row[0],
            data.FirstName=row[1],
            data.LastName=row[2],
            data.Address=row[3],
            data.save()
        return render_to_response('csv_test.html',locals())

model.py

class test(models.Model):
    PersonID = models.CharField(max_length=45)
    FirstName = models.CharField(max_length=45)
    LastName = models.CharField(max_length=45)
    Address = models.CharField(max_length=45)

here is the traceback:

• /var/www/html/env/local/lib/python2.7/site-packages/django/core/handlers/base.py in get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs
  
  
  /var/www/html/webapp/csvapp/views.py in readcsvfile
  data.FirstName=row[1],
  
  

Cheers,

Sum

[Andrew Farrell]

could you replace your function with 

def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)

        for i, row in enumerate(readata):

            try:

                data=test()
                data.PersonID=row[0],
                data.FirstName=row[1],
                data.LastName=row[2],
                data.Address=row[3],
                data.save()

            except Exception, e:

                import pdb;pdb.set_trace()

                print i

                print row

                print row[1]

                raise e
        return render_to_response('csv_test.html',locals())

And then re-run this again? When you hit the offending row of data, it should pop you into the python debugger. type 'print row<enter>' and you should see what the problem is. If not type 'next<enter>' and step though the code. 

You can execute arbitrary python from the debugger and once you figure out the problem, I encourage you to play around with it. Aside from pedantically explaining to yourself what you think going on, pdb is one of the most generally useful tools.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y74u7NDyxxG_c7rvpver7GKmg-X9CNv-ZJpmug%3Dop3LFA%40mail.gmail.com.

[Andrew Farrell]

You might have to make sure that my indentation there is correct.

Also, I noticed something else about your code: you are assigning 2-tuples to all of your data types rather than strings when you do

   data.PersonID=row[0],
   data.FirstName=row[1],
   data.LastName=row[2],
   data.Address=row[3],

You might actually be aware of this, but I once spent 4 hours debugging before I realized this on a previous project.

weapons = ('fear', 'surprise', 'fanatical devotion to the pope',) 
clearly creates a 3-item tuple. But so does

weapons = 'fear', 'surprise', 'fanatical devotion to the pope',

and in fact

weapons = 'fear',

creates a 1-item tuple.

[sum abiut]

Ok , i manage to figure out where the error was coming from in my view.py on line  readata=csv.reader(csvfile,delimiter=';', quotechar='|'). The delimiter =';' i change the semicolon to  a comma The delimiter =',' and it works.

However when the data are imported from the csv file to the model it also write some funny character to the model as shown on the image below. How to import data to model without writing this funny characters?  i want the data to be clean

view.py
def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)
        for row in readata:
            data=test()
            data.PersonID=row[0],
            data.FirstName=row[1],
            data.LastName=row[2],
            data.Address=row[3],
            data.save()
        return render_to_response('csv_test.html',locals())

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLY_BPTQieu%3D%3Dc_hg4ahAGJevp%2B5zb%3D1PWOw5GJ-4ULZ3w%40mail.gmail.com.

[Andrew Farrell]

Ah, I suppose that bit didn't get sent. Google groups only lets mail from my old gmail address through and bounces mail from the MIT address I use by default.

I noticed something else about your code: you are assigning 2-tuples to all of your data types rather than strings when you do

   data.PersonID=row[0],
   data.FirstName=row[1],
   data.LastName=row[2],
   data.Address=row[3],

You might actually be aware of this, but I once spent 4 hours debugging before I realized this on a previous project.

weapons = ('fear', 'surprise', 'fanatical devotion to the pope',) 
clearly creates a 3-item tuple. But so does

weapons = 'fear', 'surprise', 'fanatical devotion to the pope',

and in fact

weapons = 'fear',

creates a 1-item tuple.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y7ky9f7PrpnHvK70yLAMiHaw4s3U500RAdyJV25bdpbag%40mail.gmail.com.

[sum abiut]

i am importing data from csv file to the model test, so i am taking data as an instance of test(). are you saying  i should define data as a string?? and have the parameters as personid, firstname,lastname,address. Please advise

Cheers

On Sun, Apr 26, 2015 at 2:36 AM, Andrew Farrell <armors...@gmail.com> wrote:

Ah, I suppose that bit didn't get sent. Google groups only lets mail from my old gmail address through and bounces mail from the MIT address I use by default.

I noticed something else about your code: you are assigning 2-tuples to all of your data types rather than strings when you do

   data.PersonID=row[0],
   data.FirstName=row[1],
   data.LastName=row[2],
   data.Address=row[3],

You might actually be aware of this, but I once spent 4 hours debugging before I realized this on a previous project.

weapons = ('fear', 'surprise', 'fanatical devotion to the pope',) n

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLab49_vfB%3D8ysOLm22tXXXXrOsaKYGoRQvzis_1PbsX3Q%40mail.gmail.com.

[Andrew Farrell]

What I'm saying that you are defining data.LastName as a tuple and when it gets turned into a string. So if a row is

44, Barack, Obama, 1600 Pennsylvania Avenue

Then data.FirstName is going to be 

('Barack',)

Which is a single-element tuple. When it gets turned into a string, it becomes ('Barack',)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y4zOzVi_uw9jQcmmHy3GVXWukYvu-bKP3HK-G0JB_Zt0Q%40mail.gmail.com.

[sum abiut]

Hi Andrew,

Thanks heaps. do you mine explaining how to fix that.

cheers,

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLbAx_e_LexuC6AOe3LBXWoV2O9GQZpYygSHJrTFF2xagQ%40mail.gmail.com.

[Mario Gudelj]

Remove comma from each line inside the for loop.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y4-TXxS9kPQQYUpTzmPP4ts7u__4V_FdPviOE6kYvzoKg%40mail.gmail.com.

[sum abiut]

wow! Thanks heaps Mario :) . it works like a champ, Thank you James and Andrew for the directions. you guys were a great help.

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHqTbjma6H4BLtSz2ghLExnMeo1rTzBpV8gShB18TdKyCogBJQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

-