Can anyone share code for uploading csv or excel file to sqlite3 database.
BBG
PASCUAL Eric
Hi,
Have you studied the documentation of the CSV module included in Python standard library ?
You'll find there all the needed information for reading and interpreting CSV files without having to implement the raw parsing, and have there rows in a form ready to use for inserting objects in your model.
For performance's sake, it is advised to use bulk inserts or updates instead on individual saves on the Django side if your CSV files contain a lot of data.
Best
Eric
Sent: Wednesday, September 26, 2018 6:04:09 PM
To: Django users
Subject: Can anyone share code for uploading csv or excel file to sqlite3 database.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/fc5736a8-3396-491b-b265-853f46fdad87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
BBG
def upload_csv(request):
data = {}
if "GET" == request.method:
return render(request, "add_student/bulk.html", data)
# if not GET, then proceed
try:
csv_file = request.FILES["csv_file"]
if not csv_file.name.endswith('.csv'):
messages.error(request,'File is not CSV type')
return HttpResponseRedirect(reverse("add_student:upload_csv"))
#if file is too large, return
if csv_file.multiple_chunks():
messages.error(request,"Uploaded file is too big (%.2f MB)." % (csv_file.size/(1000*1000),))
return HttpResponseRedirect(reverse("add_student:upload_csv"))
file_data = csv_file.read().decode("utf-8")
lines = file_data.split("\n")
#loop over the lines and save them in db. If error , store as string and then display
for line in lines:
fields = line.split(",")
data_dict = {}
data_dict["enrollment_no"] = fields[0]
data_dict["student_name"] = fields[1]
data_dict["gender"] = fields[2]
data_dict["course"] = fields[3]
data_dict["category"] = fields[4]
data_dict["admission_year"] = fields[5]
data_dict["branch"] = fields[6]
data_dict["current_semester"] = fields[7]
data_dict["address"] = fields[8]
data_dict["city"] = fields[9]
data_dict["district"] = fields[10]
data_dict["state"] = fields[11]
data_dict["student_contact"] = fields[12]
data_dict["parent_contact"] = fields[13]
try:
form = EventsForm(data_dict)
if form.is_valid():
form.save()
else:
logging.getLogger("error_logger").error(form.errors.as_json())
except Exception as e:
logging.getLogger("error_logger").error(repr(e))
pass
except Exception as e:
logging.getLogger("error_logger").error("Unable to upload file. "+repr(e))
messages.error(request,"Unable to upload file. "+repr(e))
return HttpResponseRedirect(reverse("add_student:upload_csv"))On Thursday, September 27, 2018 at 1:59:24 PM UTC+5:30, Eric Pascual wrote:
Hi,
Have you studied the documentation of the CSV module included in Python standard library ?
You'll find there all the needed information for reading and interpreting CSV files without having to implement the raw parsing, and have there rows in a form ready to use for inserting objects in your model.
For performance's sake, it is advised to use bulk inserts or updates instead on individual saves on the Django side if your CSV files contain a lot of data.
Best
Eric
From: django...@googlegroups.com <django...@googlegroups.com> on behalf of BBG <bhavik....@gmail.com>
Sent: Wednesday, September 26, 2018 6:04:09 PM
To: Django users
Subject: Can anyone share code for uploading csv or excel file to sqlite3 database.
I want to create upload bulk data. Can anyone share code to upload csv or excel file.--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to djang...@googlegroups.com.
Derek
Andréas Kühne
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/88ea4342-f61d-4282-99d2-511f7e9f14a7%40googlegroups.com.
PASCUAL Eric
Hi,
Which error(s) did you get exactly ? It is not clear in you message whether your problem is at the CSV data processing level or at the Django one.
In case this could help, using the standard
csv module you can obtain a reader wich takes care of parsing and decoding thanks to the
reader() module function. You pass it the file object directly, without reading its content in memory. Since it provides an iterator over the data, yielding rows one at a time as tuples, this lets you load big files without any problem.
The DictReader class of the same module works more or less the same way, but returns rows as dicts which keys are the CSV columns names, making the mapping code more readable and self-documented.
Best
Eric
To: Django users
Subject: Re: Can anyone share code for uploading csv or excel file to sqlite3 database.