Import a float field using a script

[dtdave]

I have the following field in my model:

area_hectares = models.FloatField(blank=True, null=True)

In my script for importing from a csv I seem to run into the following error:

could not convert string to float

The relevant part of the script is as follows:

area_hectares=float(row[6])

Could someone please show me the error of my ways as I know I am doing something wrong(and probably very stupid)?

Thanks in advance

[Kasper Laudrup]

Hi dtdave,

On 16/10/2019 12.28, 'dtdave' via Django users wrote:
>
> In my script for importing from a csv I seem to run into the following
> error:
> could not convert string to float
> The relevant part of the script is as follows:
> area_hectares=float(row[6])
>

That is impossible to tell without seeing the contents of your CSV file,
but obviously row 6 of that file at least sometimes contains something
that cannot be converted to a float.

I'd suggest you simply print out the values of row 6 in your script to
see which values you cannot convert and then handle that. Possibly it's
an empty string, in which case you might want to handle that by
converting that to a NoneType, but that depends on your use case of course.

Kind regards,

Kasper Laudrup

[David Turner]

Thanks for your response

I am using djjango-extensions to run the script. Everything imports fine except the one field. The Category, Region, County and Iso’s are foreign keys to the property.

The data in the csv for the area_hectares does contain in some cases no data but the format for the data is:

158.9265

70

58.9

150

7200000

665.03

Here is the contents of my import script

import csv  # https://docs.python.org/3/library/csv.html

# python manage.py runscript import_data_csv

from unesco.models import Category, County, Region, Iso, Property


def run():
    # file handler
    fhand = open('unesco/whc-sites.csv')
    reader = csv.reader(fhand)
    next(reader)  # Advance past the header

    contSuccess = 0
    Category.objects.all().delete()
    County.objects.all().delete()
    Region.objects.all().delete()
    Iso.objects.all().delete()
    Property.objects.all().delete()

    for row in reader:
        print(row)

        category_object, created = Category.objects.get_or_create(name=str(row[7]))
        county_object, created = States.objects.get_or_create(name=str(row[8]))
        region_object, created = Region.objects.get_or_create(name=str(row[9]))
        iso_object, created = Iso.objects.get_or_create(name=str(row[10]))
        e, created = Property.objects.get_or_create(name=row[0], description=row[1], sale_details=row[2], year=int(row[3]), longtitude=float(row[4]), latitude=float(row[5]), area_hectares=float(row[6]), category=category_object, county=county_object, region=region_object, iso=iso_object)

        contSuccess += 1
    print(f'{str(contSuccess)} inserted successfully! ')

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/vlHtz3JtsC0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6360fe4d-e3f2-6825-e73a-ca93084667ae%40stacktrace.dk.

[Kasper Laudrup]

Hi again,

On 16/10/2019 13.02, 'David Turner' via Django users wrote:
> Thanks for your response
>

You're welcome.

> I am using djjango-extensions to run the script. Everything imports fine
> except the one field. The Category, Region, County and Iso’s are foreign
> keys to the property.
>
> The data in the csv for the area_hectares does contain in some cases no
> data but the format for the data is:
> 158.9265
> 70
> 58.9
>
> 150
> 7200000
> 665.03
>

Since no data is represented as an empty string you need to special case
handle that so that gets converted to a NoneType instead.

Instead of simply calling float(row[6]) in your import script (which
will raise the ValueError exception on empty strings) a solution could
be to write a small wrapper function that tests if the input argument is
an empty string and returns None in that case, otherwise calls the float
type converter as usual.

That function should be trivial to write, but that's not the only
solution of course.

Thinking about, I'm not 100% sure that a nullable field in Django can be
created with a NoneType (that will then set the type to Null), but I'm
fairly certain. Someone please correct me if I'm wrong.

Kind regards,

Kasper Laudrup

[David Turner]

Thanks for pointing me in the right direction.

I added a simple
try:
y = float(row[6])
except:
y = None
And changed the area_hectares=y, in my script.
Everything works fine now.

Best

> --
> You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/vlHtz3JtsC0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a7420722-8818-4c89-3eb7-4224c70dfc9d%40stacktrace.dk.

[Kasper Laudrup]

Hi again David

On 16/10/2019 14.22, 'David Turner' via Django users wrote:
> Thanks for pointing me in the right direction.
>

No problem. Happy to help.

Just a pedantic note on your solution:

> I added a simple
> try:
> y = float(row[6])
> except:
> y = None
> And changed the area_hectares=y, in my script.
> Everything works fine now.
>

While this definitely solves the issue, there are some things you might
consider.

Catch *all* exceptions as you do with the "except:" line can be quite
dangerous, since you will not only catch the TypeError that caused the
original problem, but also other exceptions like IndexError or whatever
else might raise an exception in the try block now or in the future when
you update the code.

This might be what you want, but it can cause very hard to debug errors
much later, so it's good to be aware of.

Also, while "asking for forgiveness" as this kind of exception handling
is doing is a perfectly valid pattern in Python code, it might not be
what you want here. What if the field wasn't empty but instead contained
an actual but invalid value?

Again that might be what you want, but you could consider doing
something like:

y = float(row[6]) if row[6] else None

That will explicitly set the variable if row 6 is falsy as an empty
string is.

Kind regards,

Kasper Laudrup

[David Turner]

Hi

I agree with what you are saying. And you say there are issues with my solution. However, now it is working I will make the necessary changes to improve the solution.

Many thanks for all the help.

Best
-David

> --
> You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/vlHtz3JtsC0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9e34a9cd-b47c-b26b-8dba-cd27f338d753%40stacktrace.dk.