manual import problem

[MaryJo Webster]

I'm using the manual import function for the first time since installing PANDA about three years ago. I have a  2.8gb file that I'm trying to pull in. It threw this error on the first time (see below). The "integer out of range" doesn't make sense to me because there aren't any integer fields in the file. In fact, all the columns have text qualifiers around them (that's how it came out of the MySQL server I had it in).   I sliced off the top couple dozen rows and tried importing that -- worked just fine. 

Does anyone have suggestions for other trouble-shooting I can try?

thanks,
MaryJo

Traceback (most recent call last):

  File "manage.py", line 21, in <module>

    execute_manager(settings)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 459, in execute_manager

    utility.execute()

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv

    self.execute(*args, **options.__dict__)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute

    output = self.handle(*args, **options)

  File "/opt/panda/panda/management/commands/manual_import.py", line 44, in handle

    encoding='utf-8')

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 137, in create

    return self.get_query_set().create(**kwargs)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 377, in create

    obj.save(force_insert=True, using=self.db)

  File "/opt/panda/panda/models/data_upload.py", line 72, in save

    super(DataUpload, self).save(*args, **kwargs)

  File "/opt/panda/panda/models/base_upload.py", line 46, in save

    super(BaseUpload, self).save(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 463, in save

    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 551, in save_base

    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 203, in _insert

    return insert_query(self.model, objs, fields, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1593, in insert_query

    return query.get_compiler(using=using).execute_sql(return_id)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 912, in execute_sql

    cursor.execute(sql, params)

  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute

    return self.cursor.execute(query, args)

django.db.utils.DatabaseError: integer out of range

 

--

MaryJo Webster

mjweb...@gmail.com

651-491-6576

@MaryJoWebster

[w higgins]

https://github.com/pandaproject/panda/issues/955

--
You received this message because you are subscribed to the Google Groups "PANDA Project Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to panda-project-users+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[w higgins]

Open issue. It's the file size, not a column value.

https://github.com/pandaproject/panda/issues/955

To unsubscribe from this group and stop receiving emails from it, send an email to panda-project-users+unsubscribe...@googlegroups.com.

[w higgins]

I got around issues like that by loading via the API, where you can cycle through batches.

These rather old scripts won't fit your need, but they might provided a path.

[w higgins]

https://github.com/higs4281/panda-loaders

[w higgins]

It appears this line would need to be a BigIntegerField, but I seem to recall there were other issues with large manual uplaods, so that might not do it for you.

[MaryJo Webster]

Thank you. We'll take a look at this. 

To unsubscribe from this group and stop receiving emails from it, send an email to panda-project-users+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Serdar Tumgoren]

Hey MaryJo,

I sent a patch for this back in the day. It wasn't merged, but you can always use my fork or apply the patch manually. Note that it requires a Django database schema migration.

https://github.com/pandaproject/panda/pull/944

[Serdar Tumgoren]

Sorry, a clarification and extra bit of help if you choose to patch your PANDA install.  

The Pull Request linked earlier addresses both issues that Bill mentioned: the database Integer field issue as well as providing a refinement to the system for bulk loading "large" data files. The schema migration addresses the former.

If you review the description in that pull request, there's a link to an old Google Group thread that outlines the steps necessary to perform the database migration after applying the patch.

HTH. Shout back if you have any questions or run into issues.

[MaryJo Webster]

Thank you, Serdar!  I'm going to talk with our developer about this when he's back in the office tomorrow. I might get back to you with questions.

To unsubscribe from this group and stop receiving emails from it, send an email to panda-project-users+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.