Populating Django app db with JSON data

[Sandeep Murthy]

Hi

I've tried to get the answer to this question (which is a bit open-ended) on stackoverflow without much success, which
is basically this: what is the recommended approach to populating a pre-existing Django app database table (generated
from a model and which is currently empty) with JSON data?

There seem to be several alternatives given in the Django documentation (Django 1.7 manual) which include (1) fixtures,
(2) SQL scripts, (3) data migrations.  Of these I am a bit confused by the advice in the manual which suggests that (1)
and (2) are only useful for loading initial data.  That's not what I want to do.  The data that the app needs is going to be
persistent and permanent because the app is intended to be a web query tool for a large dataset that is currently in the
form of several JSON files, each containing on average thousands of JSON objects, each object representing an entry
corresponding to a table entry in a relational db.  The data is not going to be re-loaded or change after entry, and there
is no user facility for changing the data.

The table has been created using the makemigrations and migrate tools, but is empty.  I just need to populate the
table with the JSON data.  It seems that I need to write a custom data migration script that will insert the data into the
table via the interpreter, and then I need to run python manage.py migrate.  Is this the case, and if so, are there
are examples that I could use?

Thanks in advance for any suggestions.

SM

[aRkadeFR]

Hello,

Indeed, the data migration is the best way. Check out
the documentation here:
https://docs.djangoproject.com/en/1.7/ref/migration-operations/#django.db.migrations.operations.RunPython

You write your function that will be called by the RunPython
and will load your JSON.
Migration are ordered, your first migration will create the
tables and the second (your data migration) will load your
JSON.

To create an empty migration:
./manage.py makemigrations <app> --empty

You can rename to a useful descriptive name the migration
file.

Have a good one

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/db5919c5-ace4-4556-b90e-aa47baa26552%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aRkadeFR]

“custom Python code in a historical context”
FMPOV, it means that your code will be run exactly at a certain
schema of your DB. If your migration is the 3rd one, it will
always be run after your first two migrations, thus you will
know exactly your schema.

The schema is passed to your function (as apps and schema_editor)
in order to get your Model class.

Sorry if my explanation wasn't clear, hope it will help at least.

On 03/02/2015 10:27 AM, Murthy Sandeep wrote:
> Hi
>
> thanks for the info.
>
> The docs also say that RunPython runs “custom Python code
> in a historical context”. What does that mean exactly? It seems
> related to the apps and schema_editor arguments passed to
> the custom method that will be called by RunPython - is this something
> like a snapshot of the app model that is stored when I do `python manage.py migrate`?
>
> Sandeep

>> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/54F42164.6040505%40arkade.info.

[Murthy Sandeep]

Hi

thanks for the info.

The docs also say that RunPython runs “custom Python code
in a historical context”. What does that mean exactly? It seems
related to the apps and schema_editor arguments passed to
the custom method that will be called by RunPython - is this something
like a snapshot of the app model that is stored when I do `python manage.py migrate`?

Sandeep


> On 2 Mar 2015, at 19:37, aRkadeFR <con...@arkade.info> wrote:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/54F42164.6040505%40arkade.info.

[Collin Anderson]

Hi Sandeep,

A snapshot (or at least a diff from the previous one) is stored when you run makemigrations.

Collin