Dynamic creation of models

[Marc]

Hello

 

Currently working on a project to allow researchers to upload data files into a repository. The system will only handle the import, validation and transformation of the data, so that it meets the required standards. When a file is upload the for the first time the user has to be able to define the structure of the destination based on the csv/xls. This includes key field, data types, foreign keys etc. There is no requirement for the data to be displayed or edited once in the repository by this system, currently. We are currently planning on using PostgreSQL as the database.

 

Our organisation already has another system, with a similar requirement, that is partially written by an external contractor in Ruby on Rails. The dynamic creation of the tables is no problem but it is the dynamic creation of the models and controllers that is proving difficult and is the reason why this other project is not finished.

 

Reading through the forums it looks like Django may offer a better solution through meta-programming. https://code.djangoproject.com/wiki/DynamicModels This does look like quite an old article though.

 

The other solution would be to have one table that holds all the records for all the imported files and have a JSONB (in PostgreSQL) to hold the columns that differ between files. This means that we would not be able to represent the relationships between files though. Ideally the organisation would like the data in a relational form to make subsequent analysis with other tools easier.

 

Above all it is important that the system is robust and maintainable. Any solution will of course be tested by creating a proof of concept first.

 

Is it still possible to dynamically create the models in the latest versions of Django (1.11 and 2.02)?

 

Thank you

Marc

[Jani Tiainen]

Hi,

Since you're uploading more like documents than similary structured data, why not to use more proper tool, like document database (ArangoDB would get my vote) instead of trying to do what Django is not designed for and doesn't support well with concepts like "dynamic models".

We do similar thing with certain uploadable reports and it works pretty fine.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d707e803-3dbf-43e6-95a8-13869043c280%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

[Marc]

Thank you Jani, for taking the time to respond. We are currently at the design stage so are considering all options. We have used document databases (MongoDB) for other projects and in those cases it turned out to not be good fit. I agree that at first glance a document database does feels like a good fit, but I want to explore all options. The organisation is asking for quite ambitious requirements (moon on a stick), they are going to have to make compromises I am sure. A good compromise does seem to be the JOSN column data type in both PostgreSQL and MySQL, I believe that both Ruby on Rails and Django have support

[Marc]

This is an interesting article against document databases, I am sure though that there are equally valid arguments for

http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/

[Jani Tiainen]

Hi,

That's why I suggested ArangoDB since it's really transactional like normal relational databases. It's harder to get it out of sync than for example MongoDB.

Also surprisingly ArangoDB query language is really close to SQL.

You should give it a spin. That was the reason we picked ArangoDB for out projects.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/117a9163-ad2e-4d18-b599-63f766bb5587%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

[Matemática A3K]

This can help you with the modelling:
https://github.com/sromero84/django-eav

Still, seems that the structure and its content will be tied to Django - you will have to access it through your Django app. If you want to have it in a "normalized" form in the database, what you need to write in Django is the frontend to a python import tool and then for displaying the data. In this case, you "decouple Django from the data", your app contains history about the imports and how to retrieve the info. In the first case, you access the data through your Django models which gives you the structure of the data.

HTH

 

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.