Hello everybody, I'm having an hard time to figuring this out.
I'm designing a django project based on data retrieved from a remote public REST API that will:
- Retrieve data from a REST API
- Saving the data in my models (my local db)
- Analyze and process the local data asynchronously and create statistics and trends
When a user request the data:- Check if the local data (models) are in sync with the remote, otherwise fetch the remote data and save them in the local db
- Present the data
What is the best approach to do this?
My questions:- Should I design my models replicating all the remote API fields?
- Can I use JSonField and store data in json format to speed things up?
- In this case would it be complex to calculate statistics on the data?
In other words I can't use the django ORM and foreign keys if I store data in JSON format, right?
- Where should I call the remote API?
- Considering that the remote data should "replicate" the database models, first I tough to add model methods that call the REST API inside the models
- Or I could put them in the manager, considering that when a user needs the data he/she doesn't need to know if they are local or remote
- How do I tell django to grab the remote data and put them in db before presenting them?
- Maybe this should be an asynchronous task?
Sorry for the complexity, I hope I was clear!
Thanks in advance for your help