Process Data during server initialization

[Lois Greene-Hernandez]

Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

[Scott Sawyer]

Could you run a cron job on the system to analyze the data periodically?

Sent from my iPhone

On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez <loi...@gmail.com> wrote:

Hello, 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.

[Lois Greene-Hernandez]

Well it's data that I need to populate the pages.  It's very processor intensive and I'll need to pass it back to the views once it's processed.  Can I pass data back to the views once I've run a task?  I'm looking into caching alternatives.  

Thanks

Lois

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/X1joo7QEbnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com.

[Okware Aldo]

Hi Lois, 

To expand on what Scott is saying. 

- model DB tables to store the data you need to render the graphs

- build a service to preprocess the data - this service can run periodically depending on how up to date you need the graphs to be (a microservice)

- build a single endpoint to simple fetch the data the graphs need (Django) if this is a monolith the normal Django views should do the trick

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAAGmcw0tk9fd%3DhjvpdZTpqqf9-gDDKgtW7YYfvJxVcCo1DvD4g%40mail.gmail.com.

[Shaheed Haque]

Well, it's a bit of a blunt weapon, but Python is going to run any code it finds at module scope as the process is initialised. So you could cause your heavy stuff to be run there? Or at least spawned from there? 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMEZma-P3Z6yoAJJGCWbDpd8YEWkc_kV8uS4FKDNvsR-LF-4WA%40mail.gmail.com.