problem with timeout

[Szabo, Patrick (LNG-VIE)]

Hi,

I have an operation that takes about 10 minutes to befinished. It takes that long because our DB is pretty big.
This produces a timeout.

I have tried so set the timeout in apache higher but it seems that apache is not taking this directive....just keeps timing out after 300 seconds.

Is there anything I can to to overcome this within my app ?

The exception says that a connections was aborted by the software oft he hostcomputer (translated from german): core_output_filter: writing data to the network

This is how i get the entries:

Datum being a Date and int(str(datestring).split('-')[0]) delivering a year like 2012

Buchung.objects.filter(Datum__year = int(str(datestring).split('-')[0]))


Help would be much appreciated.



. . . . . . . . . . . . . . . . . . . . . . . . . .
Ing. Patrick Szabo
Developer
LexisNexis
A-1030 Wien, Marxergasse 25

mailto:Patric...@lexisnexis.at
Tel.: +43 1 53452 1573
Fax.: +43 1 534 52 146

. . . . . . . . . . . . . . . . . . . . . . . . . .

[Avraham Serour]

you could optimize the database (creating indexes, putting in memory, using a ssd etc)

you can run the process in backgorund

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Russell Keith-Magee]

On Mon, Dec 17, 2012 at 7:34 PM, Szabo, Patrick (LNG-VIE) <patric...@lexisnexis.at> wrote:

Hi,

I have an operation that takes about 10 minutes to befinished. It takes that long because our DB is pretty big.
This produces a timeout.

I have tried so set the timeout in apache higher but it seems that apache is not taking this directive....just keeps timing out after 300 seconds.

Is there anything I can to to overcome this within my app ?

Yes. Stop doing long lived operations in a web request.

Seriously, stop. :-)

HTTP is designed to be a large number of short lived operations, not long lived operations. At the moment, you're hitting a resource limit at the HTTP server, but I'd be deeply surprised if that's the only problem you hit. HTTP just isn't designed to work in the way you've described.

If you need to perform a long lived operation, you need to structure your workflow differently. Instead of a single request that takes 10 minutes to create a response, you:

 1) Issue a request that creates a record in the database with all the details necessary to start the "job"

 2) Return to the user a success message that indicates work has started. 

 3) In the background, in a *completely separate process*, process the job.

 4) When the job is finished, insert the results into the database, and mark the job as finished

 5) If the user hits the page representing the job, they can view the results. 

The important detail here is that you get the heavy lifting *out* of the request-response cycle. How you achieve that is up to you. On the very simple end, you just have a job status page that the user manually reloads, and a cron task in the background to do the heavy lifting. On the complex end, you could use long polling or web sockets to provide the status update, and something like Celery to handle the task management.

Yours,

Russ Magee %-)

[Szabo, Patrick (LNG-VIE)]

Thanks, that does sound like a better idea and now that i think about it i saw this behaviour on many other webinterfaces already…don’t know why i didn’t think about that J

 

However I’ve noticed that it’s not the query that is taking so long, it’s the processing of the result that takes ~180 seconds…yes 10 minutes might have been a little exaggerated ;-)

 

It’s this code:

 

for buchung in buchungen:

            if str(buchung.Produkt.Bestell_Nr):

               buch_pool[str(buchung.Produkt) + " #" + str(buchung.Produkt.Bestell_Nr)].append([str(buchung.Aktivitaet), str(buchung.Minuten)])

            else:

                buch_pool[str(buchung.Produkt)].append([str(buchung.Aktivitaet), str(buchung.Minuten)])

 

You mightbe wondering why I’m even building such a weird construct. Truth is, this thing has grown over time with different devs working on it and I don’t have time to change it.

 

My guess is that this code is triggering a load of queries…any idea how to make this more efficient ?

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

 

. . . . . . . . . . . . . . . . . . . . . . . . . .

Ing. Patrick Szabo

Developer

LexisNexis

A-1030 Wien, Marxergasse 25

 

Patric...@lexisnexis.at

[Szabo, Patrick (LNG-VIE)]

Nevermind…select_related() did the trick.

226 queries cut to 4…amazing this little statement J

 

 

. . . . . . . . . . . . . . . . . . . . . . . . . .

Ing. Patrick Szabo

Developer

LexisNexis

A-1030 Wien, Marxergasse 25

 

Patric...@lexisnexis.at

Tel.: +43 1 53452 1573

Fax.: +43 1 534 52 146

. . . . . . . . . . . . . . . . . . . . . . . . . .

 

[Chris Cogdon]

Nice find!

If you turn debugging on for "django.db.backends", it will show you what SQL queries are being issued, and the time taken for each.

LOGGING['handlers']['console'] = { 'level':'DEBUG', 'class': 'logging.StreamHandler' }

LOGGING['loggers']['django.db.backends'] = { 'handlers':['console'], 'level':'DEBUG' }

(you could also just modify the existing LOGGING variable)