Django error page - MemoryError

[Cal Leeming [Simplicity Media Ltd]]

Hey,

So, we have a few clients who use Django for processing large amounts of data in a single query. 

If an exception is raised in development, the get_traceback_html() method fails with a MemoryError, and in the event that it doesn't, you end up with huge variable data print outs making the debug page somewhat tedious to use.

Temporarily, I've put a try/catch on MemoryError to generate a text based email, which at least tells us what originally happened, albeit with the useful variable information.

With all this in mind, I would like to see if anyone else thinks the following might be a good idea

• Place a try/catch for MemoryError on the exception handler to send back a simple exception traceback to the browser.
  
  
• Include a configuration settings option to limit the maximum payload it will send back to the browser per variable (i.e. maybe 500kb per stack frame, or 2kb per variable etc)
  

I'd be happy to provide a patch for this, assuming it has any chance of getting into the core.

Any thoughts/feedback would be appreciated.

Cal

[Paul McMillan]

> Place a try/catch for MemoryError on the exception handler to send back a
> simple exception traceback to the browser.

Yes, this makes sense, as long as we are sure the memory error is
raised by Django code, not user code.

> Include a configuration settings option to limit the maximum payload it will
> send back to the browser per variable (i.e. maybe 500kb per stack frame, or
> 2kb per variable etc)

I think we should select some reasonable limits for these, and
hardcode them, rather than adding a setting. Users who are debugging
the entire contents of multi-megabyte variable values on the html
debug page are doing it wrong.

-Paul

[Alex Gaynor]

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Once you get a real memory error (i.e. one indicating you're out of memory, not just some operation wouldn't work "aaaa" * sys.maxint for example), doing *any* allocation might fail, so any limits you try to put down won't work in all circumstances.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

[Cal Leeming [Simplicity Media Ltd]]

Hi Alex,

Please note, I am already using a try/catch block on MemoryError, and this does indeed resolve the problem.

I think at the very least, we should attempt to generate the text exception, and if it fails due to a particular circumstance, then it will just fall back to doing whatever it originally does.

I agree that it won't always 100% work (although I'm yet to see it fail in my tests), but generating the text based exception has a fairly small footprint, so it should work the majority of the time.

Cal

--

-- 

Cal Leeming

Technical Director | www.simplicitymedialtd.co.uk
Registered company number 7143564

Office: +44 (0)2476 980798 | Cell: +44 (0)7534 971120

[Cal Leeming [Simplicity Media Ltd]]

Comments below, apologies for the email signature on previous emails!

On Tue, Dec 20, 2011 at 7:56 PM, Paul McMillan <pa...@mcmillan.ws> wrote:

> Place a try/catch for MemoryError on the exception handler to send back a
> simple exception traceback to the browser.

Yes, this makes sense, as long as we are sure the memory error is
raised by Django code, not user code.

Agreed, this would be wrapped around get_traceback_html() most likely.

 

> Include a configuration settings option to limit the maximum payload it will
> send back to the browser per variable (i.e. maybe 500kb per stack frame, or
> 2kb per variable etc)

I think we should select some reasonable limits for these, and
hardcode them, rather than adding a setting. Users who are debugging
the entire contents of multi-megabyte variable values on the html
debug page are doing it wrong.

I completely agree.

I'm wondering if it would be better to have the variable commented out with a small message if it's above X amount, or whether it should display up to X amount, then display a small message explaining why it's been clipped.

[Cal Leeming [Simplicity Media Ltd]]

Anyone else have any thoughts on if I should submit this for consideration into the core?

Cal