Profiling HowTo

[Michele Comitini]

After some experience with the standard profiling option using web2py.py I found that  most of the profiling output while correct was of little utility.

So I developed a simple method to do a very detailed profiling.

I would change the appfactory in main to support profiling in a similar way, by using many different profiling files, one for each wsgi request.

That way we can find exactly where the code can be improved.

[BTW it seems to me that "Hello World!" without sessions is faster than it was not too long ago.]

What do you think?

Here is the way to do it:

- apply the patch below

- install runsnakerun (pip does it)

- run the following:

$ runsnake /tmp/web2py-profiling/*

8<--------------------------8<-------------------

diff --git a/gluon/main.py b/gluon/main.py

index b3c37fb..190ead8 100644

--- a/gluon/main.py

+++ b/gluon/main.py

@@ -435,6 +435,10 @@ def wsgibase(environ, responder):

         [a-zA-Z0-9_]

       - file and sub may also contain '-', '=', '.' and '/'

     """

+    import cProfile

+

+    prof = cProfile.Profile()

+    prof.enable()

 

     current.__dict__.clear()

     request = Request()

@@ -689,6 +693,8 @@ def wsgibase(environ, responder):

         return wsgibase(new_environ, responder)

     if global_settings.web2py_crontype == 'soft':

         newcron.softcron(global_settings.applications_parent).start()

+    prof.disable()

+    prof.dump_stats('/tmp/web2py-profiling/prof-' +  str(time.time()) + '.prof')

     return http_response.to(responder, env=env)

 

8<----------------------------------------8<--------------------------

[Massimo DiPierro]

Ok with the patch except that the name of the profile file should be specified by the web2py.py --profile <filename> option.

 

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michele Comitini]

Massimo,

The patch I sent is NOT OK for production it is just a patch to show how it works!  It must be moved to appfactory function first.

Then the --profile arg should become a directory (a safe default could be a dir inside web2py tree).  Every time the wsgi app is called a new profile dump would be created.  

I wonder if it would be possible to make it work for a single app and using admin.  Do you think is possible?

About the good and the bad of the method I propose in the patch: 

- BAD

  The name of the file now is created with a time.time() and this is a problem because now web2py goes so fast that there could be collisions (no kidding!).  Also the profiling directory would fill up with thousand of files.

 - GOOD

   The good thing of this approach is that, besides filename collisions, there is no need to serialize requests, profiling analysis can be done on a single run or a group of runs without stopping the server.  And last but not least profiling works under any frontend server (nginx, rocket, apache...).

mic

2013/7/29 Massimo DiPierro <massimo....@gmail.com>

[Massimo DiPierro]

Any news about this?

[Michele Comitini]

Not yet.  I had to please Apple quirks in my web2py spare time and now I am late with work for the money ;-)

I will do it as soon as I get some spare time again...

2013/8/5 Massimo DiPierro <massimo....@gmail.com>

[Massimo DiPierro]

No worries. :-)

[Niphlod]

got the method and changed a little bit the wording.

@massimo: I'll send a PR if you agree on changing the behaviour (and the variable name) "profiler_filename" to "profiler_dir", meaning that you should pass a dir instead of a file. Shouldn't break lots of things, as I assume people using the old method are close to 0. The new method is indeed more flexible and with the addition of runsnake it's a little bit more meaningful than scanning a single file with the top callers for each and every request serialized to a human-parseable table.

[Massimo Di Pierro]

Sorry for the late reply. I had missed this message. I am ok with this. I agree there is no backward compatibility issue.

[Niphlod]

ok, pr available at https://github.com/web2py/web2py/pull/170