How to profile python programs

[tweet...@gmail.com]

Hey all,

I have a py_binary rule that I would like to profile. I build the binary and then run like this:

python -m cProfile -o out.profile bazel-bin/.../my_script

If I then run:
python -m pstats out.profile

I get fairly useless results. All my time is spent in "posix.waitpid" and no other useful profile information is left.

I gather this is from Bazel's use of subprocess to actually run my script.

Is there any easy way to profile my code?

Thanks

[lcid...@gmail.com]

Sadly the current way that python is executed makes this and other scenarios complicated.

When a py_binary gets executed, bazel first calls into a trampoline python2 script, which either spawns the desired script in a subprocess or execvs the desired script directly into the bootstrapped process. The latter is the common code path.

Now in order to use pstats you (for now) have to find out whether it can a. attach to child processes, too and b. work with execv (this will become the default in future versions).

IMO the current "trampolining" could be replaced by a less invasive method but it feels to me like Google does not have any plans to do so.
Glad to be corrected here.

[halfa...@gmail.com]

This worked for me:

import cProfile
profile = cProfile.Profile()
profile.run('<CODE HERE>')
profile.dump_stats('/tmp/out.profile')