Difficulties attempting to debug a Cython program with gdb.

[Tom Swirly]

Hello, Cython people, and thanks for the help a few weeks ago.  

We've made great progress but I'm having a trouble in my C++ code, not a trouble that appears when I run the C++ code standalone, so I need to run it in gdb.

I have a very recent version of Cython installed, 0.19.2.

I followed the instructions here: http://docs.cython.org/src/userguide/debugging.html

1. Can't use the flag to setup.py

First, none of the following four command lines work:

python setup.py build_ext --pyrex-gdb

python setup.py build_ext --pyrex_gdb

python setup.py build_ext --cython-gdb

python setup.py build_ext --cython_gdb

in each case, I get an error looking like

error: option --cython_gdb not recognized

2. Instructions for using extension.Extension has incorrect syntax.

If you follow the instructions given on the page, you use commands like this:

ext = extension.Extension('source', ['source.pyx'], pyrex_gdb=True)
setup(..., ext_modules=[ext])

But this doesn't work, whether I use pyrex_gdb or cython_gdb - and looking at the code, I can't see how it would possibly work.  I get the fairly predictable error:

error: unknown file type '.pyx' (from 'echomesh.pyx')

This is easily fixed by changing setup() to look like:

setup(..., ext_modules=cythonize([ext]))

3. When I fix 2., everything builds but I get no debug data.

I keep getting the error:

No debug files were found in /development/echomesh/code/cython/build. Aborting.

Looking at the code, I can see it's looking for files starting with cython_debug - no such files appear anywhere under that directory.

I can add the cython_gdb=True flag to either the Extension constructor or the cythonize command or both, with no change.

4. How I'll proceed (if I don't get enlightenment from you)

It all works exactly the same with a virtualenv'ed Cython, so I'm going to create a clean virtualenv for this, and then start to add print statements to Cython.  I already started this but I'm out of time for tonight...

--
     /t

http://radio.swirly.com - art music radio 24/7 366/1000

[Tom Swirly]

FYI, you can see the code here.

[Tom Swirly]

Here's my steps in debugging this issue.  If you don't care about the steps, you can skip to the end, where I find what seems to be an issue...  sorry this is so long but I spent a long time trying to figure this out.

The steps

I started by taking a statement, assert False, and putting it in various places - first near the top to make sure that I was debugging the right Cython (I was), and then other places too see if they are reached.

1. CythonDebugWriter is what writes the debug information.
2. But it is never constructed...
3. the only place it's constructed is in create_pyx_pipeline, which is never called.
4. which is only called from Main.run_pipeline, which is never called.
5. which is only called from compile_single and compile_multiple, which are never called.
6. which is only called from Main.compile, which is never called(!)
7. Main.compile is hard to track down because of its generic name :-( but is called by BuildExecutable.cycompile, Dependencies.cythonize_one, and Main.main
8. ...and none of those functions are called!  (but it is somehow compiling code...)

So I've hit a dead-end there - except I now know what to look for when I'm going top-down.

Unfortunately the code goes through distutils.setup which I'm not prepared to trace through.

I know that Dependencies.cythonize IS called.  There is a call to cythonize_one from this BUT it's only called in the case that your Python DOES NOT have multiprocessing (i.e. version < 2.6).

Deceived by setup.py clean

At this point I discover that setup clean does not delete the generated .cpp files.  :-(  :-(

Overall, the "clean" target has been a disappointment - I already had to expand that class in order to get it to clean the linked output, but didn't catch the .cpp file.  By default, setup.py clean should kill everything that's been made by setup.py, shouldn't it?

Things like this are a trap for development and, as you can see, consumptive of developer time.

Back into it.

I add the .cpp to the clean "target".  This doesn't fix the issue, but now I can go back and correctly do the work I did before....

It seems that Main.compile is called and compile_multiple is called, and create_pyx_pipeline is called - but gdb_debug is NOT true at that point.  So now I have to track down why gdb_debug isn't being set - which I actually attempted before going off track.

in build_ext.py, CompilationOptions is always constructed with cython_gdb = True.  The issue is setting the gdb_debug flag.

There are only two places where gdb_debug is set - in CmdLine.parse_command_line, which is never called, and build_ext.cython_sources, which is never called.

I note that parse_command_line has another option for setting the output, namely --gdb - but this doesn't work as an argument to setup.py.  

However, parse_command_line is only called if you directly call the cython compiler from the command line (question: why are these options so completely different?)

The issue?

So it has to be the call to cython_sources that we want to make go off.  And this is only called from build_ext.build_extensions.  

Which is never called anywhere in the code - grepping for build_extensions in the Cython directory finds only the definition, not a call.

And build_extensions also makes a call to self.build_extension() - which is not defined anywhere - so build_ext.build_extensions() couldn't possible work.

How can this be?  It seems wrong.

For the moment, I'm going into the source and hard-coding gdb_debug to always be true and move forward from there.

And if you're skipping here from the start, I noted two other possible issues:

• make clean doesn't delete the .cpp's created by Cython
• the arguments to cython are quite different to the arguments you use in setup.py

[Tom Swirly]

So that didn't get me too far.

I immediately get an error telling me that 

  File "/Users/tom/cython/lib/python2.7/site-packages/Cython/Compiler/Pipeline.py", line 228, in create_pyx_pipeline

    options.output_dir)

AttributeError: 'CompilationOptions' object has no attribute 'output_dir'

Grepping for output_dir, I can see the same pattern as before: it's set in two places

• CmdLine - which isn't called from setup.py
  
• and build_ext.cython_sources, which doesn't seem to EVER be called.

My theory - something is broken in the build setup, such that build_ext.build_extensions is never called.

Looking at github leads me to believe that this is so:  https://github.com/cython/cython/search?q=build_extensions&ref=cmdform - so in the tip, build_extensions DOES seem to get called.

I'm now going to try to install what's in the current repo.  I'll keep you posted...

[Tom Swirly]

This didn't give me any joy either.

Why do I keep on this?

Honestly, if I could have debugged this with print statements in the C++ code, I'd have done it yesterday.  The issue is that I'm linking to a pretty big GUI library and when I call my code, nothing is happening (except a lot of print statements) - but if I run it as a standalone, I get a window popping up.

Instrumenting the entire very large 

What's wrong with the tip?

Going through the list:

$ python setup.py build_ext --inplace --cython_gdb

gives me the error:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

   or: setup.py --help [cmd1 cmd2 ...]

   or: setup.py --help-commands

   or: setup.py cmd --help

error: option --cython_gdb not recognized

but 

 python setup.py build_ext --inplace --cython-gdb

(note:  _ vs -)

gives me a different error

running build_ext

error: error in command line: command 'build_ext' has no such option 'cython_gdb'

Just running it without the extra command line arguments, but the cython_gdb_=True arguments to extension.Extension and setup - 

well, that works, but I'm back again to "not generating any cython_debug files".

So i'm back to where I was 8+ hours ago.

So I'm putting this away for a time, until I either get the energy to go through the same debugging process with this new repo, or I get some insight here.

My theory - this can't be done from setup.py because of some code issue - you have to do it from cython, despite the documentation....

[Andreas van Cranenburgh]

Looking at your setup.py on the github repo you linked, I see you don't import build_ext from Cython:

from Cython.Distutils import build_ext 

I make a debug build like this:

python-dbg setup.py build_ext --inplace --debug --pyrex-gdb

Hope this helps

[Tom Swirly]

On Sat, Nov 9, 2013 at 2:15 PM, Andreas van Cranenburgh <and...@unstable.nl> wrote:

Looking at your setup.py on the github repo you linked, I see you don't import build_ext from Cython:

Oh, thanks for the suggestion - but :-(  I tried it - and nothing changed.

Frankly, I don't see why that should be expected to change anything?

The symbol build_ext isn't even used in setup.py, so the result should be to bring a new symbol build_ext into the scope - but one that is not ever used.

This would prompt a warning from any IDE or a linter like, say, pylint.  You should avoid importing libraries that you don't use in your code.

It's conceivable that loading build_ext might have side effects.  But there are serious issues with this theory:

1. reading the code for build_ext,  there doesn't seem to be such a side-effect.
   
2. build_ext does get loaded anyway by the compiler - I put a print statement in to make sure - so any side-effect would go off regardless of my import.
3. It's really unclear how this would fix any of the issues I unearthed during my debugging process above.
4. it's not a good idea at all - "explicit is better than implicit".

 

from Cython.Distutils import build_ext 

I make a debug build like this:

python-dbg setup.py build_ext --inplace --debug --pyrex-gdb

 

Hope this helps

[Andreas van Cranenburgh]

On 10/11/13 16:23, Tom Swirly wrote:
> The symbol build_ext isn't even used in setup.py, so the result should be
> to bring a new symbol build_ext into the scope - but one that is not ever
> used.

It's supposed to be used. Have a look at my setup.py:

https://github.com/andreasvc/disco-dop/blob/master/setup.py

build_ext is passed on line 58. This causes the command
'python setup.py build_ext' to run the Cython version of build_ext.

Other than that I think it is important to use a Python version with
debugging symbols when calling setup.py. In Ubuntu it is part of the
package python-dbg and the command is called python-dbg; I wouldn't know
how it works on e.g. Mac.

Good luck

--
-- Andreas [ http://unstable.nl | gopher://unstable.nl ]

[Tom Swirly]

Thanks, an example is very much what I'm looking for!

That build_ext is supposed to be used is not mentioned at all in the documentation I've found - http://docs.cython.org/src/userguide/debugging.html or https://github.com/cython/cython/wiki/DebuggingTechniques

Is there another piece of documentation that I've missed?

It'd be nice to have it documented so "the next guy" didn't have to spend a couple of days debugging someone else's Python code...

[Tom Swirly]

Even worse results when I use a debug build of Python (and the other differences you suggested).

The rebuild doesn't complete - instead I get:

$ g++ -bundle -undefined dynamic_lookup -g build/temp.macosx-10.4-x86_64-2.7-pydebug/echomesh.o build/temp.macosx-10.4-x86_64-2.7-pydebug/Source/Tiny.o -L/development/echomesh/code/cython/Builds/MacOSX/build/Debug -lechomesh -o /development/echomesh/code/cython/echomesh-debug.so -framework Cocoa -framework WebKit -g

ld: targeted OS version does not support use of thread local variables in __ZNK4juce16ThreadLocalValueIPNS_6ThreadEE3getEv for architecture x86_64

clang: error: linker command failed with exit code 1 (use -v to see invocation)

This seems to be a second error, as by my reading of the code, it should have already generated the debug symbols directory cython_debug before it gets to this point, and it still has not.

You should be able to easily fix this error by adding -lpthread to the end of the link line.  And when I do it at the command line the link finishes fine.  

However, if I add pthread to libraries in setup.py, while the logging prints the correct g++ line out, I still get the same error:

$ g++ -bundle -undefined dynamic_lookup -g build/temp.macosx-10.4-x86_64-2.7-pydebug/echomesh.o build/temp.macosx-10.4-x86_64-2.7-pydebug/Source/Tiny.o -L/development/echomesh/code/cython/Builds/MacOSX/build/Debug -lechomesh  -lpthread -o /development/echomesh/code/cython/echomesh-debug.so -framework Cocoa -framework WebKit -g

ld: targeted OS version does not support use of thread local variables in __ZNK4juce16ThreadLocalValueIPNS_6ThreadEE3getEv for architecture x86_64

clang: error: linker command failed with exit code 1 (use -v to see invocation)

Yes, if I simply paste that first command line into the same terminal and press return, it does work correctly.  I assume there's a different between the library paths, even though the pthread library is in /usr/lib... but, well, I'm out of time again.

I do want to add something here.  

The first thing to suspect would be that my various build paths are messed up - but I'm extremely careful about this.  

I'm build master for two open source and one closed source project and do active development on yet another closed source project.  I'm ultra-careful with my installations, and I run everything inside virtualenv so I have a completely clean Python environment for each project (and, yes, I've tried installing Cython without virtualenv and I got exactly the same results, so now I need to remove it...)  My system library path is clean - I'm careful to install each library I use with the system that uses it. 

While I've been attempting to debug this issue, I've been doing other builds and a minor release that have been working just fine.

Thanks for your help.  I'm going to go back to the C++ and try to use print statements again, despite the size of the library I'm needed to use.

[Yury V. Zaytsev]

On Sun, 2013-11-10 at 20:08 -0500, Tom Swirly wrote:
>
> Thanks for your help. I'm going to go back to the C++ and try to use
> print statements again, despite the size of the library I'm needed to
> use.

Cython-enabled gdb is nice to have, but given that you've suffered so
much to make it work on Mac OS X, wouldn't resorting to a plain gdb much
better than using the print statements?

The upside is that you don't need anything else, but the debugging
symbols (-g). The downside, of course, is that you are not going to be
able to step through Cython code, but only generated C/C++ code.

However, I find it so clear and straightforward to understand, that I've
never actually tried any more advanced debugging facilities other than
to attach to the process and inspect variables... usually it all became
clear even after just loading a core dump.

--
Sincerely yours,
Yury V. Zaytsev

[Tom Swirly]

Somehow missed this till now.  But that was a good suggestion!

The issue is that I'm writing a library that I load and execute from Python, so it would have been fairly difficult to actually run it directly.  I had a testbed that ran apparently the same code from a C++ main - but that worked perfectly.

Still, as I'm planning to do a lot of development with this system in future, I was thinking it'd be worth the time to learn how to use GDB to debug.

We just got another collaborator who's Linux-based.  Next time we hit some hard-to-find bug, I'll get him to try this - I get the impression that people have had more success with Linux in this matter.  And I learned a lot...

I actually managed to get past the issue without even print statements - by comparing the normal startup of the C++ program and library with the startup in Python.  I'm making good progress now and hoping to very soon get rid of a ton of C++ and Python code and replace it with nothing at all (always the best fix!) - right now all my graphics and sound are in a separate process which I control through a socket and all that socket stuff and subprocess stuff will go.

So very excited!

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yury V. Zaytsev]

Hi Tom,

On Sun, 2013-11-17 at 19:05 -0500, Tom Swirly wrote:
>
> The issue is that I'm writing a library that I load and execute from
> Python, so it would have been fairly difficult to actually run it
> directly. I had a testbed that ran apparently the same code from a C++
> main - but that worked perfectly.

You don't need to execute it directly to be able to debug it with gdb.

Just make sure that you've compiled your Cython extension with debugging
symbols and your Python interpreter comes with debugging symbols as
well, then run the Python script and boldly attach to the interpreter
process, or load a resulting core dump.

As I said, you won't be able to step through the Cython code, only
through the generated C/C++ code, but I find it terribly useful at times
nevertheless.

> Still, as I'm planning to do a lot of development with this system in
> future, I was thinking it'd be worth the time to learn how to use GDB
> to debug.

I'm no great fan of interactive debugging, but knowing your way around
gdb is absolutely well worth it. At times, a backtrace with variable
states is worth a thousand words...

> We just got another collaborator who's Linux-based. Next time we hit
> some hard-to-find bug, I'll get him to try this - I get the impression
> that people have had more success with Linux in this matter. And I
> learned a lot...

Oh yeah, I'm running Linux, I should have mentioned that.

However, you should be able to use plain gdb on Mac OS just as easily.
It's the much more advanced stuff, such as Python / Cython debugging
front-ends that makes it way more difficult...

[Tom Swirly]

Indeed, I used this very technique to debug a segfault we were getting, so it came out to be very useful!

In fact, you don't even need to have your Python built with debugging symbols - I didn't realize till I'd resolved it that I was using the wrong Python.

And indeed, the chances that I'll be needed to debug the Cython code are much smaller.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Stefan Behnel]

Tom Swirly, 10.11.2013 23:20:

> Thanks, an example is very much what I'm looking for!
>
> That build_ext is supposed to be used is not mentioned at all in the
> documentation I've found -
> http://docs.cython.org/src/userguide/debugging.html or
> https://github.com/cython/cython/wiki/DebuggingTechniques
>
> Is there another piece of documentation that I've missed?

I think you missed two things. One is the very first import in the cygdb
docs example:

"""
from Cython.Distutils import extension # <<< not distutils.extension!

ext = extension.Extension('source', ['source.pyx'], pyrex_gdb=True)
setup(..., ext_modules=[ext])
"""

and the other is that Cython modules should be built by a call to
cythonize(), as described in various places, e.g. here:

http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils

It's clear that the cygdb docs don't really show how to use the two
together, though. That should be improved.

Also, the setup.py command line that they show actually relies on the old
way how Cython modules used to be built, using the "build_ext" replacement
that Andreas pointed you to. That should be removed. It's deprecated.

And, in fact, you don't even need to use the Extension class from
Cython.Distutils as you can pass the "gdb_debug=True" argument right into
cythonize(). So the way to do it is actually this:

from distutils.core import setup
from distutils.extension import Extension

ext = Extension('module', ['module.pyx'], ...)

setup(...,
ext_modules=cythonize(ext, gdb_debug=True)
)

Stefan

[Tom Swirly]

On Wed, Nov 20, 2013 at 12:51 AM, Stefan Behnel <stef...@behnel.de> wrote:

Tom Swirly, 10.11.2013 23:20:

> Thanks, an example is very much what I'm looking for!
>
> That build_ext is supposed to be used is not mentioned at all in the
> documentation I've found -
> http://docs.cython.org/src/userguide/debugging.html or
> https://github.com/cython/cython/wiki/DebuggingTechniques
>
> Is there another piece of documentation that I've missed?

I think you missed two things. One is the very first import in the cygdb
docs example:

"""
from Cython.Distutils import extension   # <<< not distutils.extension!

ext = extension.Extension('source', ['source.pyx'], pyrex_gdb=True)
setup(..., ext_modules=[ext])

Well, I am in fact using Cython.Distutils...

 

"""

and the other is that Cython modules should be built by a call to
cythonize()

I'm doing that too!

 

, as described in various places, e.g. here:

http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils

It's clear that the cygdb docs don't really show how to use the two
together, though. That should be improved.

Also, the setup.py command line that they show actually relies on the old
way how Cython modules used to be built, using the "build_ext" replacement
that Andreas pointed you to. That should be removed. It's deprecated.

And, in fact, you don't even need to use the Extension class from
Cython.Distutils as you can pass the "gdb_debug=True" argument right into
cythonize(). So the way to do it is actually this:

    from distutils.core import setup
    from distutils.extension import Extension

    ext = Extension('module', ['module.pyx'], ...)

    setup(...,
          ext_modules=cythonize(ext, gdb_debug=True)
    )

Again, I am in fact doing that...

Frankly, I've been having good results just running a non-debug Python and debug shared object in gdb, so I don't really need this now.

[Tom Swirly]

So here I am, three years later, on a (more or less) completely different project and I'm again getting the same error I can't work out.

Luckily, this time it's not blocking anything - I don't actually need to build this extension in debug, I was just trying it out - but it does give me a bit more information.

I am believing now that the unfortunate combination might be the combination of making a debug build for both the JUCE library and Cython.  I say this because that's something this project and the original project shared in common, and because the only relevant example of this error elsewhere seems to be here.

Now, I'm not targeting old OS/X versions - I'm uniformly targeting 10.9 and above - so it's not exactly this issue.  And as I said, it's not blocking me! - but I wanted to follow up for the next guy who gets blocked.

[sndr....@gmail.com]

great

but to use cython debugging on windows