Cython traceback does not work even with the most trivial example!?

[walt3k]

Hi all, I posted many weeks ago about not being able to get the traceback to give me the same and proper output seen in an equivalent python test.  It was explained to me that if the .pyx file could not be discovered then the lines from the .pyx file would not appear in the traceback.  Based on this,  I assumed my traceback problem had to do with the .pyx file not being in the correct place because we are compiling the cython code with CMake along side other C++ code, etc.   But no matter where I copied my .pyx file, I could not get the traceback to report the lines in the .pyx file as they would have been if it were a .py file.

So I decided to create the absolute simplest test to see if I could get the python and cython traceback to look the same.  I have found I cannot - the traceback is incomplete even in this super trivial example.  I could not possibly have the .pyx file in the wrong location... I think.
At the end of steps 1-6 below, you will note in the "bad output" that the func1 and func2 calls do not appear like they do in the "good output."

Can you explain why?  If you do these steps do you see different output?

1. in file config_pyx.py:
---------------------------------
from distutils.core import setup
from Cython.Build import cythonize
setup(name="testcase", ext_modules=cythonize("testcase.pyx"))

2. in testcase.pyx:
-------------------------
import traceback

def func1():
    stack=traceback.format_stack()
    stackLines=len(stack)
    for line in stack:
       print (line)

def func2():
    func1()

func2()

3. Compile testcase.pyx
----------------------------------------
python3 config_pyx.py build_ext --inplace

4. in testcython.py
-------------------------------
import testcase

5. Run testcase.pyx as a cython file and see bad output
-----------------------------------------------------------------------------------------
python3 testcython.py
 
  File "/u/walt3k/work/sandbox/cythonTestDir/testcython.py", line 1, in <module>
    import testcase

  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load

  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 1181, in exec_module

  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed


6. Run testcase.pyx as a python file and see good output
-------------------------------------------------------------------------------------------
python3 testcase.pyx

  File "/u/walt3k/work/sandbox/cythonTestDir/testcase.pyx", line 12, in <module>
    func2()

  File "/u/walt3k/work/sandbox/cythonTestDir/testcase.pyx", line 10, in func2
    func1()

  File "/u/walt3k/work/sandbox/cythonTestDir/testcase.pyx", line 4, in func1
    stack=traceback.format_stack()

[da-woods]

On 29/04/2022 04:47, walt3k wrote:

    stack=traceback.format_stack()

Cython only generates the traceback as it propagates an exception upwards. So format_stack definitely doesn't work because there's nothing to inspect at this point.

e.g.

def f1():
  raise RuntimeError()

def f2():
  f1()

def f3():
  try:
    f2()
  except RuntimeError as e:
     ...

def f4():
  f3()

If you call f4 then eventually the RuntimeError is caught in f3(). That exception has a traceback (which you can look at). The traceback contains f1, f2, and f3, but doesn't know that it was called from f4.

This is the limit of Cython's handling of tracebacks.

David

[walt3k]

I see.  As you can tell what I am looking for based on what I call the "good" output above is the callstack or backtrace for any point in the code, not just where exceptions are thrown.  Like backtrace_symbols() in C, etc.  I don't need to use traceback in Python.  I just assumed traceback was  intended for printing the call stack based on the output I saw in my Python programs.

Is there no way to get the call stack information for any point in a Cython test where the Python function names can be known?  I am fine with any solution.

Thx, Walt.

[da-woods]

On 29/04/2022 21:28, walt3k wrote:
> Is there no way to get the call stack information for any point in a
> Cython test where the Python function names can be known?  I am fine
> with any solution.

This isn't currently supported in Cython. "traceback" is intended for
what it for you want but only in Python code.

The only solution I can suggest is pretty bad - you need to wrap every
function with a Python wrapper (you can make a decorator to do it)

def makestackable(func):
    name = func.__name__
    key = "func" if name != func else "func_"
    d = {key: func}
    exec(f"""
def {name}(*args, **kwds):
    return {key}(*args, **kwds)
    """, d)
    out = d[name]
    import functools
    functools.update_wrapper(out, func)
    return out

@makestackable
def f1():
    return f2()

@makestackable
def f2():
    import traceback
    return traceback.format_stack()

That loses the line numbers, but at least preserves the names in the
traceback. You can't use the decorator on cdef functions, and you add an
inefficient extra layer. There's probably better variants of the same
idea, but I don't think you'll find anything hugely different.

[walt3k]

Thanks a bunch for the creative solution.  I am still learning python and did not know about the exec function.  
I had a couple of questions:

1. I assume this line

          key = "func" if name != func else "func_"

      is to protect against the case where the wrapped function is actually called "func".  If so, I think there a quotes missing on the comparison, i.e.

          key = "func" if name != "func" else "func_"

2. I notice I get the exact same output with or without the call to 
          functools.update_wrapper(out, func)
    I want to be sure I am not missing something.  What does that line add to the solution?

Thx again for this example.  I learned a lot while dissecting it.  :-) 

Walt.

[da-woods]

On 01/05/2022 19:35, walt3k wrote:
> I am still learning python and did not know about the exec function

It's fairly rarely useful so it's not a huge thing not to know. In
Cython specifically it's occasionally useful for running uncompiled code
(which is what I do here).

> 1. I assume this line
>           key = "func" if name != func else "func_"
>       is to protect against the case where the wrapped function is
> actually called "func".  If so, I think there a quotes missing on the
> comparison, i.e.
>           key = "func" if name != "func" else "func_"

Yes - you're right about the purpose and the missed quotea

>
> 2. I notice I get the exact same output with or without the call to
>           functools.update_wrapper(out, func)
>     I want to be sure I am not missing something.  What does that line
> add to the solution?

functools.update_wrapper just copies some minor details like the
docstring over to the wrapping function. It isn't required for the
solution to work but it's often a good idea to use that or
@functools.wraps when writing decorators.

[walt3k]

Ok, gotcha.  Thanks again for all the help, Walt.